Log playback method, data transmission method, and log playback system

Abstract

Embodiments of the present specification provide a log playback method, a data transmission method and a log playback system, wherein the log playback method comprises: obtaining a to-be-processed log and a log type of the to-be-processed log; according to the log type, screening a target processing queue from a to-be-processed queue, and adding the to-be-processed log to the target processing queue to obtain an updated to-be-processed queue, wherein the to-be-processed queue comprises a first processing queue and a second processing queue, the first processing queue is used to store logs generated by a unique index table, and the second processing queue is used to store logs generated by a non-unique index table; reading a to-be-played log from the updated to-be-processed queue, and adding the to-be-played log to a concurrent playback queue, wherein the to-be-played log is a log that can be concurrently played back in the updated to-be-processed queue; and concurrently playing back the concurrent playback log in the concurrent playback queue. The to-be-played log that can be concurrently played back is added to the concurrent playback queue, and the log playback efficiency is improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to log playback methods, data transmission methods, and log playback systems. Background Technology

[0002] Oplog is MongoDB's operation log, which records all modifications to the data stored in the database. Because Oplog doesn't record the complete content before and after data updates or deletions, only the primary key values ​​and changed fields, this mechanism means that for tables with unique indexes, the Data Transmission Service (DTS) can only perform table-level serial log replay during incremental data synchronization to avoid data consistency conflicts, as the dependencies between data cannot be determined.

[0003] However, for scenarios that do not involve unique index value conflicts (such as adding or deleting data only), serial log replay at the table level limits the data write speed, leading to delays in incremental data migration and extremely low efficiency. Therefore, an efficient log replay solution is urgently needed. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a log playback method. One or more embodiments of this specification also relate to a data transmission method, a log playback system, a log playback device, a data transmission device, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a log playback method is provided, comprising: Get the logs to be processed and their log types; Based on the log type, the target processing queue is selected from the pending queue, and the pending logs are added to the target processing queue to obtain the updated pending queue. The pending queue includes a first processing queue and a second processing queue. The first processing queue is used to store logs generated by unique index tables, and the second processing queue is used to store logs generated by non-unique index tables. Read the logs to be replayed from the updated pending queue and add them to the concurrent replay queue. The logs to be replayed are the logs that can be replayed concurrently in the updated pending queue. Perform concurrent replay of the concurrent replay logs in the concurrent replay queue.

[0006] According to a second aspect of the embodiments of this specification, a data transmission method is provided, comprising: During data transmission, the incremental log is parsed to determine the incremental data to be written. The log replay method is used to replay the pending logs containing incremental data to be written, and the log replay results are obtained. Based on the log replay results, the incremental data to be written is written to the target database.

[0007] According to a third aspect of the embodiments of this specification, a log replay system is provided, including a queue allocation unit and a concurrent replay unit; The queue allocation unit is used to obtain the logs to be processed and the log types of the logs to be processed; according to the log types, it filters the target processing queue from the queues to be processed and adds the logs to be processed to the target processing queue to obtain the updated queues to be processed. The queues to be processed include a first processing queue and a second processing queue. The first processing queue is used to store logs generated by the unique index table, and the second processing queue is used to store logs generated by the non-unique index table. The concurrent replay unit is used to read concurrent replay logs from the updated pending queue and add the concurrent replay logs to the concurrent replay queue. The logs to be replayed are the logs in the updated pending queue that can be replayed concurrently. The concurrent replay logs in the concurrent replay queue are replayed concurrently.

[0008] According to a fourth aspect of the embodiments of this specification, a log playback device is provided, comprising: The acquisition module is configured to acquire logs to be processed and the log types of the logs to be processed; The filtering module is configured to filter the target processing queue from the pending queue according to the log type, and add the pending logs to the target processing queue to obtain the updated pending queue. The pending queue includes a first processing queue and a second processing queue. The first processing queue is used to store logs generated by the unique index table, and the second processing queue is used to store logs generated by the non-unique index table. The reading module is configured to read logs to be replayed from the updated pending queue and add the logs to be replayed to the concurrent replay queue, wherein the logs to be replayed are the logs that can be replayed concurrently in the updated pending queue; The first replay module is configured to perform concurrent replay of the concurrent replay logs in the concurrent replay queue.

[0009] According to a fifth aspect of the embodiments of this specification, a data transmission apparatus is provided, comprising: The parsing module is configured to parse the incremental log during data transmission to determine the incremental data to be written. The second replay module is configured to use the log replay method to replay the pending logs containing incremental data to be written, and obtain the log replay results. The write module is configured to write incremental data to the target database based on log replay results.

[0010] According to a sixth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the log playback method provided in the first aspect above.

[0011] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the log playback method provided in the first aspect.

[0012] According to an eighth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the log playback method provided in the first aspect above.

[0013] This specification provides a log replay method according to one embodiment, comprising: obtaining logs to be processed and their log types; selecting a target processing queue from the queue to be processed based on the log type, and adding the logs to be processed to the target processing queue to obtain an updated queue to be processed, wherein the queue to be processed includes a first processing queue and a second processing queue, the first processing queue being used to store logs generated by a unique index table, and the second processing queue being used to store logs generated by a non-unique index table; reading logs to be replayed from the updated queue to be processed, and adding them to a concurrent replay queue, wherein the logs to be replayed are those in the updated queue that can be replayed concurrently; and performing concurrent replay of the logs in the concurrent replay queue. By adding logs to be processed to different processing queues according to their log types, the problems of data backlog and data interference are reduced. Furthermore, adding logs to be replayed concurrently from the queue to the concurrent replay queue enables multi-threaded concurrent log replay, improving log replay efficiency. Attached Figure Description

[0014] Figure 1 This is a flowchart of a log playback method provided in one embodiment of this specification; Figure 2 This is an architecture diagram of a log playback system provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating the processing procedure of a log playback method provided in one embodiment of this specification; Figure 4This is a flowchart illustrating the processing procedure of a queue allocation unit according to one embodiment of this specification; Figure 5 This is a flowchart illustrating the processing procedure of a concurrent playback unit according to one embodiment of this specification; Figure 6 This is a flowchart illustrating a data transmission method provided in one embodiment of this specification; Figure 7 This is a schematic diagram of the structure of a log playback device provided in one embodiment of this specification; Figure 8 This is a schematic diagram of the structure of a data transmission device provided in one embodiment of this specification; Figure 9 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0015] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0016] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0017] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0018] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0019] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0020] MongoDB is a database based on distributed file storage, designed to provide scalable, high-performance data storage solutions for web applications. MongoDB supports a very flexible data structure, using a JSON-like BSON format, allowing it to store relatively complex data types.

[0021] Unique Indexes: In MongoDB, unique indexes ensure that all values ​​in a collection are unique for a given field or combination of fields. This means that when a unique index is created for a field, each document value for that field is unique, and no two documents can have the same field value. If an attempt is made to insert or update data that violates the uniqueness constraint, the operation will fail.

[0022] DTS (Data Transmission Service) for MongoDB: This is part of the data transmission service, specifically designed for data migration, synchronization, and subscription tasks for MongoDB databases. DTS MongoDB helps users transfer data between different MongoDB instances, supporting various scenarios including full migration, incremental synchronization, and data subscription.

[0023] Incomplete logs: These are operation logs that do not record the complete content before and after data updates or deletions.

[0024] Log retrieval job: The job responsible for retrieving operation logs from the source.

[0025] Log replay job: This job is responsible for replaying operation logs on the target end.

[0026] Log files: These record all modifications made to the database, such as MongoDB's Oplog. These log files are crucial in the following scenarios: Fault recovery: When a database fails, data can be restored to its pre-failure state by replaying the operations in the log files. Master-slave replication: The master node records write operations to the log file, and slave nodes maintain data synchronization with the master by reading and replaying these log operations. Data backup and recovery: Regularly backing up log files allows data to be restored when needed by replaying the logs. Traditional log replay is usually sequential, replaying operations one by one according to the order in the log file. However, this is inefficient when processing large amounts of log records.

[0027] Concurrent replay refers to the process in a database system of reading operation records from a log file and re-executing those operations in parallel. Concurrent replay is crucial for improving data processing efficiency, especially in large-scale distributed systems.

[0028] This specification provides a log playback method, and also relates to a data transmission method, a log playback system, a log playback device, a data transmission device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0029] See Figure 1 , Figure 1 This specification shows a flowchart of a log playback method according to an embodiment, which specifically includes the following steps: Step 102: Obtain the logs to be processed and their log types.

[0030] In one or more embodiments of this specification, in information systems, particularly database management systems, software testing systems, and transaction systems, log replay can be used to re-execute previously recorded logs, thereby restoring system status, testing system functionality, testing performance bottlenecks, or troubleshooting. During log replay, the log to be processed and its log type can be obtained, and the log to be processed can be replayed based on its log type.

[0031] It should be noted that pending logs refer to operation log files that have been recorded but not yet used for playback. There can be one or more pending logs. When there are multiple pending logs, they can be retrieved according to their generation order. Pending logs can be incomplete or complete logs, depending on the actual situation; this specification does not impose any limitations on this. The log type of the pending logs is determined based on the table that generated them. For example, if the pending logs are generated by a unique index table, then the log type is a unique index table type. If the pending logs are generated by a non-unique index table, then the log type is a non-unique index table type. A non-unique index table can also be called a regular table.

[0032] In practical applications, there are multiple ways to obtain logs to be processed and their log types. The specific method should be chosen based on the actual situation, and this specification does not impose any limitations on these methods. In one possible implementation of this specification, a log fetching job can continuously fetch logs to be processed (such as Oplog) from a source database (such as MangoDB) in the order in which the logs were generated. The logs to be processed carry the log type.

[0033] In another possible implementation of this specification, the log to be processed can be obtained first, and then the log type of the log to be processed can be parsed. That is, the above-mentioned obtaining the log to be processed and the log type of the log to be processed may include the following steps: Get the logs to be processed; Parse the logs to be processed to determine the log record information of the logs to be processed; Based on the log record information, determine the log type corresponding to the log to be processed.

[0034] It's important to note that log information includes the specific content and metadata of the logs to be processed. This information typically includes, but is not limited to, the following: Timestamp: Records the time the log was generated, accurate to the second or millisecond. Log Level: Indicates the importance and severity of the log. Common log levels include debug, info, warning, error, and critical. Debug level logs record debugging information, typically used during development and testing phases to help developers find and fix problems. Info level logs record general information, such as system startup, shutdown, and routine operations, helping to understand the system's running status. Warning level logs record potential problems or situations requiring attention, but these problems will not cause system crashes or functional failures. Error level logs record errors that occur during system operation, which may cause partial functional failures or data loss. Critical level logs record serious errors or exceptions, which typically cause system crashes or prevent further operation, requiring immediate attention. Log Source: Records the name of the module, component, or service that generated the log. The log entries should include the database name and table name. The database name is the name of the database and a unique identifier used to distinguish different databases. The table name is the name of the table in the database used to store log records and is a unique identifier used to distinguish different tables within the same database. Each table typically contains records of a specific type, such as log records. Operation type: Describes the type of operation or event recorded in the log entry, such as insert, update, delete, login, and logout. Specifically, insert is used to add a new record to the database; update is used to modify an existing record in the database; delete is used to remove a record from the database; login records a successful login to the system or service; and logout records a user's exit from the system or service. Operation parameters: Record the specific parameters of the operation, such as the table name and field values ​​in the Structured Query Language (SQL) statement. Result status: Record the result of the operation, such as success, failure, or timeout. Other metadata may include additional information such as user identifier (ID), session ID, and Internet Protocol address.

[0035] In practical applications, there are multiple ways to parse logs to be processed and determine their log information. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on these methods. One possible implementation in this specification is to use regular expressions to extract log information from the logs to be processed. Another possible implementation in this specification is to use a dedicated log parsing library (such as Logstash, Fluentd, etc.) to parse the logs to be processed and determine their log information.

[0036] Furthermore, there are multiple ways to determine the log type corresponding to the log to be processed based on the log record information. The specific method should be selected according to the actual situation, and this specification does not impose any limitations on this method in the embodiments. In one possible implementation of this specification, the table that generated the log to be processed can be determined based on the database table name in the log record information, and the corresponding log type can be determined based on the table. In another possible implementation of this specification, the log type of the log to be processed can be determined based on the operation type and fields in the log record information. For example, if the log record information includes a large number of errors caused by uniqueness constraints (such as E11000 duplicate key error), it indicates that the set has a unique index, and the log type of the log to be processed is a unique index table type.

[0037] By applying the scheme of the embodiments in this specification, the log type of the log to be processed is determined based on the parsing result, thereby accurately determining the log type of the log to be processed and providing accurate information for the queue allocation process of the log to be processed.

[0038] Step 104: Based on the log type, filter out the target processing queue from the pending queue, and add the pending logs to the target processing queue to obtain the updated pending queue. The pending queue includes a first processing queue and a second processing queue. The first processing queue is used to store logs generated by unique index tables, and the second processing queue is used to store logs generated by non-unique index tables.

[0039] It should be noted that during log replay, the pending queue is used to store logs that were not used for replay. The pending queue is divided into different types based on the log type; for example, it can be divided into a first processing queue and a second processing queue. The number of first and second processing queues can be one or more, depending on the actual situation, and this specification does not limit this. For example, there are four pending queues: first processing queue 1, first processing queue 2, second processing queue 3, and second processing queue 4. The target processing queue refers to the pending queue that matches the log type of the pending log. For example, if the log type of the pending log is a non-unique index type, it means that the pending log is generated by a non-unique index table, and the second processing queue is used to store logs generated by non-unique index tables; then the target processing queue for the pending log is the second processing queue. Similarly, if the log type of the pending log is a unique index type, it means that the pending log is generated by a unique index table, and the first processing queue is used to store logs generated by unique index tables; then the target processing queue for the pending log is the first processing queue. Since there may be multiple first processing queues and second processing queues, the target processing queue can be further selected from multiple first processing queues or multiple second processing queues based on the logs to be processed. For example, the target processing queue for the logs to be processed can be selected from second processing queue 3 and second processing queue 4 as second processing queue 3.

[0040] In practical applications, when adding logs to be processed to the target processing queue, if there are multiple logs to be processed, they can be added to the target processing queue first according to the order in which they were generated. There are multiple ways to select the target processing queue from the logs to be processed, depending on the log type. The specific method should be chosen based on the actual situation, and this specification does not impose any limitations on these methods in the embodiments.

[0041] In one optional embodiment of this specification, the method for filtering the target processing queue of logs of the non-unique index table type (logs generated by non-unique index tables) is described. That is, the above-mentioned filtering of the target processing queue from the queue of logs to be processed according to the log type may include the following steps: When the log type is a non-unique index table type, read the primary key value from the log to be processed; Based on the primary key value, the target processing queue is selected from the queue to be processed, where the target processing queue is the second processing queue.

[0042] It's important to note that a non-unique index table refers to a regular table that does not contain a unique index. The primary key value (PrimaryKey) refers to the document's _id field, which is a unique identifier for each document. Each document in each collection has a unique _id field, ensuring that each document is unique within the collection and preventing duplicate document insertion. For non-unique index tables, since MongoDB's primary key value is immutable and unique, and the primary key value is recorded in the Oplog corresponding to insert, update, delete, and other operations, it can be used as the basis for queue allocation.

[0043] In practical applications, when reading primary key values ​​from a pending log, taking Oplog as an example, a MongoDB client can be used to connect to a MongoDB instance. Select the local database, where Oplog records are stored in the Oplog.rs or Oplog.ts collection. Filter Oplog records using various conditions, such as time range and operation type, parse the filtered Oplog records, and extract the primary key values.

[0044] Furthermore, when selecting the target processing queue from the pending queue based on the primary key value, a hash calculation can be performed on the log content of the pending logs to determine the queue identifier information of the target processing queue. The pending queue that matches the queue identifier information is then identified as the target processing queue for the pending logs. If there are multiple pending logs, pending logs with the same primary key value can be assigned to the same second processing queue. Pending logs in different second processing queues do not affect each other, and the logs to be replayed can be read directly from the second processing queue.

[0045] By applying the scheme of the embodiments in this specification, the target processing queue can be filtered based on the logs to be processed that are of the type of non-unique index table with primary key values, and the queue allocation can be performed efficiently and accurately.

[0046] In another optional embodiment of this specification, the method for filtering the target processing queue of logs of the unique index table type (logs generated by the unique index table) is described. That is, the above-mentioned filtering of the target processing queue from the queue of logs to be processed according to the log type may include the following steps: When the log type is a unique index table type, determine the log operation of the log to be processed, wherein the log operation includes at least one of the following: insert operation, update operation, delete operation, and key-value modification operation; Based on log operations, the target processing queue is selected from the pending queue, where the target processing queue is the first processing queue.

[0047] It should be noted that key-value modification operations refer to operations that modify unique key values. There are multiple ways to determine the log operation of a log to be processed, and the specific method should be chosen based on the actual situation. This specification does not impose any limitations on this method. In one possible implementation of this specification, the log operation of the log to be processed can be determined based on the fields in the log record information. For example, in MongoDB's Oplog, each record contains an `op` field, which indicates the type of operation. By checking this field, the specific type of log operation can be determined. The following are common operation types and their meanings: `i` represents an insert operation, `u` represents an update operation, `d` represents a delete operation, `c` represents a command operation (such as creating an index), and `n` represents no operation (usually a heartbeat operation). Furthermore, for update operations (`u`), the `o` and `o2` fields can be further checked to determine the specific update content and conditions. The `o` field contains the specific content of the update operation, such as the updated fields and new values. The `o2` field contains the query conditions for the update operation, i.e., which documents will be updated. In another possible implementation of this specification, the log operation of the log to be processed can be determined by parsing the log to be processed using a third-party log parsing library.

[0048] In practical applications, when selecting target processing queues from the pending queues based on log operations, for pending logs of unique index table type, the unique key values ​​in different logs can affect each other. Therefore, queue allocation and concurrency judgment cannot be directly based on the primary key value. Furthermore, the pending logs may not contain unique key values. Therefore, certain operations can be selected as "specified operations" to control queue allocation and log concurrency. The pending logs are categorized according to log operations, and the operations `update unique index value` and `delete` are selected as "specified operations." It is then determined whether a log operation is a specified operation, and the target processing queue is selected from the pending queue based on the determination result.

[0049] By applying the solution in the embodiments of this specification, the target processing queue can be accurately selected from the queue to be processed by filtering out the target processing queue based on log operations and specified operations, while ensuring data consistency and efficiency before and after concurrent replay.

[0050] In practical applications, when selecting the target processing queue from the pending queue based on log operations, it can be determined whether the log operation is a specified operation. In one possible implementation of this specification, if the log operation is not a specified operation, a hash calculation can be performed directly based on the log content of the pending log to determine the queue identifier information of the target processing queue, and the pending queue that matches the queue identifier information is determined as the target processing queue for the pending log.

[0051] In another possible implementation of this specification, the case where the log operation is a specified operation is described. That is, the above-mentioned filtering of the target processing queue from the pending queue based on the log operation may include the following steps: When the log operation is a specified operation, the target log operation is blocked, and the target processing queue is selected from the pending queue according to the log operation. The target log operation is processed until the first log in the first processing queue is replayed. The specified operation includes at least one of the unique index value update operation and the deletion operation. The target log operation is the log operation executed after the log operation.

[0052] It's important to note that when the log operation is a specified operation, subsequent operations on the unique index table must first be blocked; that is, the target log operation must be blocked. For example, if the current log operation to be processed is delete 1-1, and the target log operation is insert 1-1, without blocking the target log operation, delete 1-1 and insert 1-1 will be executed simultaneously, affecting data uniqueness. After blocking the target log operation, the target processing queue can be selected from the pending queue based on the log operations, and the log operations can be added to the target processing queue until the first log in the first processing queue has been replayed, indicating that delete 1-1 has been completed. At this point, subsequent target log operations can be processed.

[0053] By applying the solution in the embodiments of this specification, when the log operation is a specified operation, the subsequent target log operation is blocked until the log operation is completed, and the processing of the target log operation is resumed, thus ensuring the consistency of data in the unique index table.

[0054] Step 106: Read the logs to be replayed from the updated pending queue and add them to the concurrent replay queue. The logs to be replayed are the logs that can be replayed concurrently in the updated pending queue.

[0055] It should be noted that the concurrent replay queue is used to store logs that can be replayed concurrently; it can be understood as a collection of logs that can be replayed concurrently.

[0056] In practical applications, logs to be replayed can be read from the updated pending queue based on whether the updated pending queue includes logs whose log operation is the specified operation. Since pending logs whose log operation is the specified operation are stored in the first processing queue, it can be determined whether the first processing queue in the updated pending queue includes the first target log, and the pending logs to be replayed can be read from the updated pending queue based on the determination result. Furthermore, when adding the pending logs to be replayed to the concurrent replay queue, it can be determined whether the concurrent replay logs from the pending queue corresponding to the pending logs to be replayed have been replayed, and the determination result can be used to decide whether to add the pending logs to the concurrent replay queue.

[0057] In one optional embodiment of this specification, the above-described method of reading logs to be replayed from the queue to be processed and adding the logs to be replayed to the concurrent replay queue may include the following steps: If the first processing queue in the updated pending queue includes the first target log, the first log in the first processing queue is determined as the log to be replayed, wherein the first target log is the first log whose log operation is the specified operation; Based on the concurrent replay logs in the concurrent replay queue, add the logs to be replayed to the concurrent replay queue.

[0058] It should be noted that if the first processing queue includes the first target log, it means that the first processing queue includes the first log whose log operation is the specified operation, and the target log operation to be executed afterward has been blocked when the log operation is the specified operation. Therefore, all the first logs in the first processing queue can be determined as logs to be replayed and played back concurrently.

[0059] Furthermore, since the concurrent replay queue may contain previously existing concurrent replay logs that have not yet been replayed, directly adding all logs to be replayed to the concurrent replay queue may cause conflicts between the logs to be replayed and the concurrent replay logs that have not yet been replayed. Therefore, logs to be replayed can be added to the concurrent replay queue based on the concurrent replay logs in the concurrent replay queue.

[0060] In practical applications, there are multiple ways to add logs to be replayed to the concurrent replay queue based on the concurrent replay logs in the concurrent replay queue. The specific method should be selected according to the actual situation, and this specification does not limit this approach. In one possible implementation, the logs to be replayed are added to the concurrent replay queue when all concurrent replay logs in the concurrent replay queue have been replayed, or when the concurrent replay queue does not contain any concurrent replay logs that have not been replayed. In another possible implementation, the logs to be replayed are added to the concurrent replay queue when the concurrent replay logs from the corresponding processing queue in the concurrent replay queue have been replayed.

[0061] Applying the scheme of the embodiments of this specification, when the first processing queue includes the first target log, the first log in the first processing queue is determined as the log to be replayed, thereby accurately determining the log to be replayed. Furthermore, based on the concurrent replay log in the concurrent replay queue, the log to be replayed is added to the concurrent replay queue, avoiding log conflicts in the concurrent replay queue and ensuring log replay efficiency and uniqueness.

[0062] In another optional embodiment of this specification, the above-described reading of logs to be replayed from the queue to be processed and adding the logs to be replayed to the concurrent replay queue may include the following steps: If the first target log is not included in the first processing queue of the updated pending queue, the log to be replayed is read from the updated pending queue based on the log addition order of the updated pending queue, where the first target log is the first log for the specified operation. Based on the concurrent replay logs in the concurrent replay queue, add the logs to be replayed to the concurrent replay queue.

[0063] It should be noted that if the first processing queue does not include the first target log, it means that the first processing queue does not include the first log whose log operation is the specified operation. The log to be replayed can be directly read from the updated pending queue. That is, the log to be replayed is read from both the first and second processing queues. It is worth noting that the logs stored in the updated pending queue follow a first-in, first-out (FIFO) log reading principle; that is, the logs stored in the pending queue first are read first. For example, if the first processing queue 1 includes logs 1-3 and 1-4, where log 1-3 is the third log stored in the first processing queue 1 and log 1-4 is the fourth log stored in the first processing queue 1, then when reading the log to be replayed from the first processing queue 1, log 1-3 is read first, followed by log 1-4.

[0064] In practical applications, if there are multiple updated queues to be processed, the logs can be read in a round-robin fashion from each updated queue. For example, the updated queues to be processed include First Processing Queue 1 and First Processing Queue 2. First Processing Queue 1 includes Logs 1-3 and Logs 1-4, and First Processing Queue 2 includes Logs 2-3 and Logs 2-4. When reading logs to be replayed from First Processing Queue 1 and First Processing Queue 2, the logs can be read in the following order: first Logs 1-3, then Logs 2-3, then Logs 1-4, and finally Logs 2-4.

[0065] The solution implemented in this specification, when the first target log is not included in the first processing queue, further adds the log to be replayed to the concurrent replay queue based on the updated log addition order of the pending processing queue and the concurrent replay log in the concurrent replay queue, thereby avoiding log conflicts in the concurrent replay queue and ensuring log replay efficiency and uniqueness.

[0066] In one optional embodiment of this specification, adding the log to be replayed to the concurrent replay queue based on the concurrent replay log in the concurrent replay queue may include the following steps: Retrieve concurrent replay logs from the concurrent replay queue; Determine the identifier of the first queue corresponding to the concurrent replay logs, and determine the identifier of the second queue of logs to be replayed; Based on the first queue identifier and the second queue identifier, the target log to be replayed is selected from the logs to be replayed, wherein the second queue identifier of the target log to be replayed is different from the first queue identifier; Add the target log to be replayed to the concurrent replay queue.

[0067] It should be noted that the concurrent replay logs in the concurrent replay queue refer to logs that have not yet been concurrently replayed. Once the concurrent replay logs in the concurrent replay queue have finished replaying, they can be removed from the concurrent replay queue. The first queue identifier corresponding to the concurrent replay log refers to the queue identifier of the pending queue where the concurrent replay log was stored before entering the concurrent replay queue. The second queue identifier of the log to be replayed refers to the queue identifier of the pending queue where the log to be replayed is stored. The first queue identifier corresponding to the concurrent replay log and the second queue identifier of the log to be replayed can be determined by hash calculation based on the log content, or by parsing from the log record information.

[0068] In practical applications, after determining the first queue identifier corresponding to the concurrent replay logs and the second queue identifier of the logs to be replayed, the logs to be replayed can be added to the concurrent replay queue based on the first and second queue identifiers. Specifically, if the second queue identifier of a log to be replayed is different from both the first queue identifier corresponding to the concurrent replay logs, it means that there are no concurrent replay logs from the same waiting queue as the log to be replayed in the concurrent replay queue. This log to be replayed is the target log, and it can be directly added to the concurrent replay queue. For logs to be replayed other than the target logs, since the logs in the concurrent replay queue are continuously being replayed concurrently, the target logs can be continuously filtered from the logs to be replayed based on the first and second queue identifiers. When the concurrent replay log from the same waiting queue as the log to be replayed has been replayed, the log to be replayed is added to the concurrent replay queue.

[0069] The solution implemented in this specification determines the target log to be replayed based on the first queue identifier of the concurrent replay logs and the second queue identifier of the logs to be replayed in the concurrent replay queue. This ensures that adding the target log to be replayed to the concurrent replay queue allows for normal concurrent log replay, thereby improving log replay efficiency while ensuring data consistency before and after log replay.

[0070] Step 108: Perform concurrent replay of the concurrent replay logs in the concurrent replay queue.

[0071] By applying the solutions in the embodiments of this specification, the problems of data accumulation and data interference are reduced by adding logs to be processed to different processing queues according to log type. Furthermore, by adding logs that can be replayed concurrently from the processing queue to the concurrent replay queue, multi-threaded concurrent replay of logs is achieved, thereby improving log replay efficiency.

[0072] In practical applications, there are various ways to perform concurrent replay of concurrent replay logs in the concurrent replay queue. The specific method should be selected according to the actual situation, and the embodiments in this specification do not impose any limitations on this. In one optional embodiment of this specification, the concurrent replay logs in the concurrent replay queue can be randomly distributed to multiple log replay jobs for concurrent replay.

[0073] In another possible implementation of this specification, the concurrent replay of the concurrent replay logs in the concurrent replay queue described above may include the following steps: Determine the first queue identifier corresponding to the concurrent replay log in the concurrent replay queue; Based on the first queue identifier, the concurrent replay logs are distributed to multiple log replay jobs; Perform concurrent replay of multiple log replay jobs.

[0074] It's important to note that when distributing concurrent replay logs to multiple log replay jobs based on the first queue identifier, many log operations (especially those involving tables with unique indexes) need to be completed within a single transaction to ensure data consistency. Splitting logs from the same pending queue into different log replay jobs could lead to some operations succeeding while others fail, thus compromising transaction consistency. Therefore, when distributing concurrent replay logs to multiple log replay jobs based on the first queue identifier, concurrent replay logs from the same pending queue (with the same first queue identifier) ​​will not be split into different log replay jobs; that is, concurrent replay logs with the same first queue identifier will have the same log replay job. In practical applications, when performing concurrent replay on multiple log replay jobs, different log replay jobs can replay concurrently, but multiple concurrent replay logs within a single log replay job need to be executed sequentially.

[0075] By applying the scheme of the embodiments in this specification, concurrent replay logs with the same first queue identifier are assigned to the same log replay job according to the first queue identifier, thus ensuring data consistency.

[0076] In one optional embodiment of this specification, after performing concurrent replay on the concurrent replay logs in the concurrent replay queue as described above, the following steps may also be included: Remove concurrent replay logs that have completed concurrent replay from the concurrent replay queue.

[0077] It should be noted that once a concurrent replay log in the concurrent replay queue has completed its concurrent replay, it can be removed from the queue in a timely manner. This ensures that subsequent logs awaiting replay can enter the concurrent replay queue for concurrent replay, guaranteeing the normal execution of concurrent log replay and improving the efficiency of concurrent log replay.

[0078] See Figure 2 , Figure 2 This specification shows an architecture diagram of a log replay system according to an embodiment of the present specification. The log replay system may include a queue allocation unit 202 and a concurrent replay unit 204. The queue allocation unit 202 is used to obtain the logs to be processed and the log types of the logs to be processed; according to the log types, it filters the target processing queue from the queues to be processed and adds the logs to be processed to the target processing queue to obtain the updated queues to be processed. The queues to be processed include a first processing queue and a second processing queue. The first processing queue is used to store logs generated by the unique index table, and the second processing queue is used to store logs generated by the non-unique index table. The concurrent replay unit 204 is used to read concurrent replay logs from the updated pending queue and add the concurrent replay logs to the concurrent replay queue. The logs to be replayed are the logs in the updated pending queue that can be replayed concurrently. The concurrent replay logs in the concurrent replay queue are replayed concurrently.

[0079] By applying the scheme of the embodiments in this specification, the log replay process is optimized from single-threaded serial execution to multi-threaded concurrent execution through queue allocation unit, concurrent replay unit and different queues to be processed. While ensuring data consistency before and after log replay, the incremental migration speed of DTS MongoDB is improved.

[0080] It should be noted that the queue allocation unit 202 and the concurrent replay unit 204 are connected via a network. The network provides the medium for the communication link between the queue allocation unit 202 and the concurrent replay unit 204. The log replay system can be configured on the server side. The server side can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0081] See Figure 3 , Figure 3 This specification illustrates a flowchart of a log replay method according to an embodiment. The log replay method specifically includes: The log retrieval job continuously retrieves logs to be processed from the source database in the order in which the operation logs were generated; The queue allocation unit selects the target processing queue from the pending logs based on the log type, and adds the pending logs to the target processing queue, thus obtaining an updated pending queue. The pending queue includes a first processing queue and a second processing queue. The first processing queue stores logs generated by unique index tables, and the second processing queue stores logs generated by non-unique index tables. (See also...) Figure 3The processing queues include a first processing queue 1, a first processing queue 2, a second processing queue 3, and a second processing queue 4. The first processing queue 1 stores logs 1-3, 1-4, ..., 1-n. The first processing queue 2 stores logs 2-3, 2-4, ..., 2-n. The second processing queue 3 stores logs 3-3, 3-4, ..., 3-n. The second processing queue 4 stores logs 4-3, 4-4, ..., 4-n. Taking log 4-n as an example, log 4-n represents the nth log entry stored in the second processing queue 4.

[0082] The concurrent replay unit continuously retrieves logs to be replayed from each updated pending queue, adding logs that can be replayed concurrently to the concurrent replay queue. See also Figure 3 The concurrent replay queue includes logs 1-1 and 1-2 from the first processing queue 1, logs 2-1 and 2-2 from the first processing queue 2, logs 3-1 and 3-2 from the second processing queue 3, and logs 4-1 and 4-2 from the second processing queue 4. Since logs 1-1 and 1-2 have already been added from the first processing queue 1 to the concurrent replay queue, therefore... Figure 3 The example of the first processing queue 1 does not include log 1-1 and log 1-2. The same applies to other queues to be processed. Therefore, the embodiments in this specification will not be described in detail.

[0083] The logs in the concurrent replay queue are distributed to multiple log replay jobs to achieve concurrent replay, and the concurrent replay results are written to the target database. See also Figure 3 Multiple log replay jobs are included: Log Replay Job 1, Log Replay Job 2, Log Replay Job 3, and Log Replay Job 4. Logs 1-1 and 1-2 are assigned to Log Replay Job 1 for replay, logs 2-1 and 2-2 are assigned to Log Replay Job 2 for replay, logs 4-1 and 4-2 are assigned to Log Replay Job 3 for replay, and logs 3-1 and 3-2 are assigned to Log Replay Job 4 for replay. Log replay occurs concurrently between Log Replay Jobs 1 through 4, and logs within each Log Replay Job are replayed sequentially. After a Log Replay Job completes its replay, logs that were not replayed concurrently in the concurrent replay queue can be assigned to idle Log Replay Jobs.

[0084] The solution implemented in this specification addresses the problem that MongoDB tables with unique indexes can only perform log replay serially at the table level. By determining the concurrent replayability of incomplete logs, incomplete logs are allocated to multiple log replay jobs for concurrent and ordered replay, thereby improving the incremental migration speed of DTS MongoDB while ensuring data consistency before and after log replay.

[0085] See Figure 4 , Figure 4 This specification illustrates a flowchart of a queue allocation unit's processing procedure according to an embodiment of the present invention. Specifically, the queue allocation unit acquires logs to be processed. Based on the log type of the logs to be processed, it determines whether the logs originate from a unique index table. If not, it determines the queue identifier of the logs to be processed and allocates the logs to the second processing queue corresponding to the queue identifier. If yes, it determines whether the log operation of the logs to be processed is a specified operation (unique index value update operation and deletion operation). If not, it determines the queue identifier of the logs to be processed and allocates the logs to the first processing queue corresponding to the queue identifier. If yes, it blocks subsequent operations of the unique index table from entering the queue allocation unit, determines the queue identifier based on the specified operation, allocates the logs to the first processing queue corresponding to the queue identifier, and sends a specified operation signal to the concurrent playback unit to notify the concurrent playback unit to add all first logs in the first processing queue to the concurrent playback queue. After notifying the concurrent replay unit to add all first logs in the first processing queue to the concurrent replay queue, it determines whether the first logs in the concurrent replay queue have been replayed and cleared (removed from the concurrent replay queue). If not, it returns to notify the concurrent replay unit to add all first logs in the first processing queue to the concurrent replay queue. If so, it processes the subsequent operations of the unique index table.

[0086] By applying the scheme of the embodiments in this specification, the queue allocation unit allocates the logs to be processed from unique index tables and non-unique index tables to different queues according to different rules, thereby reducing data accumulation and mutual interference.

[0087] See Figure 5 , Figure 5This specification illustrates a flowchart of the processing procedure of a concurrent playback unit according to an embodiment of the present invention. Specifically, the concurrent playback unit determines whether a specified operation signal has been received. If so, it indicates that the first processing queue includes a first target log (the log operation being the first log of the specified operation), and all first logs in the first processing queue are identified as logs to be played back. If not, it indicates that the first processing queue does not include the first target log, and the logs to be played back are read from the updated processing queue (including the first and second processing queues) based on the log addition order of the updated processing queue. After obtaining the logs to be played back, it determines whether to add them to the concurrent playback queue. Specifically, it can be determined whether all concurrent playback logs from the processing queue corresponding to the logs to be played back have been played back completely. If not, the log to be replayed will not be added to the concurrent replay queue. The system will wait for all concurrent replay logs from the corresponding processing queue to complete their replay before returning to determine whether to add the log to be replayed to the concurrent replay queue. If yes, the log to be replayed will be added to the concurrent replay queue for concurrent replay by multiple log replay jobs. It's worth noting that after all concurrent replay logs from the corresponding processing queue have completed their replay, the completed concurrent replay logs can be removed from the concurrent replay queue. Similarly, after multiple log replay jobs have completed their concurrent replay, the completed concurrent replay logs can be removed from the concurrent replay queue.

[0088] The solution implemented in this specification introduces a "specified operation" for the unique index table through a concurrent replay unit. This limits the concurrent execution degree and execution order of the data, improves the log replay efficiency of the unique index table, and ensures data consistency before and after log replay.

[0089] See Figure 6 , Figure 6 This specification illustrates a flowchart of a data transmission method according to an embodiment, which specifically includes the following steps: Step 602: During data transmission, parse the incremental log to determine the incremental data to be written.

[0090] Step 604: Use the log replay method to replay the pending logs containing the incremental data to be written, and obtain the log replay results.

[0091] Step 606: Based on the log replay results, write the incremental data to be written to the target database.

[0092] It's important to note that incremental logs are a mechanism used during data transmission or synchronization to record data changes since the last successful transmission or synchronization. They are primarily used to improve data synchronization efficiency and reduce unnecessary data transfer. Parsing incremental logs refers to the process of reading and analyzing the records in the incremental log file. The purpose of parsing incremental logs is to identify which data has changed (added, modified, or deleted) since the last synchronization; this changed data is the incremental data to be written.

[0093] In practical applications, the implementation method of step 604 is the same as described above. Figure 1 The implementation of the log replay method shown is the same, and will not be repeated in the embodiments of this specification. There are multiple ways to parse the incremental log and determine the incremental data to be written, and the specific method should be selected according to the actual situation. This specification does not limit this method. In one possible implementation of this specification, the incremental data to be written can be determined based on the timestamp or sequence number in the incremental log. This method relies on the timestamp or globally unique sequence number of each data record to track changes. The system records the time point or sequence number position of the last synchronization. During the next synchronization, by querying all change records after that time point or sequence number, it is possible to effectively identify which data is the incremental data to be written. In another possible implementation of this specification, the incremental data to be written can be determined based on the change data capture method. All insert, update, and delete operations are read directly from the incremental log, and data changes are accurately captured based on all operations.

[0094] By applying the scheme of the embodiments of this specification, the log replay method reduces the problems of data accumulation and data interference by adding logs to be processed to different processing queues according to log type. Furthermore, by adding logs that can be replayed concurrently in the processing queue to the concurrent replay queue, multi-threaded concurrent replay of logs is realized, which improves the efficiency of log replay. Therefore, by obtaining the log replay result based on the log replay method and writing the incremental data to be written to the target database based on the log replay result, the efficiency of data transmission and writing is improved.

[0095] Corresponding to the above-described log playback method embodiments, this specification also provides log playback device embodiments. Figure 7 A schematic diagram of a log playback device according to one embodiment of this specification is shown. Figure 7 As shown, the device includes: The acquisition module 702 is configured to acquire the logs to be processed and the log types of the logs to be processed; The filtering module 704 is configured to filter the target processing queue from the pending queue according to the log type, and add the pending logs to the target processing queue to obtain the updated pending queue. The pending queue includes a first processing queue and a second processing queue. The first processing queue is used to store logs generated by the unique index table, and the second processing queue is used to store logs generated by the non-unique index table. The reading module 706 is configured to read the logs to be replayed from the updated pending queue and add the logs to be replayed to the concurrent replay queue, wherein the logs to be replayed are the logs that can be replayed concurrently in the updated pending queue; The first playback module 708 is configured to perform concurrent playback of concurrent playback logs in the concurrent playback queue.

[0096] Optionally, the filtering module 704 is further configured to read the primary key value from the log to be processed when the log type is a non-unique index table type; and to filter the target processing queue from the queue to be processed based on the primary key value, wherein the target processing queue is the second processing queue.

[0097] Optionally, the filtering module 704 is further configured to determine the log operations of the log to be processed when the log type is a unique index table type, wherein the log operations include at least one of insert operation, update operation, delete operation, and key-value modification operation; and to filter the target processing queue from the queue to be processed according to the log operations, wherein the target processing queue is the first processing queue.

[0098] Optionally, the filtering module 704 is further configured to block the target log operation when the log operation is a specified operation, and filter the target processing queue from the pending queue according to the log operation until the first log playback in the first processing queue is completed, and process the target log operation, wherein the specified operation includes at least one of a unique index value update operation and a deletion operation, and the target log operation is a log operation executed after the log operation.

[0099] Optionally, the reading module 706 is further configured to, if the first processing queue in the updated pending queue includes the first target log, determine the first log in the first processing queue as the log to be replayed, wherein the first target log is the first log whose log operation is the specified operation; and add the log to be replayed to the concurrent replay queue according to the concurrent replay log in the concurrent replay queue.

[0100] Optionally, the reading module 706 is further configured to, if the first target log is not included in the first processing queue in the updated pending queue, read the log to be replayed from the updated pending queue based on the log addition order of the updated pending queue, wherein the first target log is the first log for which the log operation is the specified operation; and add the log to be replayed to the concurrent replay queue according to the concurrent replay log in the concurrent replay queue.

[0101] Optionally, the reading module 706 is further configured to: obtain concurrent replay logs in the concurrent replay queue; determine the first queue identifier corresponding to the concurrent replay logs and determine the second queue identifier of the logs to be replayed; filter out the target logs to be replayed from the logs to be replayed based on the first queue identifier and the second queue identifier, wherein the second queue identifier of the target logs to be replayed is different from the first queue identifier; and add the target logs to be replayed to the concurrent replay queue.

[0102] Optionally, the first playback module 708 is further configured to determine the first queue identifier corresponding to the concurrent playback log in the concurrent playback queue; allocate the concurrent playback log to multiple log playback jobs according to the first queue identifier; and perform concurrent playback on the multiple log playback jobs.

[0103] Optionally, the acquisition module 702 is further configured to acquire logs to be processed; parse the logs to be processed to determine the log record information of the logs to be processed; and determine the log type corresponding to the logs to be processed based on the log record information.

[0104] Optionally, the device further includes a removal module configured to remove concurrent replay logs that have been completed in the concurrent replay queue.

[0105] By applying the solutions in the embodiments of this specification, the problems of data accumulation and data interference are reduced by adding logs to be processed to different processing queues according to log type. Furthermore, by adding logs that can be replayed concurrently from the processing queue to the concurrent replay queue, multi-threaded concurrent replay of logs is achieved, thereby improving log replay efficiency.

[0106] The above is an illustrative scheme of a log playback device according to this embodiment. It should be noted that the technical solution of this log playback device and the technical solution of the log playback method described above belong to the same concept. For details not described in detail in the technical solution of the log playback device, please refer to the description of the technical solution of the log playback method described above.

[0107] Corresponding to the above-described data transmission method embodiments, this specification also provides embodiments of data transmission devices. Figure 8 A schematic diagram of a data transmission apparatus according to one embodiment of this specification is shown. Figure 8As shown, the device includes: The parsing module 802 is configured to parse the incremental log during data transmission to determine the incremental data to be written. The second replay module 804 is configured to use the log replay method to replay the pending logs of incremental data to be written, and obtain the log replay results. The write module 806 is configured to write incremental data to the target database based on log replay results.

[0108] By applying the scheme of the embodiments of this specification, the log replay method reduces the problems of data accumulation and data interference by adding logs to be processed to different processing queues according to log type. Furthermore, by adding logs that can be replayed concurrently in the processing queue to the concurrent replay queue, multi-threaded concurrent replay of logs is realized, which improves the efficiency of log replay. Therefore, by obtaining the log replay result based on the log replay method and writing the incremental data to be written to the target database based on the log replay result, the efficiency of data transmission and writing is improved.

[0109] The above is an illustrative scheme of a data transmission device according to this embodiment. It should be noted that the technical solution of this data transmission device and the technical solution of the data transmission method described above belong to the same concept. For details not described in detail in the technical solution of the data transmission device, please refer to the description of the technical solution of the data transmission method described above.

[0110] Figure 9 A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0111] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 940 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0112] In one embodiment of this specification, the above-described components of the computing device 900 and Figure 9 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 9 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0113] The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 900 can also be a mobile or stationary server.

[0114] The processor 920 is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of the log playback method or data transmission method described above.

[0115] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the log playback method and the data transmission method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the log playback method or the data transmission method described above.

[0116] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described log playback method or data transmission method.

[0117] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the log playback method and the data transmission method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the log playback method or the data transmission method described above.

[0118] An embodiment of this specification also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps of the above-described log playback method or data transmission method.

[0119] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the log playback method and the data transmission method described above. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the log playback method or the data transmission method described above.

[0120] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0121] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0122] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0123] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0124] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A log replay method, comprising: Obtain the logs to be processed and the log type of the logs to be processed; According to the log type, a target processing queue is selected from the pending queue, and the pending logs are added to the target processing queue to obtain an updated pending queue. The pending queue includes a first processing queue and a second processing queue. The first processing queue is used to store logs generated by a unique index table, and the second processing queue is used to store logs generated by a non-unique index table. Read the logs to be replayed from the updated pending queue and add the logs to be replayed to the concurrent replay queue, wherein the logs to be replayed are the logs that can be replayed concurrently in the updated pending queue; Concurrent replay of the concurrent replay logs in the concurrent replay queue is performed.

2. The method according to claim 1, wherein selecting the target processing queue from the pending processing queue according to the log type includes: If the log type is a non-unique index table type, the primary key value is read from the log to be processed; Based on the primary key value, the target processing queue is selected from the queue to be processed, wherein the target processing queue is the second processing queue.

3. The method according to claim 1, wherein selecting the target processing queue from the pending processing queue according to the log type includes: When the log type is a unique index table type, the log operation of the log to be processed is determined, wherein the log operation includes at least one of the following: insert operation, update operation, delete operation, and key-value modification operation; Based on the log operation, the target processing queue is selected from the queue to be processed, wherein the target processing queue is the first processing queue.

4. The method according to claim 3, wherein filtering the target processing queue from the pending processing queue based on the log operation comprises: When the log operation is a specified operation, the target log operation is blocked, and the target processing queue is selected from the pending queue according to the log operation. The target log operation is processed until the first log playback in the first processing queue is completed. The specified operation includes at least one of a unique index value update operation and a deletion operation, and the target log operation is a log operation executed after the log operation.

5. The method according to claim 1, wherein reading the log to be replayed from the updated pending queue and adding the log to be replayed to the concurrent replay queue comprises: If the first processing queue in the updated pending queue includes a first target log, the first log in the first processing queue is determined as the log to be replayed, wherein the first target log is the first log whose log operation is a specified operation; The log to be replayed is added to the concurrent replay queue based on the concurrent replay log in the concurrent replay queue.

6. The method according to claim 1, wherein reading the log to be replayed from the updated pending queue and adding the log to be replayed to the concurrent replay queue comprises: If the first target log is not included in the first processing queue in the updated pending queue, the log to be replayed is read from the updated pending queue based on the log addition order of the updated pending queue, wherein the first target log is the first log for which the log operation is a specified operation; The log to be replayed is added to the concurrent replay queue based on the concurrent replay log in the concurrent replay queue.

7. The method according to claim 5 or 6, wherein adding the log to be replayed to the concurrent replay queue based on the concurrent replay log in the concurrent replay queue comprises: Retrieve the concurrent replay logs from the concurrent replay queue; Determine the first queue identifier corresponding to the concurrent replay log, and determine the second queue identifier of the log to be replayed; Based on the first queue identifier and the second queue identifier, target logs to be replayed are selected from the logs to be replayed, wherein the second queue identifier of the target logs to be replayed is different from the first queue identifier; Add the target log to be replayed to the concurrent replay queue.

8. The method according to claim 1, wherein concurrent replaying of the concurrent replay logs in the concurrent replay queue comprises: Determine the first queue identifier corresponding to the concurrent replay log in the concurrent replay queue; Based on the first queue identifier, the concurrent replay logs are distributed to multiple log replay jobs; The multiple log replay jobs are replayed concurrently.

9. The method according to claim 1, wherein obtaining the log to be processed and the log type of the log to be processed includes: Get the logs to be processed; Parse the log to be processed to determine the log record information of the log to be processed; Based on the log record information, determine the log type corresponding to the log to be processed.

10. The method according to any one of claims 1 to 9, wherein after performing concurrent replay on the concurrent replay logs in the concurrent replay queue, it further comprises: Remove the concurrent replay logs that have completed concurrent replay from the concurrent replay queue.

11. A data transmission method, comprising: During data transmission, the incremental log is parsed to determine the incremental data to be written. Using the method described in any one of claims 1 to 10, the log to be processed for writing incremental data is replayed to obtain a log replay result; Based on the log replay results, the incremental data to be written is written to the target database.

12. A log replay system, comprising a queue allocation unit and a concurrent replay unit; The queue allocation unit is used to obtain the logs to be processed and the log type of the logs to be processed; according to the log type, it filters the target processing queue from the queues to be processed, and adds the logs to be processed to the target processing queues to obtain the updated queues to be processed, wherein... The queue to be processed includes a first processing queue and a second processing queue. The first processing queue is used to store logs generated by a unique index table, and the second processing queue is used to store logs generated by a non-unique index table. The concurrent replay unit is used to read concurrent replay logs from the updated pending queue and add the concurrent replay logs to the concurrent replay queue, wherein the logs to be replayed are the logs in the updated pending queue that can be replayed concurrently; and to perform concurrent replay of the concurrent replay logs in the concurrent replay queue.

13. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

Images