Data transmission method, architecture, apparatus, and storage medium

Abstract

The application relates to a data transmission method, architecture, device and storage medium. The method comprises the following steps: reading operation records from a log of a source database; storing the logical time numbers of target operation records meeting a set condition in a set cache area, and executing database operations corresponding to the target operation records on a target database; the set condition refers to that a database operation corresponding to an operation record is normally completed, and the logical time number represents the execution sequence of the database operation corresponding to the operation record; when it is determined that the database operation corresponding to the target operation record is successfully executed on the target database, the logical time number corresponding to the target operation record is deleted from the cache area; when it is determined that breakpoint continuation is performed, the earliest logical time number representing the execution sequence in the cache area is used to determine a breakpoint position in the log, and operation records are read from the breakpoint position in the log and all operations after the operation records are read from the log of the source database are executed. Therefore, breakpoint continuation can be realized, and no data loss occurs.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a data transmission method, architecture, apparatus, and storage medium. Background Technology

[0002] Incremental migration refers to the real-time synchronization of incremental data generated in the source database to the target database. If the migration task is abnormally interrupted during the incremental migration process, a breakpoint resume mechanism needs to be enabled to ensure the continued stable operation of the migration task.

[0003] Currently, the common method for resuming interrupted data transfers in DTS (Data Transmission Service) is to use the minimum application point value of each thread on the write end as the breakpoint position, which is also the starting point for reloading the migration task. Here, the application point refers to the position in the source database log of the last successfully executed incremental data to the target database before the migration task was abnormally interrupted. For example, assuming there are two threads on the write end performing database operations on the target database in parallel, and assuming that the application point of one thread is 10 and the application point of the other thread is 8, according to the above description, 8 would be determined as the starting point for reloading the migration task.

[0004] However, the breakpoint resume method described above is only applicable to scenarios where the source database log is a binlog log. In scenarios where the source database log is a redo log, applying the above breakpoint resume method may result in data loss. Summary of the Invention

[0005] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this application provides a data transmission method, architecture, apparatus and storage medium.

[0006] Firstly, this application provides a data transmission method, the method comprising:

[0007] Read operation records from the log of the source database, the log including operation records of multiple database operations performed on the source database;

[0008] The logical timestamp of the target operation record that meets the set conditions is stored in the set cache area, and the database operation corresponding to the target operation record is executed on the target database; the set conditions refer to the normal completion of the database operation corresponding to the operation record, and the logical timestamp represents the execution order of the database operation corresponding to the operation record.

[0009] When it is determined that the database operation corresponding to the target operation record has been successfully executed on the target database, the logical time number corresponding to the target operation record is deleted from the cache.

[0010] When it is determined to resume interrupted download, the breakpoint position is determined from the log based on the earliest logical time number represented in the cache, and the operation record is read from the breakpoint position in the log, and all operations after reading the operation record from the log of the source database are executed.

[0011] In one possible implementation, storing the logical time number of the read target operation record that meets the set conditions into a set buffer includes:

[0012] Determine the operation type corresponding to the currently read operation record;

[0013] If the operation type is a DDL operation or a DML operation, the logical time number of the currently read operation record is stored in the set cache area.

[0014] If the operation type is a database transaction rollback operation, then the logical timestamps of all operation records corresponding to the current database transaction are deleted from the cache. The current database transaction is the database transaction to which the currently read operation record belongs.

[0015] In one possible implementation, the method further includes:

[0016] If the operation type corresponding to the currently read operation record is a DDL operation or a DML operation, then record the physical time number of the currently read operation record and the number of times the physical time number appears, n. The physical time number represents the execution order of the database operation corresponding to the operation record.

[0017] Determining the logical time number of the currently read operation record specifically includes:

[0018] Compare the physical time number of the currently read operation record with the physical time number of the previously read operation record to see if they are consistent.

[0019] If they are inconsistent, a key-value pair is determined as the logical time number of the currently read operation record based on the physical time number of the operation record. The key and value in the logical time number are both the physical time number.

[0020] If they match, a key-value pair is determined as the logical time number of the currently read operation record based on the physical time number of the operation record and the n corresponding to the previously read operation record. The key in the logical time number is the sum of n and 1, and the value is the physical time number.

[0021] In one possible implementation, determining the breakpoint location from the log based on the earliest logical time number representing the execution order in the cache includes:

[0022] The smallest value among the logical time numbers in the cache is determined as the target physical time number;

[0023] The operation record with the target physical time number in the log is identified as the breakpoint location.

[0024] In one possible implementation, determining to resume data transmission from a breakpoint includes:

[0025] When the set resume transmission period is reached, data resume transmission will be initiated.

[0026] In one possible implementation, the source database is an Oracle database, and the log is a redo log.

[0027] Secondly, this application provides a data transmission architecture, which includes a reading source end, a writing end, and a first message queue, wherein the reading source end is provided with a buffer.

[0028] The reading source reads operation records from the log of the source database, the log including operation records of multiple database operations performed on the source database;

[0029] The reading source end also stores the logical time number of the target operation record that meets the set conditions into the set cache area. The set conditions refer to the normal completion of the database operation corresponding to the operation record, and the logical time number represents the execution order of the database operation corresponding to the operation record.

[0030] The writing end performs the database operation corresponding to the target operation record on the target database;

[0031] The writing end also stores the logical time number of the target operation record into the first message queue when it determines that the database operation corresponding to the target operation record has been successfully executed on the target database.

[0032] The reading source end also deletes the logical time number stored in the first message queue from the cache area;

[0033] When determining to resume interrupted transmission, the source end determines the breakpoint position from the log based on the earliest logical time number represented in the cache, and starts reading operation records from the breakpoint position in the log, and executes all operations after reading the operation records from the log of the source database.

[0034] In one possible implementation, the writing end starts multiple threads to perform database operations corresponding to the target operation record on the target database.

[0035] Thirdly, this application provides a data transmission apparatus, the apparatus comprising:

[0036] A reading module is used to read operation records from the logs of the source database, the logs including operation records of multiple database operations performed on the source database;

[0037] The caching module is used to store the logical timestamps of target operation records that meet set conditions into a set cache area; the set conditions refer to the normal completion of the database operation corresponding to the operation record, and the logical timestamps represent the execution order of the database operations corresponding to the operation record.

[0038] The synchronization module is used to perform the database operations corresponding to the target operation record on the target database;

[0039] The deletion module is used to delete the logical time number corresponding to the target operation record from the cache when it is determined that the database operation corresponding to the target operation record has been successfully executed on the target database.

[0040] The breakpoint resume module is used to determine the breakpoint position from the log based on the earliest logical time number represented in the execution order in the cache when it is determined to resume the breakpoint, and to start reading operation records from the breakpoint position in the log and execute all operations after reading the operation records from the log of the source database.

[0041] Fourthly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0042] Memory, used to store computer programs;

[0043] A processor, when executing a program stored in memory, implements the steps of the data transfer method described in any of the first aspects.

[0044] Fifthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data transmission method as described in any of the first aspects.

[0045] The technical solutions provided in this application have the following advantages compared with the prior art:

[0046] The technical solution provided in this application reads operation records from the logs of the source database, stores the logical time number of the target operation record that meets the set conditions in a set cache, and executes the database operation corresponding to the target operation record on the target database. The set conditions refer to the normal completion of the database operation corresponding to the operation record, and the logical time number represents the execution order of the database operation corresponding to the operation record. When it is determined that the database operation corresponding to the target operation record has been successfully executed on the target database, the logical time number corresponding to the target operation record is deleted from the cache. Thus, the cache ultimately stores the logical time numbers of database operations that were successfully completed on the source database side but failed to be successfully executed on the target database side, that is, the logical time numbers of database operations that were not successfully synchronized. Therefore, when it is determined to perform breakpoint resume, the breakpoint position can be determined from the log based on the logical time number in the cache, that is, the starting point of the migration reload task can be determined. Furthermore, since the breakpoint position is determined from the log based on the earliest logical time number represented by the execution order in the cache, data loss can be effectively avoided, and data loss can also be effectively avoided in the scenario where the log of the source database is a redo log. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating a data transmission method provided in an embodiment of this application;

[0050] Figure 2 A schematic diagram of a data transmission architecture provided in an embodiment of this application;

[0051] Figure 3 A flowchart illustrating another data transmission method provided in an embodiment of this application;

[0052] Figure 4 A schematic diagram of a data transmission device provided in an embodiment of this application;

[0053] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] In one example, suppose that during the time interval between the start operation of database transaction A (hereinafter referred to as database operation A1) and the commit operation of database transaction A (hereinafter referred to as database operation A3), the following operations are executed in sequence: database transaction A's DDL or DML operation (hereinafter referred to as database operation A2), database transaction B (including database transaction B's start operation B1, DDL or DML operation B2, and commit operation B3), and database transaction C (including database transaction C's start operation C1, DDL or DML operation C2, and commit operation C3).

[0056] In this example, since the binlog records database operations in the order of database transaction commits, the binlog records B1, B2, B3, C1, C2, C3, A1, A2, and A3 in that order.

[0057] Assume that during incremental migration, two threads (hereinafter referred to as thread 1 and thread 2) on the DTS write end execute database operations on the target database in parallel. Also assume that the migration task is interrupted when thread 1 executes database operation A2 and thread 2 executes database operation C2. Using existing breakpoint resumption methods, since the application point C1 of thread 2 is smaller than the application point A1 of thread 1, the application point C1 of thread 2 is determined as the starting point for reloading the migration task. That is, during breakpoint resumption, the resumed data will start from C1. Therefore, in scenarios where the source database log is a binlog log, applying the existing breakpoint resumption method will not cause the loss of database operations in database transaction A.

[0058] In this example, since the redo log records database operations in the exact order of execution, the redo log records A1, A2, B1, B2, B3, C1, C2, C3, and A3 in that order.

[0059] Assume that during incremental migration, two threads (hereinafter referred to as thread 1 and thread 2) on the DTS write end execute database operations on the target database in parallel. Also assume that the migration task is interrupted when thread 1 executes database operation B2 and thread 2 executes database operation C2. According to the existing breakpoint resumption method, since the application point B1 of thread 1 is smaller than the application point C1 of thread 2, the application point B1 of thread 1 is determined as the starting point for reloading the migration task; that is, during breakpoint resumption, the resumption begins from B1.

[0060] However, in the Redo log, there are database operations A1 and A2 of database transaction A before B1. Therefore, if B1 is used as the starting point for the migration task reload, some database operations of database transaction A will be lost.

[0061] To address the issue that data loss is easily caused by using breakpoint resume methods in existing technologies when the source database logs are redo logs, this application provides a data transmission method.

[0062] The data transmission method provided in this application will be further explained below with reference to specific embodiments. These embodiments do not constitute a limitation on the technical solution of this application.

[0063] Figure 1 This is a flowchart illustrating a data transmission method provided in an embodiment of this application. Figure 1 As shown, it includes the following steps:

[0064] Step 101: Read the operation records from the source database log. The log includes operation records of multiple database operations performed on the source database.

[0065] As an example application scenario, the source database mentioned above is an Oracle database, and the corresponding log mentioned above is a redo log. In this example application scenario, the source database log includes operation records of multiple database operations performed on the source database, and the operation records of multiple database operations are arranged sequentially according to the execution order. This also means that, regardless of whether it is a committed database transaction or a rolled-back database transaction, the sequence of database operations (containing multiple database operations) contained in the database transaction is recorded in the source database log.

[0066] As an example, operation records can be read sequentially from the logs of the source database, with one operation record read at a time.

[0067] Step 102: Store the logical time number of the target operation record that meets the set conditions into the set cache area, and perform the database operation corresponding to the target operation record on the target database.

[0068] The above conditions assume that the database operation corresponding to the operation record is completed normally. Here, "normal completion" means that the database transaction to which the database operation belongs is eventually committed. It's important to note that a database transaction is a user-defined sequence of database operations and possesses atomicity. Atomicity means that a database transaction is an indivisible unit of work; either all database operations within it are executed, or none are executed. Therefore, if a database transaction is committed, it means that all database operations contained within that transaction have been completed normally. Conversely, if a database transaction is rolled back, it means that all database operations contained within that transaction have been reversed, i.e., it did not complete normally.

[0069] Furthermore, since the database operation corresponding to the target operation record was completed normally, it means that the database operation corresponding to the target operation record was successfully executed on the source database. In order to achieve data migration, the same database operation as the source database can be executed on the target database. Therefore, in this step 102, the database operation corresponding to the target operation record is executed on the target database.

[0070] Furthermore, based on the above description, normal completion means that the database transaction to which the database operation belongs is finally committed. Therefore, in this embodiment, operation records are read sequentially from the source database log. For each operation record read, the operation type corresponding to the currently read record is determined. If the operation type is a database transaction start operation, a DDL (Data Definition Language) operation, a DML (Data Manipulation Language) operation, or a database transaction rollback operation, the next operation record is read. If the operation type is a database transaction commit operation, all database operations contained in the current database transaction are determined as target operation records that meet the set conditions. Then, the database operation corresponding to the target operation record is executed on the target database. Here, the current database transaction refers to the database transaction to which the database operation corresponding to the currently read operation record belongs.

[0071] The aforementioned logical timestamps represent the execution order of the database operations corresponding to the operation records. Specifically, in practice, each operation record in the redo log corresponds to a timestamp (hereinafter referred to as the physical timestamp). This physical timestamp can represent the execution order of the database operations corresponding to the operation record; that is, the order in which database operations are executed can be distinguished by the physical timestamp. However, in practice, there may be situations where multiple database operations are executed simultaneously. In this case, multiple database operations correspond to the same physical timestamp. Therefore, it is clear that the physical timestamp alone cannot distinguish between multiple database operations executing simultaneously.

[0072] Based on this, the concept of a logical time number is introduced in this application embodiment to represent the execution order of database operations corresponding to an operation record. Furthermore, when multiple database operations are executed simultaneously, the logical time number can also distinguish between them. How the logical time number of the operation record is determined will be explained below. Figure 3 The illustrated embodiments are described in detail, but will not be elaborated upon here.

[0073] As can be seen from the description of step 102, in this embodiment of the application, the logical time number of the target operation record that is read and whose database operation is completed normally is first stored in the set cache area. At this time, the cache area stores the logical time number of the database operation that is completed normally on the source database side.

[0074] As an example, the following process can be used to store the logical timestamp of the target operation record corresponding to the normally completed database operation into a set cache: Operation records are read sequentially from the source database log. For each operation record read, the operation type is determined. If the operation type is a database transaction start operation, the next operation record is read. If the operation type is a DDL or DML operation, the logical timestamp of the currently read operation record is stored in the set cache, and the next operation record is read. If the operation type is a database transaction commit operation, the next operation record is read. If the operation type is a database transaction rollback operation, as described above, a database transaction rollback means that all database operations contained in the database transaction have not been completed normally. In this case, the logical timestamps of all operation records corresponding to the current database transaction can be deleted from the cache. Here, the current database transaction refers to the database transaction to which the database operation corresponding to the currently read operation record belongs.

[0075] Step 103: When it is determined that the database operation corresponding to the target operation record has been successfully executed on the target database, delete the logical timestamp corresponding to the target operation record from the cache.

[0076] It is understandable that by executing step 103, when it is determined that the database operation corresponding to the target operation record has been successfully executed on the target database, the logical time number corresponding to the target operation record is deleted from the cache. This allows the cache to ultimately store the logical time numbers of database operations that were successfully completed on the source database side but failed to be successfully executed on the target database side.

[0077] Step 104: When it is determined to resume interrupted download, determine the breakpoint position from the log based on the earliest logical time number represented in the cache, and start reading the operation record from the breakpoint position in the log, and return to execute step 102.

[0078] In practice, database operations that are completed normally on the source database side but fail to be executed successfully on the target database side are precisely the database operations that need to be executed again on the target database. Furthermore, in order to avoid data loss, when it is determined to resume interrupted transmission, the breakpoint position can be determined from the log based on the earliest logical time number represented in the execution order in the cache area. Then, the operation record is read from the breakpoint position in the log, and the execution step 102 is returned, thereby realizing the resumed transmission.

[0079] The technical solution provided in this application reads operation records from the logs of the source database, stores the logical time number of the target operation record that meets the set conditions in a set cache, and executes the database operation corresponding to the target operation record on the target database. The set conditions refer to the normal completion of the database operation corresponding to the operation record, and the logical time number represents the execution order of the database operation corresponding to the operation record. When it is determined that the database operation corresponding to the target operation record has been successfully executed on the target database, the logical time number corresponding to the target operation record is deleted from the cache. Thus, the cache ultimately stores the logical time numbers of database operations that were successfully completed on the source database side but failed to be successfully executed on the target database side, that is, the logical time numbers of database operations that were not successfully synchronized. Therefore, when it is determined to perform breakpoint resume, the breakpoint position can be determined from the log based on the logical time number in the cache, that is, the starting point of the migration reload task can be determined. Furthermore, since the breakpoint position is determined from the log based on the earliest logical time number represented by the execution order in the cache, data loss can be effectively avoided, and data loss can also be effectively avoided in the scenario where the log of the source database is a redo log.

[0080] Figure 2 This is a schematic diagram of a data transmission architecture provided in an embodiment of this application. For example... Figure 2 As shown, the data transmission architecture mainly consists of four parts: the reading source 21, the writing end 22, the formatter 23, and the first message queue 24.

[0081] Among them, the source read terminal 21 is used to perform operations related to the source database, such as connecting to the source database and reading the source database logs.

[0082] The write end 22 is used to perform operations related to the target database, such as connecting to the target database and performing database operations on the target database.

[0083] Formatter 23 is used for format conversion in heterogeneous data migration scenarios. For example, the data types of Oracle databases and MySQL databases are not the same, and the formatter needs to be used to convert the format of the data that needs to be synchronized between Oracle databases and MySQL databases.

[0084] The first message queue 24 serves as a communication bridge between the reading source and the writing end. Here, the communication includes the reading source sending the database operation successfully executed on the source database to the writing end.

[0085] In addition, the source end 21 is equipped with a cache area to store the logical timestamps of database operations performed on the source database.

[0086] The data transmission architecture described above also includes a second message queue (not shown in the figure), which is used by the writer to send the logical timestamp of the database operation successfully executed on the target database to the reader.

[0087] The following is combined Figure 3 The process shown is for Figure 2 The workflow of each part and the interaction between them in the data transmission architecture shown are described in detail. Figure 3 As shown, it includes the following steps:

[0088] Step 301: The source end reads the operation records from the log of the source database.

[0089] In this embodiment of the application, the reading source can sequentially read operation records from the log of the source database. For each operation record read, the following steps are executed.

[0090] Step 302: The source end determines the operation type corresponding to the currently read operation record; if the operation type is a database transaction start operation, then return to execute step 301; if the operation type is a DDL operation or DML operation, then execute step 303; if the operation type is a database transaction commit operation, then execute step 308; if the operation type is a database transaction rollback operation, then execute step 310.

[0091] In this embodiment of the application, each time the reading source reads an operation record, it determines the operation type corresponding to the currently read operation record and performs different processing according to the different operation types.

[0092] When the source end determines that the operation type of the currently read operation record is a database transaction start operation, since it does not need to perform a database transaction start operation on the target database, it does not process the currently read operation record and continues to read the next operation record from the source database log.

[0093] When the source end determines that the operation type of the currently read operation record is a DDL operation or a DML operation, it can determine the logical time number of the currently read operation record and store the logical time number of the currently read operation record in the buffer. For a detailed description of this process, please refer to the description of steps 303 to 307 below.

[0094] When the read source determines that the operation type of the currently read operation record is a database transaction commit operation, it can determine that the source database has successfully completed all database operations included in the current database transaction. Therefore, the read source can send all database operations included in the current database transaction to the write end, so that the write end can execute all database operations included in the current database transaction on the target database. Here, the current database transaction refers to the database transaction to which the database operation corresponding to the currently read operation record belongs. For a detailed description of this process, please refer to the descriptions of steps 308, 309, and 311 below.

[0095] When the source end determines that the operation type of the currently read operation record is a database transaction rollback operation, it can determine that the source database has not properly completed all database operations included in the current database transaction. Therefore, the source end can delete the logical timestamps corresponding to all database operations included in the current database transaction from the aforementioned cache. A detailed description of this process is provided in step 310 below.

[0096] Step 303: The source end records the physical time number of the currently read operation record and the number of times the physical time number appears, n.

[0097] In practice, if only one database operation is executed at a given moment, the physical time number corresponding to that database operation will only have one possible value: 1. However, if multiple database operations are executed at a given moment, the number of occurrences of the physical time numbers corresponding to those multiple database operations can have multiple possible values. For example, suppose four database operations are executed at a given moment. When the read source reads the operation records of these four database operations sequentially, the recorded occurrence counts will be 1, 2, 3, and 4, respectively.

[0098] Step 304: The source end compares the physical time number of the currently read operation record with the physical time number of the previously read operation record. If they are not consistent, proceed to step 305; if they are consistent, proceed to step 306.

[0099] Step 305: The source end determines a key-value pair as the logical time number of the currently read operation record based on the physical time number of the currently read operation record. The key and value in the logical time number are both physical time numbers; proceed to step 307.

[0100] Step 306: The source end determines a key-value pair as the logical time number of the currently read operation record based on the physical time number of the currently read operation record and the previously recorded n corresponding to the operation record. The key in the logical time number is the sum of n and 1, and the value is the physical time number.

[0101] If the physical timestamp of the currently read operation record is inconsistent with the physical timestamp of the previously read operation record, it means that the database operation corresponding to the currently read operation record was not executed simultaneously with the database operation corresponding to the previously read operation record. In this case, the physical timestamp can be used to distinguish the database operation records executed at different times. Therefore, if the physical timestamp of the currently read operation record is inconsistent with the physical timestamp of the previously read operation record, a key-value pair can be determined as the logical timestamp of the currently read operation record based on the physical timestamp of the currently read operation record, where both the key and value in the logical timestamp are physical timestamps.

[0102] Conversely, if the physical time number of the currently read operation record is the same as the physical time number of the previously read operation record, it means that the database operation corresponding to the currently read operation record was executed simultaneously with the database operation corresponding to the previously read operation record. In this case, the physical time number alone cannot distinguish between database operations executed at different times. Therefore, if the physical time number of the currently read operation record is the same as the physical time number of the previously read operation record, a key-value pair can be determined as the logical time number of the currently read operation record based on the physical time number of the currently read operation record and the n corresponding to the previously read operation record. Here, the key in the logical time number is the sum of n and 1, and the value is the physical time number.

[0103] For example, suppose four database operations are executed at a certain moment. When the read source reads the operation records of these four database operations sequentially, the occurrence counts are 1, 2, 3, and 4, respectively. The resulting logical time numbers are [1]-t, [2]-t, [3]-t, and [4]-t, respectively, where t is the physical time number. It should be noted that the default occurrence count is 0. Thus, when the read source reads the operation record of the first database operation among the four database operations, it determines a key-value pair as the logical time number of the operation record based on the physical time number and the default occurrence count of 0. The key of this logical time number is the sum of 0 and 1, which is 1.

[0104] Therefore, it can be seen that even if the physical time number of the currently read operation record is the same as the physical time number of the previously read operation record, the logical time numbers of the two operation records are also different. Thus, the database operations executed at the same time can be distinguished by the logical time number.

[0105] Step 307: The source end stores the logical time number of the currently read operation record into the set buffer area and executes step 314.

[0106] Step 308: The source end sends the operation records of all database operations included in the current database transaction and their corresponding logical timestamps to the formatter.

[0107] Step 309: The formatter performs format conversion on the received operation record and writes the format-converted operation record and the corresponding logical time number into the first message queue.

[0108] Step 310: The reading source deletes the logical timestamps of all operation records corresponding to the current database transaction from the cache.

[0109] If the operation type is a database transaction rollback operation, combine the above... Figure 1 As described in the illustrated embodiment, database transaction rollback means that all database operations contained in the database transaction have not been completed normally. In this case, the logical timestamps of all operation records corresponding to the current database transaction can be deleted from the cache.

[0110] Step 311: The writing end retrieves the operation record from the first message queue and executes the database operation corresponding to the retrieved operation record on the target database.

[0111] As described above, the read source can send operation records to the write end through a producer-consumer mechanism. After retrieving the operation records from the first message queue, the write end can execute database operations on the target database by starting multiple threads. By starting multiple threads to execute database operations on the target database, the efficiency of data synchronization can be improved.

[0112] Step 312: When the writing end determines that the database operation has been successfully executed on the target database, it stores the logical time number corresponding to the database operation into the second message queue.

[0113] As described in steps 308 and 309 above, the reading source also sends the logical time number corresponding to the operation record to the writing end. Therefore, the writing end can obtain the logical time number corresponding to the operation record. Consequently, when the writing end determines that the database operation has been successfully executed on the target database, it stores the logical time number corresponding to the database operation into the second message queue.

[0114] Step 313: The source end retrieves the logical time number from the second message queue and deletes the retrieved logical time number from the cache.

[0115] In this embodiment, the reading source can start a thread to listen to the second message queue. When a logical time number is detected in the second message queue, the logical time number can be retrieved from the second message queue and deleted from the cache. This achieves the goal of the reading source deleting the logical time number corresponding to the database operation successfully executed on the target database from the cache.

[0116] It should be noted that since logical timestamps can distinguish between database operations executed simultaneously, when a database operation has been completed on the target database, only the logical timestamp corresponding to that database operation can be deleted from the cache. Compared to only caching the physical timestamp, this avoids deleting the logical timestamps of other database operations that occurred at the same time from the cache, thus preventing errors when determining the breakpoint location later.

[0117] Step 314: When the source end determines to resume interrupted transmission, it determines the breakpoint position from the log based on the earliest logical time number represented in the cache, and starts reading operation records from the breakpoint position in the log, and executes all operations after reading the operation records from the source database log.

[0118] As an example, the source end can start a timer, and determine to resume the interrupted transmission every time the timeout period is reached, for example, 10 seconds. Through this process, periodic interrupted transmission can be achieved, that is, the source end determines to resume the data interrupted transmission when it determines that the set transmission period has been reached.

[0119] As another embodiment, in Figure 2 The illustrated data transfer architecture also includes a task manager, to which signals from other components, such as the write end, can be sent. Based on this, when a write exception occurs—that is, when an exception occurs while performing a database operation on the target database—the write end can send a signal to the task manager indicating that the migration task has encountered an error. Furthermore, upon receiving this signal, the task manager can send the signal to the read end, thus allowing the read end to determine whether to resume the interrupted transfer upon receiving the signal.

[0120] As described in steps 305 and 306 above, the value in the logical time number represents the physical time number, and the physical time number represents the execution order of database operations. Therefore, the smaller the value in the logical time number, the earlier the execution order of the database operation corresponding to that logical time number is, that is, the earlier the database operation is executed. Based on this, in this embodiment of the application, when the reading source determines to perform breakpoint resumption, it can determine the logical time number with the smallest value in the logical time number in the cache as the logical time number representing the earliest execution order. Furthermore, it determines the value in the logical time number, that is, the smallest value in the logical time number in the cache, as the target physical time number, and determines the operation record with the target physical time number in the log as the breakpoint position.

[0121] To enable those skilled in the art to more intuitively understand the technical problems that the embodiments of this application can solve, the following examples are shown:

[0122] In the example where A1, A2, B1, B2, B3, C1, C2, C3, and A3 are recorded sequentially in the Redo log, under normal circumstances, the logical timestamps corresponding to A2, B2, and C2 will be stored sequentially in the buffer of the read source, and the write end of DTS will execute database transactions B, C, and A sequentially to the target database.

[0123] Suppose the writing end successfully executes database transaction B to the target database, and a migration task is abnormally interrupted during the execution of database transaction C. According to the data transmission method provided in this application embodiment, the buffer stores the logical timestamps corresponding to A2 and C2. Since the execution order of A2 is earlier than that of C2, A2 is determined as the breakpoint position. That is, the operation record is read from the position corresponding to A2 in the log, and all operations after reading the operation record from the source database log are executed. Therefore, it can be seen that applying the data transmission method provided in this application embodiment can avoid the loss of some data operations in database transaction A.

[0124] Figure 4 This is a schematic diagram of a data transmission device provided in an embodiment of this application. Figure 4 As shown, it includes:

[0125] The reading module 41 is used to read operation records from the log of the source database, the log including operation records of multiple database operations performed on the source database;

[0126] The caching module 42 is used to store the logical time number of the target operation record that meets the set conditions into the set cache area; the set conditions refer to the normal completion of the database operation corresponding to the operation record, and the logical time number represents the execution order of the database operation corresponding to the operation record.

[0127] Synchronization module 43 is used to perform database operations corresponding to the target operation record on the target database;

[0128] The deletion module 44 is used to delete the logical time number corresponding to the target operation record from the cache when it is determined that the database operation corresponding to the target operation record has been successfully executed on the target database.

[0129] The breakpoint resume module 45 is used to determine the breakpoint position from the log based on the earliest logical time number represented in the execution order in the cache when it is determined to resume the breakpoint, and to start reading operation records from the breakpoint position in the log and execute all operations after reading the operation records from the log of the source database.

[0130] In one possible implementation, the cache module 42 includes (not shown in the figure):

[0131] The type determination submodule is used to determine the operation type corresponding to the currently read operation record;

[0132] The storage submodule is used to store the logical time number of the currently read operation record into a set cache if the operation type is a DDL operation or a DML operation.

[0133] The deletion module 44 is further configured to delete the logical timestamps of all operation records corresponding to the current database transaction from the cache if the operation type is a database transaction rollback operation, wherein the current database transaction is the database transaction to which the currently read operation record belongs.

[0134] In one possible implementation, the device further includes (not shown in the figures):

[0135] The recording module is used to record the physical time number of the currently read operation record and the number of times the physical time number appears, n, if the operation type corresponding to the currently read operation record is a DDL operation or a DML operation. The physical time number represents the execution order of the database operation corresponding to the operation record.

[0136] The determination module is used to determine the logical time number of the currently read operation record, specifically including:

[0137] Compare the physical time number of the currently read operation record with the physical time number of the previously read operation record to see if they are consistent.

[0138] If they are inconsistent, a key-value pair is determined as the logical time number of the currently read operation record based on the physical time number of the operation record. The key and value in the logical time number are both the physical time number.

[0139] If they match, a key-value pair is determined as the logical time number of the currently read operation record based on the physical time number of the operation record and the n corresponding to the previously read operation record. The key in the logical time number is the sum of n and 1, and the value is the physical time number.

[0140] In one possible implementation, the breakpoint resume module 45 is specifically used for:

[0141] The smallest logical time number in the cache is determined as the target physical time number; the operation record in the log with the target physical time number is determined as the breakpoint location.

[0142] In one possible implementation, the breakpoint resume module 45 is specifically used for:

[0143] When the set resume transmission period is reached, data resume transmission will be initiated.

[0144] In one possible implementation, the source database is an Oracle database, and the log is a redo log.

[0145] like Figure 5 As shown in the figure, this application provides an electronic device, including a processor 511, a communication interface 512, a memory 513, and a communication bus 514, wherein the processor 511, the communication interface 512, and the memory 513 communicate with each other through the communication bus 514.

[0146] Memory 513 is used to store computer programs;

[0147] In one embodiment of this application, when the processor 511 executes a program stored in the memory 513, it implements the data transmission method provided in any of the foregoing method embodiments, including:

[0148] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the data transmission method provided in any of the foregoing method embodiments.

[0149] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0150] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data transmission method, characterized in that, The method includes: Read operation records from the log of the source database, the log including operation records of multiple database operations performed on the source database; The logical time number of the target operation record that meets the set conditions is stored in the set cache area, and the database operation corresponding to the target operation record is executed on the target database. The set conditions refer to the normal completion of the database operation corresponding to the operation record. The logical time number is an identifier generated based on the physical time number of the operation record and its order of appearance in operation records with the same physical time number, which is used to uniquely represent the execution order of the database operation corresponding to the operation record. When it is determined that the database operation corresponding to the target operation record has been successfully executed on the target database, the logical time number corresponding to the target operation record is deleted from the cache. When it is determined to resume interrupted download, the breakpoint position is determined from the log based on the earliest logical time number represented in the cache, and the operation record is read from the breakpoint position in the log, and all operations after reading the operation record from the log of the source database are executed.

2. The method according to claim 1, characterized in that, The step of storing the logical time number of the target operation record that meets the set conditions into the set cache area includes: Determine the operation type corresponding to the currently read operation record; If the operation type is a Database Definition Language (DDL) operation or a Database Manipulation Language (DML) operation, then the logical timestamp of the currently read operation record is stored in the set cache area. If the operation type is a database transaction rollback operation, then the logical timestamps of all operation records corresponding to the current database transaction are deleted from the cache. The current database transaction is the database transaction to which the currently read operation record belongs.

3. The method according to claim 2, characterized in that, The method further includes: If the operation type corresponding to the currently read operation record is a DDL operation or a DML operation, then record the physical time number of the currently read operation record and the number of times the physical time number appears, n. The physical time number represents the execution order of the database operation corresponding to the operation record. Determining the logical time number of the currently read operation record specifically includes: Compare the physical time number of the currently read operation record with the physical time number of the previously read operation record to see if they are consistent. If they are inconsistent, a key-value pair is determined as the logical time number of the currently read operation record based on the physical time number of the operation record. The key and value in the logical time number are both the physical time number. If they match, a key-value pair is determined as the logical time number of the currently read operation record based on the physical time number of the operation record and the n corresponding to the previously read operation record. The key in the logical time number is the sum of n and 1, and the value is the physical time number.

4. The method according to claim 3, characterized in that, The step of determining the breakpoint location from the log based on the earliest logical time number representing the execution order in the cache includes: The smallest value among the logical time numbers in the cache is determined as the target physical time number; The operation record with the target physical time number in the log is identified as the breakpoint location.

5. The method according to claim 1, characterized in that, The determination to resume data transmission from a breakpoint includes: When the set resume transmission period is reached, data resume transmission will be initiated.

6. The method according to any one of claims 1 to 5, characterized in that, The source database is an Oracle database, and the log is a Redo log.

7. A data transmission architecture, characterized in that, The architecture includes a read source end, a write end, and a first message queue, and the read source end is equipped with a cache area; The reading source reads operation records from the log of the source database, the log including operation records of multiple database operations performed on the source database; The reading source end also stores the logical time number of the target operation record that meets the set conditions into the set cache area, and writes the target operation record into the first message queue. The set conditions refer to the normal completion of the database operation corresponding to the operation record. The logical time number is an identifier generated based on the physical time number of the operation record and its order of appearance in operation records with the same physical time number, which is used to uniquely represent the execution order of the database operation corresponding to the operation record. The writing end retrieves the target operation record from the first message queue and executes the database operation corresponding to the target operation record on the target database; The reading source end also deletes the logical time number of the target operation record from the cache when it determines that the writing end has successfully executed the database operation corresponding to the target operation record on the target database. When determining to resume interrupted transmission, the source end determines the breakpoint position from the log based on the earliest logical time number represented in the cache, and starts reading operation records from the breakpoint position in the log, and executes all operations after reading the operation records from the log of the source database.

8. The architecture according to claim 7, characterized in that, The architecture also includes: a second message queue; The writing end also writes the logical time number of the target operation record into the second message queue when it determines that the database operation corresponding to the target operation record has been successfully executed on the target database. The reading source obtains a logical time number from the second message queue and deletes the obtained logical time number from the cache.

9. A data transmission device, characterized in that, The device includes: A reading module is used to read operation records from the logs of the source database, the logs including operation records of multiple database operations performed on the source database; The caching module is used to store the logical time number of the target operation record that meets the set conditions into the set cache area; the set conditions refer to the normal completion of the database operation corresponding to the operation record; the logical time number is an identifier generated based on the physical time number of the operation record and its order of appearance in operation records with the same physical time number, which is used to uniquely represent the execution order of the database operation corresponding to the operation record. The synchronization module is used to perform the database operations corresponding to the target operation record on the target database; The deletion module is used to delete the logical time number corresponding to the target operation record from the cache when it is determined that the database operation corresponding to the target operation record has been successfully executed on the target database. The breakpoint resume module is used to determine the breakpoint position from the log based on the earliest logical time number represented in the execution order in the cache when it is determined to resume the breakpoint, and to start reading operation records from the breakpoint position in the log and execute all operations after reading the operation records from the log of the source database.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the data transmission method according to any one of claims 1-6.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data transmission method as described in any one of claims 1-6.

Images