Transaction processing method and device, equipment and storage medium

Abstract

The invention relates to a transaction processing method and device, equipment and a storage medium, and the method comprises the steps: collecting a database archiving log in a source end database, the database archiving log comprising a transaction set; in the transaction set, detecting whether a first transaction or / and a first subset exist or not; when the first transaction exists, splitting the first transaction by using a preset transaction splitting strategy to obtain a second transaction; when the first subset exists, combining transactions in the first subset based on a transaction combination strategy corresponding to the first subset to obtain a second transaction; according to the second transaction, updating the transaction set to obtain an updated transaction set; and processing the target end database according to the updated transaction set. According to the method, the data migration efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a transaction processing method, apparatus, device and storage medium. Background Art

[0002] Data migration, especially data migration between heterogeneous databases, often requires the use of data migration tools. The common method in the industry is to use the collection component to collect database archive logs in the source database, use Kafka or other middleware to temporarily store the logs, and then use the playback component to convert the data stored in the middleware into executable statements on the target side, and then process the target database based on these statements.

[0003] Due to the uncertainty of the source database, there will be various transactions in the database archive log, and some transactions will reduce the efficiency of data migration. For example, some transactions occupy a certain thread for a long time, increasing the waiting events of other transactions, resulting in reduced migration efficiency. Some transactions will frequently operate the target database, and frequent data operations will cause most of the time to be wasted on data exchange, resulting in reduced data migration efficiency. Summary of the invention

[0004] The present application provides a transaction processing method, apparatus, device and storage medium, which splits large transactions and combines small transactions or transactions corresponding to the same object, thereby avoiding the problem of the target database being occupied all the time and the target database being frequently operated, and improving data migration efficiency.

[0005] In a first aspect, the present application provides a transaction processing method, the method comprising:

[0006] Collecting database archive logs in a source database, wherein the database archive logs include a transaction set;

[0007] In the transaction set, detecting whether there is a first transaction or / and a first subset, the number of execution operations of the first transaction is greater than a first preset number, and the first subset includes multiple transactions whose number of execution operations is less than a second preset number or transactions corresponding to the same operation object;

[0008] When a first transaction exists, split the first transaction using a preset transaction splitting strategy to obtain a second transaction;

[0009] When the first subset exists, combining the transactions in the first subset based on the transaction combination strategy corresponding to the first subset to obtain a second transaction;

[0010] According to the second transaction, the transaction set is updated to obtain an updated transaction set;

[0011] The target database is processed according to the updated transaction set.

[0012] Optionally, in the transaction set, detecting whether a first subset exists includes:

[0013] In the database archive log, obtaining the operation object of each transaction in the transaction set;

[0014] Determine the target operation object according to the operation object of each transaction;

[0015] The transactions corresponding to the same target operation object are grouped into a first subset.

[0016] Optionally, the transaction combination strategy corresponding to the first subset is:

[0017] Obtaining the execution operation and time corresponding to each transaction in the first subset;

[0018] In the time corresponding to each transaction, determine a third transaction with the earliest corresponding time, and determine the initial state of the target operation object according to the third transaction;

[0019] According to the time corresponding to each transaction, the execution operations of the transaction are executed in sequence to obtain the final state of the target operation object;

[0020] The second transaction is generated according to the initial state and the final state of the target operation object.

[0021] Optionally, in the transaction set, detecting whether a first transaction exists includes:

[0022] In the transaction set, obtaining execution operations included in each transaction;

[0023] Count the execution operations included in each transaction to obtain the number of execution operations of each transaction;

[0024] The transaction whose execution operation times are greater than the first preset times is determined as the first transaction.

[0025] Optionally, in the transaction set, detecting whether a first subset exists includes:

[0026] In the transaction set, determining the number of execution operations corresponding to each transaction;

[0027] A plurality of transactions whose execution operation times are less than a second preset time are combined to obtain a first subset.

[0028] Optionally, the transaction splitting strategy is:

[0029] Determine a maximum value and a minimum value according to the number of execution operations of the transaction, a first preset number of times, and a second preset number of times;

[0030] Determine a first number in a numerical range consisting of the maximum value and the minimum value;

[0031] Evenly divide the execution operations included in the first transaction according to the first number to obtain an execution operation set;

[0032] The execution operations in the execution operation set are combined to obtain a second transaction.

[0033] Optionally, the transaction combination strategy corresponding to the first subset is:

[0034] In the first subset, determine the number of execution operations corresponding to each transaction to obtain the total number of execution operations of the first subset;

[0035] Determining a second number according to the total number of execution operations, the first preset number, and the second preset number;

[0036] Evenly divide the first subset according to the second number to obtain a second subset;

[0037] The transactions in the second subset are combined to obtain a second transaction.

[0038] In a second aspect, the present application provides a transaction processing device, the device comprising:

[0039] A collection unit, used for collecting database archive logs in a source database, wherein the database archive logs include a transaction set;

[0040] a detection unit, configured to detect whether there is a first transaction or / and a first subset in the transaction set, the execution operation number of the first transaction being greater than a first preset number, the first subset including a plurality of transactions whose execution operation number is less than a second preset number or transactions corresponding to the same operation object;

[0041] a splitting unit, configured to, when a first transaction exists, split the first transaction using a preset transaction splitting strategy to obtain a second transaction;

[0042] a combining unit, configured to combine transactions in the first subset based on a transaction combination strategy corresponding to the first subset to obtain a second transaction when the first subset exists;

[0043] an updating unit, configured to update the transaction set according to the second transaction to obtain an updated transaction set;

[0044] The processing unit is used to process the target database according to the updated transaction set.

[0045] Optionally, a detection unit for:

[0046] In the database archive log, obtaining the operation object of each transaction in the transaction set;

[0047] Determine the target operation object according to the operation object of each transaction;

[0048] The transactions corresponding to the same target operation object are grouped into a first subset.

[0049] Optionally, the device further includes a first strategy unit, wherein the first strategy unit is configured to:

[0050] Obtaining the execution operation and time corresponding to each transaction in the first subset;

[0051] In the time corresponding to each transaction, determine a third transaction with the earliest corresponding time, and determine the initial state of the target operation object according to the third transaction;

[0052] According to the time corresponding to each transaction, the execution operations of the transaction are executed in sequence to obtain the final state of the target operation object;

[0053] The second transaction is generated according to the initial state and the final state of the target operation object.

[0054] Optionally, a detection unit for:

[0055] In the transaction set, obtaining execution operations included in each transaction;

[0056] Count the execution operations included in each transaction to obtain the number of execution operations of each transaction;

[0057] The transaction whose execution operation times are greater than the first preset times is determined as the first transaction.

[0058] Optionally, a detection unit for:

[0059] In the transaction set, determining the number of execution operations corresponding to each transaction;

[0060] A plurality of transactions whose execution operation times are less than a second preset time are combined to obtain a first subset.

[0061] Optionally, the device further includes a second strategy unit, wherein the second strategy unit is configured to:

[0062] Determine a maximum value and a minimum value according to the number of execution operations of the transaction, a first preset number of times, and a second preset number of times;

[0063] Determine a first number in a numerical range consisting of the maximum value and the minimum value;

[0064] Evenly divide the execution operations included in the first transaction according to the first number to obtain an execution operation set;

[0065] The execution operations in the execution operation set are combined to obtain a second transaction.

[0066] Optionally, the device further includes a third strategy unit, and the third strategy unit is used to:

[0067] In the first subset, determine the number of execution operations corresponding to each transaction to obtain the total number of execution operations of the first subset;

[0068] Determining a second number according to the total number of execution operations, the first preset number, and the second preset number;

[0069] Evenly divide the first subset according to the second number to obtain a second subset;

[0070] The transactions in the second subset are combined to obtain a second transaction.

[0071] In a third aspect, the present application provides a transaction processing device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein the processor is configured to:

[0072] Collecting database archive logs in a source database, wherein the database archive logs include a transaction set;

[0073] In the transaction set, detecting whether there is a first transaction or / and a first subset, the number of execution operations of the first transaction is greater than a first preset number, and the first subset includes multiple transactions whose number of execution operations is less than a second preset number or transactions corresponding to the same operation object;

[0074] When a first transaction exists, split the first transaction using a preset transaction splitting strategy to obtain a second transaction;

[0075] When the first subset exists, combining the transactions in the first subset based on the transaction combination strategy corresponding to the first subset to obtain a second transaction;

[0076] According to the second transaction, the transaction set is updated to obtain an updated transaction set;

[0077] The target database is processed according to the updated transaction set.

[0078] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned transaction processing method when executed by a processor.

[0079] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art: In the embodiment of the present application, the database archive log in the source database is collected, and the database archive log includes a transaction set; in the transaction set, it is detected whether there is a first transaction or / and a first subset, the execution operation number of the first transaction is greater than the first preset number, and the first subset includes multiple transactions with execution operation times less than the second preset number or transactions corresponding to the same operation object; when the first transaction exists, the first transaction is split to obtain the second transaction; when the first subset exists, the transactions in the first subset are combined to obtain the second transaction; according to the second transaction, the transaction set is updated to obtain an updated transaction set; according to the updated transaction set, the target database is processed. In the implementation of the present application, large transactions can be split, and multiple small transactions or transactions corresponding to the same object can be combined, which avoids the problem that large transactions occupy the target database for a long time, and multiple small transactions or transactions corresponding to the same object frequently operate the target database, thereby improving the efficiency of data migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0081] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0082] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0083] Figure 1 A flowchart of a transaction processing method provided in an embodiment of the present application;

[0084] Figure 2 A schematic diagram of a flow chart of a method for determining a first subset provided in an embodiment of the present application;

[0085] Figure 3 A schematic diagram of a flow chart of a second transaction determination method provided in an embodiment of the present application;

[0086] Figure 4 A schematic diagram of a flow chart of a first transaction determination method provided in an embodiment of the present application;

[0087] Figure 5 A schematic diagram of a flow chart of another method for determining a first subset provided in an embodiment of the present application;

[0088] Figure 6 A flowchart of another second transaction determination method provided in an embodiment of the present application;

[0089] Figure 7 A flowchart of another second transaction determination method provided in an embodiment of the present application;

[0090] Figure 8 A schematic diagram of a process flow of a transaction processing device provided in an embodiment of the present application;

[0091] Fig. 9 A schematic diagram of a transaction processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0092] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0093] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0094] Data migration, especially data migration between heterogeneous databases, often requires the use of data migration tools. The common method in the industry is to use the collection component to collect database archive logs in the source database, use Kafka or other middleware to temporarily store the logs, and then use the playback component to convert the data stored in the middleware into executable statements on the target side, and then process the target database based on these statements.

[0095] Due to the uncertainty of the source database, there will be various transactions in the database archive log, and some transactions will reduce the efficiency of data migration. For example, some transactions occupy a certain thread for a long time, increasing the waiting events of other transactions, resulting in reduced migration efficiency. Some transactions will frequently operate the target database, and frequent data operations will cause most of the time to be wasted on data exchange, resulting in reduced data migration efficiency.

[0096] In summary, the present application embodiment provides a transaction processing method, which can improve the efficiency of data migration. Figure 1 As shown, the specific steps include:

[0097] Step 101: Collect database archive logs from the source database.

[0098] The database archive log includes a transaction set, which includes multiple transactions. Each transaction includes information such as execution operation, start time, end time, and operation object. The execution operation is the processing operation performed on the source database. The start time is the start time of the transaction, and the end time is the end time of the transaction.

[0099] In this step, when the data in the source database is operated based on the user operation, a transaction is generated according to the operation related information, and the transaction is written into the database archive log. The database archive log is updated at intervals. Therefore, before the database archive log is updated, the transaction processing device collects the database archive log in the source database.

[0100] It should be noted that the execution subject of the present application is a transaction processing device, which can detect the database archive log, process the detection result, and then update the database archive log based on the processing result.

[0101] Step 102: Detect whether a first transaction and / or a first subset exists in the transaction set.

[0102] Among them, the first transaction is a large transaction, and a large transaction is a transaction whose corresponding execution operation number is greater than the first preset number. The first subset includes multiple transactions, and the transactions are all small transactions. A small transaction is a transaction whose corresponding execution operation number is less than the second preset number. The first preset number is greater than the second preset number, and the first preset number and the second preset number are set by technicians based on experience, and can also be set based on other methods, which are not limited here. In addition, the first subset includes transactions of the same operation object, and the operation object is a target operation object, and the target operation object is an operation object that appears more than the preset number of times, and the operation object can be a row or column in a data table in the source database.

[0103] In this step, in the transaction set, by detecting whether there is a transaction with an execution operation number greater than a first preset number, it is detected whether there is a first transaction in the transaction set. By detecting whether there is a transaction with an execution operation number less than a first preset number, it is detected whether there is a first subset in the transaction set. By detecting whether there is a transaction corresponding to the target operation object, it is detected whether there is a first subset in the transaction set.

[0104] Step 103: When a first transaction exists, a preset transaction splitting strategy is used to split the first transaction to obtain a second transaction.

[0105] In this step, when there is a first transaction in the transaction set, the first transaction can be split to obtain a second transaction.

[0106] Furthermore, the step of splitting the first transaction is: searching for the execution operation sequence of the first transaction in the database archive log, dividing the execution operation sequence into preset parts, obtaining multiple execution operation subsequences, and determining the transaction corresponding to each execution operation subsequence as the second transaction.

[0107] For example, the execution operation sequence of the first transaction is (execute operation 1, execute operation 2, execute operation 3, execute operation 4), which is preset to be divided into 2. This execution operation sequence is divided into two execution operation sub-sequences, namely (execute operation 1, execute operation 2) and (execute operation 3, execute operation 4). Based on these two execution operation sub-sequences, a corresponding second transaction is generated.

[0108] Furthermore, in order to ensure that the generated second transaction is not a large transaction, the number of parts to be divided can be determined based on the ratio of the number of execution operations corresponding to the first transaction and the first preset number, and then the first transaction can be split into multiple second transactions based on the number of parts.

[0109] For example, after determining the ratio of the number of times the operation is performed and the first preset number of times, the integer part of the ratio is determined, and 1 is added to the integer part to obtain the number of parts to be divided.

[0110] Furthermore, a maximum value and a minimum value are determined according to the number of operations performed, a first preset number, and a second preset number, and then an integer is selected within the range formed by the maximum value and the minimum value, and the integer is determined as the number of shares to be divided.

[0111] In addition to the above, other splitting methods may also be used in the embodiments of the present application, which are not limited here.

[0112] Step 104: When the first subset exists, based on the transaction combination strategy corresponding to the first subset, the transactions in the first subset are combined to obtain a second transaction.

[0113] In this step, when there is a first subset, all transactions in the first subset can be combined to obtain a second transaction, or the transactions in the second subset can be divided into multiple parts, and the transactions in each part can be combined to obtain the second transaction.

[0114] Furthermore, the transactions are sorted according to the time information corresponding to each transaction to obtain a transaction sequence. Starting from the first transaction in the transaction sequence, the execution operation times corresponding to the first two transactions are added, and it is detected whether the sum is greater than the second preset number. If it is greater, the first two transactions are merged to obtain the second transaction, and then the remaining transactions are tested based on the above method until all transactions are merged. If it is less than or equal to, the above test is performed based on the first three transactions.

[0115] In another embodiment, in order to prevent the second transaction from being too small or too large, the number of execution operations corresponding to each transaction may be obtained, and all the execution operation numbers are added together to obtain the total number of execution operations. Then, the ratio of the total number of execution operations to the second preset number is calculated, and the integer part of the ratio is subtracted by 1 to obtain the number of shares to be divided. According to the number of shares to be divided, the first subset is evenly divided to obtain the second subset, and the transactions in each second subset are merged to obtain the second transaction.

[0116] Furthermore, the maximum value and the minimum value are determined according to the total number of operations performed, the first preset number and the second preset number, and then an integer is selected within the range formed by the maximum value and the minimum value, and the integer is determined as the number of shares to be divided.

[0117] In addition to the above, other splitting methods may also be used in the embodiments of the present application, which are not limited here.

[0118] Step 105: Update the transaction set according to the second transaction to obtain an updated transaction set.

[0119] In this step, when only the first transaction exists in the transaction set, the second transaction is used to replace the first transaction in the transaction set to obtain an updated transaction set. When only the first subset exists in the transaction set, the second transaction is used to replace the first subset in the transaction set to obtain an updated transaction set. When the first transaction and the first subset exist in the transaction set, the second transaction is used to replace the first transaction and the first subset in the transaction set to obtain an updated transaction set.

[0120] Step 106: Process the target database according to the updated transaction set.

[0121] In this step, the updated transaction set is stored in the middleware or Kafka queue. The transaction stored in the middleware or Kafka queue is then converted into a database execution statement, and the database execution statement is used to perform corresponding operations on the data in the target database to complete the update of the target database.

[0122] In an embodiment of the present application, a database archive log in a source database is collected, and the database archive log includes a transaction set; in the transaction set, it is detected whether there is a first transaction or / and a first subset, the execution operation number of the first transaction is greater than the first preset number, and the first subset includes multiple transactions whose execution operation number is less than the second preset number or transactions corresponding to the same operation object; when the first transaction exists, the first transaction is split to obtain a second transaction; when the first subset exists, the transactions in the first subset are combined to obtain a second transaction; according to the second transaction, the transaction set is updated to obtain an updated transaction set; according to the updated transaction set, the target database is processed. In the implementation of the present application, large transactions can be split and multiple small transactions can be combined, which avoids the problem that large transactions occupy the target database for a long time, and multiple small transactions or transactions corresponding to the same object frequently operate the target database, thereby improving data migration efficiency.

[0123] In an embodiment of the present application, when a certain operation object is frequently operated, there may be invalid or redundant operations in the frequent operations, and the existence of these operations will reduce the efficiency of data migration. Therefore, the transactions where the frequently operated operation objects are located can be determined in the database archive log, and these transactions can be grouped into a first subset. Therefore, an embodiment of the present application provides a method for determining a first subset, such as Figure 2 As shown, the specific steps include:

[0124] Step 201: Obtain the operation object of each transaction in the transaction set from the database archive log.

[0125] In this step, generally speaking, through the database archive log, it is possible to determine which data table or row or column in the database each transaction operates on by simple analysis, and if the specific data of each transaction operation is determined, since the log may include a large number of transactions, a large amount of calculation will be increased. Therefore, the operation object involved in this step is a certain row or column in the data table. The database archive log also includes the operation object of each transaction, so the operation object of each transaction in the transaction set can also be obtained in the database archive log.

[0126] Step 202: Determine the target operation object according to the operation object of each transaction.

[0127] The target operation object is an object that is frequently operated, for example, a row or column that is frequently operated.

[0128] In this step, when an operation object appears frequently, the operation object will be frequently operated. The number of occurrences of each operation object can be counted to detect whether the number of occurrences of each operation object exceeds the preset number of occurrences. When the number of occurrences of an operation object exceeds the preset number of occurrences, the operation object is determined as the target operation object. When the number of occurrences of an operation object does not exceed the preset number of occurrences, the operation object is not the target operation object.

[0129] The preset number of times is set by the technicians based on their experience. Since the operation object is a row or column in the data table, if the preset number of occurrences is set too small, the transaction processing state will need to frequently execute related transactions. If the preset number of occurrences is set too large, the target database will be frequently operated. Therefore, the preset number of occurrences should be set to an appropriate value. Generally speaking, it is best to set the preset number of occurrences to 5 times.

[0130] Step 203: Group the transactions corresponding to the same target operation object into a first subset.

[0131] In this step, for each target operation object, in the transaction set, transactions corresponding to the same target operation object are determined, and these transactions are grouped into a first subset.

[0132] In the embodiment of the present application, it is pre-detected whether the operation objects corresponding to each transaction in the first subset are the same. When it is detected that the operation objects corresponding to each transaction in the first subset are the same, the transaction change strategy adopted for the first subset is to generate a new transaction according to each transaction in the first subset. This new transaction is the second transaction. Therefore, the embodiment of the present application provides a second transaction determination method. Figure 3 As shown, the specific steps include:

[0133] Step 301: Obtain the execution operation and time corresponding to each transaction in the first subset.

[0134] In this step, the execution operation and time corresponding to each transaction in the first subset are obtained from the database archive log.

[0135] Step 302: Determine the third transaction with the earliest corresponding time in the time corresponding to each transaction, and determine the initial state of the target operation object according to the third transaction.

[0136] In this step, the third transaction with the earliest corresponding time is found in the time corresponding to each transaction. According to the position of the third transaction in the database archive log, the state of the target operation object is analyzed at the position, and the state is determined as the initial state of the target operation object.

[0137] Step 303: Execute the execution operation of each transaction according to the time information corresponding to each transaction to obtain the final state of the target operation object.

[0138] In this step, the occurrence order of the transactions is determined according to the time information corresponding to each transaction, and the execution operations of the transactions are executed in sequence according to the occurrence order to obtain the execution results of the target operation objects.

[0139] Step 304: Generate an optimal transaction according to the initial state and the final state of the target operation object, and determine the optimal transaction as the second transaction.

[0140] In this step, the optimal transaction is the transaction that performs the least operations to convert the target operation object from the initial state to the final state. The merger involved in this step involves not only the merger of transactions, but also the merger of execution operations, so that all operations on the target operation object are completed in one transaction.

[0141] In addition, when executing Figure 3 Before the embodiment, the first subset can be further verified, and the specific steps include: further analyzing the operation object of each transaction to determine the specific operation data of each transaction, and dividing the transactions corresponding to the same specific operation data together to obtain a new first subset. Since the original first subset generally includes a small number of transactions, only calculating the specific operation data of these transactions does not increase too much calculation amount. Therefore, the first subset can be further divided according to the specific operation data of each transaction to obtain a new first subset.

[0142] In the embodiment of the present application, the transaction processing device can also obtain the execution operation of each transaction in the transaction set, determine the number of execution operations corresponding to each transaction, and determine whether each transaction is the first transaction according to the number of execution operations corresponding to each transaction. Therefore, the embodiment of the present application provides a first transaction determination method, which is a further limitation of step 102, such as Figure 4 As shown, the specific steps are:

[0143] Step 401: Obtain the execution operations included in each transaction in the transaction set.

[0144] Here, one execution operation is one operation performed on the data in the source database. For example, one execution operation is one change to the source database.

[0145] In this step, the database archive log records the execution operations included in each transaction, so that the execution operations included in each transaction can be obtained in the database archive log.

[0146] Furthermore, in order to avoid the same transaction being processed repeatedly, Figure 2 After the first subset is obtained, it is also necessary to remove the first subset from the transaction set and determine the first transaction in the transaction set.

[0147] Step 402: Count the execution operations included in each transaction to obtain the number of execution operations of each transaction.

[0148] In this step, for each transaction, the execution operations included in the transaction are counted to obtain the number of execution operations of the transaction.

[0149] Step 403: determine a transaction whose execution operation times are greater than a first preset times as a first transaction.

[0150] The first preset number can be set by technicians based on experience or determined based on other methods, which are not limited here. For example, collect transactions that take too long, determine the number of execution operations included in these transactions, and determine the average of these execution operation numbers as the first preset number.

[0151] In this step, a transaction whose execution operation times are greater than a first preset times is determined as a first transaction.

[0152] In an embodiment of the present application, in the database archive log, the number of execution operations corresponding to each transaction is determined, and the transactions whose number of execution operations is less than the second preset number are combined to obtain a first subset. Therefore, an embodiment of the present application provides a method for determining a first subset, such as Figure 5 As shown, the specific steps include:

[0153] Step 501: Determine the number of execution operations corresponding to each transaction in the transaction set.

[0154] In this step, the database archive log records the execution operations included in each transaction, so that the execution operations included in each transaction can be obtained in the database archive log. For each transaction, the execution operations included in the transaction are counted to obtain the number of execution operations of the transaction.

[0155] Step 502: Combine multiple transactions whose execution operation times are less than a second preset number to obtain a first subset.

[0156] The second preset number of times may be set by a technician based on experience or may be determined based on other methods, which are not limited here. For example, transactions that take too short a time are collected, the number of execution operations included in these transactions is determined, and the average value of these execution operation numbers is determined as the second preset number of times. The second preset number of times is less than the first preset number of times.

[0157] In this step, in the transaction set, all transactions whose execution operation times are less than a second preset times are detected, and these transactions are combined to obtain a first subset.

[0158] Furthermore, in order to avoid the same transaction being processed repeatedly, Figure 2 After the first subset is added, it is also necessary to remove the first subset from the transaction set and execute Figure 5 Steps shown.

[0159] Furthermore, generally speaking, there are often multiple transactions in the log whose operation times are less than the second preset number, but there are exceptions. Therefore, it is also necessary to detect the number corresponding to the transactions whose operation times are less than the second preset number. When the number is greater than the preset number, these transactions are combined into a first subset. When the number is less than the preset number, the first subset is not formed based on these transactions.

[0160] In the embodiment of the present application, when the first transaction is a large transaction, the first transaction can also be split to obtain multiple second transactions. Therefore, the embodiment of the present application provides a second transaction determination method, such as Figure 6 As shown, the specific steps include:

[0161] Step 601: determine a maximum value and a minimum value according to the number of execution operations of the first transaction, a first preset number, and a second preset number.

[0162] In this step, in order to prevent the number of execution operations corresponding to the split transaction from being greater than the first preset number, after obtaining the execution operations included in the first transaction, the execution operations included in the first transaction are counted to obtain the number of execution operations corresponding to the first transaction, and finally, the ratio of the number of execution operations corresponding to the first transaction to the first preset number is calculated, and the ratio is the minimum value. The ratio of the number of execution operations corresponding to the first transaction to the second preset number is calculated, and the ratio is the maximum value.

[0163] Step 602: Determine a first number in a numerical range consisting of a maximum value and a minimum value.

[0164] In this step, an integer is selected between the minimum value and the maximum value, and the integer is determined as the first number.

[0165] Step 603: Evenly divide the execution operations included in the first transaction according to the first number to obtain an execution operation set.

[0166] The number of execution operations of the first transaction is less than a first preset number.

[0167] In this step, the execution operations included in the first transaction are evenly divided according to the first number to obtain multiple execution operation sets.

[0168] Step 604: Combine the execution operations in the execution operation set to obtain a second transaction.

[0169] In this step, for each execution operation set, all execution operations in the execution operation set are obtained, and all execution operations are combined to obtain a second transaction.

[0170] In the embodiment of the present application, it is detected whether the transactions in the first subset are all small transactions. When the transactions in the first subset are all small transactions, the transaction change strategy adopted for the first subset is to divide the first subset into multiple parts and generate a second transaction based on each part. Therefore, the embodiment of the present application also provides a second transaction determination method, which method is as follows: Figure 7 As shown, the specific steps include:

[0171] Step 701: In a first subset, determine the number of execution operations corresponding to each transaction, and obtain the total number of execution operations of the first subset.

[0172] In this step, for each transaction in the first subset, the number of execution operations corresponding to each transaction is obtained from the database archive log. Then, the number of execution operations corresponding to all transactions is added up to obtain the total number of execution operations.

[0173] Step 702: Determine a second number according to the total number of times the operation is performed, the first preset number, and the second preset number.

[0174] In this step, in order to prevent the number of execution operations corresponding to the split transaction from being greater than the first preset number, after obtaining the execution operations included in the first transaction, the execution operations included in the first transaction are counted to obtain the number of execution operations corresponding to the first transaction. Finally, the ratio of the number of execution operations corresponding to the first transaction to the first preset number is calculated, and the ratio is the minimum value. The ratio of the number of execution operations corresponding to the first transaction to the second preset number is calculated, and the ratio is the maximum value. In the numerical range composed of the maximum value and the minimum value, in this step, an integer is selected between the minimum value and the maximum value, and the integer is determined as the second number.

[0175] Step 703: Evenly divide the first subset according to the second number to obtain a second subset.

[0176] The second number is the number of the second subset, and the second subset includes multiple transactions.

[0177] In this step, the first subset is divided into a second target number of second subsets.

[0178] Step 704: Combine the transactions in the second subset to obtain a second transaction.

[0179] In this step, all transactions in the second subset are obtained, and all transactions are combined to obtain the second transaction.

[0180] like Figure 8 As shown, an embodiment of the present application provides a transaction processing device, which corresponds to the method embodiment and specifically includes:

[0181] A collection unit 801 is used to collect database archive logs in a source database, where the database archive logs include a transaction set;

[0182] A detection unit 802 is configured to detect whether there is a first transaction or / and a first subset in the transaction set, the execution operation number of the first transaction is greater than a first preset number, and the first subset includes multiple transactions whose execution operation number is less than a second preset number or transactions corresponding to the same operation object;

[0183] A splitting unit 803 is used for, when there is a first transaction, splitting the first transaction using a preset transaction splitting strategy to obtain a second transaction;

[0184] A combining unit 804 is configured to combine the transactions in the first subset to obtain a second transaction based on the transaction combination strategy corresponding to the first subset when the first subset exists;

[0185] An updating unit 805, configured to update the transaction set according to the second transaction to obtain an updated transaction set;

[0186] The processing unit 806 is used to process the target database according to the updated transaction set.

[0187] Optionally, the detection unit 802 is used to:

[0188] In the database archive log, obtaining the operation object of each transaction in the transaction set;

[0189] Determine the target operation object according to the operation object of each transaction;

[0190] The transactions corresponding to the same target operation object are grouped into a first subset.

[0191] Optionally, the device further includes a first strategy unit 807, and the first strategy unit 807 is configured to:

[0192] Obtaining the execution operation and time corresponding to each transaction in the first subset;

[0193] In the time corresponding to each transaction, determine a third transaction with the earliest corresponding time, and determine the initial state of the target operation object according to the third transaction;

[0194] According to the time corresponding to each transaction, the execution operations of the transaction are executed in sequence to obtain the final state of the target operation object;

[0195] The second transaction is generated according to the initial state and the final state of the target operation object.

[0196] Optionally, the detection unit 802 is used to:

[0197] In the transaction set, obtaining execution operations included in each transaction;

[0198] Count the execution operations included in each transaction to obtain the number of execution operations of each transaction;

[0199] The transaction whose execution operation times are greater than the first preset times is determined as the first transaction.

[0200] Optionally, the detection unit 802 is used to:

[0201] In the transaction set, determining the number of execution operations corresponding to each transaction;

[0202] A plurality of transactions whose execution operation times are less than a second preset time are combined to obtain a first subset.

[0203] Optionally, the device further includes a second strategy unit 808, and the second strategy unit 808 is configured to:

[0204] Determine a maximum value and a minimum value according to the number of execution operations of the transaction, a first preset number of times, and a second preset number of times;

[0205] Determine a first number in a numerical range consisting of the maximum value and the minimum value;

[0206] Evenly divide the execution operations included in the first transaction according to the first number to obtain an execution operation set;

[0207] The execution operations in the execution operation set are combined to obtain a second transaction.

[0208] Optionally, the device further includes a third strategy unit 809, and the third strategy unit 809 is configured to:

[0209] In the first subset, determine the number of execution operations corresponding to each transaction to obtain the total number of execution operations of the first subset;

[0210] Determining a second number according to the total number of execution operations, the first preset number, and the second preset number;

[0211] Evenly divide the first subset according to the second number to obtain a second subset;

[0212] The transactions in the second subset are combined to obtain a second transaction.

[0213] like Fig. 9 As shown, the embodiment of the present application provides a transaction processing device, including a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0214] Memory 903, used for storing computer programs;

[0215] In one embodiment of the present application, the processor 901 is used to implement the transaction processing method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 903, including:

[0216] Collecting database archive logs in a source database, wherein the database archive logs include a transaction set;

[0217] In the transaction set, detecting whether there is a first transaction or / and a first subset, the number of execution operations of the first transaction is greater than a first preset number, and the first subset includes multiple transactions whose number of execution operations is less than a second preset number or transactions corresponding to the same operation object;

[0218] When a first transaction exists, split the first transaction using a preset transaction splitting strategy to obtain a second transaction;

[0219] When the first subset exists, combining the transactions in the first subset based on the transaction combination strategy corresponding to the first subset to obtain a second transaction;

[0220] According to the second transaction, the transaction set is updated to obtain an updated transaction set;

[0221] The target database is processed according to the updated transaction set.

[0222] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps performed by the transaction processing method provided in any of the aforementioned method embodiments are implemented.

[0223] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0224] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0225] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0226] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A transaction processing method, characterized in that: The method comprises: Collecting database archive logs in a source database, wherein the database archive logs include a transaction set; In the transaction set, detecting whether there is a first transaction or / and a first subset, the number of execution operations of the first transaction is greater than a first preset number, and the first subset includes multiple transactions whose number of execution operations is less than a second preset number or transactions corresponding to the same operation object; When a first transaction exists, split the first transaction using a preset transaction splitting strategy to obtain a second transaction; When the first subset exists, combining the transactions in the first subset based on the transaction combination strategy corresponding to the first subset to obtain a second transaction; According to the second transaction, the transaction set is updated to obtain an updated transaction set; The target database is processed according to the updated transaction set.

2. The method according to claim 1, characterized in that: In the transaction set, detecting whether a first subset exists includes: In the database archive log, obtaining the operation object of each transaction in the transaction set; Determine the target operation object according to the operation object of each transaction; The transactions corresponding to the same target operation object are grouped into a first subset.

3. The method according to claim 2, characterized in that: The transaction combination strategy corresponding to the first subset is: Obtaining the execution operation and time corresponding to each transaction in the first subset; In the time corresponding to each transaction, determine a third transaction with the earliest corresponding time, and determine the initial state of the target operation object according to the third transaction; According to the time corresponding to each transaction, the execution operations of the transaction are executed in sequence to obtain the final state of the target operation object; The second transaction is generated according to the initial state and the final state of the target operation object.

4. The method according to claim 1 or 2, characterized in that: In the transaction set, detecting whether a first transaction exists includes: In the transaction set, obtaining execution operations included in each transaction; Count the execution operations included in each transaction to obtain the number of execution operations of each transaction; The transaction whose execution operation times are greater than the first preset times is determined as the first transaction.

5. The method according to claim 4, characterized in that: In the transaction set, detecting whether a first subset exists includes: In the transaction set, determining the number of execution operations corresponding to each transaction; A plurality of transactions whose execution operation times are less than a second preset time are combined to obtain a first subset.

6. The method according to claim 5, characterized in that: The transaction splitting strategy is: Determine a maximum value and a minimum value according to the number of execution operations of the first transaction, a first preset number of times, and a second preset number of times; Determine a first number in a numerical range consisting of the maximum value and the minimum value; Evenly divide the execution operations included in the first transaction according to the first number to obtain an execution operation set; The execution operations in the execution operation set are combined to obtain a second transaction.

7. The method according to claim 5, characterized in that: The transaction combination strategy corresponding to the first subset is: In the first subset, determine the number of execution operations corresponding to each transaction to obtain the total number of execution operations of the first subset; Determining a second number according to the total number of execution operations, the first preset number, and the second preset number; Evenly divide the first subset according to the second number to obtain a second subset; The transactions in the second subset are combined to obtain a second transaction.

8. A transaction processing device, characterized in that: The device comprises: A collection unit, used for collecting database archive logs in a source database, wherein the database archive logs include a transaction set; a detection unit, configured to detect whether there is a first transaction or / and a first subset in the transaction set, the execution operation number of the first transaction being greater than a first preset number, the first subset including a plurality of transactions whose execution operation number is less than a second preset number or transactions corresponding to the same operation object; a splitting unit, configured to, when a first transaction exists, split the first transaction using a preset transaction splitting strategy to obtain a second transaction; a combining unit, configured to combine transactions in the first subset based on a transaction combination strategy corresponding to the first subset to obtain a second transaction when the first subset exists; an updating unit, configured to update the transaction set according to the second transaction to obtain an updated transaction set; The processing unit is used to process the target database according to the updated transaction set.

9. A transaction processing device, characterized in that: include: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein the processor is configured to: Collecting database archive logs in a source database, wherein the database archive logs include a transaction set; In the transaction set, detecting whether there is a first transaction or / and a first subset, the number of execution operations of the first transaction is greater than a first preset number, and the first subset includes multiple transactions whose number of execution operations is less than a second preset number or transactions corresponding to the same operation object; When a first transaction exists, split the first transaction using a preset transaction splitting strategy to obtain a second transaction; When the first subset exists, combining the transactions in the first subset based on the transaction combination strategy corresponding to the first subset to obtain a second transaction; According to the second transaction, the transaction set is updated to obtain an updated transaction set; The target database is processed according to the updated transaction set.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the transaction processing method according to any one of claims 1 to 7 is implemented.

Images