Transaction log analysis method of database and related device

Abstract

The embodiment of the invention provides a transaction log analysis method of a database and a related device, and relates to the technical field of data processing. The method comprises the steps that all change records in a to-be-analyzed transaction log are read in sequence through a reading thread; under the condition that the currently read change record is the first change record of a new transaction, a translation thread is allocated to the new transaction, the translation thread is used for translating the change record corresponding to the transaction, and the new transaction is a transaction which is not read before the to-be-analyzed transaction logs are sequentially read; translating the translated transactions of the distributed translation threads in parallel through the distributed translation threads, and packaging data obtained after translation under the condition that the translation threads finish translation, so as to complete the analysis of the transaction log. According to the method, when the transaction log containing the multiple transactions is analyzed, the overall analysis speed of the transaction log can be increased.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a method for parsing transaction logs of a database and related devices. Background Art

[0002] In the process of managing or applying a database, it is usually necessary to parse the transaction logs of the database to obtain the actual data change situations corresponding to each change record in the transaction logs.

[0003] The existing method for parsing transaction logs is to sequentially read each change record in the transaction logs one by one. When all the change records of a certain transaction are read, the method starts to translate and encapsulate the change records of this transaction to complete the parsing of this transaction. In this way, when all the transactions in the transaction logs are parsed, the parsing of this transaction logs is completed.

[0004] In some scenarios, the transaction logs may contain multiple transactions, and each transaction includes a certain number of change records with unequal quantities. The arrangement order of these change records may be interleaved and not arranged by concentrating each transaction's change records. Therefore, for such transaction logs containing multiple transactions, if the existing parsing method is sampled, it will take a long time to parse all the transactions in the logs, resulting in a low speed of parsing the transaction logs. Summary of the Invention

[0005] Embodiments of this application provide a method for parsing transaction logs of a database and related devices, which can improve the speed of parsing transaction logs when parsing transaction logs containing multiple transactions.

[0006] In a first aspect, embodiments of this application provide a method for parsing transaction logs of a database. The method includes: sequentially reading each change record in the transaction logs to be parsed through a reading thread; when the currently read change record is the first change record of a new transaction, allocating a translation thread for the new transaction, where the translation thread is used to translate the change records corresponding to the transaction, and the new transaction is a transaction that has not been read since starting to sequentially read the transaction logs to be parsed; parallelly translating the various transactions in translation of the allocated translation threads through the allocated translation threads, and encapsulating the data obtained after translation when the translation thread finishes translation, to complete the parsing of the transaction logs.

[0007] In a possible implementation, allocating a translation thread for the new transaction includes: creating a snapshot according to the master data dictionary of the database, and determining the snapshot as the data dictionary for the translation thread to perform data translation, where the master data dictionary is a set of information characterizing the current database structure information of the database; and allocating a translation thread including the data dictionary for the new transaction according to the data dictionary.

[0008] In a possible implementation, the method further includes: for any transaction being translated, when the currently read change record is the change record of the commit event of the any transaction being translated, continuously translating until the change record of the commit event is translated.

[0009] In a possible implementation, the translation thread includes a defined change cache for storing change records of data definition language. The continuously translating until the change record of the commit event is translated includes: for the any transaction being translated, when the currently read change record is the change record of the data definition language of the any transaction being translated, storing the change record of the data definition language in the defined change cache, and updating the data dictionary of the translation thread according to the change record of the data definition language; continuously translating until the change record of the commit event is translated, and updating the master data dictionary of the database according to the change records of the data definition language stored in the defined change cache.

[0010] In a possible implementation, the translation thread includes a blocking queue for storing change records in a blocking queue manner. The method further includes: for any new transaction, sequentially storing the first change record of the read any new transaction and other change records into the blocking queue; and sequentially translating the change records stored in the blocking queue through the translation thread of the any new transaction.

[0011] In a possible implementation, the method further includes: for any transaction being translated, when the currently read change record is the change record of the rollback event of the any transaction being translated, stopping translation and destroying the cached data in the translation thread of the any transaction being translated, and ending the translation of the any transaction being translated.

[0012] Second aspect, an embodiment of the present application provides a transaction log parsing device for a database. The device includes: a reading module, configured to sequentially read each change record in the transaction log to be parsed through a reading thread; an allocation module, configured to, when the currently read change record is the first change record of a new transaction, allocate a translation thread for the new transaction, where the translation thread is used to translate the change records corresponding to the transaction, and the new transaction is a transaction that has not been read since the transaction log to be parsed is sequentially read from the beginning; a processing module, configured to parallelly translate each in-translation transaction of the allocated translation threads through the allocated translation threads, and encapsulate the data obtained after translation when the translation thread ends the translation, so as to complete the parsing of the transaction log.

[0013] In a possible implementation manner, the allocation module is specifically configured to: create a snapshot according to the main data dictionary of the database, and determine the snapshot as the data dictionary for the translation thread to perform data translation, where the main data dictionary is a set of information characterizing the current database structure information of the database; allocate a translation thread including the data dictionary for the new transaction according to the data dictionary.

[0014] In a possible implementation manner, the processing module is further configured to: for any in-translation transaction, when the currently read change record is the change record of the commit event of the any in-translation transaction, continue translation until the change record of the commit event is translated.

[0015] In a possible implementation manner, the translation thread includes a defined change cache, and the defined change cache is used to store change records of data definition language. The processing module is specifically configured to: for any in-translation transaction, when the currently read change record is the change record of the data definition language of the any in-translation transaction, store the change record of the data definition language in the defined change cache, and update the data dictionary of the translation thread according to the change record of the data definition language; continue translation until the change record of the commit event is translated, and update the main data dictionary of the database according to the change records of the data definition language stored in the defined change cache.

[0016] In a possible implementation manner, the translation thread includes a blocking queue, and the blocking queue is used to store change records in a blocking queue manner. The processing module is further configured to: for any new transaction, sequentially store the first change record and other change records of the read any new transaction into the blocking queue; sequentially translate the change records stored in the blocking queue through the translation thread of the any new transaction.

[0017] In a possible implementation, the processing module is further configured to: for any in-translation transaction, when the currently read change record is the change record of the rollback event of the any in-translation transaction, stop translation and destroy the cached data in the translation thread of the any in-translation transaction, and end the translation of the any in-translation transaction.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the above first aspect and / or various possible implementations of the first aspect.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the above first aspect and / or various possible implementations of the first aspect.

[0021] The method and related device for parsing a transaction log of a database provided by the embodiments of the present application. In this method, a reading thread sequentially reads each change record in the transaction log. When a new transaction that has not been read is read, a translation thread can be allocated for the new transaction. Through this translation thread, translation of the first change record of the transaction and each subsequent change record of the transaction read can be started in a timely manner. Therefore, for each new transaction read during the reading process, an independent translation thread can be allocated for it and it can immediately enter the translation process. Then, each transaction does not have to wait until all the change records of the transaction are read before starting translation, which can shorten the waiting time for the transaction to start translation. And, when the transaction log includes multiple transactions, parallel translation and encapsulation of multiple transactions can be realized through the respective translation threads of each transaction, and the parsing of all transactions can be completed faster through parallel processing. Therefore, the method of the present application can effectively improve the speed of parsing the transaction log while shortening the time required for translating the change records of each transaction. Description of the Drawings

[0022] The drawings here are incorporated into the description and form a part of this description, showing embodiments consistent with the present application, and are used together with the description to explain the principles of the present application.

[0023] Figure 1 It is a schematic diagram of incremental data synchronization provided by an embodiment of the present application;

[0024] Figure 2 Schematic diagram of the method for parsing the transaction log of an existing database

[0025] Figure 3 Flow schematic diagram of the method for parsing the transaction log of a database provided by an embodiment of the present application

[0026] Figure 4 Schematic diagram of the working process of the translation thread provided by an embodiment of the present application

[0027] Figure 5 Schematic diagram of the working process of the reading thread provided by an embodiment of the present application

[0028] Figure 6 Schematic diagram of the structure of the device for parsing the transaction log of a database provided by an embodiment of the present application

[0029] Figure 7 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application

[0030] Through the above-mentioned drawings, specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments Detailed implementation manners

[0031] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims

[0032] In the technical solution of the embodiment of the present application, the collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information and other processing all comply with the provisions of relevant laws and regulations and do not violate public order and good customs

[0033] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for the user to select authorization or rejection

[0034] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0035] In the embodiments of the present application, if words such as "first" and "second" are used, they are for distinguishing the same items or similar items with basically the same functions and roles. For example, the first electronic device and the second electronic device are only for distinguishing different electronic devices, and do not limit their sequence. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0036] In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0037] In the management and application scenarios of a database, it is usually necessary to perform operations such as data synchronization on the data in the database. For example, a data synchronization tool is used to synchronize the data in the target database according to the data in the source database, so that the data in the target database is consistent with the data in the source database.

[0038] Exemplarily, when using a data synchronization tool for real-time data synchronization, it can include three stages. Among them, the first stage can be to perform an initial load on the existing data to obtain a basis point for data synchronization; the second stage can be to perform incremental data synchronization based on the synchronization basis point established by the initial data load; the third stage can be to periodically compare and verify the source data and target data of the data synchronization to avoid data loss during the data synchronization process. The second stage and the third stage can be in a long-term parallel execution state.

[0039] Figure 1 For the schematic diagram of incremental data synchronization provided by the embodiments of the present application, during the incremental data synchronization process in the second stage, incremental data can be obtained by analyzing the logs of the source database, thereby realizing real-time data synchronization. Specifically, as Figure 1As shown in the figure, the online log or archived log of the source database can be parsed by the source-side synchronization software to obtain the incremental data of the source database. For example, based on the transaction log of the source database, the changes in data addition, deletion, and modification can be obtained by processes such as translating and encapsulating the change records in the transaction log. These change situations can reflect the incremental data. These change situations are converted into a specific message format inside the synchronization software in units of transactions. After filtering these transactions, the filtered incremental data can be obtained. Through the private transmission protocol of the synchronization software, each transaction can be sent to the target-side synchronization software of the target database. The target-side synchronization software can restore the obtained transaction log into structured query language (SQL) statements supported by the target database and execute them on the target database, so that the data in the target database can be synchronized to be consistent with the data in the source database.

[0040] Figure 2 It is a schematic diagram of an existing method for parsing the transaction log of a database, as Figure 2 shown, the transaction log of the database may contain multiple transactions, such as transaction a, transaction b, and transaction c, etc. Each transaction may also include multiple change records. For example, transaction a - SQL1, transaction a - SQL2, and transaction a - commit are all change records of transaction a; transaction b - SQL1, transaction b - SQL2, and transaction b - rollback are all change records of transaction b. Among them, SQL1 and SQL2 represent two different SQL statements; commit represents commit. For example, the change record transaction a - commit represents a change record for executing the commit operation on transaction a, and this change record records the commit event, indicating that all changes of transaction a have been executed; rollback represents rollback. For example, the change record transaction b - rollback represents a change record for executing the rollback operation on transaction b, and this change record records the rollback event, indicating that all changes of transaction b have not been executed.

[0041] As Figure 2 shown, during the parsing process of the transaction log, the entire transaction log needs to be read and stored in the reordering cache. The process from the change records of a transaction being deposited in the cache to being taken out can be called reordering. The transaction log of the database may include multiple change records. Due to concurrent operations of data connections, the multiple change records in a certain transaction may not be continuously sorted in the transaction log. For example, the change records of transaction a, transaction b, and transaction c may be interleaved. Therefore, during the incremental synchronization process in the second stage, after each change record of a transaction is read, it needs to be classified and stored in the cache according to the identity document (id) of the transaction.

[0042] After reading the commit event of a certain transaction, obtain the list of transaction change records corresponding to the transaction from the cache, and translate each change record of the transaction one by one according to the list until the last change record of the transaction is translated, and then encapsulate the translated data. After the entire transaction is processed, delete all change records of this transaction from the cache. After reading the rollback event of a certain transaction, directly delete all change records of this transaction from the cache.

[0043] In the translation stage of parsing, the change records of the transaction log are translated into actual data changes. This stage depends on the relevant meta-information of the data table or the fields in the data table, and this meta-information can be a data dictionary. A data dictionary can be understood as a collection of concepts or system tables used in a database to store information about the database structure. It contains the metadata of database objects, such as data definitions of tables, columns, and indexes, as well as information about user permissions and relationship schemas. The translation stage of parsing the log depends on the data dictionary.

[0044] There is a visibility problem for each transaction relative to other transactions. It can be understood that due to the difference in the commit times of different transactions, there is a problem of differentiation in the data dictionary when parsing different transactions. Due to visibility, the existing transaction log parsing methods will perform translations in the order of the commit order of the parsed transactions. Accordingly, it can be analyzed that the time required for the existing method to parse the transaction log is: parsing transaction time = transaction log reading time + transaction translation time; where the transaction log reading time = the end time of reading the transaction - the start time of reading the transaction.

[0045] When the execution time of a transaction is long and / or the number of change records involved is large, this transaction can be called a large transaction. When using the existing method to parse large transactions in the source-side transaction log, reading all change records of the large transaction and translating each change record are executed sequentially, which is a serial execution method. Therefore, it takes a long time to parse each large transaction. If a transaction log includes a relatively large number of transactions, or a relatively large number of transactions among them are large transactions, the overall parsing time of the transaction log is long, resulting in a low overall parsing speed when parsing the transaction log.

[0046] In view of this, an embodiment of the present application provides a method for parsing transaction logs of a database. This method allocates translation threads to the newly read transactions to minimize the waiting time for each transaction to start translation, and combines the method of parallel processing of multiple transactions to improve the speed of parsing transaction logs. Specifically, this method uses a reading thread to sequentially read each change record in the transaction log. When a new transaction that has not been read is read, a translation thread can be allocated to this new transaction, and through this translation thread, the first change record of this transaction and each subsequent change record of this transaction read can be started for translation in a timely manner. Therefore, for each new transaction read during the reading process, an independent translation thread can be allocated to it and immediately enter the translation process, so that each transaction does not have to wait until all the change records of this transaction are read before starting translation, which can shorten the waiting time for the transaction to start translation. Moreover, when the transaction log includes multiple transactions, parallel translation and encapsulation of multiple transactions can be achieved through the respective translation threads of each transaction, and the parsing of all transactions can be completed faster through the parallel processing method. Therefore, the method of the present application can effectively improve the speed of parsing transaction logs while shortening the time required to translate the change records of each transaction.

[0047] The technical solution of the present application will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0048] Figure 3 It is a schematic flowchart of the method for parsing transaction logs of a database provided by an embodiment of the present application. The execution subject of this method can be an electronic device with corresponding data storage capabilities and computing capabilities, such as a computer, a server, or a server cluster, etc. As Figure 3 shown, this method includes:

[0049] S301, sequentially read each change record in the transaction log to be parsed through a reading thread.

[0050] Exemplarily, the transaction log can be any log used to record transactions in the database. The transaction log can be an online log or an archived log. The change records in the transaction log can include records of change operations such as adding, deleting, or modifying data in the database or data table, and can also include records of committing or rolling back the change operations.

[0051] For example, during a certain database management process, for Data Table A in the database, through SQL1 statement, a command is issued to add a row of data to Data Table A; through SQL2 statement, a command is issued to modify a field value in a row of data in Data Table A; through SQL3 statement, a command is issued to delete a row of data in Data Table A. Then these three change operations will form three change records of a transaction in this management process. After that, if the command statements of these three change operations are executed, change records of the commit event of this transaction will also be formed after the change records of the three change operations. On the contrary, if the command statements of these three change operations are not executed, change records of the rollback event of this transaction will be formed after the change records of the three change operations.

[0052] Similarly, during the database management process, a transaction log including multiple change records of multiple transactions can be formed, and any transaction log that needs to be parsed can be a transaction log to be parsed.

[0053] The reading thread can be a working thread used to perform reading operations on the change records in the transaction log, and can read the specific data of each change record. If the transaction log includes multiple change records, the reading thread can read each change record in sequence according to the arrangement order of each change record.

[0054] Exemplarily, in the transaction log, the change records of different transactions may be interleaved. If the transaction log is sliced into multiple data blocks and a reading thread is assigned to each data block to achieve parallel reading of the change records, there may be a problem that multiple reading threads cannot read any transaction in the order of the change records of that transaction, which may cause subsequent translation and encapsulation to fail to execute properly.

[0055] For example, the transaction log is split into two data blocks from a certain middle position, and two reading threads are configured to perform parallel reading on these two data blocks. After reading for a period of time, assume that the first reading thread has only read a part of the change records of Transaction A, and there are other change records of Transaction A in this data block. At this time, if the second reading thread reads the change record of the commit event of Transaction A, and then the first reading thread reads other change records of Transaction A. In this way, for Transaction A, the change records of Transaction A read are stored in sequence in the cache, but the change record of the commit event is inserted in the middle of the change records, so the order of the change records of Transaction A in the cache is disordered, which will lead to incorrect translation or inability to translate Transaction A subsequently, and will also cause the inability to properly encapsulate the translated content, resulting in parsing failure.

[0056] Therefore, when each change record in the transaction log to be parsed is sequentially read by a reading thread, the problem of disordered reading and disordered cache order of the change records of each transaction will not occur, and each transaction can be normally translated and encapsulated.

[0057] S302. When the currently read change record is the first change record of a new transaction, allocate a translation thread for the new transaction. The translation thread is used to translate the change records corresponding to the transaction. The new transaction is a transaction that has not been read since the transaction log to be parsed is sequentially read from the beginning.

[0058] Exemplarily, each transaction can have its own transaction identification identifier, such as the id of the transaction, etc. After starting to parse a transaction log, if the transaction identification identifier of the currently read change record corresponds to the transaction identification identifier that is encountered for the first time after the start of parsing, then the transaction is the new transaction, and the first change record of the transaction read is the first change record of the new transaction. When reading each subsequent change record, the change records of the transaction can be cached in sequence according to the transaction identification identifier of the transaction, so as to perform sequential translation on the change records of the transaction.

[0059] Exemplarily, the translation thread can be a working thread that translates the change record into an actual data change. For example, the translation thread can transcribe any change record according to the metadata of the database and / or data table object, so as to convert the change record into an actual data change. The translation thread allocated for the new transaction can continuously translate all the change records of the transaction.

[0060] Since the transaction log can include multiple transactions, and the change records of each transaction may be interleaved, a translation thread pool including multiple translation threads can be preset, so as to pre-configure multiple translation threads to be allocated in the translation thread pool. When the reading thread reads the first change record of each new transaction, a translation thread can be quickly allocated for each new transaction, thereby shortening the waiting time for each transaction to start translation.

[0061] S303. Through each allocated translation thread, parallelly translate each translating transaction for which the translation thread has been allocated, and encapsulate the data obtained after translation when the translation thread ends the translation, thereby completing the parsing of the transaction log.

[0062] Exemplarily, a translating transaction refers to a transaction for which a translation thread has been allocated and the translation thread has not translated all the change records to be translated, and can be understood as a transaction that is in the process of being translated.

[0063] A large transaction can be understood as a transaction with a relatively long execution time and / or involving a relatively large number of change records. For example, a relatively long execution time can be understood as an execution time greater than 30 minutes, and a relatively large number of change records involved can be understood as the number of change records being greater than 30, etc.

[0064] In a transaction log including multiple transactions, if there are a relatively large number of large transactions, then there are relatively many transactions in the translation process at the same time. Through each allocated translation thread, the change records of each transaction being translated can be translated in parallel, realizing parallel transaction translation.

[0065] For any transaction, after translating each change record of the transaction, the data obtained after translation can be obtained, and these data can be actual data changes. After the translation thread finishes translating all the change records of the transaction, the data obtained after translation of the transaction can be encapsulated. After each translation thread finishes translating and encapsulating all transactions, the parsing of the transaction log is completed.

[0066] The method for parsing a transaction log of a database provided by an embodiment of the present application. In this method, a reading thread sequentially reads each change record in the transaction log. When a new transaction that has not been read is read, a translation thread can be allocated for the new transaction. Through the translation thread, the first change record of the transaction and each subsequent change record of the transaction read can be translated in a timely manner. Therefore, for each new transaction read during the reading process, an independent translation thread can be allocated for it and it can immediately enter the translation process. Then, each transaction does not have to wait until all the change records of the transaction are read before starting translation, which can shorten the waiting time for the transaction to start translation. Moreover, in the case where the transaction log includes multiple transactions, parallel translation and encapsulation of multiple transactions can be realized through the respective translation threads of each transaction, and the parsing of all transactions can be completed faster through the parallel processing method. Therefore, the method of the present application can effectively improve the speed of parsing the transaction log while shortening the time required to translate the change records of each transaction.

[0067] In a possible implementation manner, when allocating a translation thread for a new transaction, specifically, it can be: creating a snapshot according to the main data dictionary of the database, and determining the snapshot as the data dictionary for the translation thread to use for data translation. The main data dictionary is a set of information representing the current database structure information of the database; allocating a translation thread including the data dictionary for the new transaction according to the data dictionary.

[0068] Exemplarily, the main data dictionary can be a set of information representing the current database structure information of the database. For example, the main data dictionary can include all metadata representing database objects in the current state. The metadata can be, for example, data definitions such as tables, columns, and indexes, as well as information such as user permissions and relationship schemas. The metadata can be understood as database structure information.

[0069] Exemplarily, through the snapshot function of the database management system, a snapshot can be created for the master data dictionary of the database to configure a data dictionary for each translation thread. By creating a snapshot of the master data dictionary, a static copy identical to the content of the master data dictionary can be obtained, and this static copy is the data dictionary. The data dictionary can be applied to the translation thread to guide the translation thread to translate each change record to be translated.

[0070] It should be understood that in the case where the metadata of the database object changes, the master data dictionary will also change accordingly. Therefore, the data dictionaries assigned to different translation threads may be different. By updating the master data dictionary in a timely manner according to the change situation of the database structure information, the correct data dictionary can be assigned to the translation thread to correctly guide the translation thread to translate the change record.

[0071] In an embodiment of the present application, when parallelly translating the change records of multiple in-translation transactions, a data dictionary can be assigned to each translation thread, and this data dictionary is obtained by creating a snapshot of the master data dictionary of the database. In this way, when multiple translation threads translate in parallel, dirty reads can be avoided, resulting in incorrect data after translation, and the accuracy and reliability of the parsing result can be improved.

[0072] Exemplarily, since the translation process of the translation thread lags behind the reading process of the reading thread, and the time required to translate a change record may be longer than the time required to read a change record, for any transaction, there will be a situation where all the change records of the transaction are read first and then all the change records of the transaction are translated.

[0073] In a possible implementation manner, the method further includes: for any in-translation transaction, when the currently read change record is the change record of the commit event of any in-translation transaction, continue to translate until the change record of the commit event is translated.

[0074] Exemplarily, when the reading thread reads the change record, it can identify the specific content of the currently read change record to identify whether the change record is the change record of the commit event. For example, the reading thread can determine whether it is the change record of the commit event by checking whether the keyword "commit" is included in the change record. If it is identified that the keyword "commit" is included in the change record, then the change record is determined to be the change record of the commit event.

[0075] In a possible implementation, the method further includes: for any in-translation transaction, when the currently read change record is the change record of a rollback event of any in-translation transaction, stop translation and destroy the cached data in the translation thread of any in-translation transaction, and end the translation of any in-translation transaction.

[0076] Exemplarily, when the reading thread reads the change record, it can identify the specific content of the currently read change record to identify whether the change record is the change record of a rollback event. For example, the reading thread can determine whether it is the change record of a rollback event by checking whether the keyword "rollback" is included in the change record. If it is identified that the change record includes "rollback", then the change record is determined to be the change record of a rollback event.

[0077] Under normal circumstances, the last change record of a transaction is either the change record of a commit event or the change record of a rollback event. A commit event indicates that the change operations corresponding to the change records of the transaction have all been executed, while a rollback event indicates that the change operations corresponding to the change records of the transaction have not been executed.

[0078] Therefore, for any in-translation transaction, when the currently read change record is the change record of a commit event of any in-translation transaction, the translation of the change records of any in-translation transaction can be continued until the change record of the commit event is reached. In this way, for a transaction whose change operations have all been executed, the change records of the transaction can be continuously translated to completely obtain the actual data changes corresponding to the change operations of the transaction. On the contrary, for any in-translation transaction, when the currently read change record is the change record of a rollback event of any in-translation transaction, by stopping translation and destroying the cached data in the translation thread of any in-translation transaction and ending the translation of any in-translation transaction, the translation of the change operations that ultimately have not been executed can be stopped in a timely manner, the computing resources required by the translation thread can be stopped from being continuously consumed, the cache space can be released, and the storage and computing overhead can be saved.

[0079] In a possible implementation, the translation thread includes a definition change cache, and the definition change cache is used to store the change records of data definition language. When continuously translating until the change record of the commit event is reached, it specifically includes: for any in-translation transaction, when the currently read change record is the change record of the data definition language of any in-translation transaction, store the change record of the data definition language in the definition change cache, and update the data dictionary of the translation thread according to the change record of the data definition language; continuously translate until the change record of the commit event is reached, and update the main data dictionary of the database according to the change records of the data definition language stored in the definition change cache.

[0080] Exemplarily, a definition change cache corresponding to a translation thread can be set for any translation thread, and the definition change cache can be understood as a cache space for storing change records of data definition language.

[0081] Exemplarily, the change operations on data can include change operations of data definition language (DDL) and change operations of data manipulation language (DML). Among them, the change operations of DDL can be understood as mainly used to change the structure and definition of a database, such as operations of creating, modifying, or deleting tables and other database objects. These operations are usually one-time and affect the metadata of the database. The change operations of DML can be understood as mainly used to change the data content in the database, such as inserting, modifying, or deleting data. These operations are usually performed frequently and affect the actual data in the database.

[0082] The change record that records the change operation of any DDL is the change record of data definition language; the change record that records the change operation of any DML is the change record of data manipulation language.

[0083] For any translation transaction, when the currently read change record is the change record of the data definition language of any translation transaction, the change record of the data definition language can be stored in the definition change cache, and the data dictionary of the translation thread can be updated according to the change record of the data definition language.

[0084] Exemplarily, when the reading thread reads the current change record, it can determine whether the change record is a change record of DDL by identifying whether the change record includes keywords related to DDL. For example, by identifying whether it includes keywords for creating a table or keywords for modifying the table structure, etc., it can be determined whether the change record is a change record of DDL. When it is determined that the current change record is a change record of DDL, a copy of the change record of DDL can be stored in the definition change cache corresponding to the translation thread of this transaction. After storing a change record of DDL in the definition change cache, the data dictionary of this translation thread can be updated correspondingly according to the change operation executed by the change record of DDL, so that the updated data dictionary can continuously maintain effective guidance for translation.

[0085] Continue translating until the change record of the commit event is reached, and then the main data dictionary of the database can be updated accordingly according to all the change records of the data definition language stored in the definition change cache and the change operations performed by each change record on the structure and definition of the database.

[0086] Since the embodiments of the present application are directed to transactions that have executed a commit operation, the DDL change operations corresponding to the change records of each DDL of the transaction have taken effect. By updating the main data dictionary of the database according to the change records of each data definition language stored in the definition change cache, the main data dictionary can be unified with the substantial modifications made to the database structure and definition, enabling the main data dictionary to correctly represent the current database structure information of the database and keeping the main data dictionary valid. Based on this, the data dictionary created through the updated main data dictionary can effectively guide the translation work of subsequent translation threads, avoid translation errors caused by dirty reads of subsequent translation threads, and improve the accuracy of translation change records and the reliability of transaction log parsing results.

[0087] In a possible implementation manner, the translation thread includes a blocking queue, and the blocking queue is used to store change records in a blocking queue manner. The method further includes: for any new transaction, sequentially storing the first change record and other change records of any new transaction read into the blocking queue; sequentially translating the change records stored in the blocking queue through the translation thread of any new transaction.

[0088] Exemplarily, the blocking queue may be a fixed-length blocking queue, and the number of change records cached by the fixed-length blocking queue may be preset according to the available space size of the cache space. For example, a fixed-length blocking queue storing n change records may be preset for the translation thread of a certain transaction, so as to orderly cache and sequentially process each change record through the fixed-length blocking queue, and n may be any positive integer.

[0089] When the reading thread reads the first change record and other change records of any new transaction, they can be sequentially stored into the blocking queue in the order in which the change records are read. If it is a fixed-length blocking queue, the read change records can be deposited when the fixed-length blocking queue is not full; when the fixed-length blocking queue is full, it can wait until the change record at the head of the queue is extracted and then continue to deposit.

[0090] In this embodiment, by setting a blocking queue in the translation thread, the difference between the translation speed of the translation thread and the reading speed of the reading thread can be balanced. Moderately controlling the speed of reading and caching each change record through the blocking queue is beneficial to reasonably adjusting the process of processing each change record of a large transaction in the case of limited cache space, preventing the overflow of change records or the data obtained after translation due to limited cache space, avoiding data loss and translation errors, and improving the reliability of transaction log parsing.

[0091] Exemplarily, in the method provided by the embodiments of the present application, the entire parsing thread may include a reading thread for reading transaction logs and several translation threads for translating the change records of transactions. Next, throughFigure 4 And Figure 5 Further introduce the working processes of the translation thread and the reading thread respectively.

[0092] Figure 4 It is a schematic diagram of the working process of the translation thread provided by the embodiment of this application. A data dictionary can be included in the translation thread to ensure visibility isolation when translating this transaction; a fixed-length blocking queue can also be included to store the change records to be translated of this transaction; a reordering cache that can automatically overflow to disk can also be included to store the data obtained after translation after translation; a DDL change cache, that is, a defined change cache, can also be included to store the change records corresponding to all DDL events in each change record of this transaction.

[0093] Such as Figure 4 shown, initialization can initialize each storage resource and computing resource, configure each parameter required for the parsing process, etc. For the translation thread of any transaction, within the period of translating this transaction, this translation thread is only responsible for translating each change record of this transaction.

[0094] After initializing and allocating a translation thread to a new transaction, this translation thread can determine whether there are change records to be translated. If not, it waits for the reading thread to continue reading until change records to be translated appear. If so, it pulls a change record to be translated and translates it through the data dictionary. Specifically, corresponding processing can be performed according to the type of the change record to be translated.

[0095] When the change record is a change record of a DML event, this change record can be stored in the reordering cache for translation; when the change record is a change record of a DDL event, the data dictionary in this translation thread can be updated according to this change record, and this change record is added to the DDL change cache of this translation thread; when the change record is a change record of a commit event or a rollback event, mark the end of this transaction, clean up the storage resources and computing resources of this translation thread and exit the loop, ending this translation thread.

[0096] Figure 5 It is a schematic diagram of the working process of the reading thread provided by the embodiment of this application. As Figure 5 shown, after initialization, the reading thread starts to read the transaction log to be parsed. It can first determine whether there are change records to be processed in this transaction log. If not, it exits the reading; if so, it continues to read.

[0097] During the process of reading the transaction log, when the first change record of a new transaction is read, a translation thread is allocated for the new transaction, and a snapshot can be created from the main data dictionary and configured as the data dictionary for the translation thread. The translation thread is started, resources and queues are initialized, and the first change record can be stored in the blocking queue of the translation thread for translation. Continue to read each change record of the transaction log. When the currently read change record is a change record of a transaction being translated, corresponding processing can be performed according to the type of the change record.

[0098] For example, if the currently read change record is a transaction being translated, this change record is stored in the blocking queue of the corresponding translation thread. If the change record is a change record corresponding to a commit event or a rollback event, the following additional processing is required:

[0099] In the first case, if it is a change record of a commit event, wait for the translation thread to finish translation, that is, until the change record of the commit event of this transaction is translated. After that, all the translated data in the reordering cache corresponding to this transaction is sequentially extracted to the encapsulation step. Then, it is judged whether there is change data in the DDL change cache of the translation thread. If so, the main data dictionary is updated according to the change data in the DDL change cache, that is, all DDL changes of this transaction are applied to the main data dictionary. After that, all the cached data in the translation thread can be destroyed. If not, that is, there is no change data in the DDL change cache of the translation thread, then all the cached data in the translation thread can be destroyed.

[0100] In the second case, if it is a change record of a rollback event, stop the translation thread. After that, all the cached data in the translation thread is destroyed.

[0101] In the third case, if it is a change record of other change types, that is, a change record that is neither a commit event nor a rollback event, such as a change record of a DDL event or a DML event, then continue to read the next change record.

[0102] The method for parsing the transaction log of the database provided by the embodiments of the present application realizes parallel processing of transaction log reading and translation on the basis of making each transaction visible to the data dictionary, and can greatly improve the performance of transaction log parsing. Since each transaction's translation thread has an independent data dictionary, this can prevent dirty reads during parallel translation of different transactions and cause translation errors, improving the reliability of parsing. In addition, reading the transaction log and translating the transaction are in parallel processing. The reading and translation processes of each transaction have changed from a serial mode to a pipelined parallel mode, effectively utilizing computing resources and improving the overall parsing speed. For the parsing of transaction logs including large transactions, the improvement effect of the parsing speed is particularly obvious.

[0103] Figure 6The following is a schematic structural diagram of the transaction log parsing device for the database provided by the embodiments of the present application. As Figure 6 shown, the embodiments of the present application provide a transaction log parsing device for a database. The device includes:

[0104] A reading module 601, configured to sequentially read each change record in the transaction log to be parsed through a reading thread;

[0105] An allocation module 602, configured to, when the currently read change record is the first change record of a new transaction, allocate a translation thread for the new transaction. The translation thread is used to translate the change records corresponding to the transaction. The new transaction is a transaction that has not been read since the transaction log to be parsed is sequentially read from the beginning;

[0106] A processing module 603, configured to parallelly translate each in-translation transaction of the allocated translation threads through the allocated translation threads, and encapsulate the data obtained after translation when the translation thread ends the translation, so as to complete the parsing of the transaction log.

[0107] In a possible implementation manner, the allocation module 602 is specifically configured to: create a snapshot according to the master data dictionary of the database, and determine the snapshot as the data dictionary used by the translation thread for data translation. The master data dictionary is a set of information representing the current database structure information of the database; allocate a translation thread including the data dictionary for the new transaction according to the data dictionary.

[0108] In a possible implementation manner, the processing module 603 is further configured to: for any in-translation transaction, when the currently read change record is the change record of the commit event of any in-translation transaction, continue to translate until the change record of the commit event is translated.

[0109] In a possible implementation manner, the translation thread includes a defined change cache, and the defined change cache is used to store the change records of the data definition language. The processing module 603 is specifically configured to: for any in-translation transaction, when the currently read change record is the change record of the data definition language of any in-translation transaction, store the change record of the data definition language in the defined change cache, and update the data dictionary of the translation thread according to the change record of the data definition language; continue to translate until the change record of the commit event is translated, and update the master data dictionary of the database according to the change records of the data definition language stored in the defined change cache.

[0110] In a possible implementation, the translation thread includes a blocking queue, which is used to store change records in the manner of a blocking queue. The processing module 603 is further configured to: for any new transaction, sequentially store the first change record read for any new transaction and other change records into the blocking queue; and sequentially translate the change records stored in the blocking queue through the translation thread of any new transaction.

[0111] In a possible implementation, the processing module 603 is further configured to: for any transaction being translated, in the case where the currently read change record is a change record of a rollback event of any transaction being translated, stop translation and destroy the cached data in the translation thread of any transaction being translated, and end the translation of any transaction being translated.

[0112] The transaction log parsing device of the database provided by the embodiments of the present application can be used to execute the technical solutions of the database transaction log parsing method in any of the above embodiments of the present application. The implementation principles and technical effects are similar, and will not be elaborated here in this embodiment.

[0113] Figure 7 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 7 shown, the electronic device of this embodiment may include: at least one processor 701; and a memory 702 communicatively connected to the at least one processor; wherein, the memory 702 stores instructions executable by the at least one processor 701, and when the instructions are executed by the at least one processor 701, the electronic device executes the method in any of the above embodiments.

[0114] Optionally, the memory 702 can be either independent or integrated with the processor 701.

[0115] The implementation principles and technical effects of the electronic device provided by this embodiment can be referred to the foregoing embodiments, and will not be elaborated here.

[0116] The embodiments of the present application further provide a computer-readable storage medium, in which computer-executable instructions are stored. When the processor executes the computer-executable instructions, the method in any of the foregoing embodiments is implemented.

[0117] The embodiments of the present application further provide a computer program product, including a computer program, which when executed by a processor, implements the method in any of the foregoing embodiments.

[0118] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0119] The integrated modules implemented in the form of software function modules can be stored in a computer-readable storage medium. The above software function modules stored in a storage medium include several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute some steps of the methods described in the various embodiments of the present application.

[0120] It should be understood that the above processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk memory, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disc, etc.

[0121] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic memory, flash memory, a magnetic disk or an optical disk. The storage medium can be any available medium accessible by a general-purpose or special-purpose computer.

[0122] An exemplary storage medium is coupled to the processor, enabling the processor to read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application-specific integrated circuit. Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a master device.

[0123] It should be noted that in this document, the terms "including", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitations, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including such element.

[0124] The serial numbers of the above embodiments of the present application are only for description and do not represent the superiority or inferiority of the embodiments.

[0125] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general-purpose hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0126] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be similarly included in the patent protection scope of the present application.

[0127] Those skilled in the art will readily conceive of other embodiments of the embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the embodiments of the present application, which follow the general principles of the embodiments of the present application and include known common knowledge or conventional technical means in the technical field not disclosed in the embodiments of the present application.

[0128] It should be understood that the embodiments of the present application are not limited to the exact structures already described and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the embodiments of the present application is only limited by the appended claims.

Claims

1. A database transaction log parsing method, characterized in that: The method comprises: Each change record in the transaction log to be parsed is read in sequence by a reading thread; In the case where the change record currently being read is the first change record of a new transaction, a translation thread is allocated to the new transaction, and the translation thread is used to translate the change record corresponding to the transaction, and the new transaction is a transaction that has not been read since the transaction log to be parsed was read sequentially from the beginning; Through each allocated translation thread, each translation transaction of the allocated translation thread is translated in parallel, and when the translation thread finishes the translation, the data obtained after the translation is encapsulated to complete the parsing of the transaction log.

2. The method according to claim 1, characterized in that The allocating a translation thread to the new transaction comprises: Creating a snapshot according to a master data dictionary of the database, and determining the snapshot as a data dictionary used by a translation thread for data translation, wherein the master data dictionary is an information set representing current database structure information of the database; A translation thread including the data dictionary is allocated to the new transaction according to the data dictionary.

3. The method according to claim 2, characterized in that The method further comprises: For any transaction being translated, if the currently read change record is the change record of the commit event of any transaction being translated, the translation is continued until the translation reaches the change record of the commit event.

4. The method according to claim 3, characterized in that The translation thread includes a definition change cache, the definition change cache is used to store the change record of the data definition language, and the continuous translation until the change record of the submission event is translated includes: For any of the translation-in-progress transactions, if the currently read change record is a change record of the data definition language of any of the translation-in-progress transactions, the change record of the data definition language is stored in the definition change cache, and the data dictionary of the translation thread is updated according to the change record of the data definition language; The translation is continued until the change record of the submission event is translated, and the master data dictionary of the database is updated according to the change record of the data definition language stored in the definition change cache.

5. The method according to any one of claims 1 to 4, characterized in that: The translation thread includes a blocking queue, and the blocking queue is used to store change records in a blocking queue manner. The method further includes: For any new transaction, the first change record and other change records of any new transaction read are sequentially stored in the blocking queue; The change records stored in the blocking queue are translated in sequence by the translation thread of any new transaction.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: For any transaction in translation, when the currently read change record is the change record of the rollback event of any transaction in translation, stop translation and destroy cache data in the translation thread of any transaction in translation, and end the translation of any transaction in translation.

7. A database transaction log parsing device, characterized in that: The device comprises: A reading module is used to sequentially read each change record in the transaction log to be parsed through a reading thread; An allocation module, configured to allocate a translation thread to a new transaction when the currently read change record is the first change record of the new transaction, wherein the translation thread is configured to translate the change record corresponding to the transaction, and the new transaction is a transaction that has not been read since the transaction log to be parsed was first read in sequence; The processing module is used to translate the various translation transactions of the allocated translation threads in parallel through the allocated translation threads, and to encapsulate the data obtained after the translation when the translation thread finishes the translation, so as to complete the parsing of the transaction log.

8. An electronic device, characterized in that: include: Memory and processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images