A data processing method and device

Abstract

The present invention discloses a data processing method and device, and relates to the field of computer technology. A specific implementation of the method includes: obtaining the redo log file of the master node of the database; parsing the redo log file of the master node to obtain log data and its corresponding identification information, storing the log data in the physical transaction, and writing the physical transaction into the local redo log file; obtaining the physical data page corresponding to the physical transaction from the local redo log file according to the identification information, and performing corresponding data processing according to the physical data page. This implementation significantly reduces the data synchronization delay and data synchronization cost of the master and slave nodes of the database, and improves the scalability of data synchronization and the system performance of the database.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and device. Background Art

[0002] In the master-slave model of a relational database (such as MySQL), one master node corresponds to one or more slave nodes, and the master node is responsible for writing and the slave node is responsible for reading. This can solve the bottleneck of single-point load. Even if the master node locks the table or crashes, the normal operation of the business can be guaranteed by reading the slave node. To achieve the above situation, it is necessary to ensure data synchronization between the master node and the slave node of the database.

[0003] The existing methods mainly use binary log (binary log, referred to as binlog, after the transaction is committed, the master node server sends the generated binlog file to the slave node server. After receiving the binlog, the slave node server applies the logical log generated by the master node to its own database system, thereby realizing the master-slave data synchronization function) and redo log (redo log technology, recording the physical changes to the actual data file (what changes are made to the data at what location in the data file). Only when the redo log is stored on the disk of the slave node will the actual physical data page modifications be stored on the disk, thereby realizing master-slave data synchronization).

[0004] There are at least the following problems in the prior art:

[0005] Existing business processing methods have technical problems such as long data synchronization delay between master and slave nodes, high data synchronization cost, poor scalability, and impact on database system performance. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a data processing method and device, which can significantly reduce the data synchronization delay of the master and slave nodes of the database, reduce the data synchronization cost, improve the scalability of data synchronization, and improve the system performance of the database.

[0007] To achieve the above object, according to a first aspect of an embodiment of the present invention, a data processing method is provided, comprising:

[0008] Get the redo log file of the database master node;

[0009] Parse the redo log file of the master node to obtain log data and its corresponding identification information, store the log data in the physical transaction, and write the physical transaction into the local redo log file;

[0010] The physical data page corresponding to the physical transaction is obtained from the local redo log file according to the identification information, and corresponding data processing is performed according to the physical data page.

[0011] Furthermore, obtaining the redo log file of the database master node also includes:

[0012] Determine the initial consistent points corresponding to the master node and the slave node of the database, obtain the redo log file after the log sequence number corresponding to the initial consistent point, and store the redo log file in the cache corresponding to the slave node.

[0013] Further, the redo log file of the master node is parsed to obtain log data and corresponding identification information, the log data is stored in the physical transaction, and the physical transaction is written into the local redo log file, further comprising:

[0014] Traverse and obtain the redo log files in the cache, classify and parse the redo log files according to the log type, extract identification information and log data, store the log data in the physical transaction, and write the physical transaction to the local redo log file; wherein the identification information includes the tablespace number and the data page number.

[0015] Further, storing the log data into the physical transaction and writing the physical transaction into the local redo log file also includes:

[0016] The log data is stored in a physical transaction, and the physical transaction corresponding to a single log type and the physical transaction corresponding to multiple log types are determined;

[0017] Log sequence numbers are assigned to physical transactions corresponding to a single log type according to a fixed log data length, and log sequence numbers are assigned to physical transactions corresponding to multiple log types according to actual log data lengths;

[0018] Write the log data to the local redo log file according to the log sequence number corresponding to the physical transaction.

[0019] Further, obtaining the physical data page corresponding to the physical transaction from the local redo log file according to the identification information also includes:

[0020] According to the correspondence between the identification information in the hash table and the physical transaction, the physical data page corresponding to the physical transaction is obtained from the local redo log file; wherein the hash table is created after the step of storing the log data in the physical transaction.

[0021] Furthermore, in the case where a write operation is generated in the local redo log file, the method further includes:

[0022] Intercept write operations and determine the source of the write operations;

[0023] Correct the write status according to the source of the operation.

[0024] Furthermore, it also includes:

[0025] At least one of the following methods is used to correct the slave node transaction state corresponding to the undo log: prohibiting slave node transaction regeneration and clearing the rollback queue.

[0026] According to a second aspect of an embodiment of the present invention, there is provided a data processing device, including:

[0027] The file acquisition module is used to obtain the redo log files of the database master node;

[0028] The parsing module is used to parse the redo log file of the master node to obtain log data and its corresponding identification information, store the log data in the physical transaction, and write the physical transaction into the local redo log file;

[0029] The data processing module is used to obtain the physical data page corresponding to the physical transaction from the local redo log file according to the identification information, and perform corresponding data processing according to the physical data page.

[0030] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0031] one or more processors;

[0032] a storage device for storing one or more programs,

[0033] When one or more programs are executed by one or more processors, the one or more processors implement any of the above-mentioned data processing methods.

[0034] According to a fourth aspect of an embodiment of the present invention, there is provided a computer-readable medium on which a computer program is stored. When the program is executed by a processor, any of the above-mentioned data processing methods is implemented.

[0035] An embodiment of the above invention has the following advantages or beneficial effects: because the redo log file of the master node of the database is obtained; the redo log file of the master node is parsed and processed to obtain log data and its corresponding identification information, the log data is stored in the physical transaction, and the physical transaction is written to the local redo log file; the physical data page corresponding to the physical transaction is obtained from the local redo log file according to the identification information, and the corresponding data processing is performed according to the physical data page. The technical problems of long data synchronization delay between the master and slave nodes, high data synchronization cost, poor scalability, and impact on the system performance of the database in the existing business processing methods are overcome, thereby achieving the technical effect of significantly reducing the data synchronization delay between the master and slave nodes of the database, reducing the data synchronization cost, improving the scalability of data synchronization, and improving the system performance of the database.

[0036] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0038] Figure 1 is a schematic diagram of the main flow of a data processing method provided according to a first embodiment of the present invention;

[0039] Figure 2a is a schematic diagram of the main flow of a data processing method provided according to a second embodiment of the present invention;

[0040] Figure 2b yes Figure 2a A schematic diagram of the framework corresponding to the method;

[0041] Figure 3 is a schematic diagram of main modules of a data processing device provided according to an embodiment of the present invention;

[0042] Figure 4 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0043] Figure 5 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0045] Figure 1 is a schematic diagram of the main flow of the data processing method provided according to the first embodiment of the present invention; Figure 1 As shown, the data processing method provided by the embodiment of the present invention mainly includes:

[0046] Step S101, obtaining the redo log file of the database master node.

[0047] The redo log file (redolog) is used to record the physical changes to the actual data file (what data location in the data file is changed in what way). Only when the redolog is stored on the disk will the actual physical data page modification be stored on the disk. This design makes MySQL data writing very safe. Even if the database crashes halfway through writing, the redolog that has been stored on the disk can be used to perform local MySQL Crash Recovery.

[0048] Further, according to an embodiment of the present invention, the above-mentioned obtaining of the redo log file of the database master node also includes:

[0049] Determine the initial consistent points corresponding to the master node and the slave node of the database, obtain the redo log file after the log sequence number corresponding to the initial consistent point, and store the redo log file in the cache corresponding to the slave node.

[0050] Since the inconsistent database or the inconsistent points of the consistent database cannot apply the data in the redo log file, maintaining the initial consistent point is the basis for implementing the present application (according to a specific implementation of an embodiment of the present invention, the initial consistent point is implemented through the MySQL 8.0 cloning function).

[0051] Specifically, according to an embodiment of the present invention, by recording the LSN (Log Sequence Number, used to indicate the position of a specific log file record in a log file) of the consistent site, the LSN is used as the initial LSN, and the database slave node initiates a replication request to the master node to obtain a redolog log after the start LSN, and records the end LSN of this redolog, and stores the redo log file in the cache corresponding to the slave node. Through the above settings, the redolog of the master node database can be successfully redone and applied as database data in the slave node database through subsequent operations in the form of redolog physical replication, solving the system performance degradation and master-slave delay problems caused by the use of binlog logical replication in the existing method.

[0052] Preferably, according to an embodiment of the present invention, the redo log file after the initial consistent point is obtained through the communication protocol and communication request agreed upon by the database slave node and the master node.

[0053] Step S102, parse the redo log file of the master node to obtain log data and its corresponding identification information, store the log data into the physical transaction, and write the physical transaction into the local redo log file.

[0054] Specifically, according to an embodiment of the present invention, the above-mentioned parsing process of the redo log file in the cache follows the design rules of the storage engine Innodb, that is, mtr (Mini-transaction, mini transaction, also known as physical transaction) is used as the operation unit and mini log (MLOG, Mini-log) is used as the parsing unit to parse and process the redo log file of the primary node, store the log data in the physical transaction, and write the physical transaction into the local redo log file.

[0055] Further, according to an embodiment of the present invention, the above-mentioned parsing and processing of the redo log file of the master node to obtain log data and its corresponding identification information, storing the log data in the physical transaction, and writing the physical transaction into the local redo log file also includes:

[0056] Traverse and obtain the redo log files in the cache, classify and parse the redo log files according to the log type, extract identification information and log data, store the log data in the physical transaction, and write the physical transaction to the local redo log file; wherein the above identification information includes the tablespace number and the data page number.

[0057] After the step of storing the redo log file in the cache corresponding to the slave node, activate the parsing thread (logparser) to implement the parsing process of the redo log using the parsing thread. According to a specific implementation of an embodiment of the present invention, the redo log file in the cache is continuously traversed, and if there is no data (i.e., there is no redo log file in the cache), it is dormant and waiting; if there is, the redo log file is obtained, and the redo log file is classified and parsed according to the log type, and the table space number (space_id) and the data page number (page_no) (i.e., identification information) and log data are extracted. After the log data is stored in the physical transaction, the physical transaction is written to the local redo log file. Through the above settings, the redo log file of the master node is parsed and constructed as a locally available physical transaction, which lays a good foundation for the subsequent slave node to apply the data of the physical transaction, and at the same time significantly reduces the data synchronization delay of the master and slave nodes of the database and reduces the data synchronization cost.

[0058] Preferably, according to an embodiment of the present invention, the above-mentioned storing log data into physical transactions and writing physical transactions into local redo log files further includes:

[0059] The log data is stored in the physical transaction, and the physical transaction corresponding to the single log type and the physical transaction corresponding to the multiple log types are determined;

[0060] Log sequence numbers are assigned to physical transactions corresponding to a single log type according to a fixed log data length, and log sequence numbers are assigned to physical transactions corresponding to multiple log types according to actual log data lengths;

[0061] Write the log data into the local redo log file according to the log sequence number corresponding to the physical transaction.

[0062] According to an embodiment of the present invention, after the redo log file is parsed with the mini log as the parsing unit, it will be divided into a single log type and multiple log types and stored in the physical transaction. For the physical transaction of the single log type, the log sequence number corresponding to the log data length can be directly allocated; and the physical transaction of the multiple log types increases the length to be allocated according to the length of each parsing unit during the parsing process. It is necessary to wait until all the mini units are parsed and then allocate the corresponding log sequence number according to the real-time log data length. Through the above settings, the log sequence number pre-allocation technology is combined with the log system design corresponding to the physical transaction of the storage engine itself, and the log sequence number is pre-allocated for the physical transaction to be applied (referring to the physical data page in the application physical transaction), thereby ensuring the continuity and orderliness of the subsequent application of the physical transaction.

[0063] Step S103, obtaining the physical data page corresponding to the physical transaction from the local redo log file according to the identification information, and performing corresponding data processing according to the physical data page.

[0064] A redo log is usually a physical log that records physical modifications to data pages rather than how a particular row or rows are modified. It is used to restore physical data pages after submission (restore data pages, and can only be restored to the last submitted position).

[0065] Specifically, according to an embodiment of the present invention, the above method of obtaining a physical data page corresponding to a physical transaction from a local redo log file according to identification information further includes:

[0066] According to the correspondence between the identification information in the hash table and the physical transaction, the physical data page corresponding to the physical transaction is obtained from the local redo log file; wherein the hash table is created after the step of storing the log data in the physical transaction.

[0067] After the step of writing log data into the local redo log file according to the embodiment of the present invention, the application thread is activated to apply the physical transaction to the buffer pool using the application thread. Specifically, since the parsed physical transaction is stored in the hash table according to the table space number and the data page number, the physical data page can be obtained from the cache according to the table space number and the data page number. If it exists, it is directly applied. If it does not exist, the page is first read from the disk. Through the above settings, the scalability of data synchronization is improved and the system performance of the database is improved.

[0068] According to the embodiment of the present invention, due to the disorder of the hash table, the application of the redo log file must be orderly. The advantage of the hash table is that data can be applied according to the table space number and the data page number, reducing the update frequency of the physical data page in the buffer pool and reducing the number of disk reads. At the same time, the orderliness of the application of the redo log file is also replaced by using a linear application technology, that is, a linear storage structure is used to replace the hash table structure to store physical transactions.

[0069] Further, according to an embodiment of the present invention, in the case where a write operation is generated in a local redo log file, the method further includes:

[0070] Intercept write operations and determine the source of the write operations;

[0071] Correct the write status according to the source of the operation.

[0072] For any MySQL instance, it is necessary to store the metadata information of its own database. Since MySQL 8.0, MySQL's metadata is completely stored by the Innodb storage engine, so all metadata modifications will be written to the redolog. Any free writing of local redolog will cause inconsistencies between the local physical data page and the physical data page changes recorded in the master node redolog. This inconsistency may cause data application failure at best, or directly damage the physical file and make the database unavailable at worst. Therefore, through the above settings, the dual guarantee of external truncation and internal verification is adopted to ensure the consistency of node data, thereby ensuring the applicability of the local redo log file of the slave node and improving the scalability of data synchronization.

[0073] Preferably, according to an embodiment of the present invention, the method further includes:

[0074] At least one of the following methods is used to correct the slave node transaction state corresponding to the undo log: prohibiting slave node transaction regeneration and clearing the rollback queue.

[0075] Undo tablespace (undo space), also known as undo log (undolog), is an important means for Innodb to ensure data transactionality. Undolog and redolog cooperate with each other to ensure the consistency of database transactions when transactions are committed, rolled back, and crashed and recovered. However, undolog itself will also generate redolog during the execution process. If this process occurs when the redolog data application needs to be executed from the node, the consequence of free writing will inevitably occur. Through the above-mentioned settings, in addition to the above-mentioned means of external truncation and internal verification, the state of the undo tablespace is corrected by taking multiple means such as preventing the slave node transaction from regenerating and clearing the rollback queue. According to a specific implementation of an embodiment of the present invention, after the slave node is restarted, the unfinished transaction will be regenerated, and then the undolog will be executed to prohibit its regeneration, and the content executed by the undo (undo) operation can all be obtained by the application of redolog, and the same principle is adopted for clearing the rollback queue. Through the above-mentioned settings, it is ensured that the state of the slave node is consistent with the state of the tablespace, thereby ensuring the consistency of the log sequence number, and thus significantly improving the system performance of the database.

[0076] According to the technical solution of the embodiment of the present invention, the redo log file of the master node of the database is obtained; the redo log file of the master node is parsed to obtain log data and its corresponding identification information, the log data is stored in the physical transaction, and the physical transaction is written to the local redo log file; the physical data page corresponding to the physical transaction is obtained from the local redo log file according to the identification information, and the corresponding data processing is performed according to the physical data page. Therefore, the technical problems of long data synchronization delay between the master and slave nodes, high data synchronization cost, poor scalability and impact on the system performance of the database in the existing business processing methods are overcome, thereby achieving the technical effect of significantly reducing the data synchronization delay between the master and slave nodes of the database, reducing the data synchronization cost, improving the scalability of data synchronization, and improving the system performance of the database.

[0077] Figure 2a is a schematic diagram of the main flow of the data processing method provided according to the second embodiment of the present invention; Figure 2a As shown, the data processing method provided by the present invention mainly includes:

[0078] Step S201, determine the initial consistent points corresponding to the master node and the slave node of the database, obtain the redo log file after the log sequence number corresponding to the initial consistent point, and store the redo log file in the cache corresponding to the slave node.

[0079] Since inconsistent databases or inconsistent sites of consistent databases cannot apply data in redo log files, maintaining the initial consistent site is the basis for implementing the present application. According to a specific implementation of an embodiment of the present invention, the initial consistent site is implemented through the MySQL 8.0 cloning function.

[0080] Specifically, according to an embodiment of the present invention, by recording the LSN (Log Sequence Number, used to indicate the position of a specific log file record in a log file) of the consistent site, the LSN is used as the initial LSN, and the database slave node initiates a replication request to the master node to obtain a redolog log after the start LSN, and records the end LSN of this redolog, and stores the redo log file in the cache corresponding to the slave node. Through the above settings, the redolog of the master node database can be successfully redone and applied as database data in the slave node database through subsequent operations in the form of redolog physical replication, solving the system performance degradation and master-slave delay problems caused by the use of binlog logical replication in the existing method.

[0081] Preferably, according to an embodiment of the present invention, the redo log file after the initial consistent point is obtained through the communication protocol and communication request agreed upon by the database slave node and the master node.

[0082] Step S202, traverse and obtain the redo log files in the cache, classify and parse the redo log files according to the log type, and extract identification information and log data.

[0083] Specifically, according to an embodiment of the present invention, the above-mentioned parsing process of the redo log file in the cache follows the design rules of the storage engine Innodb, that is, mtr (Mini-transaction, mini transaction, also known as physical transaction) is used as the operation unit and mini log (MLOG, Mini-log) is used as the parsing unit to parse and process the redo log file of the primary node, store the log data in the physical transaction, and write the physical transaction into the local redo log file.

[0084] After the step of storing the redo log file in the cache corresponding to the slave node, activate the parsing thread (logparser) to implement the parsing process of the redo log using the parsing thread. According to a specific implementation of an embodiment of the present invention, the redo log file in the cache is continuously traversed, and if there is no data (i.e., there is no redo log file in the cache), it is dormant and waiting; if there is, the redo log file is obtained, and the redo log file is classified and parsed according to the log type, and the table space number (space_id) and the data page number (page_no) (i.e., identification information) and log data are extracted. After the log data is stored in the physical transaction, the physical transaction is written to the local redo log file. Through the above settings, the redo log file of the master node is parsed and constructed as a locally available physical transaction, which lays a good foundation for the subsequent slave node to apply the data of the physical transaction, and at the same time significantly reduces the data synchronization delay of the master and slave nodes of the database and reduces the data synchronization cost.

[0085] Step S203: Store the log data into a physical transaction, and determine a physical transaction corresponding to a single log type and a physical transaction corresponding to multiple log types.

[0086] Step S204 , allocating log sequence numbers to physical transactions corresponding to a single log type according to a fixed log data length, and allocating log sequence numbers to physical transactions corresponding to multiple log types according to actual log data lengths.

[0087] According to an embodiment of the present invention, after the redo log file is parsed with mini-logs as parsing units, it will be divided into single log type and multiple log types and stored in physical transactions. For physical transactions of single log type, the log sequence number corresponding to the log data length can be directly assigned; while for physical transactions of multiple log types, the length to be allocated is increased incrementally according to the length of each parsing unit during the parsing process. It is necessary to wait until all mini-units are parsed and then assign the corresponding log sequence number according to the real-time log data length.

[0088] Step S205, writing the log data into the local redo log file according to the log sequence number corresponding to the physical transaction.

[0089] Through the above settings, the log sequence number pre-allocation technology is combined with the log system design corresponding to the physical transaction of the storage engine itself, and the log sequence number is pre-allocated for the physical transaction to be applied (referring to the physical data page in the application of the physical transaction), thereby ensuring the continuity and orderliness of the subsequent application of the physical transaction.

[0090] Step S206, according to the correspondence between the identification information in the hash table and the physical transaction, obtain the physical data page corresponding to the physical transaction from the local redo log file.

[0091] A redo log is usually a physical log that records physical modifications to data pages rather than how a particular row or rows are modified. It is used to restore physical data pages after submission (restore data pages, and can only be restored to the last submitted position).

[0092] After the step of writing log data into the local redo log file according to the embodiment of the present invention, the application thread is activated to apply the physical transaction to the buffer pool using the application thread. Specifically, since the parsed physical transaction is stored in the hash table according to the table space number and the data page number, the physical data page can be obtained from the cache according to the table space number and the data page number. If it exists, it is directly applied. If it does not exist, the page is first read from the disk. Through the above settings, the scalability of data synchronization is improved and the system performance of the database is improved.

[0093] Step S207, performing corresponding data processing according to the physical data page.

[0094] Specifically, Figure 2b As shown in the figure, the Master node is the database master node, the Slave node is the slave node, the logparser is the parsing thread, and the log applier is the application thread.

[0095] According to the embodiment of the present invention, due to the disorder of the hash table, the application of the redo log file must be orderly. The advantage of the hash table is that data can be applied according to the table space number and the data page number, reducing the update frequency of the physical data page in the buffer pool and reducing the number of disk reads. At the same time, the orderliness of the application of the redo log file is also replaced by using a linear application technology, that is, a linear storage structure is used to replace the hash table structure to store physical transactions.

[0096] Further, according to an embodiment of the present invention, in the case where a write operation is generated in a local redo log file, the method further includes:

[0097] Intercept write operations and determine the source of the write operations;

[0098] Correct the write status according to the source of the operation.

[0099] For any MySQL instance, it is necessary to store the metadata information of its own database. Since MySQL 8.0, MySQL's metadata is completely stored by the Innodb storage engine, so all metadata modifications will be written to the redolog. Any free writing of local redolog will cause inconsistencies between the local physical data page and the physical data page changes recorded in the master node redolog. This inconsistency may cause data application failure at best, or directly damage the physical file and make the database unavailable at worst. Therefore, through the above settings, the dual guarantee of external truncation and internal verification is adopted to ensure the consistency of node data, thereby ensuring the applicability of the local redo log file of the slave node and improving the scalability of data synchronization.

[0100] Preferably, according to an embodiment of the present invention, the method further includes:

[0101] At least one of the following methods is used to correct the slave node transaction state corresponding to the undo log: prohibiting slave node transaction regeneration and clearing the rollback queue.

[0102] Undo tablespace (undo space), also known as undo log (undolog), is an important means for Innodb to ensure data transactionality. Undolog and redolog cooperate with each other to ensure the consistency of database transactions when transactions are committed, rolled back, and crashed and recovered. However, undolog itself will also generate redolog during the execution process. If this process occurs when the redolog data application needs to be executed from the node, the consequence of free writing will inevitably occur. Through the above-mentioned settings, in addition to the above-mentioned means of external truncation and internal verification, the state of the undo tablespace is corrected by taking multiple means such as preventing the slave node transaction from regenerating and clearing the rollback queue. According to a specific implementation of an embodiment of the present invention, after the slave node is restarted, the unfinished transaction will be regenerated, and then the undolog will be executed to prohibit its regeneration, and the content executed by the undo (undo) operation can all be obtained by the application of redolog, and the same principle is adopted for clearing the rollback queue. Through the above-mentioned settings, it is ensured that the state of the slave node is consistent with the state of the tablespace, thereby ensuring the consistency of the log sequence number, and thus significantly improving the system performance of the database.

[0103] According to the technical solution of the embodiment of the present invention, the redo log file of the master node of the database is obtained; the redo log file of the master node is parsed to obtain log data and its corresponding identification information, the log data is stored in the physical transaction, and the physical transaction is written to the local redo log file; the physical data page corresponding to the physical transaction is obtained from the local redo log file according to the identification information, and the corresponding data processing is performed according to the physical data page. Therefore, the technical problems of long data synchronization delay between the master and slave nodes, high data synchronization cost, poor scalability and impact on the system performance of the database in the existing business processing methods are overcome, thereby achieving the technical effect of significantly reducing the data synchronization delay between the master and slave nodes of the database, reducing the data synchronization cost, improving the scalability of data synchronization, and improving the system performance of the database.

[0104] Figure 3 is a schematic diagram of main modules of a data processing device provided according to an embodiment of the present invention; Figure 3 As shown, the data processing device 300 provided in this embodiment of the present invention mainly includes:

[0105] The file acquisition module 301 is used to acquire the redo log file of the database master node.

[0106] The redo log file (redolog) is used to record the physical changes to the actual data file (what data location in the data file is changed in what way). Only when the redolog is stored on the disk will the actual physical data page modification be stored on the disk. This design makes MySQL data writing very safe. Even if the database crashes halfway through writing, the redolog that has been stored on the disk can be used to perform local MySQL Crash Recovery.

[0107] Further, according to an embodiment of the present invention, the file acquisition module 301 is further used to:

[0108] Determine the initial consistent points corresponding to the master node and the slave node of the database, obtain the redo log file after the log sequence number corresponding to the initial consistent point, and store the redo log file in the cache corresponding to the slave node.

[0109] Since the inconsistent database or the inconsistent points of the consistent database cannot apply the data in the redo log file, maintaining the initial consistent point is the basis for implementing the present application (according to a specific implementation of an embodiment of the present invention, the initial consistent point is implemented through the MySQL 8.0 cloning function).

[0110] Specifically, according to an embodiment of the present invention, by recording the LSN (Log Sequence Number, used to indicate the position of a specific log file record in a log file) of the consistent site, the LSN is used as the initial LSN, and the database slave node initiates a replication request to the master node to obtain a redolog log after the start LSN, and records the end LSN of this redolog, and stores the redo log file in the cache corresponding to the slave node. Through the above settings, the redolog of the master node database can be successfully redone and applied as database data in the slave node database through subsequent operations in the form of redolog physical replication, solving the system performance degradation and master-slave delay problems caused by the use of binlog logical replication in the existing method.

[0111] Preferably, according to an embodiment of the present invention, the redo log file after the initial consistent point is obtained through the communication protocol and communication request agreed upon by the database slave node and the master node.

[0112] The parsing module 302 is used to parse the redo log file of the master node to obtain log data and its corresponding identification information, store the log data into a physical transaction, and write the physical transaction into a local redo log file.

[0113] Specifically, according to an embodiment of the present invention, the above-mentioned parsing process of the redo log file in the cache follows the design rules of the storage engine Innodb, that is, mtr (Mini-transaction, mini transaction, also known as physical transaction) is used as the operation unit and mini log (MLOG, Mini-log) is used as the parsing unit to parse and process the redo log file of the primary node, store the log data in the physical transaction, and write the physical transaction into the local redo log file.

[0114] Further, according to an embodiment of the present invention, the above-mentioned parsing module 302 is also used for:

[0115] Traverse and obtain the redo log files in the cache, classify and parse the redo log files according to the log type, extract identification information and log data, store the log data in the physical transaction, and write the physical transaction to the local redo log file; wherein the above identification information includes the tablespace number and the data page number.

[0116] After the step of storing the redo log file in the cache corresponding to the slave node, activate the parsing thread (logparser) to implement the parsing process of the redo log using the parsing thread. According to a specific implementation of an embodiment of the present invention, the redo log file in the cache is continuously traversed, and if there is no data (i.e., there is no redo log file in the cache), it is dormant and waiting; if there is, the redo log file is obtained, and the redo log file is classified and parsed according to the log type, and the table space number (space_id) and the data page number (page_no) (i.e., identification information) and log data are extracted. After the log data is stored in the physical transaction, the physical transaction is written to the local redo log file. Through the above settings, the redo log file of the master node is parsed and constructed as a locally available physical transaction, which lays a good foundation for the subsequent slave node to apply the data of the physical transaction, and at the same time significantly reduces the data synchronization delay of the master and slave nodes of the database and reduces the data synchronization cost.

[0117] Preferably, according to an embodiment of the present invention, the above-mentioned parsing module 302 is further used for:

[0118] The log data is stored in a physical transaction, and the physical transaction corresponding to a single log type and the physical transaction corresponding to multiple log types are determined;

[0119] Log sequence numbers are assigned to physical transactions corresponding to a single log type according to a fixed log data length, and log sequence numbers are assigned to physical transactions corresponding to multiple log types according to actual log data lengths;

[0120] Write the log data to the local redo log file according to the log sequence number corresponding to the physical transaction.

[0121] According to an embodiment of the present invention, after the redo log file is parsed with the mini log as the parsing unit, it will be divided into a single log type and multiple log types and stored in the physical transaction. For the physical transaction of the single log type, the log sequence number corresponding to the log data length can be directly allocated; and the physical transaction of the multiple log types increases the length to be allocated according to the length of each parsing unit during the parsing process. It is necessary to wait until all the mini units are parsed and then allocate the corresponding log sequence number according to the real-time log data length. Through the above settings, the log sequence number pre-allocation technology is combined with the log system design corresponding to the physical transaction of the storage engine itself, and the log sequence number is pre-allocated for the physical transaction to be applied (referring to the physical data page in the application physical transaction), thereby ensuring the continuity and orderliness of the subsequent application of the physical transaction.

[0122] The data processing module 303 is used to obtain the physical data page corresponding to the physical transaction from the local redo log file according to the identification information, and perform corresponding data processing according to the physical data page.

[0123] Specifically, according to an embodiment of the present invention, the data processing module 303 is further used to:

[0124] According to the correspondence between the identification information in the hash table and the physical transaction, the physical data page corresponding to the physical transaction is obtained from the local redo log file; wherein the hash table is created after the step of storing the log data in the physical transaction.

[0125] After the step of writing log data into the local redo log file according to the embodiment of the present invention, the application thread is activated to apply the physical transaction to the buffer pool using the application thread. Specifically, since the parsed physical transaction is stored in the hash table according to the table space number and the data page number, the physical data page can be obtained from the cache according to the table space number and the data page number. If it exists, it is directly applied. If it does not exist, the page is first read from the disk. Through the above settings, the scalability of data synchronization is improved and the system performance of the database is improved.

[0126] According to the embodiment of the present invention, due to the disorder of the hash table, the application of the redo log file must be orderly. The advantage of the hash table is that data can be applied according to the table space number and the data page number, reducing the update frequency of the physical data page in the buffer pool and reducing the number of disk reads. At the same time, the orderliness of the application of the redo log file is also replaced by using a linear application technology, that is, a linear storage structure is used to replace the hash table structure to store physical transactions.

[0127] Further, according to an embodiment of the present invention, the data processing device 300 further includes a write state correction module. When a write operation is generated in a local redo log file, the write state correction module is used to:

[0128] Intercept write operations and determine the source of the write operations;

[0129] Correct the write status according to the source of the operation.

[0130] For any MySQL instance, it is necessary to store the metadata information of its own database. Since MySQL 8.0, MySQL's metadata is completely stored by the Innodb storage engine, so all metadata modifications will be written to the redolog. Any free writing of local redolog will cause inconsistencies between the local physical data page and the physical data page changes recorded in the master node redolog. This inconsistency may cause data application failure at best, or directly damage the physical file and make the database unavailable at worst. Therefore, through the above settings, the dual guarantee of external truncation and internal verification is adopted to ensure the consistency of node data, thereby ensuring the applicability of the local redo log file of the slave node and improving the scalability of data synchronization.

[0131] Preferably, according to an embodiment of the present invention, the data processing device 300 further includes a module for correcting the transaction state of a slave node, which is used to:

[0132] At least one of the following methods is used to correct the slave node transaction state corresponding to the undo log: prohibiting slave node transaction regeneration and clearing the rollback queue.

[0133] Undo tablespace (undo space), also known as undo log (undolog), is an important means for Innodb to ensure data transactionality. Undolog and redolog cooperate with each other to ensure the consistency of database transactions when transactions are committed, rolled back, and crashed and recovered. However, undolog itself will also generate redolog during the execution process. If this process occurs when the redolog data application needs to be executed from the node, the consequence of free writing will inevitably occur. Through the above-mentioned settings, in addition to the above-mentioned means of external truncation and internal verification, the state of the undo tablespace is corrected by taking multiple means such as preventing the slave node transaction from regenerating and clearing the rollback queue. According to a specific implementation of an embodiment of the present invention, after the slave node is restarted, the unfinished transaction will be regenerated, and then the undolog will be executed to prohibit its regeneration, and the content executed by the undo (undo) operation can all be obtained by the application of redolog, and the same principle is adopted for clearing the rollback queue. Through the above-mentioned settings, it is ensured that the state of the slave node is consistent with the state of the tablespace, thereby ensuring the consistency of the log sequence number, and thus significantly improving the system performance of the database.

[0134] According to the technical solution of the embodiment of the present invention, the redo log file of the master node of the database is obtained; the redo log file of the master node is parsed to obtain log data and its corresponding identification information, the log data is stored in the physical transaction, and the physical transaction is written to the local redo log file; the physical data page corresponding to the physical transaction is obtained from the local redo log file according to the identification information, and the corresponding data processing is performed according to the physical data page. Therefore, the technical problems of long data synchronization delay between the master and slave nodes, high data synchronization cost, poor scalability and impact on the system performance of the database in the existing business processing methods are overcome, thereby achieving the technical effect of significantly reducing the data synchronization delay between the master and slave nodes of the database, reducing the data synchronization cost, improving the scalability of data synchronization, and improving the system performance of the database.

[0135] Figure 4 An exemplary system architecture 400 is shown to which the data processing method or data processing device according to the embodiment of the present invention can be applied.

[0136] like Figure 4As shown, the system architecture 400 may include terminal devices 401, 402, 403, a network 404 and a server 405 (this architecture is only an example, and the components included in the specific architecture may be adjusted according to the specific application). The network 404 is used to provide a medium for a communication link between the terminal devices 401, 402, 403 and the server 405. The network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0137] Users can use terminal devices 401, 402, 403 to interact with server 405 through network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, 403, such as data processing applications, data synchronization applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0138] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0139] The server 405 may be a server that provides various services, such as a server that processes data (only as an example) received by the user using the terminal devices 401, 402, and 403. The server may analyze and process the received redo log files and other data, and feed back the processing results (such as log data and its corresponding identification information - only as an example) to the terminal device.

[0140] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 405 , and accordingly, the data processing device is generally disposed in the server 405 .

[0141] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0142] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of a terminal device or a server suitable for implementing an embodiment of the present invention. Figure 5 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0143] like Figure 5As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0144] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0145] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.

[0146] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0147] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a file acquisition module, a parsing module and a data processing module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the file acquisition module may also be described as a "module for acquiring redo log files of a database master node".

[0149] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining the redo log file of the database master node; parsing the redo log file of the master node to obtain log data and its corresponding identification information, storing the log data in the physical transaction, and writing the physical transaction into the local redo log file; obtaining the physical data page corresponding to the physical transaction from the local redo log file according to the identification information, and performing corresponding data processing according to the physical data page.

[0150] According to the technical solution of the embodiment of the present invention, the redo log file of the master node of the database is obtained; the redo log file of the master node is parsed to obtain log data and its corresponding identification information, the log data is stored in the physical transaction, and the physical transaction is written to the local redo log file; the physical data page corresponding to the physical transaction is obtained from the local redo log file according to the identification information, and the corresponding data processing is performed according to the physical data page. Therefore, the technical problems of long data synchronization delay between the master and slave nodes, high data synchronization cost, poor scalability and impact on the system performance of the database in the existing business processing methods are overcome, thereby achieving the technical effect of significantly reducing the data synchronization delay between the master and slave nodes of the database, reducing the data synchronization cost, improving the scalability of data synchronization, and improving the system performance of the database.

[0151] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: Get the redo log file of the database master node; Parsing the redo log file of the master node to obtain log data and its corresponding identification information, storing the log data in a physical transaction, and writing the physical transaction into a local redo log file; Acquire a physical data page corresponding to the physical transaction from the local redo log file according to the identification information, and perform corresponding data processing according to the physical data page; The parsing process of the redo log file of the master node to obtain log data and its corresponding identification information, storing the log data in a physical transaction, and writing the physical transaction into a local redo log file also includes: Traverse and obtain the redo log files in the cache, classify and parse the redo log files according to the log type, extract identification information and log data, store the log data in the physical transaction, and write the physical transaction into the local redo log file; wherein the identification information includes the table space number and the data page number.

2. The data processing method according to claim 1, characterized in that: The step of obtaining the redo log file of the database master node also includes: Determine the initial consistent points corresponding to the master node and the slave node of the database, obtain the redo log file after the log sequence number corresponding to the initial consistent point, and store the redo log file in the cache corresponding to the slave node.

3. The data processing method according to claim 1, characterized in that: Storing the log data into a physical transaction, and writing the physical transaction into a local redo log file, further comprising: The log data is stored in a physical transaction, and a physical transaction corresponding to a single log type and a physical transaction corresponding to multiple log types are determined; Allocate a log sequence number for the physical transaction corresponding to the single log type according to the fixed log data length, and allocate a log sequence number for the physical transaction corresponding to the multiple log types according to the actual log data length; The log data is written into a local redo log file according to the log sequence number corresponding to the physical transaction.

4. The data processing method according to claim 1, characterized in that: The step of obtaining the physical data page corresponding to the physical transaction from the local redo log file according to the identification information further includes: According to the correspondence between the identification information in the hash table and the physical transaction, the physical data page corresponding to the physical transaction is obtained from the local redo log file; wherein the hash table is created after the step of storing the log data in the physical transaction.

5. The data processing method according to claim 1, characterized in that: In the case where the local redo log file generates a write operation, the method further includes: intercepting the write operation and determining an operation source of the write operation; The write status is corrected according to the operation source.

6. The data processing method according to claim 5, characterized in that: Also includes: At least one of the following methods is used to correct the slave node transaction state corresponding to the undo log: prohibiting slave node transaction regeneration and clearing the rollback queue.

7. A data processing device, characterized in that: include: The file acquisition module is used to obtain the redo log files of the database master node; A parsing module, used for parsing the redo log file of the master node to obtain log data and its corresponding identification information, storing the log data in a physical transaction, and writing the physical transaction into a local redo log file; A data processing module, configured to obtain a physical data page corresponding to the physical transaction from the local redo log file according to the identification information, and perform corresponding data processing according to the physical data page; The parsing module is also used for: Traverse and obtain the redo log files in the cache, classify and parse the redo log files according to the log type, extract identification information and log data, store the log data in the physical transaction, and write the physical transaction into the local redo log file; wherein the identification information includes the table space number and the data page number.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images