Implementation method, device, equipment and storage medium of high-availability database system

Abstract

The present invention relates to the field of database technology, and discloses a method, apparatus, device, and storage medium for implementing a high-availability database system. The method comprises: detecting in real time whether a fault occurs during the operation of the main computing node; when a fault occurs during the operation of the main computing node, hot-upgrading the physical configuration specifications of the standby computing node to the same physical configuration specifications as the main computing node; performing a rapid crash recovery operation on the database instance running on the standby computing node to upgrade the standby computing node to a new main computing node; and when performing the rapid crash recovery operation, using the new main computing node to provide database services to the outside world. The database system of the present invention adopts an asymmetric master-slave node architecture, which further reduces the operating costs of the database while ensuring the reliable operation of the database system.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to an implementation method, apparatus, device and storage medium of a high-availability database system. Background Art

[0002] Both hardware and software failures are possible in the entire hardware and software stack of a database system. If it is a stand-alone database system, system failures require a long time to process (the database system takes a long time to restart, or manual intervention is required), resulting in prolonged database service unavailability. Obviously, a stand-alone database system cannot meet the application requirements of an enterprise-level database.

[0003] High-availability database systems are one of the mainstream solutions for enterprise-level databases on the market. They work by automatically handling and quickly recovering from database failures through redundant database nodes and automatic disaster recovery mechanisms. Even if a node in the database system becomes unavailable due to a problem, the database system can still provide normal database services to the outside world as a whole.

[0004] Traditional high-availability database systems require identical hardware configurations (CPU, memory, disk, etc.) between the primary and backup nodes. This means they are deployed with hardware equivalents (symmetrical master-slave nodes). Typical high-availability database systems on the market, such as MySQL Cluster and Oracle DataGuard, utilize a dual-node architecture with one primary and one backup node. While achieving minimal downtime and higher availability, these high-availability database systems incur several times the hardware cost of stand-alone database systems. Summary of the Invention

[0005] The main purpose of the present invention is to provide a method, device, equipment and storage medium for implementing a high-availability database system, aiming to solve the technical problem of high operating costs of traditional high-availability databases.

[0006] A first aspect of the present invention provides a method for implementing a high-availability database system, the high-availability database system comprising at least one primary computing node, a distributed storage, and a standby computing node, wherein the physical configuration specifications of the standby computing node are lower than those of the primary computing node, the method comprising:

[0007] Detect in real time whether a failure occurs during the operation of the main computing node;

[0008] When a failure occurs in the main computing node during operation, hot-upgrading the physical configuration specifications of the backup computing node to the same physical configuration specifications as the main computing node;

[0009] Performing a fast crash recovery operation on the database instance running on the standby computing node to upgrade the standby computing node to a new master computing node;

[0010] A fast crash recovery operation is performed on the database instance, and when the fast crash recovery operation is performed, the new primary computing node is used to provide database services externally.

[0011] Optionally, in a first implementation of the first aspect of the present invention, after upgrading the standby computing node to a new master computing node, the method further includes:

[0012] A new standby computing node is started, and a master-standby relationship is established between the new master computing node and the new standby computing node.

[0013] Optionally, in a second implementation of the first aspect of the present invention, performing a fast crash recovery operation on the database instance running on the standby computing node to upgrade the standby computing node to a new master computing node includes:

[0014] Loading unsynchronized redo logs from the redo log files in the distributed storage, parsing the unsynchronized redo logs, and storing the parsing results in a preset hash table;

[0015] Reading a physical page corresponding to preset necessary data from the distributed storage to the memory of the standby computing node for version update, and writing the updated physical page to the distributed storage, wherein the necessary data includes: data dictionary metadata, maximum transaction ID, and globally unique ID;

[0016] Performing version recovery on the necessary data based on the updated physical page and the updated rollback log in the physical page, in units of DDL operations, so that the version of the necessary data remains consistent with that at the time when the primary computing node fails, wherein after the version of the necessary data is recovered, the high-availability database system provides external services;

[0017] According to the updated physical page and the updated rollback log in the physical page, version recovery is performed on the table record of the high-availability database system in units of transactions, so that the version of the table record is consistent with that when the main computing node fails.

[0018] Optionally, in a third implementation of the first aspect of the present invention, when performing the crash recovery operation, using the new primary computing node to provide external database services includes:

[0019] When performing the crash recovery operation, receiving a database service request sent by an external client through the SQL engine where the new primary computing node is located, and determining a target table record to be processed by the database service request and a target physical page containing the target table record;

[0020] Initiate a first acquisition request for the target table record to a table record engine through the SQL engine, and initiate a second acquisition request for the target physical page containing the target table record to a physical page engine through the table record engine;

[0021] If the target physical page does not exist in the physical page cache pool of the physical page engine, initiating a third acquisition request for the target physical page to the distributed storage through the physical page engine;

[0022] When the distributed storage responds to the third acquisition request and returns the target physical page, intercepting the target physical page for version update, and storing the target physical page after the version update in the physical page cache pool;

[0023] Reading the target physical page from the physical page cache pool by the physical page engine and returning the target physical page to the table record engine in response to the second acquisition request;

[0024] Determining, by the table record engine, whether there is an unprocessed pending transaction in the target table record to be read in the target physical page;

[0025] If there is no unprocessed pending transaction, the target table record is returned to the SQL engine through the table record engine to respond to the first acquisition request; if there is an unprocessed pending transaction, the corresponding rollback log is loaded, and the unprocessed pending transaction is rolled back or committed using the loaded rollback log, and the target table record is returned to the SQL engine through the table record engine to respond to the first acquisition request;

[0026] Based on the target table record, the database service request is processed by the SQL engine, and the processing result is returned to the external client.

[0027] Optionally, in a fourth implementation of the first aspect of the present invention, parsing the unsynchronized redo log and storing the parsing result in a preset hash table includes:

[0028] Parsing the unsynchronized redo log to obtain a physical page number of the unsynchronized redo log;

[0029] A key-value pair consisting of the physical page number and the redo log is constructed with the physical page number as a keyword and the redo log corresponding to the physical page of the physical page number as a value, and the key-value pair is inserted into a preset hash table.

[0030] Optionally, in a fifth implementation of the first aspect of the present invention, when the distributed storage responds to the third acquisition request and returns the target physical page, intercepting the target physical page for version update, and storing the target physical page after the version update in the physical page cache pool includes:

[0031] When the distributed storage responds to the third acquisition request and returns the target physical page, intercepting the target physical page;

[0032] Searching the hash table according to the physical page number of the target physical page to obtain unapplied redo logs in the target physical page;

[0033] According to the unapplied redo log in the target physical page, the target physical page is updated in version, and the target physical page after the version update is stored in the physical page cache pool.

[0034] Optionally, in a sixth implementation of the first aspect of the present invention, the method for implementing the high-availability database system further includes:

[0035] After the high-availability database system provides external services, the physical pages of other data except the necessary data are updated according to the redo log of the physical page corresponding to the hash table, so that the physical page version of the other data is consistent with that when the main computing node fails.

[0036] A second aspect of the present invention provides an implementation device for a high-availability database system, the high-availability database system comprising at least one primary computing node, a distributed storage, and a standby computing node, wherein the physical configuration specifications of the standby computing node are lower than those of the primary computing node, the implementation device for the high-availability database system comprising:

[0037] A detection module is used to detect in real time whether a fault occurs during the operation of the main computing node;

[0038] An upgrade module, configured to hot-upgrade the physical configuration specifications of the standby computing node to the same physical configuration specifications as the primary computing node when a failure occurs during operation of the primary computing node;

[0039] A crash recovery module, configured to perform a fast crash recovery operation on the database instance running on the standby computing node, so as to upgrade the standby computing node to a new primary computing node;

[0040] The service module is used to adopt the new main computing node to provide database services externally when executing the fast crash recovery operation.

[0041] Optionally, in a first implementation of the second aspect of the present invention, the device for implementing the high-availability database system further includes:

[0042] The establishment module is used to start a new standby computing node and establish a master-standby relationship between the new master computing node and the new standby computing node.

[0043] Optionally, in a second implementation of the second aspect of the present invention, the crash recovery module is specifically configured to:

[0044] Loading unsynchronized redo logs from the redo log files in the distributed storage, parsing the unsynchronized redo logs, and storing the parsing results in a preset hash table;

[0045] Reading physical pages corresponding to preset necessary data from the distributed storage into the memory of the standby computing node for version update, and writing the updated physical pages to the distributed storage, wherein the necessary data includes: data dictionary metadata, maximum transaction ID, and globally unique ID;

[0046] Performing version recovery on the necessary data based on the updated physical page and the updated rollback log in the physical page, in units of DDL operations, so that the version of the necessary data remains consistent with that at the time when the primary computing node fails, wherein after the version of the necessary data is recovered, the high-availability database system provides external services;

[0047] According to the updated physical page and the updated rollback log in the physical page, version recovery is performed on the table record of the high-availability database system in units of transactions, so that the version of the table record is consistent with that when the main computing node fails.

[0048] Optionally, in a third implementation of the second aspect of the present invention, the service module is specifically configured to:

[0049] When performing the crash recovery operation, receiving a database service request sent by an external client through the SQL engine where the new primary computing node is located, and determining a target table record to be processed by the database service request and a target physical page containing the target table record;

[0050] Initiate a first acquisition request for the target table record to a table record engine through the SQL engine, and initiate a second acquisition request for the target physical page containing the target table record to a physical page engine through the table record engine;

[0051] If the target physical page does not exist in the physical page cache pool of the physical page engine, initiating a third acquisition request for the target physical page to the distributed storage through the physical page engine;

[0052] When the distributed storage responds to the third acquisition request and returns the target physical page, intercepting the target physical page for version update, and storing the target physical page after the version update in the physical page cache pool;

[0053] Reading the target physical page from the physical page cache pool by the physical page engine and returning the target physical page to the table record engine in response to the second acquisition request;

[0054] Determining, by the table record engine, whether there is an unprocessed pending transaction in the target table record to be read in the target physical page;

[0055] If there is no unprocessed pending transaction, the target table record is returned to the SQL engine through the table record engine to respond to the first acquisition request; if there is an unprocessed pending transaction, the corresponding rollback log is loaded, and the unprocessed pending transaction is rolled back or committed using the loaded rollback log, and the target table record is returned to the SQL engine through the table record engine to respond to the first acquisition request;

[0056] Based on the target table record, the database service request is processed by the SQL engine, and the processing result is returned to the external client.

[0057] Optionally, in a fourth implementation of the second aspect of the present invention, the crash recovery module is further configured to:

[0058] The unsynchronized redo log is parsed to obtain a physical page number of the unsynchronized redo log; a key-value pair consisting of the physical page number and the redo log is constructed using the physical page number as a keyword and the redo log corresponding to the physical page of the physical page number as a value, and the key-value pair is inserted into a preset hash table.

[0059] Optionally, in a fifth implementation of the second aspect of the present invention, the service module is further configured to:

[0060] When the distributed storage responds to the third acquisition request and returns the target physical page, the target physical page is intercepted; according to the physical page number of the target physical page, the hash table is retrieved to obtain the unapplied redo log in the target physical page; according to the unapplied redo log in the target physical page, the target physical page is version updated, and the target physical page after the version update is stored in the physical page cache pool.

[0061] Optionally, in a sixth implementation of the second aspect of the present invention, the crash recovery module is further configured to:

[0062] After the high-availability database system provides external services, the physical pages of other data except the necessary data are updated according to the redo log of the physical page corresponding to the hash table, so that the physical page version of the other data is consistent with that when the main computing node fails.

[0063] A third aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein instructions are stored in the memory;

[0064] The at least one processor calls the instructions in the memory to enable the computer device to execute the above-mentioned implementation method of the high-availability database system.

[0065] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored thereon, which, when executed by a processor, implement the above-mentioned method for implementing a high-availability database system.

[0066] The present invention provides a method, apparatus, device, and storage medium for implementing a high-availability database system, including a primary computing node and a backup computing node with asymmetric hardware configuration specifications, and distributed storage. The present invention implements a low-cost, asymmetric primary and backup node high-availability database system. When the primary computing node fails, the physical configuration specifications of the backup computing node are hot-upgraded to the same physical configuration specifications as the primary computing node, thereby upgrading the backup computing node to a new primary computing node. This ensures that the backup computing node quickly takes over after the primary computing node fails and simultaneously provides normal database services to the outside world. The backup computing node uses lower physical configuration specifications, thereby reducing the operating costs of the database from a hardware perspective. At the same time, the present invention can quickly implement external services, greatly reducing the impact of internal failures on external access. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A flowchart of an embodiment of a method for implementing a high-availability database system of the present invention is provided;

[0068] Figure 2 This is a technical architecture diagram of an embodiment of a high-availability database system of the present invention;

[0069] Figure 3 This is a schematic diagram of the implementation process of an embodiment of a high-availability database system of the present invention;

[0070] Figure 4This is a flow chart of an embodiment of a method for implementing a high-availability database system of the present invention for performing a rapid crash recovery operation on a database instance;

[0071] Figure 5 This is a flow chart of an embodiment of a method for implementing a high-availability database system of the present invention for providing external database services when performing a fast crash recovery operation;

[0072] Figure 6 A schematic diagram of functional modules of an embodiment of an implementation device of a high-availability database system of the present invention;

[0073] Figure 7 A schematic diagram of functional modules of another embodiment of an implementation device of a high-availability database system of the present invention;

[0074] Figure 8 The figure is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0075] Embodiments of the present invention provide an implementation method, apparatus, device and storage medium for a high-availability database system. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or that are inherent to these processes, methods, products or devices.

[0076] It should be noted that the implementation method of the high-availability database in the present invention is applicable to all types of database systems in principle, such as MySQL, PGSQL, MongoDB, etc. In different types of database systems, the basic concepts of the database may have some differences, which will not be introduced in detail here.

[0077] Before introducing the embodiments of the present invention, some basic concepts in database technology are introduced using the MySQL database system as an example:

[0078] Table: A database system stores the basic structure of the same type of data. A table stores multiple records, each of which consists of several fields.

[0079] Physical page: A data object that stores table records. The size of a physical page is fixed (e.g., 16KB). A physical page can store multiple records from a table.

[0080] Page Buffer Pool: A physical page buffer pool maintained in memory by the database system.

[0081] Transaction: A database transaction contains multiple SQL statements. At the same time, the database transaction mechanism will ensure that these SQL statements are either all executed correctly (if all SQL statements in the transaction are executed, the transaction is successfully committed) or none of them are executed (if some SQL statements in the transaction are not executed successfully, the transaction is not successfully committed or the transaction is rolled back).

[0082] Redo log: The database system generates a corresponding redo log for each physical page modification operation. The redo log faithfully records the transaction's modifications to the physical page.

[0083] Undolog: The database system generates a corresponding rollback log for each table record inserted, modified, or deleted. This log records the version of the table record before the transaction. After the transaction commits, these rollback logs are deleted. When a transaction is rolled back, the database system uses these rollback logs to roll back the modified table record, undoing the changes made by the transaction.

[0084] It should be noted that the rollback log is also stored in the physical page, and a physical page stores multiple rollback logs; that is, the physical page (Page) is not only a data object for storing table records (Record), but also a data object for storing rollback logs (Undolog).

[0085] SQL engine: SQL engine data is cached data that does not need to be persisted, so there is no need to restore SQL engine data during crash recovery.

[0086] Table Record Engine: The table record engine manages and maintains data called table records. During crash recovery, the table records must be restored to maintain transaction-level consistency at the time of the system failure.

[0087] Physical Page Engine: The physical page engine manages and maintains data called physical pages. During crash recovery, the physical page data must be restored to maintain consistency with the time of the system failure.

[0088] Data dictionary: The data managed and maintained by the data dictionary is the metadata of the library and table indexes. The metadata itself exists in the form of table records and is stored in physical pages.

[0089] For ease of understanding, a detailed description is given below in conjunction with the embodiments of the present invention.

[0090] The high-availability database system in the embodiment of the present invention includes at least one primary computing node, a distributed storage, and a standby computing node. The physical configuration specification of the standby computing node is lower than that of the primary computing node.

[0091] See also Figure 1 and Figure 2 , Figure 1 This is a flow chart of an embodiment of a method for implementing a high-availability database system according to the present invention. Figure 2 The following is a technical architecture diagram of an embodiment of a high-availability database system of the present invention. In this embodiment, the implementation method of the high-availability database system includes the following steps:

[0092] S10: Detect in real time whether a failure occurs during the operation of the main computing node;

[0093] When a high-availability database system operates normally, the primary compute node with high physical configuration specifications receives and processes client read and write requests, and writes physical pages and redo logs to distributed storage. At the same time, whenever the primary compute node commits a transaction, it notifies the standby compute node of the redo log position of the transaction through the TCP channel and protocol. After receiving the latest redo log position information, the standby compute node with low physical configuration specifications reads the redo log up to that point from distributed storage based on the position information, then parses and caches it. At the same time, based on the redo log position information, it discards expired redo logs cached by the standby compute node.

[0094] When the primary computing node of a high-availability database system fails, it stops writing and reading data to and from distributed storage (writing Pages and Redologs) because the primary computing node is in an unavailable state. At the same time, redo log position notifications (Redolog Position) between the primary computing node and the standby computing node also stop.

[0095] S20: When a failure occurs in the main computing node during operation, hot-upgrading the physical configuration specifications of the backup computing node to the same physical configuration specifications as the main computing node;

[0096] When the primary computing node fails, the management and control system of the high-availability database first hot-upgrades the physical configuration specifications (CPU and memory) of the standby computing node. For example, it upgrades the standby computing node equipped with a 1-core CPU and 2GB of memory to the same 16-core CPU and 32GB of memory as the primary computing node. Then, it sends a standby upgrade command to the standby computing node. After receiving the command, the standby computing node adjusts the relevant configuration of the database (for example, the size of the physical page cache pool, etc.).

[0097] S30: Performing a fast crash recovery operation on the database instance running on the standby computing node to upgrade the standby computing node to a new master computing node;

[0098] The physical configuration specifications of the standby computing node are hot upgraded. During the upgrade process, the standby database instance is kept running normally. At the same time, the access permission to the distributed storage is changed from read-only to read-write. Then, a fast crash recovery operation is performed. After the fast crash recovery is completed, the standby database instance is upgraded to the primary database instance, and the standby computing node is upgraded to the new primary computing node.

[0099] Optionally, in one embodiment, after upgrading the standby computing node to a new master computing node, the method further includes:

[0100] Start a new standby computing node and establish a master-slave relationship between the new master computing node and the new standby computing node.

[0101] After the new primary computing node provides external services, in order to achieve disaster recovery for the new primary computing node, a new backup computing node needs to be started and a master-slave relationship (master-slave cooperation relationship) is established with the new primary computing node. The new backup computing node adopts a lower physical configuration specification than the new primary computing node. After the master-slave relationship between the new primary computing node and the new backup computing node is established, a new low-cost asymmetric master-slave node high-availability database system is regenerated.

[0102] See also Figure 3 , Figure 3 This is a schematic diagram of the implementation process of the high-availability database system of the present invention. The implementation process of the high-availability database system of the present invention is divided into four stages: ① Normal operation of the high-availability database system; ② Failure of the primary computing node of the high-availability database system; ③ Upgrade of the backup computing node to the primary computing node; ④ Establishment of a relationship between the new backup computing node and the new primary computing node.

[0103] Specifically, ① represents the normal operation of an asymmetric master-slave high-availability database system. At this point, the master node receives and processes client read and write requests, writing physical pages and redo logs to distributed storage. At the same time, whenever the master node commits a transaction, it notifies the standby node of the redo log position (RedologPosition) of the transaction advance via the TCP channel and protocol. After receiving the latest redo log position information, the standby node reads the redo log up to that point from distributed storage based on this position information, then parses and caches it. At the same time, based on the redo log position information, it discards expired redo logs cached by the standby node.

[0104] ② represents the state of the high-availability database system when the primary compute node fails. The primary compute node is unavailable due to the failure and stops writing and reading data to and from distributed storage (writing pages and redologs). Simultaneously, redo log position notifications between the primary and standby compute nodes also cease.

[0105] ③ represents the process of upgrading a standby compute node to become the new primary compute node. When a primary compute node fails, the high-availability database's management and control system first performs a hot upgrade of the standby node's configuration (CPU and memory) and then issues a "standby-to-primary" command to the standby node. Upon receiving this command, the standby node first adjusts the database's configuration (for example, the size of the physical page cache pool) and then performs a rapid crash recovery operation. Once this operation is complete, the standby node becomes the new primary compute node and can begin providing external services. The entire process, from detecting the primary compute node failure to the new primary node providing external services, takes less than 30 seconds.

[0106] ④ indicates that the new standby computing node has rejoined the high-availability database system. After the new main computing node provides external services, in order to achieve disaster recovery for the main computing node, a new standby computing node needs to be started and establish a relationship with the new main computing node. Figure 1 The same master-slave relationship (master-slave coordination relationship) is used. When the new slave computing node is started and the master-slave relationship (master-slave coordination relationship) between the new master computing node and the new slave computing node is established, a new low-cost asymmetric master-slave node high-availability database system is regenerated.

[0107] S40: When executing the fast crash recovery operation, the new primary computing node is used to provide database services externally.

[0108] The present invention provides an implementation method of a high-availability database system, which can realize a high-availability database system with asymmetric master-slave nodes. When the master computing node fails, the physical configuration specifications of the standby computing node are hot-upgraded to the same physical configuration specifications as the master computing node, the distributed storage is mounted to the standby computing node and a new database instance is started to upgrade the standby computing node to a new master computing node to provide services to the outside world. This method not only ensures that the standby computing node can quickly take over after the master computing node fails and provide database services normally to the outside world, but also reduces the operating costs of the database because the standby computing node adopts lower physical configuration specifications.

[0109] See also Figure 4 , Figure 4 This is a flow chart of an embodiment of a method for implementing a high-availability database system of the present invention to perform a rapid crash recovery operation on a database instance, specifically comprising the following steps:

[0110] S401: Loading unsynchronized redo logs from the distributed stored redo log files, parsing the unsynchronized redo logs, and storing the parsing results in a preset hash table;

[0111] During the operation of the high-availability database system of the present invention, not only will the main computing node run one database instance (such as a MySQL server process, hereinafter referred to as the main database instance), but the standby computing node will also run one database instance (such as a MySQL server process, hereinafter referred to as the standby database instance). The two database instances will share the underlying distributed storage, and at the same time set sharing restrictions: the distributed storage only allows the main database instance to write, and the standby database instance can only read but not write. At the same time, the main and standby database instances will communicate in real time, and the main database instance will tell the standby database instance the redo log location it has pushed to. Then, after receiving the location, the standby database instance will read some redo logs from the underlying distributed storage based on this location.

[0112] Because the primary compute node has failed, the primary database instance no longer pushes log positions. Therefore, after a primary compute node fails, unsynchronized redo logs must be loaded from the distributed redo log files. This means loading redo logs starting from the last synchronized redo log position of the primary database instance to the very end of the distributed redo log files. Furthermore, the loaded unsynchronized redo logs are parsed, and the parsing results are stored in a pre-set hash table.

[0113] Optionally, in one embodiment, parsing the unsynchronized redo log and storing the parsing result in a preset hash table includes:

[0114] Parsing the unsynchronized redo log to obtain a physical page number of the unsynchronized redo log;

[0115] A key-value pair consisting of the physical page number and the redo log is constructed with the physical page number as a keyword and the redo log corresponding to the physical page of the physical page number as a value, and the key-value pair is inserted into a preset hash table.

[0116] Specifically, in the preset hash table where the key-value pairs are inserted, redo logs with the same physical page number will be stored in a one-way linked list; when the physical page number of a physical page is specified, all unapplied redo logs of the physical page can be found, thereby quickly achieving consistency between the standby computing node and the primary computing node at the time of failure. Further combined with data recovery work, the database can maintain crash recovery while providing external services.

[0117] S402: Reading physical pages corresponding to preset necessary data from the distributed storage into the memory of the standby computing node for version update, and writing the updated physical pages to the distributed storage, wherein the necessary data includes: data dictionary metadata, maximum transaction ID, and globally unique ID;

[0118] Specifically, for a high-availability database system to provide normal external services—that is, for SQL read and write requests initiated by clients to be correctly processed—it relies on two conditions: necessary data, such as data dictionary metadata, maximum transaction ID, and globally unique IDs; and physical pages that are consistent at the time of the failure, including record and undolog pages. As long as these two conditions are met, the high-availability database system can provide external services.

[0119] Therefore, it is necessary to read the physical pages corresponding to the preset necessary data from the distributed storage to the memory of the standby computing node for version update, and then write the updated physical pages to the distributed storage.

[0120] At this point, not only are the physical pages in the distributed storage consistent with the primary computing node at the time of the failure; the rollback log is also implicitly consistent with the time of the failure (because the rollback log is stored in the physical pages, and when the physical pages are consistent, the rollback log is naturally consistent).

[0121] S403: Based on the updated physical page and the updated rollback log in the physical page, performing version recovery on the necessary data in units of DDL operations, so that the version of the necessary data is consistent with that at the time when the primary computing node fails, wherein after the version recovery of the necessary data is performed, the high-availability database system provides external services;

[0122] Specifically, based on the updated physical pages and rollback logs in S402, the data dictionary metadata is restored, in units of DDL (Data Definition Language) operations, to maintain transactional consistency with the time of the failure. Once the data dictionary metadata is restored, the database system can perform read and write operations using the correct metadata. Once the necessary data is restored to the same consistency as at the time of the primary compute node failure, the highly available database system can begin providing external services.

[0123] Optionally, in one embodiment, the method for implementing a high-availability database system further includes:

[0124] S404: After the high-availability database system provides external services, the physical pages of other data except the necessary data are updated according to the redo logs of the physical pages corresponding to the hash table, so that the physical page versions of the other data are consistent with those when the main computing node fails.

[0125] In this embodiment, after the high-availability database system resumes external services, it further updates the physical pages corresponding to the remaining unrecovered data based on the redo logs written to the physical pages corresponding to the physical page numbers in the hash table. Specifically, a thread group for refreshing the physical pages with redo logs is started, and the remaining unrefreshed physical pages are refreshed using the redo logs in the hash table. The refreshed physical pages are then written to the distributed storage, gradually ensuring that the physical pages in the distributed storage are consistent with those at the time of the primary database instance failure.

[0126] S405: Performing version recovery on the table record of the high-availability database system in units of transactions based on the updated physical page and the updated rollback log in the physical page, so that the version of the table record remains consistent with that at the time when the primary computing node fails;

[0127] The version recovery operation for table records in a high-availability database system can be performed asynchronously, that is, the table records can be recovered while providing services; this allows the high-availability database to provide normal external services while performing crash recovery operations, without having to wait for the lengthy crash recovery process to complete.

[0128] See also Figure 5 , Figure 5 This is a flow chart of an embodiment of a method for implementing a high-availability database system of the present invention for providing database services externally when performing a fast crash recovery operation, specifically including the following steps:

[0129] S411: When performing a crash recovery operation, receiving a database service request sent by an external client through the SQL engine where the new primary computing node is located, and determining a target table record to be processed by the database service request and a target physical page containing the target table record;

[0130] Specifically, when an external client sends an SQL statement (database service request) to the SQL engine, the SQL engine generates an execution plan for the statement and executes it, and determines the target table records that need to be processed and the target physical pages containing the target table records.

[0131] S412: Initiate a first acquisition request for the target table record to a table record engine through the SQL engine, and initiate a second acquisition request for a target physical page containing the target table record to a physical page engine through the table record engine.

[0132] Specifically, during the execution process, the target table record is obtained by calling the table record engine, and the target physical page is obtained by calling the physical page engine through the table record engine.

[0133] S413: If the target physical page does not exist in the physical page cache pool of the physical page engine, initiate a third acquisition request for the target physical page to the distributed storage through the physical page engine;

[0134] Specifically, when performing a crash recovery operation, the entire database system is in an initialization state, and the target physical page does not exist in the physical page cache pool in the physical page engine. At this time, it is necessary to send a get request to the distributed storage through the physical page engine to try to read the physical page.

[0135] S414: When the distributed storage responds to the third acquisition request and returns the target physical page, intercepting the target physical page for version update, and storing the target physical page after the version update in the physical page cache pool;

[0136] Specifically, when the distributed storage responds to the acquisition request of the physical page engine and returns the target physical page (the target physical page at this time is the version before the failure moment), it intercepts the target physical page and updates it (the target physical page at this time is the version of the main computing node at the failure moment), and stores the updated target physical page in the physical page cache pool.

[0137] S415: reading the target physical page from the physical page cache pool through the physical page engine and returning the target physical page to the table record engine to respond to the second acquisition request;

[0138] Specifically, after the updated target physical page is stored in the physical page cache pool, the physical page engine reads the target physical page from the physical page cache pool and returns it to the table record engine in response to the second acquisition request in step S412.

[0139] S416: Determine, by the table record engine, whether there is any unprocessed pending transaction in the target table record to be read in the target physical page;

[0140] Specifically, the table record engine determines the target table record to be read in the target physical page;

[0141] If it is impossible to determine whether the target table record is the table record at the time of the failure, the target table record is directly returned to the SQL engine for reading and writing;

[0142] If there is an unprocessed suspended transaction in the target table record being read (this means that the table record is the table record at the time of the failure), the target table record will not be returned to the SQL engine (if the SQL engine needs to write), or the table record version before the failure will be found in the rollback log and returned to the SQL engine (if the SQL engine needs to read).

[0143] S417: If there is no unprocessed pending transaction, the target table record is returned to the SQL engine via the table record engine to respond to the first acquisition request; if there is an unprocessed pending transaction, the corresponding rollback log is loaded, and the unprocessed pending transaction is rolled back or committed using the loaded rollback log, and the target table record is returned to the SQL engine via the table record engine to respond to the first acquisition request;

[0144] S418: Based on the target table record, the database service request is processed by the SQL engine, and the processing result is returned to the external client.

[0145] Optionally, in one embodiment, step S414 specifically includes:

[0146] When the distributed storage responds to the third acquisition request and returns the target physical page, intercepting the target physical page;

[0147] Searching the hash table according to the physical page number of the target physical page to obtain unapplied redo logs in the target physical page;

[0148] According to the unapplied redo log in the target physical page, the target physical page is updated in version, and the target physical page after the version update is stored in the physical page cache pool.

[0149] Specifically, by intercepting the target physical page and searching in the hash table according to the physical page number of the target physical page, the unapplied redo log can be quickly and accurately obtained in the target physical page, and the target physical page can be further updated (updated to the version of the main computing node at the time of the failure).

[0150] The implementation method of the high-availability database system in this embodiment can realize a high-availability database system that can quickly perform crash repair while providing database services to the outside world. This is different from traditional high-availability database systems that need to wait for crash recovery to complete before providing services to the outside world.

[0151] The above describes the implementation method of the high-availability database system in the embodiment of the present invention. The following describes the implementation device of the high-availability database system in the embodiment of the present invention. Figure 6 , Figure 6 This is a functional module diagram of an embodiment of a device for implementing a high-availability database system of the present invention. In this embodiment, the device for implementing a high-availability database system includes:

[0152] Detection module 401, used to detect in real time whether a fault occurs during the operation of the main computing node;

[0153] An upgrade module 402 is configured to hot-upgrade the physical configuration specifications of the backup computing node to the same physical configuration specifications as the primary computing node when a failure occurs during operation of the primary computing node;

[0154] A crash recovery module 403 is configured to perform a fast crash recovery operation on the database instance running on the standby computing node to upgrade the standby computing node to a new master computing node;

[0155] The service module 404 is configured to use the new primary computing node to provide external database services when executing the fast crash recovery operation.

[0156] Optionally, in one embodiment, the crash recovery module 403 is specifically configured to:

[0157] Loading unsynchronized redo logs from the redo log files in the distributed storage, parsing the unsynchronized redo logs, and storing the parsing results in a preset hash table;

[0158] Reading physical pages corresponding to preset necessary data from the distributed storage into the memory of the standby computing node for version update, and writing the updated physical pages to the distributed storage, wherein the necessary data includes: data dictionary metadata, maximum transaction ID, and globally unique ID;

[0159] Performing version recovery on the necessary data based on the updated physical page and the updated rollback log in the physical page, in units of DDL operations, so that the version of the necessary data remains consistent with that at the time when the primary computing node fails, wherein after the version of the necessary data is recovered, the high-availability database system provides external services;

[0160] According to the updated physical page and the updated rollback log in the physical page, version recovery is performed on the table record of the high-availability database system in units of transactions, so that the version of the table record is consistent with that when the main computing node fails.

[0161] Optionally, in one embodiment, the crash recovery module 403 is further configured to:

[0162] The unsynchronized redo log is parsed to obtain a physical page number of the unsynchronized redo log; a key-value pair consisting of the physical page number and the redo log is constructed using the physical page number as a keyword and the redo log corresponding to the physical page of the physical page number as a value, and the key-value pair is inserted into a preset hash table.

[0163] Optionally, in one embodiment, the service module 404 is specifically configured to:

[0164] When performing the crash recovery operation, receiving a database service request sent by an external client through the SQL engine where the new primary computing node is located, and determining a target table record to be processed by the database service request and a target physical page containing the target table record;

[0165] Initiate a first acquisition request for the target table record to a table record engine through the SQL engine, and initiate a second acquisition request for the target physical page containing the target table record to a physical page engine through the table record engine;

[0166] If the target physical page does not exist in the physical page cache pool of the physical page engine, initiating a third acquisition request for the target physical page to the distributed storage through the physical page engine;

[0167] When the distributed storage responds to the third acquisition request and returns the target physical page, intercepting the target physical page for version update, and storing the target physical page after the version update in the physical page cache pool;

[0168] Reading the target physical page from the physical page cache pool by the physical page engine and returning the target physical page to the table record engine in response to the second acquisition request;

[0169] Determining, by the table record engine, whether there is an unprocessed pending transaction in the target table record to be read in the target physical page;

[0170] If there is no unprocessed pending transaction, the target table record is returned to the SQL engine through the table record engine to respond to the first acquisition request; if there is an unprocessed pending transaction, the corresponding rollback log is loaded, and the unprocessed pending transaction is rolled back or committed using the loaded rollback log, and the target table record is returned to the SQL engine through the table record engine to respond to the first acquisition request;

[0171] Based on the target table record, the database service request is processed by the SQL engine, and the processing result is returned to the external client.

[0172] Optionally, in one embodiment, the service module 404 is further configured to:

[0173] When the distributed storage responds to the third acquisition request and returns the target physical page, the target physical page is intercepted; according to the physical page number of the target physical page, the hash table is retrieved to obtain the unapplied redo log in the target physical page; according to the unapplied redo log in the target physical page, the target physical page is version updated, and the target physical page after the version update is stored in the physical page cache pool.

[0174] The embodiment of the present invention realizes a high-availability database system with low-cost asymmetric master-slave nodes. When the master computing node fails, the physical configuration specifications of the standby computing node are hot-upgraded to the same physical configuration specifications as the master computing node, the distributed storage is mounted to the standby computing node and a new database instance is started to upgrade the standby computing node to a new master computing node to provide services externally. This not only ensures that the standby computing node can quickly take over after the master computing node fails and provide database services normally to the outside world, but also reduces the operating costs of the database because the standby computing node adopts lower physical configuration specifications.

[0175] Please refer to Figure 7 , Figure 7 FIG. 2 is another functional module diagram of an implementation device of a high-availability database system according to the present invention. In this embodiment, the implementation device of a high-availability database system further includes:

[0176] Detection module 401, used to detect in real time whether a fault occurs during the operation of the main computing node;

[0177] An upgrade module 402 is configured to hot-upgrade the physical configuration specifications of the backup computing node to the same physical configuration specifications as the primary computing node when a failure occurs during operation of the primary computing node;

[0178] A crash recovery module 403 is configured to perform a fast crash recovery operation on the database instance running on the standby computing node to upgrade the standby computing node to a new master computing node;

[0179] The service module 404 is configured to use the new primary computing node to provide external database services when executing the fast crash recovery operation.

[0180] Establishing module 405 is used to start a new standby computing node and establish a master-standby relationship between the new master computing node and the new standby computing node.

[0181] Optionally, in one embodiment, the crash recovery module 403 is further configured to:

[0182] After the high-availability database system provides external services, the physical pages of other data except the necessary data are updated according to the redo log of the physical page corresponding to the hash table, so that the physical page version of the other data is consistent with that when the main computing node fails.

[0183] The implementation device of the high-availability database system in this embodiment can realize a high-availability database system that can quickly perform crash repair while providing database services to the outside world. This is different from traditional high-availability database systems that need to wait for crash recovery to complete before providing services to the outside world.

[0184] The above describes in detail the implementation device of the high-availability database system in the embodiment of the present invention from the perspective of modular functional entities. The following describes in detail the computer device in the embodiment of the present invention from the perspective of hardware processing.

[0185] Please refer to Figure 8 , Figure 8 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. The computer device 500 may vary greatly due to different configurations or performance, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. The memory 520 and the storage medium 530 may be temporary storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), each module may include a series of instruction operations in the computer device 500. Furthermore, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the computer device 500.

[0186] The computer device 500 may further include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 8 The illustrated computer device structure does not limit the computer device and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0187] The present invention also provides a computer device, which includes a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor executes the steps of the implementation method of the high-availability database system in the above-mentioned embodiments.

[0188] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the steps of the implementation method of the high-availability database system.

[0189] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0190] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., various media that can store program code.

[0191] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for implementing a high-availability database system, characterized in that: The high-availability database system includes at least one primary computing node, a distributed storage, and a standby computing node, wherein the physical configuration specification of the standby computing node is lower than the physical configuration specification of the primary computing node, and the implementation method of the high-availability database system includes: Detect in real time whether a failure occurs during the operation of the main computing node; When a failure occurs in the main computing node during operation, hot-upgrading the physical configuration specifications of the backup computing node to the same physical configuration specifications as the main computing node; Performing a fast crash recovery operation on the database instance running on the standby computing node to upgrade the standby computing node to a new master computing node; When performing the fast crash recovery operation, using the new primary computing node to provide database services externally; Wherein, when performing the fast crash recovery operation, using the new primary computing node to provide database services externally includes: When performing the fast crash recovery operation, receiving a database service request sent by an external client through the SQL engine where the new primary computing node is located, and determining a target table record to be processed by the database service request and a target physical page containing the target table record; Initiate a first acquisition request for the target table record to a table record engine through the SQL engine, and initiate a second acquisition request for the target physical page containing the target table record to a physical page engine through the table record engine; If the target physical page does not exist in the physical page cache pool of the physical page engine, initiating a third acquisition request for the target physical page to the distributed storage through the physical page engine; When the distributed storage responds to the third acquisition request and returns the target physical page, intercepting the target physical page for version update, and storing the target physical page after the version update in the physical page cache pool; Reading the target physical page from the physical page cache pool by the physical page engine and returning the target physical page to the table record engine in response to the second acquisition request; Determining, by the table record engine, whether there is an unprocessed pending transaction in the target table record to be read in the target physical page; If there is no unprocessed pending transaction, the target table record is returned to the SQL engine through the table record engine to respond to the first acquisition request; if there is an unprocessed pending transaction, the corresponding rollback log is loaded, and the unprocessed pending transaction is rolled back or committed using the loaded rollback log, and the target table record is returned to the SQL engine through the table record engine to respond to the first acquisition request; Based on the target table record, the database service request is processed by the SQL engine, and the processing result is returned to the external client.

2. The method for implementing a high-availability database system according to claim 1, wherein: After upgrading the standby computing node to a new master computing node, the method further includes: A new standby computing node is started, and a master-standby relationship is established between the new master computing node and the new standby computing node.

3. The method for implementing a high-availability database system according to claim 1 or 2, wherein: The performing a fast crash recovery operation on the database instance running on the standby computing node to upgrade the standby computing node to a new master computing node includes: Loading unsynchronized redo logs from the redo log files in the distributed storage, parsing the unsynchronized redo logs, and storing the parsing results in a preset hash table; Reading physical pages corresponding to preset necessary data from the distributed storage into the memory of the standby computing node for version update, and writing the updated physical pages to the distributed storage, wherein the necessary data includes: data dictionary metadata, maximum transaction ID, and globally unique ID; Performing version recovery on the necessary data based on the updated physical page and the updated rollback log in the physical page, in units of DDL operations, so that the version of the necessary data remains consistent with that at the time when the primary computing node fails, wherein after the version of the necessary data is recovered, the high-availability database system provides external services; According to the updated physical page and the updated rollback log in the physical page, version recovery is performed on the table record of the high-availability database system in units of transactions, so that the version of the table record is consistent with that when the main computing node fails.

4. The method for implementing a high-availability database system according to claim 3, wherein: Parsing the unsynchronized redo log and storing the parsing result in a preset hash table includes: Parsing the unsynchronized redo log to obtain a physical page number of the unsynchronized redo log; A key-value pair consisting of the physical page number and the redo log is constructed with the physical page number as a keyword and the redo log corresponding to the physical page of the physical page number as a value, and the key-value pair is inserted into a preset hash table.

5. The method for implementing a high-availability database system according to claim 3, wherein: When the distributed storage responds to the third acquisition request and returns the target physical page, intercepting the target physical page for version update, and storing the target physical page after the version update in the physical page cache pool includes: When the distributed storage responds to the third acquisition request and returns the target physical page, intercepting the target physical page; Searching the hash table according to the physical page number of the target physical page to obtain unapplied redo logs in the target physical page; According to the unapplied redo log in the target physical page, the target physical page is updated in version, and the target physical page after the version update is stored in the physical page cache pool.

6. The method for implementing a high-availability database system according to claim 3, wherein: The implementation method of the high-availability database system also includes: After the high-availability database system provides external services, the physical pages of other data except the necessary data are updated according to the redo log of the physical page corresponding to the hash table, so that the physical page version of the other data is consistent with that when the main computing node fails.

7. A device for implementing a high-availability database system, characterized in that: The high-availability database system includes at least one primary computing node, a distributed storage, and a standby computing node, wherein the physical configuration specification of the standby computing node is lower than the physical configuration specification of the primary computing node, and the implementation device of the high-availability database system includes: A detection module is used to detect in real time whether a fault occurs during the operation of the main computing node; An upgrade module, configured to hot-upgrade the physical configuration specifications of the standby computing node to the same physical configuration specifications as the primary computing node when a failure occurs during operation of the primary computing node; A crash recovery module, configured to perform a fast crash recovery operation on the database instance running on the standby computing node, so as to upgrade the standby computing node to a new primary computing node; A service module, configured to provide database services externally using the new primary computing node when executing the fast crash recovery operation; The service module is specifically used for: When performing the fast crash recovery operation, a database service request sent by an external client is received through the SQL engine where the new primary computing node is located, and a target table record to be processed by the database service request and a target physical page containing the target table record are determined; a first acquisition request for the target table record is initiated from the SQL engine to a table record engine, and a second acquisition request for the target physical page containing the target table record is initiated from the table record engine to a physical page engine; If the target physical page does not exist in the physical page cache pool of the physical page engine, a third acquisition request for the target physical page is initiated to the distributed storage through the physical page engine; when the distributed storage responds to the third acquisition request and returns the target physical page, the target physical page is intercepted for version update, and the target physical page after the version update is stored in the physical page cache pool; the target physical page is read from the physical page cache pool by the physical page engine and returned to the table record engine to respond to the second acquisition request; and the table record engine determines whether there is an unprocessed pending transaction in the target table record to be read in the target physical page; If there is no unprocessed pending transaction, the target table record is returned to the SQL engine through the table record engine to respond to the first acquisition request; if there is an unprocessed pending transaction, the corresponding rollback log is loaded, and the unprocessed pending transaction is rolled back or committed using the loaded rollback log, and then the target table record is returned to the SQL engine through the table record engine to respond to the first acquisition request; based on the target table record, the database service request is processed by the SQL engine, and the processing result is returned to the external client.

8. A computer device, characterized in that: The computer device includes: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the computer device to execute the implementation method of the high-availability database system according to any one of claims 1 to 6.

9. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the method for implementing a high-availability database system according to any one of claims 1 to 6 is implemented.

Images