Database node recovery method and device, computer equipment, medium and product

Abstract

The invention relates to a database node recovery method and device, computer equipment, a medium and a product. The method comprises the following steps: in response to a node exception event of a database cluster, obtaining a node recovery process adaptive to the node exception event; the node recovery process is used for recovering nodes in a database cluster indicated by the node exception event; the node recovery process comprises a plurality of sub-processes with a process execution sequence; according to the process execution sequence, determining a current sub-process from the sub-processes with the execution states being the to-be-executed states included in the multiple sub-processes in sequence; obtaining and storing input data of the current sub-process, executing the current sub-process based on the input data, and obtaining an execution result of the current sub-process and output data of the current sub-process; and when the execution result meets a preset condition, storing the execution result and the output data, and updating the execution state of the current sub-process into an executed state. By adopting the method, the database cluster stability can be improved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a database node recovery method, device, computer equipment, medium and product. Background Art

[0002] With the development of computer technology, the application of databases is becoming more and more extensive. When enterprises manage data, they use database clusters to meet the needs of large-scale data storage and processing. Data is stored on multiple nodes of the database cluster to provide high availability, scalability and performance advantages. In order to ensure data security, maintain business continuity and business benefits, the disaster recovery function of the database cluster is extremely important to prevent data loss.

[0003] However, in the prior art, after an abnormality occurs in a node of a database cluster, there is a problem of poor stability of the database cluster during the node recovery process. Summary of the invention

[0004] Based on this, it is necessary to provide a database node recovery method, device, computer equipment, medium and product that can improve the stability of the database cluster in response to the above technical problems.

[0005] In a first aspect, the present application provides a database node recovery method, comprising:

[0006] In response to a node abnormality event of a database cluster, a node recovery process adapted to the node abnormality event is obtained; the node recovery process is used to recover the node in the database cluster indicated by the node abnormality event; the node recovery process includes a plurality of sub-processes with a process execution order;

[0007] According to the process execution order, the current sub-process is determined from the sub-processes whose execution status is a pending execution status included in the multiple sub-processes;

[0008] Obtaining and storing input data of the current sub-process, executing the current sub-process based on the input data, and obtaining an execution result of the current sub-process and output data of the current sub-process;

[0009] When the execution result meets the preset condition, the execution result and the output data are stored, and the execution state of the current sub-process is updated to an executed state.

[0010] In a second aspect, the present application also provides a database node recovery device, including:

[0011] A response module, configured to respond to a node abnormality event of a database cluster and obtain a node recovery process adapted to the node abnormality event; the node recovery process is used to recover the node in the database cluster indicated by the node abnormality event; the node recovery process includes a plurality of sub-processes with a process execution order;

[0012] An execution module is used to determine the current sub-process from the sub-processes whose execution status is to be executed in the multiple sub-processes in accordance with the process execution order; obtain the input data of the current sub-process and store it, execute the current sub-process based on the input data, obtain the execution result of the current sub-process and the output data of the current sub-process; when the execution result meets the preset conditions, store the execution result and the output data, and update the execution status of the current sub-process to the executed status.

[0013] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0014] In response to a node abnormality event of a database cluster, a node recovery process adapted to the node abnormality event is obtained; the node recovery process is used to recover the node in the database cluster indicated by the node abnormality event; the node recovery process includes a plurality of sub-processes with a process execution order;

[0015] According to the process execution order, the current sub-process is determined from the sub-processes whose execution status is a pending execution status included in the multiple sub-processes;

[0016] Obtaining and storing input data of the current sub-process, executing the current sub-process based on the input data, and obtaining an execution result of the current sub-process and output data of the current sub-process;

[0017] When the execution result meets the preset condition, the execution result and the output data are stored, and the execution state of the current sub-process is updated to an executed state.

[0018] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0019] In response to a node abnormality event of a database cluster, a node recovery process adapted to the node abnormality event is obtained; the node recovery process is used to recover the node in the database cluster indicated by the node abnormality event; the node recovery process includes a plurality of sub-processes with a process execution order;

[0020] According to the process execution order, the current sub-process is determined from the sub-processes whose execution status is a pending execution status included in the multiple sub-processes;

[0021] Obtaining and storing input data of the current sub-process, executing the current sub-process based on the input data, and obtaining an execution result of the current sub-process and output data of the current sub-process;

[0022] When the execution result meets the preset condition, the execution result and the output data are stored, and the execution state of the current sub-process is updated to an executed state.

[0023] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0024] In response to a node abnormality event of a database cluster, a node recovery process adapted to the node abnormality event is obtained; the node recovery process is used to recover the node in the database cluster indicated by the node abnormality event; the node recovery process includes a plurality of sub-processes with a process execution order;

[0025] According to the process execution order, the current sub-process is determined from the sub-processes whose execution status is a pending execution status included in the multiple sub-processes;

[0026] Obtaining and storing input data of the current sub-process, executing the current sub-process based on the input data, and obtaining an execution result of the current sub-process and output data of the current sub-process;

[0027] When the execution result meets the preset condition, the execution result and the output data are stored, and the execution state of the current sub-process is updated to an executed state.

[0028] The above-mentioned database node recovery method, device, computer equipment, medium and product respond to the node abnormality event of the database cluster, obtain the node recovery process adapted to the node abnormality event, so that the node recovery can be automatically started when the node abnormality event occurs; according to the process execution order, the current sub-process is determined from the sub-processes whose execution status is the pending execution status contained in multiple sub-processes in sequence, and combined with the subsequent steps, multiple sub-processes can be executed in sequence according to the process execution order. Moreover, since the input data of the current sub-process is stored before the execution of the current sub-process, after the execution result and output data are obtained by executing the current sub-process based on the input data, when the execution result meets the preset conditions, the execution result and output data are stored, and the execution status of the current sub-process is updated to the executed status. In this way, even if a secondary abnormality occurs in the database cluster during the node recovery process, the sub-process can be continued to be executed at the breakpoint of the sub-process or the sub-process can be quickly rolled back, so as to avoid the secondary abnormality causing data confusion in the recovery process and causing the database cluster to crash, thereby ensuring the stability of the database cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0030] FIG1 is an application environment diagram of a database node recovery method in one embodiment;

[0031] FIG2 is a schematic diagram of a process flow of a database node recovery method in one embodiment;

[0032] FIG3 is a schematic diagram of an example of a data center architecture in one embodiment;

[0033] FIG4 is a schematic diagram showing an example of the distribution of active and standby nodes in a production center according to an embodiment;

[0034] FIG5 is a schematic diagram of a node recovery process in one embodiment;

[0035] FIG6 is a schematic diagram of a process of initializing a preconfigured physical machine in one embodiment;

[0036] FIG7 is a schematic diagram of a node master-slave switching process in one embodiment;

[0037] FIG8 is a schematic diagram of a cluster load balancing process in one embodiment;

[0038] FIG9 is a schematic diagram of a node redo standby process in one embodiment;

[0039] FIG10 is a schematic diagram of a process for updating metadata in one embodiment;

[0040] FIG11 is a schematic diagram of a working step execution flow in one embodiment;

[0041] FIG12 is a schematic diagram of the execution logic of multiple sub-processes in one embodiment;

[0042] FIG13 is a schematic diagram of a database management system architecture in one embodiment;

[0043] FIG14 is a structural block diagram of a database node recovery device in one embodiment;

[0044] FIG. 15 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] The database node recovery method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the database cluster 102 communicates with the console 104 of the database management system through the network. The data storage system can store the data that the console 104 needs to process. The console 104 of the database management system can respond to the node abnormality event of the database cluster, obtain the node recovery process adapted to the node abnormality event, and execute the node recovery process. Among them, the database cluster 102 can include a node group set, the node group set includes multiple node groups, each node group includes multiple nodes, the node can be installed in a physical machine, and one physical machine can be installed with one or more nodes. The physical machine is a computer system composed of physical hardware, and the physical machine can specifically be a physical server. The database management system is used to manage the database cluster 102. The database management system may include a console 104 and a data storage system. The console 104 is a computer program for control and management running on a management server. The management server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0047] In an exemplary embodiment, as shown in FIG2 , a database node recovery method is provided, which is described by taking the method applied to the console 104 in FIG1 as an example, and includes the following steps 202 to 208 . Among them:

[0048] Step 202, in response to a node abnormality event of the database cluster, obtain a node recovery process adapted to the node abnormality event; the node recovery process is used to recover the node in the database cluster indicated by the node abnormality event; the node recovery process includes multiple sub-processes with a process execution order.

[0049] Among them, the node abnormality event is an event in which an abnormality occurs in a node in the database cluster. The occurrence of an abnormality means that the node stops providing services, and the node is in an abnormal state. The abnormality may be caused by a failure of the node itself, a crash of the physical machine where the node is located, a power outage of the physical machine where the node is located, or other reasons. The node abnormality event can be an automatically triggered event, for example, when the console detects that there is a node abnormality in the database cluster; wherein, the console can detect whether the node in the database cluster is abnormal by means of heartbeat detection, or heartbeat detection can be performed between the nodes in the database cluster, and the nodes in the normal state notify the console when determining that there is a node in the abnormal state.

[0050] The node in the database cluster indicated by the node abnormal event is the node in an abnormal state represented by the node abnormal event. The node abnormal event may include different types of node abnormal events, and different types of node abnormal events are adapted to different node recovery processes. One type of node abnormal event may be that the node abnormal event represents that the node is in an abnormal state while the physical machine where the node is located is in a normal state. Another type of node abnormal event may be that the physical machine is represented to be in an abnormal state. Among them, when the node abnormal event represents that the physical machine is in an abnormal state, the nodes installed on the physical machine in the abnormal state are all in an abnormal state.

[0051] The node recovery process contains multiple sub-processes with a process execution order, and each sub-process can include multiple work steps. In some scenarios, a sub-process can also be called a FLOW, and a work step can also be called a STEP. By executing the node recovery process, that is, executing multiple sub-processes in the node recovery process, the node in an abnormal state can be recovered. The process execution order is the execution order of multiple sub-processes.

[0052] The database cluster may be a distributed database cluster. The nodes in the database cluster may include a GTM node (Global Transaction Management, global transaction management node), a CN node (Coordinator Node, coordination node) and a DN node (Data Node, data node).

[0053] For example, the console of the database management system can obtain a node recovery process adapted to the node abnormality event from the data storage system of the database management system in response to the node abnormality event of the database cluster. The data storage system can be an ETCD system (a distributed key-value storage system released by SoreOs), Redis (RemoteDictionary Server, a remote dictionary service, an open source key-value storage system), or others.

[0054] In one embodiment, the console may obtain multiple sub-processes adapted to the node abnormal event and the process execution order of the multiple sub-processes, and arrange the multiple sub-processes into a node recovery process according to the process execution order.

[0055] Step 204 , according to the process execution order, determine the current sub-process from the sub-processes whose execution status is the waiting-to-be-executed status included in the multiple sub-processes.

[0056] After the node recovery process is acquired, before step 204, the execution status of each of the multiple sub-processes may be defaulted to a pending execution status. The process execution order may be the ordering position of each sub-process in the multiple sub-processes.

[0057] Exemplarily, the console may set the execution status of each sub-process among multiple sub-processes to a pending state, determine the first sub-process among the multiple sub-processes according to the process execution order, determine the first sub-process as the current sub-process, execute steps 206 and 208, and from the sub-processes whose execution status is a pending state included in the multiple sub-processes, determine the next sub-process according to the process execution order, update the next sub-process to the current sub-process, execute steps 206 and 208, until the execution status of each of the multiple sub-processes is an executed state.

[0058] Step 206, obtaining and storing the input data of the current sub-process, executing the current sub-process based on the input data, and obtaining the execution result of the current sub-process and the output data of the current sub-process.

[0059] The input data may be the data required to execute the current sub-process. The input data may be obtained based on the output data of the previous sub-process, or may be pre-configured. The execution result indicates whether the execution of the sub-process is successful or not. The execution result may include a result indicating a successful execution or a result indicating a failed execution, and may also include the number of executions. The output data may be the data passed to the subsequent process after the execution of the current sub-process is completed.

[0060] Exemplarily, the console of the database management system can obtain the input data of the current sub-process and store it in the global data pool, execute the current sub-process based on the input data, and obtain the execution result of the current sub-process and the output data of the current sub-process. Among them, the global data pool can be a temporary storage space set in the data storage system of the database management system. The global data pool is used to store the data that needs to be stored during the execution of the node recovery process. The global data pool can be used to store the execution status, input data, execution results and output data of each sub-process included in the node recovery process. Each sub-process can include multiple work steps with a step execution order. In the process of executing the current sub-process, the multiple work steps included in the current sub-process can be executed according to the corresponding step execution order.

[0061] In one embodiment, after step 202 and before step 204 , the console may create and initialize a global data pool in the data storage system.

[0062] Step 208: When the execution result meets the preset condition, the execution result and output data are stored, and the execution status of the current sub-process is updated to the executed status.

[0063] The preset condition is a condition that must be met to determine that the current sub-process is completed. The preset condition can be that the execution result indicates that the execution is successful, or that the execution result indicates that the execution fails and the number of executions reaches the preset number of times. The preset number of times is 3, 4 or other. The number of executions is the number of times the current sub-process is executed.

[0064] In one embodiment, after determining the current sub-process, the current sub-process can be stored in the global data pool; after the current sub-process is executed for the first time, when the execution result indicates that the execution failed, the current sub-process can be obtained from the global data pool, and the steps of executing the current sub-process based on the input data can be returned and re-executed until the execution result indicates that the execution is successful, or the execution result indicates that the execution failed and the number of executions reaches a preset number. In addition to the number of executions, the number of retries can also be used to determine, and the number of retries refers to the number of times the input data of the current sub-process is returned and stored, and the steps of executing the current sub-process based on the input data are re-executed.

[0065] In one embodiment, illustratively, when the execution result meets the preset conditions, the console of the database management system can store the execution result and output data in the global data pool, and update the execution status of the current sub-process recorded in the global data pool from the unexecuted state to the executed state. The execution result of the current sub-process can be marked as SUCCESS in the global data pool to indicate successful execution, and marked as FALL to indicate failed execution.

[0066] In the above database node recovery method, in response to a node abnormality event of a database cluster, a node recovery process adapted to the node abnormality event is obtained, so that node recovery can be automatically started when a node abnormality event occurs; according to the process execution order, the current sub-process is determined in turn from the sub-processes whose execution status is a pending execution state contained in multiple sub-processes, and combined with subsequent steps, multiple sub-processes can be executed in turn according to the process execution order. Moreover, since the input data of the current sub-process is stored before the execution of the current sub-process, after the execution result and output data are obtained by executing the current sub-process based on the input data, when the execution result meets the preset conditions, the execution result and output data are stored, and the execution status of the current sub-process is updated to an executed state. In this way, even if a secondary abnormality occurs in the database cluster during the node recovery process, the sub-process can be continued to be executed at the breakpoint of the sub-process or the sub-process can be quickly rolled back, so as to avoid the secondary abnormality causing data confusion in the recovery process and causing the database cluster to crash, thereby ensuring the stability of the database cluster.

[0067] In an exemplary embodiment, a database cluster includes a set of node groups, each node group in the set of node groups includes a master node and two backup nodes; in the same node group, the backup node synchronizes data from the master node; wherein, the master node and two backup nodes contained in any node group are distributed in three data centers one by one, and any master node of the node group set is distributed in a production center among the three data centers; the three data centers include two production centers and one disaster recovery center, the two production centers are in the same city, and the disaster recovery center is in a different location from the two production centers.

[0068] Among them, the node group set is a set formed by multiple node groups. The multiple node groups may include at least one GTM node group, at least one CN node group, and at least one DN node group. The GTM node group contains three GTM nodes. The CN node group contains three CN nodes. The DN node group contains three DN nodes. The specifications of the GTM node can be 4-core CPU (central processing unit) and 16G memory, which can be abbreviated as 4C16G. The specifications of the CN node or DN node can be 8-core CPU and 64G memory.

[0069] Each node group contains three nodes, one is the master node and the other two are standby nodes. The master node is the node that plays the main role and is used to provide write and query functions. The write function is used to add, delete, and modify, and the object can be data or database structure. The standby node is the node that plays the standby role and is used to provide query functions. The standby node synchronizes data from the master node of the node group to which it belongs, so that when the master node is in an abnormal state, the standby node can be upgraded to the master node of the node group to which it belongs, to ensure the normal operation of the node group.

[0070] The GTM node is used to allocate global transaction IDs and manage global transaction status. In distributed transactions, when a transaction spans multiple nodes, the GTM node ensures the consistency and isolation of the transaction. The GTM node allocates a unique XID (transaction ID) to the transaction and tracks the PREPARE and COMMIT status of the transaction.

[0071] The CN node is the entry point for client applications to interact with the database cluster. The application sends SQL (Structured Query Language) requests through the CN node. The CN node is responsible for parsing the request, executing the query plan, and distributing the query to the corresponding DN node for execution. The CN node also handles the logic of transactions, such as starting transactions, committing transactions, or rolling back transactions, and communicates with the GTM node to manage the global transaction status. The CN node is also responsible for merging the results returned by the DN nodes and ultimately returning the data to the client. In a query, if the data is distributed on multiple DN nodes, the CN node will be responsible for the integration and aggregation of the data.

[0072] DN nodes are used to actually store data. Data is hashed or copied to different DN nodes, and each DN node is responsible for storing and retrieving a portion of the data. When a CN node receives a query request, it sends the SQL statement to the corresponding DN node for execution based on the data distribution strategy. DN nodes are only responsible for processing the data they manage, and do not participate in the coordination of global transactions. Instead, they follow the instructions of the CN nodes to perform local transaction operations. During the two-phase commit (2PC) process, the DN node will perform the pre-commit (PREPARE) and commit (COMMIT) operations of the local transaction according to the instructions received from the CN node.

[0073] The data center is a place that provides an environment for the physical machines where nodes are installed. The production center is the main role for providing services, while the disaster recovery center is a secondary role. Nodes are distributed in a data center, which means that the nodes are installed in the physical machines of the data center. Agents can be run in physical machines, and the console can issue instructions to the Agents. The Agents control the nodes installed in the physical machines according to the instructions issued by the console.

[0074] Any node group contains one master node and two backup nodes, which are distributed in three data centers. This means that the nodes contained in any node group are not distributed in the same data center, and each of the three data centers has one node of the node group. Any master node of the node group set is distributed in one production center among the three data centers, which means that the master node is only distributed in the production center and not in the disaster recovery center.

[0075] The two production centers are located in the same city, which means that the geographical locations of the two production centers are in the same administrative region, such as provincial administrative regions, prefecture-level administrative regions or other administrative regions. The disaster recovery center is located in different places from the two production centers, which means that the disaster recovery center is located in different administrative regions from the two production centers. Referring to the schematic diagram of the data center architecture example shown in FIG3, the data center may include production center 1 and production center 2 in the same city, and a disaster recovery center located in a different place from the production center. The master node of the database cluster is distributed in production center 1 or production center 2, and the disaster recovery center is distributed with backup nodes; data is synchronously replicated between the nodes of production center 1 and production center 2 (that is, data synchronization is performed synchronously), and the disaster recovery center asynchronously replicates data from production center 1 and production center 2 (that is, data synchronization is performed asynchronously); when a user accesses the database cluster through an access portal, the user's request is mainly routed to the production center, and the disaster recovery center can be used as a backup, and the disaster recovery center can play a role when an abnormality occurs in the production center.

[0076] In this embodiment, each node group includes a master node and two backup nodes, and in the same node group, the backup node synchronizes data from the master node and can back up the data of the master node. Moreover, the master node and two backup nodes contained in any node group are distributed in three data centers one by one, and any master node of the node group set is distributed in a production center among the three data centers. The two production centers are in the same city, and the disaster recovery center is in a different location from the two production centers, forming a high-availability architecture and creating conditions for improving the stability of the database cluster.

[0077] In an exemplary embodiment, the node group set may include one GTM node group, three CN node groups and three DN node groups, the three CN node groups may include CN1 node group, CN2 node group and CN3 node group, the three DN node groups may include DN1 node group, DN2 node group and DN3 node group, and each node group includes one master node and two backup nodes. Each production center may be provided with three physical machines, and the distribution of each node in the two production centers is shown in the example schematic diagram of the master and backup node distribution of the production center shown in FIG4.

[0078] Among them, in the same physical machine, different nodes use Cgroup (a function of the Linux operating system kernel, which is used to limit, control and separate the resources of a process group) to isolate resources. Production Center 1 can be equipped with physical machines M1, M2 and M3. M1 is installed with the master node of the GTM node group, the master node of the CN1 node group and a standby node of the DN3 node group, M2 is installed with the master node of the CN2 node group and a standby node of the DN2 node group, and M3 is installed with a standby node of the CN3 node group and the master node of the DN1 node group.

[0079] Production center 2 may be provided with physical machines M4, M5, and M6. M4 is installed with a standby node of the GTM node group, a standby node of the CN1 node group, and a master node of the DN3 node group, M2 is installed with a standby node of the CN2 node group and a master node of the DN2 node group, and M3 is installed with a master node of the CN3 node group and a standby node of the DN1 node group. In addition, a standby node of each of the GTM node group, the CN1 node group, the CN2 node group, the CN3 node group, the DN1 node group, the DN2 node group, and the DN3 node group is installed in the physical machine of the disaster recovery center.

[0080] In an exemplary embodiment, a node abnormality event characterizes that the original physical machine of the first production center among the two production centers is in an abnormal state, causing the node installed on the original physical machine to be in an abnormal state; the node installed on the original physical machine includes at least a part of the nodes distributed by the first production center; the node recovery process adapted to the node abnormality event includes multiple sub-processes with a process execution order, which are, in sequence, initializing a preconfigured physical machine; using the preconfigured physical machine to replace the original physical machine; performing master-slave switching on the nodes of the node group to which the master node in the node installed on the original physical machine belongs; performing load balancing based on the nodes in the database cluster that are in a normal state; reinstalling the nodes on the preconfigured physical machine according to the nodes installed on the original physical machine; and updating metadata based on the physical machine information of the preconfigured physical machine.

[0081] Among them, the first production center is the production center where the node is abnormal. The original physical machine is the physical machine in the first production center that is in an abnormal state. The original physical machine is in an abnormal state, which may cause all nodes installed on the original physical machine to be in an abnormal state and unable to provide services to the outside world. Therefore, a new physical machine needs to be used to replace the original physical machine. When the node abnormality event indicates that the original physical machine of the first production center among the two production centers is in an abnormal state, the second production center (one of the two production centers except the first production center) and the disaster recovery center are both in normal state.

[0082] Each production center can prepare multiple pre-configured physical machines for replacement in advance. When a node abnormality event is triggered, a pre-configured physical machine that matches the number and machine specifications of the original physical machine can be selected. Referring to the node recovery process diagram shown in Figure 5, in this embodiment, the multiple sub-processes included in the adapted node recovery process in the process execution order can be, in order, 1) initializing the pre-configured physical machine, 2) performing master-slave switching on the nodes of the node group to which the master node belongs in the node installed on the original physical machine (this sub-process can be referred to as node master-slave switching), 3) performing load balancing based on the nodes in the normal state in the database cluster (this sub-process can be referred to as cluster load balancing), 4) reinstalling the nodes on the pre-configured physical machine according to the nodes installed on the original physical machine (this sub-process can be referred to as node redo standby), and 5) performing metadata update based on the physical machine information of the pre-configured physical machine (this sub-process can be referred to as updating metadata).

[0083] The preconfigured physical machine can also be prepared by the operation and maintenance personnel after the node abnormal event is triggered. The operation and maintenance personnel can determine the number and machine specifications of the abnormal original physical machines and prepare preconfigured physical machines that match the number and machine specifications of the original physical machines. For example, the physical machines M1~M3 of the production center 1 shown in Figure 4 are all in an abnormal state, so three preconfigured physical machines need to be prepared, and then the three preconfigured physical machines are initialized by the console. Referring to the schematic diagram of the initialization preconfigured physical machine process shown in Figure 6, when the sub-process is to initialize the preconfigured physical machine, the multiple work steps included can be, 1) install the preconfigured dependencies, 2) install the preconfigured Agent (agent program). Among them, in this sub-process, the input data may include the machine identifier of the preconfigured physical machine (such as the machine name and IP address), the preconfigured dependencies, and the preconfigured Agent. The output data may include the machine identifier of the preconfigured physical machine that has completed the initialization. The dependency is a program or library that the Agent needs to rely on to run. Dependencies such as crontab scheduled task service (a tool in Linux), initialization Agent running user or others.

[0084] Referring to the schematic diagram of the node master-slave switching process shown in FIG7 , in the case where the sub-process is to perform master-slave switching on the nodes of the node group to which the master node belongs in the node installed on the original physical machine, the multiple work steps included may be, 1) determining the master node in the node installed on the original physical machine, 2) performing master-slave switching on the nodes of the node group to which the master node belongs. Among them, in this sub-process, the input data may include the mapping relationship between the machine identifier of the original physical machine and the node identifier (such as the node number, name, etc.) of the installed node, and the node information of each node in the database cluster (such as the master and standby identity of the node, the node group to which it belongs, etc.). The output data may include the node information updated after the master-slave switching is completed. This work step 1) can be understood as counting the master nodes that need to be switched, that is, the master nodes in an abnormal state. In work step 2), the master node determined in work step 1) is downgraded to a standby node, and the standby node in a normal state in the node group to which the master node belongs is upgraded to a new master node (the standby nodes in a normal state distributed in the production center are upgraded first).

[0085] In this embodiment, through a detailed node recovery process, node recovery can be automatically achieved after an abnormality occurs in a physical machine or even in the entire production center, and the user is unaware of it, which can reduce manual operations and thus reduce the consumption of operation and maintenance resources.

[0086] In an exemplary embodiment, the node group set includes multiple coordination node groups, and the above-mentioned load balancing based on the nodes in the database cluster that are in a normal state includes: in response to a database request, querying the free tokens in the token bucket; when it is queried that there are free tokens in the token bucket, determining the request type of the database request; when the request type is a query type, forwarding the database request to the coordination node with the best load among the coordination nodes distributed in the disaster recovery center; when the request type is a write type, forwarding the database request to the coordination node with the best load and serving as the master node among the coordination nodes distributed in the two production centers.

[0087] Among them, a database request is a request for calling a database for data processing. When the request type of the database request is a query type, the database request is used to query data, such as for viewing the database structure, for querying specific data stored in the database. When the request type of the database request is a write type, the database request is used to write data, such as for adding, deleting or modifying the database structure, and for adding, deleting or modifying specific data stored in the database. The database request can specifically be an SQL request.

[0088] The token bucket is a virtual container for storing tokens. The total number of tokens in the token bucket is used to limit the number of requests. One database request can occupy one token. The total number of tokens can be preset, for example, to 100. Free tokens are unoccupied tokens in the token bucket.

[0089] The nodes distributed by the disaster recovery center are standby nodes. During the execution of the node recovery process, the disaster recovery center handles query-type database requests. The coordination node group is also called the CN node group. The coordination node group includes multiple coordination nodes (CN nodes), such as three coordination nodes. The load of the coordination node can be determined by the node connection pool utilization rate of the coordination node, as well as the CPU utilization rate, memory utilization rate, and bandwidth utilization rate of the physical machine where the coordination node is located. It can also be indirectly determined by the node connection pool idle rate of the coordination node, as well as the CPU idle rate, memory idle rate, and bandwidth idle rate of the physical machine where the coordination node is located.

[0090] Exemplarily, referring to the cluster load balancing process diagram shown in Figure 8, the console can respond to the database request for the virtual IP of the database cluster, query the free tokens in the token bucket, and determine the request type of the database request when it is found that there are free tokens in the token bucket; when the request type is a query type, it can be determined to be forwarded to the disaster recovery center. At this time, the coordination node with the best load among the coordination nodes distributed in the disaster recovery center can be determined, and the database request is forwarded to the coordination node with the best load in the disaster recovery center; when the request type is a write type, it can be determined to be forwarded to the production center. At this time, the coordination node with the best load and serving as the master node among the coordination nodes distributed in the two production centers can be determined, and the database request is forwarded to the coordination node with the best load and serving as the master node in the production center.

[0091] The virtual IP is the IP address displayed by the database cluster. The client application can send database requests to the virtual IP without paying attention to the actual IP address in the database cluster that processes the database requests. Even if some nodes in the database cluster are abnormal, the user will not be aware of it. In some scenarios, the virtual IP can also be called VIP.

[0092] In one embodiment, when determining the coordination node with the best load, the coordination nodes distributed by the disaster recovery center or the coordination nodes distributed by the two production centers can be used as candidate coordination nodes. The console can determine the weight value of the candidate coordination node based on the node connection pool idle rate of each candidate coordination node, as well as the CPU idle rate, memory idle rate and bandwidth idle rate of the physical machine where the candidate coordination node is located, and use the candidate coordination node with the largest weight value as the coordination node with the best load. Among them, the optimal load can be understood as the minimum load. The weight value of the candidate coordination node can be calculated according to the following formula (1).

[0093] Formula (1)

[0094] in, The weight value that can represent the candidate coordination node, Can characterize CPU idle rate, Can characterize the memory idle rate, It can characterize the bandwidth idle rate. It can represent the idle rate of the node connection pool.

[0095] In one embodiment, when it is found that there are no free tokens in the token bucket, the database request is suspended until it is found that there are free tokens in the token bucket.

[0096] In this embodiment, the database request peak is reduced during the node recovery period by means of a token bucket, so as to avoid processing too many database requests during the node recovery period, thereby improving the stability of the database cluster. Moreover, the disaster recovery center processes query-type database requests, which can share part of the pressure on the production center and avoid excessive processing pressure on the production center during the node recovery period. For write-type database requests, the database request can also be processed according to the load of each coordination node in the production center, thereby improving the load utilization of the database cluster and the cluster stability.

[0097] In an exemplary embodiment, the above-mentioned reinstallation of nodes in the preconfigured physical machine according to the nodes installed on the original physical machine may include: according to the node information of the nodes installed on the original physical machine, setting nodes that are adapted to the node information in the preconfigured physical machine; the node information includes the correspondence between the node and the node group to which it belongs; instructing each node set in the preconfigured physical machine to obtain and store the data directory from the master node in the node group to which the node belongs; each node set in the preconfigured physical machine is a standby node; when the node completes storing the data directory, instructing the node to obtain and store incremental data from the master node in the node group to which it belongs, the incremental data is the data newly added by the master node in the node group to which the node belongs during the process of copying the data directory; when each node set in the preconfigured physical machine completes obtaining and storing the incremental data from the master node in the node group to which it belongs, it is determined that the reinstallation of the node in the preconfigured physical machine is completed.

[0098] Among them, the node information may include node identification, master-slave identity, node group to which it belongs, node specifications or others. The node identification is used to uniquely identify a node, which can be a node number, node name or others. The master-slave identity indicates whether the node is a master node or a standby node. After the node master-slave switching sub-process, all nodes installed on the original physical machine are standby nodes. Setting a node that is adapted to the node information in the pre-configured physical machine can be understood as setting a node in the pre-configured physical machine with the same node information as the node installed on the original physical machine and empty data.

[0099] For example, referring to the node redo standby process diagram shown in FIG9 , the console can set nodes adapted to the node information in the preconfigured physical machine according to the node information of the nodes installed on the original physical machine; instruct each node set in the preconfigured physical machine to obtain and store the data directory from the master node in the node group to which the node belongs (this step can be called basic data directory recovery), and instruct the master node in the node group to which the node belongs to save the incremental data in the form of a WAL log (Write-Ahead Log) on ​​the disk; when the node completes storing the data directory, instruct the node to obtain the incremental data in the form of a WAL log from the master node in the node group to which it belongs and replay the storage; when each node set in the preconfigured physical machine completes obtaining and storing the incremental data from the master node in the node group to which it belongs, it is determined that the node is reinstalled in the preconfigured physical machine. Among them, before performing node redo standby, the load of the nodes set in the preconfigured physical machine can be calculated, and the nodes with the best load are preferentially redone for standby.

[0100] In this embodiment, a node that is adapted to the node information of the node installed on the original physical machine is set in the preconfigured physical machine. Then, the node in the preconfigured physical machine obtains and stores the data directory from the master node, and then obtains the incremental data generated in the process of storing the data directory. In this way, the node is reinstalled and the data is restored. The subsequently reinstalled node can be put into use, thereby restoring the performance of the database cluster.

[0101] In one embodiment, referring to the metadata update process diagram as shown in Figure 10, the above-mentioned metadata update based on the physical machine information of the preconfigured physical machine includes: updating the physical machine information of the preconfigured physical machine to the data storage system; in the data storage system, binding the node information of the node installed in the preconfigured physical machine to the preconfigured physical machine.

[0102] The metadata may be physical machine information and a binding relationship between the physical machine information and the node information. The physical machine information may include a machine name, an IP address, an SSH (Secure Shell) port, or others. The data storage system may be an ETCD system. The node information may be bound to an IP address of a preconfigured physical machine.

[0103] In an exemplary embodiment, when a node abnormality event indicates that a node is in an abnormal state while the physical machine where the node is located is in a normal state, the multiple sub-processes included in the adapted node recovery process may be, in sequence, 1) counting the nodes in the abnormal state; 2) performing master-slave switching on the nodes in the node group to which the master node in the node in the abnormal state belongs; 3) performing load balancing based on the nodes in the normal state in the database cluster; 4) reinstalling the nodes in the abnormal state; and 5) performing metadata updates based on the reinstalled nodes.

[0104] In an exemplary embodiment, each sub-process includes multiple work steps with a step execution order; the above-mentioned execution of the current sub-process based on the input data to obtain the execution result of the current sub-process and the output data of the current sub-process includes: according to the step execution order of the multiple work steps included in the current sub-process, based on the input data, the multiple work steps included in the current sub-process are executed in sequence, and the step execution result corresponding to each work step is obtained and stored; when the multiple work steps included in the current sub-process are executed, when the multiple work steps included in the current sub-process are executed, the execution result of the current sub-process and the output data of the current sub-process are generated.

[0105] The step execution order is the execution order of multiple work steps. Input data can be data required to execute the current sub-process. Output data can be data passed to subsequent processes after the current sub-process is executed. After the multiple work steps included in the current sub-process are executed, output data can be generated according to the pre-configured output data requirements.

[0106] For example, when the current sub-process is to initialize a preconfigured physical machine, multiple work steps may include 1) installing preconfigured dependencies, and 2) installing a preconfigured Agent. The input data may include the machine identifier of the preconfigured physical machine (such as the machine name and IP address), preconfigured dependencies, and preconfigured Agent. Then, the work step of installing the preconfigured dependencies is performed based on the machine identifier and preconfigured dependencies of the preconfigured physical machine, which can be understood as installing the preconfigured dependencies in the preconfigured physical machine represented by the machine identifier. The output data requirement may be to output the machine identifier of the physical machine that has completed initialization, and the generated output data may include the machine identifier of the preconfigured physical machine that has completed initialization.

[0107] The step execution result is used to indicate whether the work step is executed successfully. The step execution result can be stored in the above-mentioned global data pool of the data storage system. The step execution result can be a result that indicates the successful execution of the step, or a result that indicates the failed execution of the step. After the execution of multiple work steps contained in the current sub-process is completed, when the execution results of each of the multiple work steps in the current sub-process indicate that the step is executed successfully, the execution result of the current sub-process can be a result that indicates the successful execution; when the execution result of any work step in the current sub-process indicates that the step fails to execute, the execution result of the current sub-process can be a result that indicates the failed execution.

[0108] For example, referring to the work step execution flow diagram shown in FIG11, the console can execute multiple work steps contained in the current sub-process in sequence based on the input data according to the step execution order; when executing each work step (STEP), if the execution is successful, the work step is marked as SUCCESS, indicating that the work step is successfully executed; if the execution fails, the work step is marked as FALL, indicating that the work step fails to execute; if there is a next work step of the currently executed work step among multiple work steps, the next work step is obtained and continued to execute until multiple work steps are executed, and the execution result of the current sub-process and the output data of the current sub-process are generated. The output data of the current sub-process can be used as the input data of the next sub-process.

[0109] In this embodiment, according to the step execution order and based on the input data, multiple work steps included in the current sub-process are executed in sequence, and the execution result of the current sub-process and the output data of the current sub-process are generated. The execution of the current sub-process can be completed, and the corresponding step execution result is stored after executing each work step. Even if a secondary disaster occurs to the database cluster during the step execution process, it can be quickly restored based on the stored data.

[0110] The above node recovery process is explained with a specific disaster recovery scenario: as shown in Figure 4, the physical machines M1~M3 of the production center 1 are all in an abnormal state, so three pre-configured physical machines are required. At the same time, it can be determined that the master nodes distributed in the production center 1 of the GTM node group, CN1 node group, CN2 node group and DN1 node group are in an abnormal state, and the backup nodes distributed in the production center 1 of the DN3 node group, DN2 node group and CN3 node group are in an abnormal state. These nodes in abnormal states cannot provide external services; the nodes distributed in the production center 2 and the disaster recovery center are all in normal state.

[0111] In conjunction with Figures 4 and 6, the console first executes the sub-process of initializing the pre-configured physical machines, and can initialize the three pre-configured physical machines. Further, the console can execute the sub-process of node master-slave switching, specifically, downgrading the master nodes distributed in the production center 1 of the GTM node group, CN1 node group, CN2 node group and DN1 node group to standby nodes, and upgrading the standby nodes distributed in the production center 2 of the GTM node group, CN1 node group, CN2 node group and DN1 node group to master nodes. At this time, all nodes distributed in the production center 2 are master nodes, and all nodes distributed in the disaster recovery center are standby nodes.

[0112] Further, the console can execute the sub-process of cluster load balancing. Specifically, when the user sends a database request for the database cluster, there are 43 idle tokens in the token bucket (the total number of tokens is set to 100, and 3 tokens can be generated per second), and one of the idle tokens is taken to process the database request; when the request type of the database cluster is a query type, the database request will be routed to the CN node with the best load distributed by the disaster recovery center; when the request type of the database cluster is other types except the query type (it can be a write type), after calculation, it is determined that the load rate of M4 and M5 of production center 2 reaches 50%, that is, the idle rate is 50%, and the load rate of M6 of production center 2 is 20%, that is, the idle rate is 80%. Combined with the connection pool idle rate of the master node in the CN node group in production center 2 (the master nodes of CN1 node group, CN2 node group, and CN3 node group respectively), it can be determined that the master node of CN3 node group has the highest weight, so the database request is routed to the master node of CN3 node group in production center 2. It can be understood that the sub-process of cluster load balancing can be carried out continuously.

[0113] Furthermore, the console can execute the sub-process of node redo standby. Specifically, the three pre-configured physical machines can be updated to M1, M2, and M3. The node settings of M1, M2, and M3 after the update must match the node settings of M1, M2, and M3 before the update, that is, one node of each of the three GTM node group, CN1 node group, and DN3 node group is set in M1, one node of each of the CN2 node group and DN2 node group is set in M2, and one node of each of the CN3 node group and DN1 node group is set in M3. Since the node master-slave switching has been performed, the nodes set in M1, M2, and M3 are all used as standby nodes. After the node settings are completed, each node in M1, M2, and M3 copies the data directory from the master node in the node group to which it belongs, and then completes the incremental data recovery operation by replaying the WAL log.

[0114] Finally, the console can execute the sub-process of updating metadata. Taking M1's information as an example, before the node recovery process, M1's IP was 192.xxx.0.2. After the replacement, the machine IP was 192.xxx.0.2. 192.xxx.0.2 was updated to ETCD. Then the node information was bound, and the IP of each node on M1 was also changed to 192.xxx.1.2.

[0115] In one embodiment, referring to the multiple sub-process execution logic diagram as shown in Figure 12, the above-mentioned database node recovery method may include: the console may respond to the node abnormality event of the database cluster, obtain multiple sub-processes (each sub-process may be called FLOW) adapted to the node abnormality event and the process execution order of the multiple sub-processes, and arrange the multiple sub-processes into a node recovery process according to the process execution order.

[0116] The console can set the execution status of each sub-process in multiple sub-processes to the pending state; create and initialize a global data pool in the ETCD system; determine the current sub-process from the sub-processes in the pending state included in the multiple sub-processes according to the process execution order, and store the current sub-process in the global data pool; obtain the input data of the current sub-process from the ETCD system and store it in the global data pool, and execute the current sub-process based on the input data.

[0117] When the execution is successful, the console can mark the execution result of the current sub-process as SUCCESS, obtain the execution result indicating the successful execution, generate the output data of the current sub-process, and store the execution result and the output data in the global data pool.

[0118] When the execution fails, the console can obtain the current sub-process from the global data pool, return to the steps of executing the current sub-process based on the input data and re-execute it; if the number of re-executions (number of retries) reaches 3 times and the execution still fails, the execution result of the current sub-process can be marked as FALL, the execution result representing the execution failure can be obtained, the output data of the current sub-process is generated, and the execution result and output data are stored in the global data pool.

[0119] If there is a next sub-process of the current sub-process among multiple sub-processes, the next sub-process is updated to the current sub-process, and the step of storing the current sub-process in the global data pool is returned to continue execution until all sub-processes are executed.

[0120] In a specific application scenario, referring to the database management system architecture diagram shown in FIG13, the database management system may be a relational database management system, which may include a configuration library, a console, and an ETCD system. Users may manage the database cluster through the console. The database management system may be deployed in one or more physical machines different from the database cluster.

[0121] The configuration library is used to record the information and user information of the entire life cycle of the console managing the database instance, and also includes some configuration items when the console is started. The information of the entire life cycle of the database instance includes activation, unsubscription, and running status.

[0122] The console may include a backup and recovery module, a process control module, a load balancing module, a high availability management module and others. The backup and recovery module can respond to node abnormality events and instruct the process control module to execute the above database node recovery method. During the process of the process control module executing the database node recovery method, the console can instruct the load balancing module to achieve cluster load balancing. The high availability management module can respond to node abnormality scenarios according to the high availability strategy set by the user (such as business priority, data priority).

[0123] When the business priority strategy is adopted, the database WAL log adopts the asynchronous replication mode, that is, the data of the standby node may be delayed relative to the primary node. When the primary node of a group of nodes (such as the DN1 node group) fails, the database management system immediately promotes the standby node of the DN1 node group to the primary node to ensure that the database cluster can continue to provide external services without considering some potential data loss problems during the switch.

[0124] When the data priority strategy is adopted, the database WAL log adopts the synchronous replication mode, that is, the data of the master and slave nodes are consistent. When the master node of a group of nodes (such as the DN1 node group) fails, the database management system will immediately try to pull up the failed master node of the DN1 node group. If it cannot be pulled up, it will switch the master from the slave.

[0125] The database cluster may include a storage base and a database instance area. The database instance area may be configured with a virtual IP (VIP) and a node group set, the node group set includes multiple node groups, and the multiple node groups include, for example, a GTM node group, a CN1 node group, a CN2 node group, a CN3 node group, a DN1 node group, a DN2 node group, and a DN3 node group.

[0126] The storage base saves snapshots of the database and supports restoring the database to a certain time point after a database disaster occurs. It is up to the user to decide whether to restore it or not. It can also be used as a last resort when both the production center and the disaster recovery center are down. The storage base can be an object storage base compatible with the S3 protocol. The database cluster can regularly encrypt and back up the data of each node in the database instance area to the storage base. For example, the regularity can be every 24 hours. In this way, when the disaster recovery plan is started, the database ensures that it can restore at least the full amount of data from one day ago.

[0127] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the process execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0128] Based on the same inventive concept, the embodiment of the present application also provides a database node recovery device for implementing the database node recovery method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more database node recovery device embodiments provided below can refer to the limitations of the database node recovery method above, and will not be repeated here.

[0129] In an exemplary embodiment, as shown in FIG. 14 , a database node recovery device 1400 is provided, including: a response module 1410 and an execution module 1420, wherein:

[0130] The response module 1410 is used to respond to a node abnormality event of the database cluster and obtain a node recovery process adapted to the node abnormality event; the node recovery process is used to recover the node in the database cluster indicated by the node abnormality event; the node recovery process includes multiple sub-processes with a process execution order.

[0131] The execution module 1420 is used to determine the current sub-process from the sub-processes whose execution status is to be executed in the multiple sub-processes in accordance with the process execution order; obtain the input data of the current sub-process and store it, execute the current sub-process based on the input data, obtain the execution result of the current sub-process and the output data of the current sub-process; when the execution result meets the preset conditions, store the execution result and output data, and update the execution status of the current sub-process to the executed status.

[0132] In an exemplary embodiment, a database cluster includes a set of node groups, each node group in the set of node groups includes a master node and two backup nodes; in the same node group, the backup node synchronizes data from the master node; wherein, the master node and two backup nodes contained in any node group are distributed in three data centers one by one, and any master node of the node group set is distributed in a production center among the three data centers; the three data centers include two production centers and one disaster recovery center, the two production centers are in the same city, and the disaster recovery center is in a different location from the two production centers.

[0133] In an exemplary embodiment, a node abnormality event characterizes that the original physical machine of the first production center among the two production centers is in an abnormal state, causing the node installed on the original physical machine to be in an abnormal state; the node installed on the original physical machine includes at least a part of the nodes distributed by the first production center; the node recovery process adapted to the node abnormality event includes multiple sub-processes with a process execution order, which are, in sequence, initializing a preconfigured physical machine; using the preconfigured physical machine to replace the original physical machine; performing master-slave switching on the nodes of the node group to which the master node in the node installed on the original physical machine belongs; performing load balancing based on the nodes in the database cluster that are in a normal state; reinstalling the nodes on the preconfigured physical machine according to the nodes installed on the original physical machine; and updating metadata based on the physical machine information of the preconfigured physical machine.

[0134] In an exemplary embodiment, the node group set includes multiple coordination node groups, each coordination node group includes three coordination nodes; the database node recovery device 1400 also includes a load balancing module, which is used to query the free tokens in the token bucket in response to the database request; when it is found that there are free tokens in the token bucket, the request type of the database request is determined; when the request type is a query type, the database request is forwarded to the coordination node with the best load among the coordination nodes distributed in the disaster recovery center; when the request type is a write type, the database request is forwarded to the coordination node with the best load and serving as the master node among the coordination nodes distributed in the two production centers.

[0135] In an exemplary embodiment, the database node recovery device 1400 also includes a node installation module, which is used to set a node that is adapted to the node information in the preconfigured physical machine according to the node information of the node installed on the original physical machine; the node information includes the node group to which the node belongs; each node set in the preconfigured physical machine is instructed to obtain and store the data directory from the master node in the node group to which the node belongs; each node set in the preconfigured physical machine is a standby node; when the node completes storing the data directory, it is instructed that the node obtains and stores incremental data from the master node in the node group to which it belongs, and the incremental data is the data newly added by the master node in the node group to which the node belongs during the process of copying the data directory at the node; when each node set in the preconfigured physical machine completes obtaining and storing incremental data from the master node in the node group to which it belongs, it is determined that the node is reinstalled in the preconfigured physical machine.

[0136] In an exemplary embodiment, each sub-process includes multiple work steps with a step execution order; the execution module 1420 is also used to execute the multiple work steps included in the current sub-process in sequence according to the step execution order of the multiple work steps included in the current sub-process based on the input data, obtain and store the step execution results corresponding to each work step; when the multiple work steps included in the current sub-process are executed, generate the execution results of the current sub-process and the output data of the current sub-process.

[0137] Each module in the above database node recovery device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0138] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG15. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data to be stored when executing the database node recovery method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a database node recovery method is implemented.

[0139] Those skilled in the art will understand that the structure shown in FIG. 15 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0140] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0141] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0142] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0143] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0144] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0145] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0146] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A database node recovery method, characterized in that: The method comprises: In response to a node abnormality event of a database cluster, a node recovery process adapted to the node abnormality event is obtained; the node recovery process is used to recover the node in the database cluster indicated by the node abnormality event; the node recovery process includes a plurality of sub-processes with a process execution order; According to the process execution order, the current sub-process is determined from the sub-processes whose execution status is a pending execution status included in the multiple sub-processes; Obtaining and storing input data of the current sub-process, executing the current sub-process based on the input data, and obtaining an execution result of the current sub-process and output data of the current sub-process; When the execution result meets the preset condition, the execution result and the output data are stored, and the execution state of the current sub-process is updated to an executed state.

2. The method according to claim 1, characterized in that The database cluster includes a node group set, each node group in the node group set includes a master node and two standby nodes; in the same node group, the standby node synchronizes data from the master node; Among them, one master node and two backup nodes included in any node group are distributed in three data centers one by one, and any master node of the node group set is distributed in a production center among the three data centers; The three data centers include two production centers and one disaster recovery center. The two production centers are in the same city, and the disaster recovery center is in a different location from the two production centers.

3. The method according to claim 2, characterized in that The node abnormal event indicates that the original physical machine of the first production center among the two production centers is in an abnormal state, so that the node installed by the original physical machine is in an abnormal state; the node installed by the original physical machine includes at least a part of the nodes distributed by the first production center; the node recovery process adapted to the node abnormal event includes multiple sub-processes with process execution order, which are, Initializing a preconfigured physical machine; the preconfigured physical machine is used to replace the original physical machine; Performing master-slave switching on the nodes of the node group to which the master node belongs among the nodes installed on the original physical machine; Perform load balancing based on nodes in the database cluster that are in a normal state; Reinstall nodes on the preconfigured physical machine according to the nodes installed on the original physical machine; Metadata is updated based on the physical machine information of the preconfigured physical machine.

4. The method according to claim 3, characterized in that The node group set includes a plurality of coordination node groups, each coordination node group includes three coordination nodes; and the load balancing based on the nodes in the database cluster that are in a normal state includes: In response to a database request, querying a token bucket for free tokens; When it is found that there are free tokens in the token bucket, determine the request type of the database request; When the request type is a query type, the database request is forwarded to a coordination node with the best load among the coordination nodes distributed in the disaster recovery center; When the request type is a write type, the database request is forwarded to the coordination node with the best load and serving as the master node among the coordination nodes distributed in the two production centers.

5. The method according to claim 3, characterized in that: The reinstalling of nodes on the preconfigured physical machine according to the nodes installed on the original physical machine includes: According to the node information of the node installed in the original physical machine, a node adapted to the node information is set in the pre-configured physical machine; the node information includes the node group to which the node belongs; Instructing each node set in the pre-configured physical machine to obtain and store a data directory from a master node in a node group to which the node belongs; each node set in the pre-configured physical machine is a standby node; When the node completes storing the data directory, instruct the node to obtain and store incremental data from a master node in the node group to which it belongs, wherein the incremental data is data newly added by the master node in the node group to which the node belongs during the process of the node copying the data directory; When each node set in the preconfigured physical machine completes obtaining and storing incremental data from a master node in the node group to which it belongs, it is determined that the reinstallation of the node in the preconfigured physical machine is completed.

6. The method according to any one of claims 1 to 5, characterized in that: Each sub-process includes a plurality of working steps with a step execution order; executing the current sub-process based on the input data to obtain the execution result of the current sub-process and the output data of the current sub-process includes: According to the step execution order of the multiple work steps included in the current sub-process and based on the input data, the multiple work steps included in the current sub-process are executed in sequence, and the step execution result corresponding to each work step is obtained and stored; When the multiple working steps included in the current sub-process are executed, the execution result of the current sub-process and the output data of the current sub-process are generated.

7. A database node recovery device, characterized in that: The device comprises: A response module, configured to respond to a node abnormality event of a database cluster and obtain a node recovery process adapted to the node abnormality event; the node recovery process is used to recover the node in the database cluster indicated by the node abnormality event; the node recovery process includes a plurality of sub-processes with a process execution order; An execution module is used to determine the current sub-process from the sub-processes whose execution status is to be executed in the multiple sub-processes in accordance with the process execution order; obtain the input data of the current sub-process and store it, execute the current sub-process based on the input data, obtain the execution result of the current sub-process and the output data of the current sub-process; when the execution result meets the preset conditions, store the execution result and the output data, and update the execution status of the current sub-process to the executed status.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images