Database cluster management control method and device, equipment and storage medium

Abstract

The application discloses a database cluster management control method and device, equipment and a storage medium, relates to the technical field of cluster management, realizes automatic recovery of the database cluster, and improves the high availability of the cluster. When the database cluster is abnormal, the management node can detect and automatically repair in a timely manner, without waiting for manual repair by relevant personnel, thereby reducing the time required for abnormal repair, improving the availability of the database cluster, and providing higher available database services to users. Moreover, the distributed database provided by the embodiment of the application comprises at least two management nodes and a plurality of pairs of data nodes, each pair of data nodes comprising a primary data node and a standby data node, so that data backup can be realized, and the reliability of data storage is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to the field of cluster management technology, providing a method, apparatus, device, and storage medium for managing and controlling a database cluster. Background Technology

[0002] Distributed databases consist of multiple interconnected databases managed by a distributed database management cluster. Due to their higher performance, distributed databases have an increasingly broad application prospect.

[0003] However, some anomalies are inevitable during the operation of distributed databases. Currently, when a database anomaly occurs, it usually requires manual detection and repair. Moreover, the manual recovery process is complicated and inefficient. Since distributed databases themselves carry the function of storing and processing business data, the business losses caused while waiting for manual recovery are unacceptable, and the user's database experience is also poor.

[0004] Therefore, how to achieve automatic recovery of distributed databases is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a database cluster management and control method, apparatus, device, and storage medium for implementing automatic recovery functionality of a distributed database.

[0006] On the one hand, a management and control method for a database cluster is provided, applied to any management node in a distributed database cluster, wherein the cluster includes at least two management nodes storing the cluster's metadata, and multiple pairs of data nodes storing data, each pair of data nodes including a primary data node and a backup data node; the method includes:

[0007] Triggered by the anomaly recovery command, the test command corresponding to the database operation function is executed, and the corresponding execution result is obtained; wherein, the execution result indicates whether the test command was successfully executed;

[0008] Based on the execution result and the role status information of this node, determine whether this node is in an abnormal state; wherein, the role status information is used to indicate whether this node is a primary management node or a backup management node;

[0009] If it is determined that this node is in an abnormal state, then based on the abnormal scenario conditions corresponding to each preset abnormal scenario, the target abnormal scenario that this node satisfies is determined.

[0010] Obtain the anomaly recovery strategy configured for the target anomaly scenario, and perform anomaly recovery processing based on the anomaly recovery strategy to restore the database operation function.

[0011] In one possible implementation, based on the execution result and the role status information of this node, determining whether the database operation function of this node is in an abnormal state includes:

[0012] If the execution result indicates that the number of times the test instruction failed to execute exceeds the set threshold, then the database operation function of this node is determined to be in an abnormal state.

[0013] If the execution result indicates that the number of times the test command failed to execute does not exceed the set threshold, then the network environment of this node is checked to confirm whether there is any abnormality in the network environment.

[0014] In one possible implementation, the method further includes:

[0015] Monitor whether heartbeat information is received from other management nodes within a preset time period;

[0016] If no response is received, perform offline checks on other management nodes to determine if they are in offline mode.

[0017] If it is determined that other management nodes are in the offline mode, then the operating mode of this node is switched to single-node mode.

[0018] On the one hand, a management and control device for a database cluster is provided, applied to any management node in a distributed database cluster, wherein the cluster includes at least two management nodes storing the cluster's metadata, and multiple pairs of data nodes storing data, each pair of data nodes including a primary data node and a backup data node; the device includes:

[0019] An anomaly detection unit is used to execute test instructions corresponding to database operation functions based on the triggering of an anomaly recovery instruction, and obtain corresponding execution results; wherein, the execution results indicate whether the test instructions were successfully executed; and, based on the execution results and the role status information of this node, to determine whether this node is in an abnormal state; wherein, the role status information is used to indicate whether this node is a primary management node or a backup management node;

[0020] An abnormal scene detection unit is used to determine the target abnormal scene that the node meets based on the abnormal scene conditions corresponding to each preset abnormal scene if it is determined that the node is in an abnormal state.

[0021] An anomaly recovery unit is used to obtain an anomaly recovery strategy configured for the target anomaly scenario and to perform anomaly recovery processing based on the anomaly recovery strategy in order to restore the database operation function.

[0022] In one possible implementation, the anomaly detection unit is specifically used for:

[0023] Based on the triggering of the abnormal recovery command, and according to the role status information of the at least two management nodes, it is determined whether there are multiple management nodes as master management nodes.

[0024] If there are no multiple management nodes as the primary management node, the test commands corresponding to the database operation functions will be executed to obtain the corresponding execution results.

[0025] In one possible implementation, the cluster provides services externally through a connection pool; the anomaly recovery unit is further configured to:

[0026] If there are multiple management nodes as master management nodes, then the external service function of the connection pool shall be stopped;

[0027] Based on the data operation progress of the multiple management nodes, determine the target management node with the latest data operation time from among the multiple management nodes;

[0028] If this node is not the target management node, then the operation data of this node is sent to the target management node, so that the target management node can restart the external service function of the connection pool after re-executing the corresponding data operation based on the operation data.

[0029] In one possible implementation, the anomaly recovery unit is further configured to:

[0030] If it is determined that the database operation function is in a normal state, then check whether the external service function corresponding to the connection pool of the cluster is in an abnormal state.

[0031] If the external service function is in an abnormal state, the abnormal recovery strategy configured for the external service function is obtained, and abnormal recovery processing is performed based on the abnormal recovery strategy.

[0032] In one possible implementation, the anomaly detection unit is specifically used for:

[0033] If the execution result indicates that the number of times the test instruction failed to execute exceeds the set threshold, then the database operation function of this node is determined to be in an abnormal state.

[0034] If the execution result indicates that the number of times the test command failed to execute does not exceed the set threshold, then the network environment of this node is checked to confirm whether there is any abnormality in the network environment.

[0035] In one possible implementation, the device further includes a role state switching unit for:

[0036] Based on the connection status between this node and the database, determine the current role status information of this node;

[0037] If the role status information is a backup management node, then perform an address unbinding operation to remove the binding relationship between this node and the virtual access address of the database;

[0038] When the database operation function of the cluster is detected to be in an abnormal state, determine whether this node meets the preconditions for switching to the master management node;

[0039] If the conditions are met, perform a role status switch operation to switch this node to the primary management node.

[0040] In one possible implementation, the device further includes a lock management unit for:

[0041] In response to a lock request message triggered by a target operation performed on the cluster, the corresponding lock is acquired from the local node; wherein the lock request message is used to request the allocation of a lock for the target operation, and the target operation is any operation performed on the cluster;

[0042] If the lock request message is triggered by a local application in this node, then a lock acquisition request is sent to other management nodes besides this node;

[0043] If all other management nodes besides this node successfully return the lock, then output an indication message indicating that the lock request message successfully acquired the lock.

[0044] In one possible implementation, the lock management unit is specifically used for:

[0045] If the lock request message is triggered by a local application in this node, then it is confirmed whether the cluster is currently in single-node mode; wherein, the single-node mode indicates that all management nodes other than this node are in offline mode.

[0046] If it is not in single-node mode, other management nodes besides this node will initiate lock acquisition requests;

[0047] If in single-node mode, an indication message indicating that the lock request message successfully acquired the lock is output.

[0048] In one possible implementation, the lock management unit is further configured to:

[0049] Monitor whether heartbeat information is received from other management nodes within a preset time period;

[0050] If no response is received, perform offline checks on other management nodes to determine if they are in offline mode.

[0051] If it is determined that other management nodes are in offline mode, then the operating mode of this node is switched to the single-node mode.

[0052] In one possible implementation, the lock management unit is further configured to:

[0053] Iterate through the lock information recorded in this node until all lock information has been traversed. For each lock record, perform the following operations:

[0054] Based on the process identifier in the lock information, determine whether the target process corresponding to the lock information still exists;

[0055] If the target process no longer exists and is currently in single-node mode, release the lock corresponding to the lock information in this node;

[0056] If the target process no longer exists and is not currently in single-node mode, then the corresponding management node is requested to release the lock corresponding to the lock information.

[0057] On one hand, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above methods.

[0058] On the one hand, a computer storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the above methods.

[0059] On one hand, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and executes the computer program, causing the computer device to perform the steps of any of the methods described above.

[0060] In this embodiment, by executing test instructions corresponding to database operation functions, the system determines whether the node is in an abnormal state based on the execution results and the node's role status information. When in an abnormal state, it can determine the target abnormal scenario that the node meets based on the abnormal scenario conditions corresponding to various preset abnormal scenarios. This allows it to obtain the corresponding abnormal recovery strategy configured for the target abnormal scenario and execute abnormal recovery processing based on the strategy to restore the database operation functions. Therefore, when an anomaly occurs in the database cluster, the management node can promptly detect and automatically repair it without waiting for manual repair, reducing the time required for anomaly repair and improving the availability of the database cluster to provide users with a more available database service. Furthermore, the distributed database provided in this embodiment includes at least two management nodes and multiple pairs of data nodes. Each pair of data nodes includes a primary data node and a backup data node, thereby enabling data backup and improving the reliability of data storage. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0062] Figure 1 A schematic diagram of the architecture of a distributed database cluster provided in an embodiment of this application;

[0063] Figure 2 This is a schematic diagram of the structure of the management node provided in an embodiment of this application;

[0064] Figure 3 A flowchart illustrating the cluster switching process provided in an embodiment of this application;

[0065] Figure 4 A flowchart illustrating the anomaly recovery process provided in this application embodiment;

[0066] Figure 5 Another flowchart illustrating the anomaly recovery process provided in this application embodiment;

[0067] Figure 6 A schematic diagram illustrating the process of requesting a lock during implementation of an embodiment of this application;

[0068] Figure 7 A schematic diagram illustrating the process of synchronizing the online status of management nodes as provided in this embodiment of the application;

[0069] Figure 8 A schematic diagram illustrating the process of releasing the lock provided in an embodiment of this application;

[0070] Figure 9 A schematic diagram of a database cluster management and control device provided in an embodiment of this application;

[0071] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0073] To facilitate understanding of the technical solutions provided in the embodiments of this application, some key terms used in the embodiments of this application will be explained below:

[0074] Role status: Role status refers to the status of the management nodes included in the database cluster. The role status involved in this application embodiment mainly includes the primary management node status, the standby management node status, and other statuses of non-primary management nodes or standby management nodes. The functions that the management nodes can perform are different in different role statuses.

[0075] Management Nodes: Management nodes are used to implement the management functions of the distributed database and store its metadata. Management nodes can include a primary management node and standby management nodes. The primary management node provides the main functions of the distributed database, i.e., it provides database services externally. The standby management node is in a standby state and can switch to become the primary management node when necessary. A single management node can be either a primary or a standby management node, and this switching can be achieved through a specific mechanism.

[0076] Database operation functions: Database operation functions mainly include the operations that need to be performed on the data, such as querying, writing or modifying. Generally speaking, database operation functions are for management nodes. When a management node is the primary management node, it can have database operation functions to provide database services to the outside world.

[0077] Currently, distributed databases offer higher performance than traditional databases, leading to their wider application. However, distributed databases inevitably experience anomalies during operation. When these anomalies occur, maintenance personnel are required for repair and repair. Manual recovery is cumbersome and inefficient, which is particularly problematic for distributed databases, which are responsible for storing and processing business data. The resulting business losses are unacceptable.

[0078] Based on this, this application provides a database cluster management and control method. This method can periodically detect and repair anomalies. Specifically, by executing test instructions corresponding to database operation functions, and based on the execution results and the role status information of the node, it is determined whether the node is in an abnormal state. When in an abnormal state, based on the abnormal scenario conditions corresponding to various preset abnormal scenarios, the target abnormal scenario that the node meets can be determined. The corresponding anomaly recovery strategy configured for the target abnormal scenario is then obtained, and anomaly recovery processing is executed based on the anomaly recovery strategy to restore the database operation functions. Therefore, when an anomaly occurs in the database cluster, the management node can detect it promptly and perform corresponding automatic repairs without waiting for manual repairs, reducing the time required for anomaly repairs and improving the availability of the database cluster to provide users with more available database services. Furthermore, the distributed database provided in this application includes at least two management nodes and multiple pairs of data nodes. Each pair of data nodes includes a primary data node and a backup data node, thereby enabling data backup and improving the reliability of data storage.

[0079] In addition, to increase the high availability of the distributed database, each management node in this embodiment periodically checks the status of the cluster, i.e., whether it is providing services normally. If the master management node cannot provide services, and this node meets the conditions for switching to the master management node, it will automatically trigger the switch to the master management node to provide services to the outside world, thus ensuring the high availability of the database service.

[0080] In this embodiment of the application, in order to avoid cluster anomalies or even data corruption caused by simultaneous exclusive operations, a distributed lock is provided. When performing a certain operation, it is necessary to apply for locks from all management nodes. Only when all management nodes' locks are acquired at the same time can the operation be allowed, so as to avoid cluster anomalies caused by multiple management nodes executing at the same time.

[0081] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application are applicable. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0082] The solution provided in this application can be applied to distributed database scenarios. For example... Figure 1 The diagram shown is a schematic of the architecture of a distributed database cluster provided in an embodiment of this application. The database cluster architecture consists of a client, a connection pool, at least two management nodes, and multiple pairs of storage nodes. Each pair of storage nodes includes a pair of storage nodes, one as the primary storage node and the other as the backup storage node, to back up the data and ensure data reliability.

[0083] The client can be deployed on terminal devices to provide users with database-related services. Management nodes store metadata for the database tables. Each management node can be an independent physical server, a server cluster or distributed cluster composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms, but is not limited to these. Storage nodes store the actual data and can be any device with data storage capabilities. These can be independent physical servers, server clusters or distributed clusters composed of multiple physical servers.

[0084] A connection pool is middleware located between the management node and database clients, used to provide database services. Specifically, it receives database operation requests from clients and forwards them to the current master management node for processing. In one possible implementation, a single-machine pgpool connection pool can be used, utilizing only the connection pool functionality provided by pgpool and not relying on pgpool for cluster switching.

[0085] It should be noted that the database cluster management and control method in this application embodiment can be executed by any management node. Each management node may include one or more processors, memory, and I / O interfaces for interaction. The memory of each management node may store the program instructions required for execution in the database cluster management and control method provided in this application embodiment. When these program instructions are executed by the processor, they can be used to implement the database anomaly recovery process provided in this application embodiment.

[0086] In one possible implementation, the distributed database cluster may include two management nodes, one as the primary management node and the other as the backup management node, and of course, the two can be switched when necessary.

[0087] In this embodiment, the client, connection pool, at least two management nodes, and multiple pairs of storage nodes can communicate directly or indirectly through one or more networks. This network can be a wired network or a wireless network; for example, the wireless network can be a mobile cellular network or a Wireless-Fidelity (WIFI) network, or any other possible network. This embodiment does not limit the types of networks used.

[0088] See Figure 2 The diagram shown is a structural schematic of the management node provided in an embodiment of this application. Figure 2 This example uses two management nodes; the same logic applies when there are more management nodes. Each management node can include a High Availability (HA) module, an anomaly recovery module, and a cluster control module. The functions of each module are as follows:

[0089] (1) HA module

[0090] The HA module is primarily used to monitor the cluster status and trigger cluster failover when the primary management node fails, switching the standby management node to the primary management node to ensure high availability. The HA module periodically checks the role status of this node, which is categorized as primary management node (represented by "primary"), standby management node (represented by "mirror"), and other roles (represented by "other"). Different operations are performed based on the different role statuses. For example, if the node is identified as "primary," the virtual access address of the externally serving database will be bound to this node, and the count of failed cluster status retrievals will be reset. Alternatively, if the node is identified as "mirror," the virtual access address will be unbound from the node, and a check will be performed to determine if a failover needs to be triggered. Furthermore, if the node is identified as "other," the virtual access address will be unbound from the node, and the count of failed cluster status retrievals will be reset.

[0091] (2) Anomaly Recovery Module

[0092] The anomaly recovery module is primarily used for recovery from common anomaly scenarios, enabling real-time automatic anomaly recovery and preventing database service interruptions. Specifically, management nodes can cyclically execute anomaly recovery checks, performing different operations depending on their role and status. When a split-brain scenario is detected, the system determines which management node can act as the primary management node based on data and write-ahead logs, and the other node is reassigned as the backup management node. When management nodes are in a normal state, the connection pool status is further checked; the primary management node checks and maintains data nodes, while the backup management node checks and maintains the backup management node's service status. When an anomaly occurs, a pre-defined recovery strategy can be attempted.

[0093] (3) Cluster control module

[0094] The cluster control module is primarily used to manage cluster startup, shutdown, and recovery operations to prevent cluster anomalies or even data corruption caused by simultaneous exclusive operations. The module implements distributed lock management and allocation, and mainly includes a request receiving thread, a heartbeat sending thread, a heartbeat receiving thread, and a periodic check thread. The request receiving thread receives request messages from the local application and another management node. If it's a request from the local application, it synchronously sends the request to the other management node. A successful request is only considered successful if both the local and remote requests are successful. The heartbeat sending and receiving threads are used to synchronize the online status between the two management nodes. If one management node loses its heartbeat, the other management node pings the peer management node. If it determines the peer management node is offline, it switches to single-node mode. That is, if all other management nodes are offline, it runs in single-node mode. If a heartbeat is received, it switches back to dual-node mode, and the heartbeat message includes the current node's lock information. The periodic check thread checks whether the local process holding the lock is still alive based on the lock information in its memory. If the process has crashed and hasn't released the lock, it releases the lock. When it is necessary to operate on the cluster, a lock needs to be requested from the cluster control module of this node. The management node maintains the lock information using a singleton pattern to achieve sharing between threads.

[0095] The specific functions performed by each of the above modules will be described in detail in the subsequent method embodiments, so they will not be elaborated on here.

[0096] The following describes the database cluster management and control method provided by the exemplary embodiments of this application in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0097] In the operation of a distributed database cluster, the primary management node provides services. If the primary management node fails, the cluster will be unable to provide database services. Therefore, it is necessary to switch the standby management node to the primary management node to ensure high availability of the cluster. This process can be implemented by the HA module mentioned above, which can run on any management node. See also... Figure 3 The diagram shown illustrates the cluster switching process provided in this embodiment of the application. This process can be executed by any management node, and its specific implementation process is as follows:

[0098] Step 301: Determine the current role status information of this node.

[0099] In this embodiment, the HA module can periodically detect whether a cluster switch is triggered, or it can detect whether a switch is to be performed based on the triggering of cluster events.

[0100] The role status includes three types: primary, mirror, and other. The management node can determine its own role status by trying to execute the functions that each role status can perform and based on the execution results.

[0101] In one possible implementation, considering that only the primary management node providing database services can connect to the database and access data nodes normally, the management node can determine its current role status based on the connection status between itself and the database. That is, the management node can attempt to connect to the database; if the connection is successful, it indicates that the node is the primary management node; if the connection fails, it indicates that the node is a backup management node, i.e., in a mirror state; otherwise, the role status can be determined as "other".

[0102] In one possible implementation, the management node can determine its current role status information based on relevant configuration fields in its own configuration information. These configuration fields are used to characterize the current role status information of the node.

[0103] Step 302: If the role status information indicates that this node is primary, then perform the address binding operation to bind the virtual access address of the database to this node.

[0104] In practical applications, after the address binding operation is performed, the virtual access address of the database is bound to this node. When a client accesses the virtual access address, the connection pool can distribute the corresponding access request to this node for processing, so as to realize the database access function.

[0105] Specifically, performing address binding operations can refer to modifying the mapping relationship between a virtual access address and its corresponding management node, so that the connection pool can identify the current master management node based on this mapping relationship when distributing access requests.

[0106] Step 303: Reset the cluster anomaly count.

[0107] The cluster failure count is represented by failed_times. Resetting failed_times to zero indicates that the current cluster's external database service function is normal.

[0108] Step 304: If the role status information indicates that this node is a mirror, then perform the address unbinding operation to remove the binding relationship between this node and the virtual access address of the database.

[0109] When this node is a mirror, it means that this node does not need to implement the functions of the master node. In order to avoid connection pool distribution errors, it is necessary to unbind the virtual access address of this node from the database. In this way, when the connection pool distributes access requests, it can distribute them to the corresponding master management node according to the binding relationship, and avoid distributing them to the backup management node, thereby ensuring the normal operation of database access.

[0110] Step 305: Check whether the cluster's database operation function is in an abnormal state.

[0111] In this embodiment of the application, when the database operation function is abnormal, it is impossible to operate the database normally, which means that the current master management node is in an abnormal state and cannot perform normal work. Therefore, it may be necessary to switch the role state to ensure the high availability of the cluster.

[0112] In one possible implementation, the database operation function can be determined by attempting to access it and the result of the access. If access is successful, the database operation function is considered to be functioning normally; conversely, if access is unsuccessful, the database operation function is considered to be functioning abnormally.

[0113] In one possible implementation, it is also possible to obtain information from the master management node about whether it is in an abnormal state through state-keeping communication with the master management node.

[0114] Step 306: When the database operation function is in an abnormal state, update the cluster error count, that is, increment failed_times by 1.

[0115] Step 307: Determine if the cluster anomaly count has reached the upper limit.

[0116] Step 308: If the cluster anomaly count reaches the upper limit, determine whether this node meets the prerequisites for switching to the primary management node.

[0117] Specifically, the preconditions refer to whether the node meets the switching check items, that is, whether the node is in some special scenario. For example, when the node is performing a restart operation, it cannot provide services normally, thus determining that the preconditions are not met.

[0118] Step 309: If the prerequisites are met, perform a role status switching operation to switch this node to the primary management node.

[0119] Step 310: When the database operation function is in a normal state, reset the cluster exception count.

[0120] Step 311: If the role status information indicates that this node is "other", then perform the address unbinding operation to remove the binding relationship between this node and the virtual access address of the database.

[0121] Step 312: Reset the cluster anomaly count.

[0122] It's important to note that the state switching of the HA module described above is triggered by the standby management node. Since the original primary management node may be offline at this time, this operation must be performed on the standby management node. Therefore, during the execution of the above process, it is also subject to constraints from the cluster control module. That is, an operation can only be executed if a lock is acquired. For example, when triggering a switch, a lock must be acquired before execution can proceed. This prevents multiple management nodes from switching to the primary management node. Of course, in practical applications, there may be situations where the original primary management node is offline. In such cases, it is impossible to acquire a lock from the original primary management node. In this situation, the cluster can be changed to single-node mode to facilitate a normal switchover.

[0123] See Figure 4 The diagram shown illustrates a flowchart of an anomaly recovery process provided in this application embodiment. This process can be implemented using the aforementioned anomaly recovery module, enabling recovery from some common anomaly scenarios to achieve real-time anomaly recovery and avoid business interruption. The specific implementation process is as follows:

[0124] Step 401: Based on the triggering of the exception recovery instruction, execute the test instruction corresponding to the database operation function and obtain the corresponding execution result; wherein, the execution result indicates whether the test instruction was successfully executed.

[0125] In this embodiment of the application, the abnormal recovery process can be executed periodically. In this case, the abnormal recovery instruction can be a pre-set periodic instruction that is triggered at predetermined intervals. Alternatively, the abnormal recovery process can also be executed based on condition triggering. In this case, when the triggering condition is met, the execution of the abnormal recovery instruction is triggered to perform the abnormal recovery process.

[0126] Specifically, the test command is used to test whether this node can successfully execute database operation functions. If it can be executed successfully, it indicates that this node can implement the database service functions that the main management node can implement. If it cannot be executed successfully, it indicates that this node cannot implement the database service functions that the main management node can implement, that is, it is not the main management node.

[0127] Step 402: Based on the execution result and the role status information of this node, determine whether this node is in an abnormal state; wherein, the role status information is used to indicate whether this node is a primary management node or a backup management node.

[0128] Among them, the role status information can represent the current actual role status of this node, so as to determine whether the execution result matches the role status indicated by the role status information, and thus determine whether this node is in an abnormal state.

[0129] Specifically, normally, the primary management node is able to successfully execute database operations. Therefore, when the role status information indicates that this node is the primary management node, but the execution result indicates that the database operation function cannot be successfully executed, it indicates that this node is in an abnormal state; or, when the role status information indicates that this node is the primary management node, and the execution result indicates that the database operation function is successfully executed, it indicates that this node is in a normal state.

[0130] Similarly, if the standby management node cannot successfully execute the database operation function, then when the role status information indicates that this node is a standby management node and the execution result indicates that the database operation function cannot be successfully executed, it means that this node is in a normal state; or, when the role status information indicates that this node is a standby management node, but the execution result indicates that the database operation function is successfully executed, it means that this node is in an abnormal state.

[0131] Step 403: If it is determined that this node is in an abnormal state, then based on the abnormal scenario conditions corresponding to each preset abnormal scenario, determine the target abnormal scenario that this node satisfies.

[0132] Step 404: Obtain the exception recovery strategy configured for the target exception scenario, and execute exception recovery processing based on the exception recovery strategy to restore the database operation function.

[0133] In this embodiment, abnormal scenario conditions and corresponding abnormal recovery strategies can be pre-configured for each abnormal scenario. When the node is in an abnormal state, it can be matched with each preset abnormal scenario to determine the target abnormal scenario that matches it. Then, the abnormal recovery strategy configured for the target abnormal scenario can be used for abnormal recovery.

[0134] Specifically, during the matching process, the system can match the current abnormal data of a node against each preset abnormal scenario one by one to determine if the current abnormal data matches the conditions of the current preset abnormal scenario. If a match is found, the matching process stops; if no match is found, the system continues to match against the next preset abnormal scenario until all preset abnormal scenarios have been matched. If no preset abnormal scenario can be successfully matched, an alert can be triggered to relevant personnel so that they can take timely corrective action.

[0135] See Figure 5 The diagram shown is another flowchart illustrating the anomaly recovery process provided in this application embodiment. The specific implementation process is as follows:

[0136] Step 501: Obtain the role status information of each management node.

[0137] Specifically, the role status information of other management nodes besides the local node can be obtained through synchronization between various management nodes.

[0138] Step 502: Determine if there are multiple management nodes as master management nodes.

[0139] Specifically, when an abnormal recovery command is triggered, it can be determined whether there are multiple management nodes as master management nodes based on the role status information of each management node. Normally, during normal operation, only one management node is allowed to provide database services. When there are multiple management nodes as master management nodes, it may cause errors in the database data. This phenomenon is called split-brain.

[0140] Step 503: If there are multiple management nodes as master management nodes, stop the external service function of the connection pool.

[0141] Specifically, if a split-brain phenomenon is detected, in order to prevent multiple master management nodes from operating on the database and causing database anomalies, the external service function of the connection pool should be stopped first to prevent new data from being written to the database.

[0142] Step 504: Based on the data operation progress of multiple management nodes, determine the target management node with the latest data operation time from among the multiple management nodes.

[0143] Taking a scenario with two management nodes as an example, when a split-brain phenomenon occurs, both this node and the other management node are master management nodes. This node can then send a request to the other management node to obtain the data operation progress of the other management node. Based on the data operation progress of itself and the other management node, it can select one as the final master management node.

[0144] In one possible implementation, the target management node with the latest data operation time can be used as the final master management node.

[0145] Step 505: If this node is not the target management node, then this node performs a backup and redo.

[0146] Step 506: Restart the connection pool function.

[0147] In other words, if a primary management node is selected, and this node is not the target management node, a backup and redo operation is required. Specifically, the operation data can be sent to the selected target management node, allowing the target management node to re-execute the corresponding data operations based on the operation data and then restart the connection pool's external service functions to continue providing database services.

[0148] In this embodiment of the application, considering that multiple management nodes may operate on the database service simultaneously when a split-brain phenomenon occurs, which may lead to database operation conflicts, the above-mentioned anomaly repair process can be used to repair the split-brain phenomenon in a timely manner, thereby avoiding conflicts in database operations and data confusion in the database, and improving the reliability of database management.

[0149] Step 507: If the determination result of step 502 is negative, that is, there are no multiple management nodes as master management nodes, then determine whether this node is in an abnormal state.

[0150] Considering the significant impact of split-brain phenomena on databases, in order to avoid data confusion caused by split-brain phenomena, it is necessary to first determine whether the current system is experiencing split-brain phenomena before testing the database operation functions of this node. Only when it is determined that there is no split-brain phenomenon, this node needs to continue to check whether it is in an abnormal state. That is, it can execute the test instructions corresponding to the database operation functions, and then determine whether this node is in an abnormal state based on the corresponding execution results and its own role status information.

[0151] Step 508: If this node is not in a normal state, then reset the number of times the test command is executed; where the number of times the test command is executed can be represented as retryCheckMaster.

[0152] Step 509: Check the connection pool status; if abnormal, perform abnormal recovery processing.

[0153] In this embodiment, when the management node is in a normal state, it will continue to check for other anomalies. Since the connection pool is used for communication between the client and the management node, in order to ensure smooth communication between the client and the management node, the connection pool status will also be checked. This allows the connection pool to be repaired in a timely manner when an anomaly occurs, preventing the client from being unable to connect to the management node and improving the reliability of the database service function.

[0154] Specifically, the primary management node will check and maintain the data nodes, while the backup management node will check and maintain the service status of the backup management node.

[0155] Specifically, this node can attempt to connect to the connection pool to detect whether the external service function corresponding to the cluster's connection pool is in an abnormal state. If the external service function is in an abnormal state, it can obtain the abnormal recovery policy configured for the external service function and execute the abnormal recovery process based on the abnormal recovery policy.

[0156] Step 510: Determine whether this node is the primary management node.

[0157] Step 511: If yes, that is, this node is the master management node, then check whether there is any abnormality in the data node. If there is an abnormality, perform abnormality recovery processing on the data node.

[0158] Step 512: If no, meaning this node is a backup management node, then check again whether there is an anomaly in this node. If there is, try to perform anomaly recovery processing according to the anomaly type of this node.

[0159] Step 513: If the determination result of step 507 is yes, that is, when this node is in an abnormal state, determine whether the number of times the test instruction execution failed exceeds the set threshold.

[0160] In this embodiment of the application, if the execution result determines that the node is in an abnormal state, then retryCheckMaster is incremented by one. When the value of retryCheckMaster is greater than the set threshold, it indicates that the node is indeed in an abnormal state. Multiple determinations are made to avoid misjudgment and improve accuracy.

[0161] Step 514: If the condition is exceeded, then based on the condition, select the corresponding preset abnormal scenario abnormality recovery strategy for recovery processing. For example, when a data node is abnormal, perform abnormal recovery on the data node; or, if a node switching failure occurs, restore the node to the state that the node prefers.

[0162] Step 515: If the limit is not exceeded, check the network environment of this node to confirm whether there are any abnormalities. For example, check whether the bandwidth of the network card link aggregation can meet the service requirements.

[0163] By periodically performing the above-mentioned anomaly detection, the management node can promptly repair any anomalies that occur, thereby avoiding database service interruptions and improving service reliability.

[0164] In this embodiment, considering that the operation of the management node may affect other management nodes or even the entire cluster, and given that the distributed architecture makes the management nodes independent of each other, simultaneous exclusive operations may cause cluster anomalies or even data corruption. For example, the switching involved in the HA module mentioned above is only triggered when the management node is a standby management node, and considering that the primary management node may be offline, this operation must be performed on the standby management node. The detection and recovery process for data nodes in the anomaly recovery module must also be performed on the primary management node, because the standby management node may also be offline. The above-mentioned operations will affect the operation of the entire cluster, but since they are not on the same node, conflicts cannot be resolved using local locks. Similarly, there are user-issued service start-ups and shutdowns, so a distributed lock is needed. However, conventional distributed lock implementations rely on distributed lock middleware, and the high availability of the middleware itself becomes a problem in two-node scenarios. Therefore, this embodiment provides a simplified distributed lock management module.

[0165] See Figure 6 The diagram shown is a schematic of the lock application process in an embodiment of this application. This process can be handled by the request receiving thread of the cluster control module, which can receive request messages from local applications and other management nodes. If it is a request from a local application, the request will be sent synchronously to other management nodes. If both the local and remote nodes succeed, the lock application is considered successful.

[0166] Step 601: Receive lock request message.

[0167] In this embodiment of the application, as described above, in order to avoid the impact of exclusive operations, a lock needs to be requested before the operation can be performed, and the operation can only be executed smoothly when the request is successful.

[0168] Step 602: Acquire the corresponding lock from this node.

[0169] Furthermore, in response to a lock request message triggered by a target operation performed on the cluster, the corresponding lock is acquired from the local node. The lock request message is used to request the allocation of a lock for the target operation, which can be any operation performed on the cluster. Of course, in practical applications, some operations can be selected as target operations instead of all operations. For other operations that are not selected as target operations, they can be executed directly without waiting for lock allocation.

[0170] Step 603: Determine whether the lock request message was triggered by a local application in this node.

[0171] If the local lock is successfully acquired, it is necessary to determine whether the lock request was triggered by a local application. A local application can refer to an operation performed on this node or an operation that this node needs to execute. In other words, it can be considered to be triggered by a local application, such as restarting this node. If it is not triggered by a local application, it should be triggered by another management node. Then, return a success message and proceed to step 608.

[0172] Step 604: If the lock request message is triggered by a local application on this node, confirm whether the cluster is currently in single-node mode.

[0173] Step 605: If it is not a single-node mode, then initiate a lock acquisition request to other management nodes besides this node.

[0174] To ensure normal operation even when other nodes malfunction, this application also provides a single-node mode. In single-node mode, the node can operate independently without considering other nodes. Therefore, if the local node is determined to be the trigger, it is necessary to request a lock from other management nodes. Thus, it is necessary to confirm whether the current mode is single-node. If other management nodes are offline or unable to communicate, it is not necessary to request a lock from them. If it is not single-node mode, it is necessary to request a lock from other management nodes. If it is single-node mode, a success message is returned, and the process proceeds to step 608.

[0175] Step 606: Determine whether other management nodes have successfully allocated locks.

[0176] Step 607: If unsuccessful, release the local lock and proceed to step 609; if successful, return a success message and proceed to step 608.

[0177] Step 608: Output a lock request message to obtain a lock success indication message.

[0178] Step 609: Output an indication message indicating that the lock request failed to obtain the indication information.

[0179] Specifically, if the lock is successfully acquired, the target operation can be executed smoothly; otherwise, if the lock acquisition fails, the target operation will refuse to respond.

[0180] In this embodiment, the management nodes need to synchronize their online status. If a management node loses its heartbeat, other management nodes can ping the peer node. If they confirm that the peer node is offline, they will switch to single-node mode. If they obtain heartbeat information, they will switch back from single-node mode to dual-node mode. At the same time, the heartbeat information will include the lock information of the current node, so that the operation of the cluster can be carried out normally.

[0181] See Figure 7 The diagram illustrates the process of synchronizing the online status of management nodes in an embodiment of this application. This process can be handled by the heartbeat sending and receiving threads of the cluster control module. It should be noted that the following synchronization process is only an example of a single synchronization process using two management nodes. In actual applications, it can be performed once or in multiple loops; this embodiment does not impose any limitations on this.

[0182] Step 701: Send a heartbeat information acquisition request to the peer management node.

[0183] Step 702: Determine whether the heartbeat information of the peer management node has been obtained.

[0184] Step 703: If the heartbeat information of the peer management node has been obtained, then update the heartbeat information of the peer management node.

[0185] Step 704: If the current mode is single-node, switch to dual-node mode.

[0186] Step 705: If the heartbeat information of the peer management node is not obtained, try to obtain it multiple times, and determine whether the continuous duration of not obtaining the heartbeat information exceeds the preset duration threshold, such as whether the heartbeat information has not been received for 30 consecutive seconds.

[0187] Step 706: Determine if the peer management node is offline. For example, you can confirm whether the peer node is offline by pinging it.

[0188] Step 707: If it is determined that it is offline, then mark itself as a single-node mode.

[0189] The above process is the same for any management node, so it will not be described in detail again.

[0190] In this embodiment, it is necessary to periodically check whether the local lock-holding process is still alive. If the process crashes without releasing the lock, the lock needs to be released. Furthermore, when an application needs to operate the cluster, it only needs to request the lock from the local cluster control module. The management node maintains lock information using a singleton pattern to achieve thread-to-thread sharing. The above process can be implemented through a periodic check thread in the cluster control module. See also... Figure 8 The diagram shows a process flow chart for releasing the lock provided in this application embodiment.

[0191] Step 801: Based on the lock information in the heartbeat information of the peer management node, remove the locks in this node that were applied for by the peer management node and whose time exceeds the preset duration threshold, but which have already been released by the peer management node.

[0192] Step 802: Traverse the lock information recorded in the current management module until the end of the traversal of all lock information is reached.

[0193] Step 803: Based on the process identification information in the lock information of this node, such as IP and PID information, check whether the target process for which this node has applied for the lock still exists. If it does, it cannot be released, and then jump to step 802 to traverse the next lock information.

[0194] Step 804: If it does not exist, the lock needs to be released. First, it is necessary to determine whether the current node is in single-node mode.

[0195] Step 805: If yes, release the lock corresponding to the lock information in this node.

[0196] Step 806: If not, request the release of the lock corresponding to the lock information from the peer management node.

[0197] Through the above process, while performing heartbeat detection, lock information can also be sent to the peer, thereby releasing locks that are no longer needed, reducing the resource consumption of locks, and improving the resource utilization of the cluster.

[0198] It should be noted that the above example is mainly based on two management nodes, but the above method can also be applied to scenarios with multiple management nodes. For example, when switching to single-node mode, it is necessary to determine that other management nodes are offline. Therefore, this application will not elaborate on the case of multiple management nodes.

[0199] In this embodiment of the application, considering that no lock waiting process is set in the above process, it is impossible to handle deadlock scenarios with different lock levels or different lock resources. However, in actual use, it can be extended according to actual needs to add lock waiting. In this way, the periodic checking thread can find deadlock information from the lock information of the two nodes and unlock according to certain rules, so that requests that will not conflict can be responded to at the same time.

[0200] In summary, this application embodiment implements a stable automatic recovery and high availability scheme for database cluster anomalies. By performing different recovery operations for different scenarios on the primary and backup management nodes, the system's high availability is ensured. It can flexibly respond to various anomalies, greatly reduce manpower input, and has good scalability. Furthermore, through self-implemented two-node distributed lock management, it resolves potential conflicts between recovery operations and user operations.

[0201] Please see Figure 9 Based on the same inventive concept, this application also provides a database cluster management and control device 90, applied to any management node in a distributed database cluster. The cluster includes at least two management nodes storing cluster metadata and multiple pairs of data nodes storing data, each pair of data nodes including a primary data node and a backup data node; the device includes:

[0202] Anomaly detection unit 901 is used to execute test instructions corresponding to database operation functions based on the triggering of anomaly recovery instructions, and obtain corresponding execution results; wherein, the execution results indicate whether the test instructions were successfully executed; and, based on the execution results and the role status information of this node, to determine whether this node is in an abnormal state; wherein, the role status information is used to indicate whether this node is a primary management node or a backup management node;

[0203] The abnormal scene detection unit 902 is used to determine the target abnormal scene that the node satisfies based on the abnormal scene conditions corresponding to each preset abnormal scene if it is determined that the node is in an abnormal state.

[0204] The anomaly recovery unit 903 is used to obtain the anomaly recovery strategy configured for the target anomaly scenario and to perform anomaly recovery processing based on the anomaly recovery strategy in order to restore the database operation function.

[0205] In one possible implementation, the anomaly detection unit 901 is specifically used for:

[0206] Based on the triggering of the abnormal recovery command, and according to the role status information of at least two management nodes, it is determined whether there are multiple management nodes as master management nodes.

[0207] If there are no multiple management nodes as the primary management node, the test commands corresponding to the database operation functions will be executed to obtain the corresponding execution results.

[0208] In one possible implementation, the cluster provides services externally through a connection pool; then the anomaly recovery unit 903 is further used for:

[0209] If there are multiple management nodes as master management nodes, then the external service function of the connection pool will be stopped.

[0210] Based on the data operation progress of multiple management nodes, determine the target management node with the latest data operation time from among the multiple management nodes;

[0211] If this node is not the target management node, the operation data of this node will be sent to the target management node so that the target management node can restart the external service function of the connection pool after re-executing the corresponding data operation based on the operation data.

[0212] In one possible implementation, the anomaly recovery unit 903 is further configured to:

[0213] If it is determined that the database operation function is in a normal state, then check whether the external service function corresponding to the cluster's connection pool is in an abnormal state.

[0214] If the external service function is in an abnormal state, the abnormal recovery policy configured for the external service function is obtained, and abnormal recovery processing is performed based on the abnormal recovery policy.

[0215] In one possible implementation, the anomaly detection unit 901 is specifically used for:

[0216] If the execution result indicates that the number of times the test command failed to execute exceeds the set threshold, then the database operation function of this node is determined to be in an abnormal state.

[0217] If the execution result indicates that the number of times the test command failed to execute has not exceeded the set threshold, then the network environment of this node will be checked to confirm whether there is any abnormality in the network environment.

[0218] In one possible implementation, the device further includes a role state switching unit 904, for:

[0219] Based on the connection status between this node and the database, determine the current role status information of this node;

[0220] If the role status information is a standby management node, then perform an address unbinding operation to remove the binding relationship between this node and the database's virtual access address;

[0221] When the cluster's database operation function is detected to be in an abnormal state, determine whether this node meets the prerequisites for switching to the primary management node.

[0222] If the conditions are met, perform a role status switch operation to switch this node to the primary management node.

[0223] In one possible implementation, the device further includes a lock management unit 905, for:

[0224] In response to a lock request message triggered by a target operation performed on the cluster, acquire the corresponding lock from this node; wherein, the lock request message is used to request the allocation of a lock for the target operation, which is any operation performed on the cluster;

[0225] If the lock request message is triggered by a local application in this node, then a lock acquisition request is sent to other management nodes other than this node;

[0226] If all other management nodes besides this node successfully return the lock, then output an indication lock request message indicating that the lock was successfully obtained.

[0227] In one possible implementation, the lock management unit 905 is specifically used for:

[0228] If the lock request message is triggered by a local application on this node, then confirm whether the cluster is currently in single-node mode; where single-node mode indicates that all management nodes other than this node are in offline mode.

[0229] If it is not in single-node mode, other management nodes besides this node will initiate lock acquisition requests;

[0230] If in single-node mode, an indication message indicating successful lock acquisition is output.

[0231] In one possible implementation, the lock management unit 905 is further configured to:

[0232] Monitor whether heartbeat information is received from other management nodes within a preset time period;

[0233] If no response is received, perform offline checks on other management nodes to determine if they are in offline mode.

[0234] If it is determined that other management nodes are in offline mode, then switch the operating mode of this node to single-node mode.

[0235] In one possible implementation, the lock management unit 905 is further configured to:

[0236] Iterate through the lock information recorded in this node until all lock information has been traversed. For each lock record, perform the following operations:

[0237] Based on the process identifier in the lock information, determine whether the target process corresponding to the lock information still exists;

[0238] If the target process no longer exists and the current state is single-node mode, release the locks corresponding to the lock information in this node;

[0239] If the target process no longer exists and is not currently in single-node mode, then request the corresponding management node to release the lock corresponding to the lock information.

[0240] The above-mentioned device enables a stable automatic recovery and high availability solution for database cluster anomalies. By performing different recovery operations for different scenarios on the primary and backup management nodes, the system's high availability is ensured. It can flexibly respond to various anomalies, greatly reduce manpower input, and has good scalability. Furthermore, through self-implemented two-node distributed lock management, it resolves potential conflicts between recovery operations and user operations.

[0241] This device can be used to execute the methods shown in the various embodiments of this application. Therefore, the functions that each functional module of this device can achieve can be referred to the description of the foregoing embodiments, and will not be repeated here.

[0242] Please see Figure 10 Based on the same technical concept, embodiments of this application also provide a computer device. In one embodiment, the computer device can be... Figure 1 The management node shown is a computer device such as Figure 10 As shown, it includes a memory 1001, a communication module 1003, and one or more processors 1002.

[0243] The memory 1001 is used to store computer programs executed by the processor 1002. The memory 1001 may mainly include a program storage area and a data storage area. The program storage area may store operation clusters and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0244] Memory 1001 may be volatile memory, such as random-access memory (RAM); memory 1001 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1001 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1001 may be a combination of the above-described memories.

[0245] The processor 1002 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1002 is used to implement the aforementioned database cluster management and control method when calling computer programs stored in the memory 1001.

[0246] The communication module 1003 is used to communicate with terminal devices and other servers.

[0247] This application embodiment does not limit the specific connection medium between the memory 1001, communication module 1003, and processor 1002. This application embodiment... Figure 10 The memory 1001 and the processor 1002 are connected via a bus 1004, and the bus 1004 is in Figure 10 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. Bus 1004 can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 10 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0248] The memory 1001 stores a computer storage medium, which stores computer-executable instructions. The computer-executable instructions are used to implement the database cluster management and control method of the embodiments of this application. The processor 1002 is used to execute the database cluster management and control method of the above embodiments.

[0249] Based on the same inventive concept, embodiments of this application also provide a storage medium storing a computer program that, when run on a computer, causes the computer to execute the steps in the database cluster management and control method according to various exemplary embodiments of this application described above.

[0250] In some possible implementations, various aspects of the database cluster management and control method provided in this application can also be implemented in the form of a computer program product, which includes a computer program. When the program product is run on a computer device, the computer program is used to cause the computer device to perform the steps in the database cluster management and control method according to various exemplary embodiments of this application described above. For example, the computer device can perform the steps of each embodiment.

[0251] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, a cluster, device, or apparatus of electrical, magnetic, optical, electromagnetic, infrared, or semiconductor materials, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0252] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on a computer device. However, the program product of this application is not limited thereto. In this application, the readable storage medium may be any tangible medium that contains or stores a program, and the computer program included therein may be used by or in conjunction with a command execution cluster, apparatus, or device.

[0253] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution cluster, apparatus, or device.

[0254] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0255] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages.

[0256] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0257] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0258] Those skilled in the art will understand that embodiments of this application can be provided as methods, clusters, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0259] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0260] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A management and control method for a database cluster, characterized in that, The method is applied to any management node in a distributed database cluster, wherein the cluster includes at least two management nodes storing the cluster's metadata, and multiple pairs of data nodes storing data, each pair of data nodes including a primary data node and a backup data node; the method includes: Triggered by the anomaly recovery command, the test command corresponding to the database operation function is executed, and the corresponding execution result is obtained; wherein, the execution result indicates whether the test command was successfully executed; Based on the execution result and the role status information of this node, determine whether this node is in an abnormal state; wherein, the role status information is used to indicate whether this node is a primary management node or a backup management node; If it is determined that this node is in an abnormal state, then based on the abnormal scenario conditions corresponding to each preset abnormal scenario, the target abnormal scenario that this node meets the abnormal scenario conditions is determined. Each preset abnormal scenario includes: an abnormal scenario where the role status of at least one node is inconsistent with its own database operation function, and / or, an abnormal scenario where there are multiple main management nodes among the at least two management nodes. Obtain the anomaly recovery strategy configured for the target anomaly scenario, and perform anomaly recovery processing based on the anomaly recovery strategy to restore the database operation function; wherein, performing anomaly recovery processing based on the anomaly recovery strategy to restore the database operation function includes: in an anomaly scenario where there are multiple master management nodes among the at least two management nodes, determining one of the multiple master management nodes as the master management node based on the data and write-ahead logs, and configuring the remaining master management nodes as backup management nodes.

2. The method as described in claim 1, characterized in that, Triggered by the exception recovery command, the test commands corresponding to the database operation functions are executed, and the corresponding execution results are obtained, including: Based on the triggering of the abnormal recovery command, and according to the role status information of the at least two management nodes, it is determined whether there are multiple management nodes as master management nodes. If there are no multiple management nodes as the primary management node, the test commands corresponding to the database operation functions will be executed to obtain the corresponding execution results.

3. The method as described in claim 2, characterized in that, The cluster provides services externally through a connection pool; then, after determining whether there are multiple management nodes as master management nodes based on the role status information of the at least two management nodes triggered by an anomaly recovery command, the method further includes: If there are multiple management nodes as master management nodes, then the external service function of the connection pool shall be stopped; Based on the data operation progress of the multiple management nodes, determine the target management node with the latest data operation time from among the multiple management nodes; If this node is not the target management node, then the operation data of this node is sent to the target management node, so that the target management node can restart the external service function of the connection pool after re-executing the corresponding data operation based on the operation data.

4. The method as described in claim 1, characterized in that, After determining whether the database operation function of this node is in an abnormal state based on the execution result and the role status information of this node, the method further includes: If it is determined that the database operation function is in a normal state, then check whether the external service function corresponding to the connection pool of the cluster is in an abnormal state. If the external service function is in an abnormal state, the abnormal recovery strategy configured for the external service function is obtained, and abnormal recovery processing is performed based on the abnormal recovery strategy.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Based on the connection status between this node and the database, determine the current role status information of this node; If the role status information is a backup management node, then perform an address unbinding operation to remove the binding relationship between this node and the virtual access address of the database; When the database operation function of the cluster is detected to be in an abnormal state, determine whether this node meets the preconditions for switching to the master management node; If the conditions are met, perform a role status switch operation to switch this node to the primary management node.

6. The method according to any one of claims 1 to 4, characterized in that, The method further includes: In response to a lock request message triggered by a target operation performed on the cluster, the corresponding lock is acquired from the local node; wherein the lock request message is used to request the allocation of a lock for the target operation, and the target operation is any operation performed on the cluster; If the lock request message is triggered by a local application in this node, then a lock acquisition request is sent to other management nodes besides this node; If all other management nodes besides this node successfully return the lock, then output an indication message indicating that the lock request message successfully acquired the lock.

7. The method as described in claim 6, characterized in that, If the lock request message is triggered by a local application on this node, then a lock acquisition request is initiated to other management nodes besides this node, including: If the lock request message is triggered by a local application in this node, then it is confirmed whether the cluster is currently in single-node mode; wherein, the single-node mode indicates that all management nodes other than this node are in offline mode; If it is not in single-node mode, other management nodes besides this node will initiate lock acquisition requests; If in single-node mode, an indication message indicating that the lock request message successfully acquired the lock is output.

8. The method as described in claim 6, characterized in that, If the heartbeat information carries lock information from other management nodes, then the method further includes: Iterate through the lock information recorded in this node until all lock information has been traversed. For each lock record, perform the following operations: Based on the process identifier in the lock information, determine whether the target process corresponding to the lock information still exists; If the target process no longer exists and is currently in single-node mode, release the lock corresponding to the lock information in this node; If the target process no longer exists and is not currently in single-node mode, then the corresponding management node is requested to release the lock corresponding to the lock information.

9. A management and control device for a database cluster, characterized in that, The apparatus is applied to any management node in a distributed database cluster, the cluster comprising at least two management nodes storing the cluster's metadata, and multiple pairs of data nodes storing data, each pair of data nodes including a primary data node and a backup data node; the apparatus includes: An anomaly detection unit is used to execute test instructions corresponding to database operation functions based on the triggering of an anomaly recovery instruction, and obtain corresponding execution results; wherein, the execution results indicate whether the test instructions were successfully executed; and, based on the execution results and the role status information of this node, to determine whether this node is in an abnormal state; wherein, the role status information is used to indicate whether this node is a primary management node or a backup management node; An abnormal scenario detection unit is used to determine the target abnormal scenario that the node meets based on the abnormal scenario conditions corresponding to each preset abnormal scenario if it is determined that the node is in an abnormal state. The preset abnormal scenarios include: at least one node's role status is inconsistent with its own database operation function, and / or, there are multiple main management nodes in the at least two management nodes. An anomaly recovery unit is used to obtain an anomaly recovery strategy configured for the target anomaly scenario and to perform anomaly recovery processing based on the anomaly recovery strategy to restore the database operation function; wherein, the step of performing anomaly recovery processing based on the anomaly recovery strategy to restore the database operation function includes: in an anomaly scenario where there are multiple master management nodes among the at least two management nodes, determining one of the multiple master management nodes as the master management node based on the data and write-ahead logs, and configuring the remaining master management nodes as backup management nodes.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images