Method for detecting down server, high availability recovery method and device of database

Abstract

The application provides a method for detecting a down server, a high-availability recovery method and device for a database. The method comprises: obtaining server state information of a plurality of servers of a server cluster, wherein the server state information is collected by a detection client deployed on the servers and uploaded to a detection server; when the server state information of a target server in the server cluster meets a preset abnormal condition, it is determined that the target server is suspected to be down; and a port state of a specified port of the target server suspected to be down is detected by any one or more detection server nodes of the detection server, and whether the target server is down is determined according to a result of the port state detection. By using the method, a down server in the server cluster can be quickly found.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of database, and particularly relates to a method for detecting a down server, a high-availability recovery method and device of a database. BACKGROUND

[0002] This section is intended to provide background or context to the embodiments of the application recited in the claims. The description herein does not constitute admission that the subject matter disclosed herein is prior art to the present application.

[0003] With the development of modern computer science technology, databases have gradually become the foundation and core of computer information systems and are widely used in the telecommunications, finance, government and other industries. The high availability of database applications has attracted more and more attention. In the prior art, there are many types of high-availability modes for relational databases, and the common ones include one master and multiple slaves, master-slave, MGR cluster, cold backup and shared storage, etc. The above high-availability architectures have their advantages and disadvantages and are suitable for different business scenarios, but no matter which high-availability architecture, the availability will be reduced during the period of server downtime.

[0004] The master-slave architecture is one of the most widely used high-availability architectures for databases. This architecture consists of two database instances, a master database (referred to as a master) and a standby database (referred to as a slave). Under normal circumstances, the master database provides services to the outside world. If there is a change to the data on the master database, the change will be written to the transaction log before being written to the database storage. The logs generated by the master database server are continuously sent to the slave database server (referred to as the standby) through a network connection. After receiving the logs, the slave database server replays them to synchronize its data with the master database server. When the master database server fails and cannot provide services to the outside world, it needs to be switched to the slave database to continue providing services, thereby achieving high availability. This database architecture is widely accepted and used in production environments due to its simple and flexible configuration, high resource utilization, and high success rate of master-slave switching.

[0005] After the failure of the master server in the master-slave architecture, the original slave database is used to provide services to the outside world. Until the master server is restored, the database is a single point of service to the outside world, and this process may last for several hours or even longer. During this period, the high availability is completely lost, which poses a great risk.

[0006] In a large-scale data center, server downtime or hardware failure occurs every day. After the server fails, it is necessary to quickly identify the down server and quickly restore the original master-slave architecture of the database to reduce the time of single-point operation of the database. SUMMARY

[0007] Aiming at the problems in the prior art, a down server detection method, a database high-availability recovery method, device and computer readable storage medium are provided, which can solve the above problems.

[0008] The application provides the following solutions.

[0009] In a first aspect, a down server detection method is provided, which is applied to a detection server, the detection server includes a plurality of cluster-type detection server nodes, the detection server is connected to a plurality of detection clients, the plurality of detection clients are respectively deployed on a plurality of servers in a server cluster, and the method includes: acquiring server state information of the plurality of servers in the server cluster, the server state information is collected by the detection client deployed on the server and uploaded to the detection server; when the server state information corresponding to a target server in the server cluster meets a preset abnormal condition, it is judged that the target server is suspected to be down; any one or more detection server nodes of the detection server perform port state detection on a specified port of the target server suspected to be down, and whether the target server is down is determined according to the result of the port state detection.

[0010] In an embodiment, the judging that the target server is suspected to be down further includes: when it is detected that the server state information corresponding to the target server meets the preset abnormal condition, an abnormal decision request is initiated to a plurality of detection server nodes in the detection server, the abnormal decision request is used to judge whether the server state of the target server is abnormal; when more than a preset proportion of the detection server nodes in the detection server judge that the server state of the target server is abnormal, it is judged that the target server is suspected to be down.

[0011] In an embodiment, the port state detection further includes: detecting the port state of the specified port on the target server through an ICMP protocol.

[0012] In an embodiment, the specified port includes an SSH port and / or a docker port.

[0013] In an embodiment, the preset abnormal condition includes: the detection server node in the detection server receives server abnormal information of the target server; and / or the detection server does not receive a heartbeat uploaded by the detection client deployed on the target server within a heartbeat period exceeding a preset number of times.

[0014] In an embodiment, the port state detection on the specified port of the target server suspected to be down further comprises: continuously detecting the specified port of the target server suspected to be down at a set frequency; determining that the target server is down when the port detection fails continuously for more than a set number of times; and determining that the target server is normal when the port detection fails continuously for less than a set number of times.

[0015] In an embodiment, when the port detection fails continuously for more than a set number of times, the method further comprises: querying the down events and / or recovery events corresponding to the target server within a specified time range; generating a down event corresponding to the target server when there is no down event of the target server, and / or when the number of down events and recovery events corresponding to the target server is consistent.

[0016] In an embodiment, when the port detection fails continuously for less than a set number of times, the method further comprises: querying the down events and / or recovery events corresponding to the target server within a specified time range; determining that the target server corresponding to the target server has recovered from down to normal when the number of down events and recovery events corresponding to the target server is inconsistent.

[0017] In a second aspect, a detection server for a down server is provided, which is configured to perform the method of any one of claims 1-8. The detection server comprises a plurality of detection server nodes, and is connected to a plurality of detection clients respectively deployed on a plurality of servers in a server cluster. The detection server is configured to: acquire server state information of the plurality of servers in the server cluster, the server state information being collected by the detection clients deployed on the servers and uploaded to the detection server; determine that a target server in the server cluster is suspected to be down when the server state information corresponding to the target server satisfies a preset abnormal condition; perform port state detection on a specified port of the target server suspected to be down by any one of the detection server nodes of the detection server, and determine whether the target server is down according to a result of the port state detection.

[0018] In a third aspect, a high-availability recovery method of a database is provided. The method comprises: determining a down server, wherein the down server is deployed with a first database container; searching for a migration server meeting a reconstruction condition in a server cluster where the down server is located; creating a second server container on the migration server, wherein the second server container has the same container configuration as the first database container; obtaining a database configuration file of the first database container, parsing to obtain database configuration information, persisting the database configuration information into the second server container, and performing database initialization in the first server container; obtaining a backup image of the first database container, and performing database restoration in the second database container according to the backup image; and building a replication relationship from the second database container to a current primary container to synchronize data, wherein the current primary container has a primary-backup relationship with the first database container before the down server is down.

[0019] In an embodiment, the method comprises: determining the down server from the server cluster by using the method of the first aspect.

[0020] In an embodiment, determining the down server where the down event occurs comprises: periodically querying a database management platform to determine a server range; and querying server down events in the server range to determine the down server.

[0021] In an embodiment, after determining the down server, the method further comprises: querying a container type of the first database container deployed on the down server in the database management platform; and determining whether to allow reconstruction of the first database container according to the container type.

[0022] In an embodiment, searching for the migration server meeting the reconstruction condition in the server cluster where the down server is located further comprises: determining the migration server meeting the reconstruction condition from the server cluster according to the dynamic index evaluation factor and / or the static index evaluation factor of each server, wherein the static index evaluation factor comprises one or more of: a business dimension index, a software dimension index, and a resource allocation rate; and the dynamic index evaluation factor comprises one or more of: a CPU usage rate, a CPU fluctuation rate, a CPU maximum value within a single day, a memory usage rate, a disk usage rate, a disk input / output value per second, a disk maximum delay, and a network card traffic.

[0023] In an embodiment, the method further comprises: periodically obtaining the static evaluation factor of each server in the server cluster; and when detecting that a server down occurs in the server cluster, obtaining the dynamic index evaluation factor of each server in the server cluster in real time.

[0024] In an embodiment, further comprising: determining the migration server satisfying the reestablishment condition from the server cluster according to a weighted sum result of one or more static index evaluation factors and one or more dynamic index evaluation factors.

[0025] In an embodiment, the database configuration file of the first database container is acquired, and database configuration information is parsed, and the method further comprises: converting parameters in the queried database configuration file into a plurality of key-value pair data; filtering the plurality of key-value pair data according to a database full parameter template to obtain the database configuration information; and storing the key-value pair data as the database configuration information in a cache database (REDIS); wherein the filtering the plurality of key-value pair data according to the database full parameter template comprises: when the database full parameter template does not contain a key in any key-value pair data, discarding the any key-value pair.

[0026] In an embodiment, the container configuration comprises one or more of an image name, a container name, CPU, memory, a file system, and an IP address of the container.

[0027] In a fourth aspect, a high-availability recovery apparatus of a database is provided, which is configured to perform the method of the third aspect, and the apparatus comprises: a down server determination module configured to determine a down server on which a first database container is deployed; a migration server determination module configured to find a migration server satisfying a reestablishment condition in a server cluster in which the target server is located; a container creation module configured to create a second server container on the migration server, the second server container having a same container configuration as the first database container; a database initialization module configured to acquire a database configuration file of the first database container, parse database configuration information, persist the database configuration information into the second server container, and perform database initialization in the first server container; a database restoration module configured to acquire a backup image of the first database container, and perform database restoration in the second database container according to the backup image; and a database synchronization module configured to build a replication relationship of the second database container to a current primary database container to synchronize data, wherein the current primary database container and the first database container have a primary-backup relationship before the target server is down.

[0028] In a fifth aspect, a down server detection apparatus is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the first aspect.

[0029] In a sixth aspect, a high-availability recovery apparatus of a database is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the third aspect.

[0030] In a seventh aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores a program, and when the program is executed by a multi-core processor, the multi-core processor performs the method of the first aspect or the method of the second aspect.

[0031] One of the advantages of the above-mentioned embodiments is that the above-mentioned method for detecting a down server can improve the efficiency of detecting a down server in a large-scale server cluster and reduce the false positive rate. It can be understood that finding a down server is the basis for high-availability recovery, and therefore the present embodiment can improve the efficiency of high-availability recovery. Through the above-mentioned method for high-availability recovery of a database, the original high-availability architecture of the database can be recovered without waiting for the server to recover from failure, thereby improving the overall high-availability capability of the database service.

[0032] Other advantages of the present application will be described in more detail in conjunction with the following description and drawings.

[0033] It should be understood that the above description is only a summary of the technical solutions of the present application, so that the technical means of the present application can be more clearly understood, and the above and other purposes, features and advantages of the present application can be implemented according to the content of the description. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0034] The advantages and benefits described herein, as well as other advantages and benefits, will be apparent to those of ordinary skill in the art upon reading the following detailed description of exemplary embodiments. The drawings are for purposes of illustrating exemplary embodiments and are not considered to be limiting of the present application. Moreover, the same reference numerals are used throughout the various drawings to represent similar components. In the drawings:

[0035] Figure 1 A system architecture diagram of a down server detection system according to an embodiment of the present application;

[0036] Figure 2 A flowchart of a down server detection method according to an embodiment of the present application;

[0037] Figure 3 A flowchart of a down server detection method according to another embodiment of the present application;

[0038] Figure 4A flowchart of a high-availability recovery method of a database according to an embodiment of the present application;

[0039] Figure 5 A structural diagram of a master-slave relationship of a database according to an embodiment of the present application;

[0040] Figure 6 A structural diagram of a high-availability recovery device of a database according to an embodiment of the present application;

[0041] Figure 7 A structural diagram of a down server detection device of a database according to an embodiment of the present application;

[0042] Figure 8 A structural diagram of a high-availability recovery device of a database according to an embodiment of the present application.

[0043] In the drawings, identical or corresponding reference signs indicate identical or corresponding parts. DETAILED DESCRIPTION

[0044] Exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be embodied in various forms without being limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0045] In the description of embodiments of the present application, it should be understood that terms such as "include" or "have" are intended to indicate that there is existence of the features, numbers, steps, actions, parts, or combinations thereof described in the specification, and do not exclude the possibility that one or more other features, numbers, steps, actions, parts, or combinations thereof exist.

[0046] Unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" herein is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can mean that A exists alone, A and B exist together, and B exists alone.

[0047] The terms "first", "second", and the like are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second", and the like can explicitly or implicitly include one or more of the features. In the description of embodiments of the present application, unless otherwise specified, the meaning of "multiple" is two or more.

[0048] All the codes in the present application are exemplary, and those skilled in the art can think of various modifications according to the programming language used, specific requirements and personal habits, etc. without departing from the idea of the present application.

[0049] In addition, it should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] Figure 1 A structural schematic diagram of a detection system of a down server according to an embodiment of the present application is shown, which includes a detection server 20 and a plurality of detection clients (31-36), wherein the plurality of detection clients are respectively deployed on a plurality of servers (11-16) in a server cluster 10, and the detection component is used to quickly detect a server that may be down from the plurality of servers in the server cluster 10.

[0051] Figure 2 A flowchart of a detection method of a down server according to an embodiment of the present application is shown, which is used to quickly find a down server from a large-scale server cluster. In the flowchart, from the perspective of equipment, the execution subject can be one or more electronic devices; from the perspective of program, the execution subject can be a program loaded on the electronic devices correspondingly. In the embodiment, the execution subject of the method can be Figure 1 the detection server 20 in the shown embodiment.

[0052] As shown in Figure 2 The method provided by the embodiment can include the following steps:

[0053] S202, obtain server state information of a plurality of servers of a server cluster.

[0054] Referring to Figure 1 For example, the server state information of the server (11-16) is periodically collected by the plurality of detection clients (31-36) and uploaded to the detection server 20. The server state information can be, for example, the state of the server, the memory usage, the heartbeat version number and the like.

[0055] S204, when the server state information corresponding to a target server in the server cluster meets a preset abnormal condition, it is judged that the target server is suspected to be down.

[0056] Referring to Figure 1 The S204 can be performed inside the detection server 20. Specifically, the server state information of each server can be queried in a loop. For any server, if the server state information of the server can be queried to be in a normal state, the subsequent detection does not need to be triggered, so as to improve the efficiency.

[0057] In an embodiment, the preset abnormal condition includes: any one of the probe server nodes in the probe server receives server abnormal information of the target server; and / or, each of the probe server nodes in the probe server does not receive the heartbeat uploaded by the probe client deployed on the target server within a preset number of heartbeat cycles. It can be understood that both of the above conditions can indicate that the target server may be down to a certain extent.

[0058] S206, any one or more probe server nodes in the probe server perform port state detection on the specified port of the target server suspected to be down, and determine whether the target server is down according to the result of the port state detection.

[0059] Reference Figure 1 For example, if the probe server determines that the server 13 is suspected to be down through S204, the probe server can perform port detection on the server 13 every ten seconds for a total of six times. Then, the result of the port detection is used to determine whether the server 13 is a down server.

[0060] In an embodiment, the port state of the specified port on the target server can be detected through the ICMP protocol. Optionally, the specified port can include an SSH port and / or a docker port.

[0061] It can be understood that the probe client itself is abnormal or the network fluctuation between the probe server and the probe client can also cause a normal server to be mistakenly determined as a server suspected to be down. In this case, the embodiment triggers the probe server to detect the listening state of multiple ports on the server suspected to be down through the ICMP protocol when the probe server determines that a certain server is suspected to be down. For example, if the detected ports are not in the listening state, it is determined that the server is down. On the contrary, if the detected ports are normally in the listening state, it is determined that the server is not down.

[0062] Figure 3 is a flowchart of a method for detecting a down server according to another exemplary embodiment of the present application. In this embodiment, the process of S204 is further described in detail based on the embodiment shown in Figure 2

[0063] As shown in Figure 3 The method provided by the embodiment can include the following steps:

[0064] S2041, when it is detected that the server state information corresponding to the target server satisfies the preset abnormal condition, an abnormality determination request is initiated to multiple probe server nodes in the probe server.

[0065] ​The abnormality judgment request is used to judge whether the server state of the target server is abnormal.

[0066] S2042, when more than a preset proportion of the probe server nodes in the probe server judge that the server state of the target server is abnormal, it is judged that the target server is suspected to be down.

[0067] Specifically, since the probe server includes a plurality of cluster-type probe server nodes, the server state information collected by the probe client is randomly or regularly distributed to one of the probe server nodes of the probe server, and whether the preset abnormal condition is met is judged by the probe server node. However, since the probe server node itself can also fail, the abnormal condition detected by it can be caused by its own failure. In view of this, an abnormality judgment request can be initiated to a plurality of other probe server nodes in the probe server, so that the other probe server nodes judge whether the target server is normal according to the server state information of the target server. For example, an odd number of probe server nodes can be deployed in the probe server, and all the probe server nodes are caused to judge the server state information of the target server, and when more than half of the probe server nodes judge that the server is abnormal, the target server is identified as suspected to be down.

[0068] By adopting the scheme of the embodiment, the performance problem of large-scale server cluster detection, false positives caused by single probe server node abnormality and probe client itself abnormality can be solved. Through long-term practice in a production environment, in a server cluster with a scale of more than 2000 servers, the state of the server can be updated every second, the false positive rate of the server down is less than one thousandth, and the CPU, memory and network resource consumption of the probe server and the client can be ignored.

[0069] Figure 3 is a flowchart of a server down detection method according to another exemplary embodiment of the present application. The embodiment is based on the embodiment shown in Figure 2 The process of S206 is further described in detail.

[0070] In an embodiment, the port state detection on the specified port of the target server suspected to be down in S206 further includes: continuously detecting the specified port of the target server suspected to be down at a set frequency; when the port detection fails continuously more than a set number of times, it is judged that the target server is down; on the contrary, when the port state detection fails continuously less than a set number of times, it is judged that the target server is normal.

[0071] For example, the specified port on the target server is probed every ten seconds, a total of six times. After the probe is completed, if there are no consecutive 4 times of probe failure, it is considered that the target server is currently in a normal state. Similarly, if there are consecutive 4 times and more of probe failure, it is considered that the target server is currently in a down state.

[0072] It can be understood that the target server currently in a down state can also have two cases, 1, the target server has been in a down state, 2, the target server has changed from normal to down state.

[0073] In view of the above case, in an embodiment, when the port probe fails consecutively more than a set number of times, the target server corresponding down event and / or recovery event in a specified time range is further queried; in the case that the target server does not exist a down event, and / or, the number of the target server corresponding down event and recovery event is consistent, it can be seen that the target server has changed from normal to down state, at this time, the down event corresponding to the target server needs to be generated to update the state of the target server. On the contrary, in the case that the number of the target server corresponding down event and recovery event is inconsistent, it can be seen that the target server has been in a down state.

[0074] Similarly, the target server currently in a normal state can still have two cases, 1, the target server has been in a normal state, 2, the target server has recovered from down to normal state.

[0075] In view of the above case, in an embodiment, when the port probe fails consecutively does not exceed a set number of times, the target server identified down event and / or recovery event in a specified time range can be further queried. In the case that the number of the target server corresponding down event and recovery event is inconsistent, it is determined that the target server has recovered from down to normal, at this time, the recovery event corresponding to the target server is generated to update the state of the target server.

[0076] Based on the same technical concept, the embodiment of the present application also provides a down server detection device for executing the down server detection method provided by any of the above embodiments.

[0077] Reference Figure 1 The detection device includes a detection service end, which includes a plurality of detection service end nodes, the detection service end is connected to a plurality of detection clients, and the plurality of detection clients are respectively deployed on a plurality of servers in a server cluster.

[0078] The probe server is configured to: acquire server state information of a plurality of servers of a server cluster, the server state information being collected by a probe client deployed on the server and uploaded to the probe server; when the server state information of a target server in the server cluster meets a preset abnormal condition, judging that the target server is suspected to be down; performing port state detection on a specified port of the target server suspected to be down by any one probe server node of the probe server, and determining whether the target server is down according to a result of the port state detection.

[0079] In this way, according to the scheme of the embodiment of the application, the efficiency of detecting the down server can be improved, and the false positive rate can be reduced.

[0080] It should be noted that the device in the embodiment of the application can implement each process of the embodiments of the foregoing method, and achieve the same effects and functions, which will not be described here.

[0081] The embodiment of the application further provides a high-availability recovery method of a database, which is used for rebuilding a server database after detecting that a server in a server cluster is down.

[0082] Server down refers to that the server is out of service due to unexpected failure, and cannot log in, execute commands or provide services. There are many reasons for server down, such as hardware failure, insufficient system resources, system vulnerability bug and the like, which can cause server down. In the server down scenario, the database deployed on the down server needs to be recovered as soon as possible, and the main factors affecting the time consumption of the high-availability architecture of the database recovery include the time for the server to recover to normal, the time for the operation and maintenance personnel to handle, the time for the database backup and restoration, and other time. The server generally needs to be recovered to normal by sending engineers to the site to replace or repair hardware to exclude failure, and according to different service levels, the time consumption is generally more than 2 hours, and there are many uncontrollable factors, and even more than 24 hours, and the average is about 4 hours. After the server is recovered to normal, the database operation and maintenance personnel need to intervene to recover the database instance, check the database state and perform data synchronization during the failure, and the average time consumption is about 1 hour. If the database needs to be backed up and restored during the recovery process, the time for the backup and restoration is also a long time-consuming process. Taking 200G data as an example, the backup and restoration and data transmission are estimated to consume about 1 hour, and if there are multiple databases on a server, the backup and restoration of multiple databases are needed, and the time consumption will also increase.

[0083] Figure 4A flowchart of a high-availability recovery method of a database according to an embodiment of the present application is shown, which is used to solve the above problems, i.e., to quickly rebuild the database on the down server to other servers. In the flowchart, the execution subject can be one or more electronic devices from the perspective of the device, and the execution subject can be the programs loaded on the electronic devices from the perspective of the program.

[0084] As shown in Figure 4 The method provided by the embodiment can include the following steps:

[0085] S402, determine a down server, the down server being deployed with a first database container;

[0086] The down server can be determined from the server cluster by using the above-mentioned detection method of the down server. Of course, other solutions can also be used.

[0087] Specifically, first, the server range can be queried on the database management platform every detection period (the period can be defined, and the default is once every minute). It can be understood that, due to the existence of server warehousing, warehousing, and deactivation, the available database server range is also dynamically changing. Then, the server down event can be queried. For example, in the above-mentioned detection method of the down server, if the server down is detected, the down event will be written. Then, according to the information of the queried down event, the above-mentioned server range is matched, and if a certain server can be matched, it means that there is a down server in the server range.

[0088] Optionally, in order to be compatible with the existence of some database containers on the server that do not need to be rebuilt, the container type of the first database container deployed on the down server can be further queried in the database management platform after S402; whether the first database container is allowed to be rebuilt is determined according to the container type.

[0089] The container type refers to the classification according to the software type deployed in the container. Since the processing methods of different types of containers may be different, some database containers do not need to be rebuilt, and therefore, if it is found that the container type of the first database container deployed on the down server is the above-mentioned container type that does not need to be rebuilt, the rebuilding process is stopped. Otherwise, the rebuilding is continued.

[0090] S404, find a migration server that meets the rebuilding condition in the server cluster where the down server is located.

[0091] It can be understood that many factors need to be considered when selecting the migration server, including server resource allocation rate (CPU, memory, and disk), resource usage rate (response time, throughput), and available state (hardware state, running state).

[0092] In an embodiment, to determine the most suitable migration server, a migration server satisfying the reconstruction condition can be determined from the server cluster according to the dynamic index evaluation factor and / or the static index evaluation factor of each server.

[0093] In actual application, a migration server satisfying the reconstruction condition can be determined from the server cluster according to the weighted sum result of one or more static index evaluation factors and one or more dynamic index evaluation factors.

[0094] The static index evaluation factor includes one or more of the following: a business dimension index, a software dimension index, a running module dimension index, a resource allocation rate, etc. Among them, the business dimension index mainly refers to the importance of the project undertaken by each server, the dependence on high availability performance. In actual operation, different proportions can be set according to the importance of the evaluation factor, such as for a database server that is more sensitive to resource allocation rate, the relevant evaluation factor proportion can be set to 40%, so the static index value is set to = (business dimension index * 100 * 0.3) + (software dimension index * 100 * 0.1) + (running module dimension * 100 * 0.2) + (resource allocation rate * 100 * 0.4).

[0095] The dynamic index evaluation factor includes one or more of the following: CPU (CPU usage, CPU fluctuation rate, CPU maximum value within a single day), memory usage, disk (disk usage, disk IOPS (Input / Output Operations Per Second) value and disk maximum delay), network card traffic (maximum inflow value, maximum outflow value) and other indicators. In actual operation, different proportions can be set according to the importance of the evaluation factor, such as for a database server that is more sensitive to disk, the disk-related evaluation factor proportion can be set to 60%, so the dynamic index value is set to = [CPU usage * 0.6 + CPU fluctuation rate * 0.3 + CPU maximum value within a single day * 0.1) * 100 * 0.3] + (memory usage * 1 * 100 * 0.1) + [(disk usage * 0.7 + disk IOPS value / 200 * 0.2 + disk maximum delay / 100 * 0.1) * 100 * 0.6].

[0096] In a large-scale server cluster, if all indicators trigger acquisition after server downtime, and whether the server is the target server for this migration is determined according to the acquired information, it will consume a lot of time. Therefore, in this embodiment, the static index evaluation factor of each server in the server cluster can be acquired periodically. And when detecting that a server in the server cluster has gone down, the dynamic index evaluation factor of each server in the server cluster is acquired in real time.

[0097] The dynamic index evaluation factor obtains the latest data after the downtime and performs calculation, the static index evaluation factor is relatively stable, and the change is extremely small, so the calculation can be performed once per hour. Finally, according to the dynamic index and the static index, the application will give a comprehensive score to each server in the cluster. After the server downtime, the available server is directly selected according to the score, and the amount of information obtained and the amount of calculation after the downtime are minimized to quickly select the target server for the database reconstruction.

[0098] S406、In the migration server, a second server container is created, and the second server container has the same container configuration as the first database container;

[0099] In an embodiment, the container configuration includes one or more of an image name, a container name, CPU, memory, a file system, and an IP address of the container.

[0100] S408, a database configuration file of the first database container is obtained, database configuration information is parsed and obtained, the database configuration information is persisted in the second server container, and database initialization is performed in the first server container;

[0101] Since the database configuration file is a normal text file and does not have a strong format requirement, the configuration file may contain garbled characters due to improper editing or other reasons, so that the configuration file is actually unusable and difficult to find. Using such a configuration file, the database cannot be normally started, which brings many problems to the large-scale database operation and maintenance. In order to realize the rapid database reconstruction on other machines, it is crucial to obtain the latest and correct configuration of the database. If the traditional method of directly synchronizing the configuration file to the configuration center is used, the database may not be started after being reconstructed on other servers due to garbled characters in the source library configuration file and other reasons, and the operation and maintenance personnel need to be involved, which cannot realize full automation and cannot quickly recover.

[0102] To solve the above problems, in an embodiment, S408 can specifically include:

[0103] The parameters in the queried database configuration file are converted into a plurality of key-value pair (KEY / VALUE) data;

[0104] The plurality of key-value pair data is filtered according to a database full parameter template to obtain database configuration information;

[0105] The key-value pair data is stored in a cache database (REDIS) as the database configuration information.

[0106] The plurality of key-value pair data is filtered according to a database full parameter template to obtain database configuration information;

[0107] In the embodiment, the parameters in the database configuration file are converted into key-value pair (KEY / VALUE) data and stored in the cache database (REDIS), a full-amount parameter template of the database is loaded, the name of the database configuration parameter is obtained, and if the key (KEY) is not configured in the parameter template, it is directly discarded. In this way, the problem of garbled code or invalid parameters in the configuration file can be effectively filtered.

[0108] In the reconstruction, the key-value pair (KEY / VALUE) data is obtained through the cache database (REDIS), the database configuration file is generated, and is persisted to the new container. The cache database (REDIS) has good query and write performance, and can realize simultaneous query and update of large-scale database parameters.

[0109] S410, a backup image of the first database container is obtained, and database restoration is performed in the second database container according to the backup image;

[0110] S412, a replication relationship of the second database container to the current master container is built to synchronize data, wherein the current master container and the first database container have a master-slave relationship before the server on which the first database container is located is down.

[0111] It can be understood that, with reference to Figure 5 In the high-availability architecture, the first database container on the down server will exist at least one server container having a master-slave relationship on another server. Before the down server is down, the first database container on the down server can be a master container or a backup container. In the high-availability architecture, after the down server is down, the server container on another server having a master-slave relationship with the down server remains a master container or is upgraded to a master container. The application utilizes the above S402-S410 to deploy the second database container in the migration server to replace the first database container, and build a replication relationship of the second database container to the current master container to synchronize data. At this time, the second database container serves as a backup container of the master container.

[0112] The master-slave replication data can be periodically checked for delay. Once the data of the new database is synchronized with the master container, the new database has the ability to provide services to the outside, that is, the original high-availability architecture of the database is restored.

[0113] The technical scheme provided by the embodiment can quickly restore the original high-availability architecture of the database in the case of database server downtime, can quickly restore the original primary-backup architecture in the case of downtime of one server in the primary-backup architecture database and not being restored, and solves the problem of single point of the database. The overall high availability of the database in the primary-backup architecture can be obviously improved, and the primary-backup architecture database is applicable to scenarios with higher requirements for high availability. The embodiment is also applicable to high-availability quick recovery of databases in a primary-secondary architecture, and the original high-availability architecture of the database can be restored without waiting for the server to recover from failure, thereby improving the overall high-availability capability of the database service.

[0114] Based on the same technical concept, the embodiment of the application further provides a high-availability recovery device of a database, which is used to execute the high-availability recovery method of the database provided by any of the above embodiments.

[0115] As shown in Figure 6 , the device comprises:

[0116] The down server determination module 602 is configured to determine a down server, and the down server is deployed with a first database container;

[0117] The migration server determination module 604 is configured to find a migration server meeting a reconstruction condition in a server cluster where the target server is located;

[0118] The container creation module 606 is configured to create a second server container on the migration server, and the second server container has the same container configuration as the first database container;

[0119] The database initialization module 608 is configured to obtain a database configuration file of the first database container, parse the database configuration information, persist the database configuration information into the second server container, and perform database initialization in the first server container;

[0120] The database restoration module 610 is configured to obtain a backup image of the first database container, and perform database restoration in the second database container according to the backup image;

[0121] The database synchronization module 612 is configured to build a replication relationship from the second database container to a current primary database container to synchronize data, wherein the current primary database container has a primary-backup relationship with the first database container before downtime of the target server.

[0122] It should be noted that the device in the embodiment of the application can implement each process of the embodiments of the foregoing method and achieve the same effects and functions, which will not be described here.

[0123] In the description of the specification, the description of the terms "some possible implementation", "some embodiment", "example", "specific example", or "some example" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the description of the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0124] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0125] Any process or method descriptions in flow charts or described elsewhere herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for performing specific logic functions or steps in the process, and that the various embodiments of the present application include additional implementations in which the order of steps can be changed, including use of an alternate order, simultaneous performance, or overlapping performance of steps, and that one or more steps can be performed or omitted, as appropriate, by a person skilled in the art.

[0126] Regarding the method flowchart of the embodiments of the present application, some operations are described as different steps executed in a certain order. Such flowcharts are illustrative rather than limiting. Some steps described herein can be grouped together and performed in a single operation, some steps can be divided into multiple sub-steps, and some steps can be performed in an order different from that shown herein. Each step shown in the flowchart can be implemented in any way by any circuit structure and / or tangible mechanism (for example, by software running on a computer device, hardware (for example, processor or chip implemented logic functions), etc., and / or any combination thereof) in any manner.

[0127] Figure 7 For the detection device of the down server according to an embodiment of the present application, for executing Figure 2The device shown in the method for detecting a down server, the device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the above embodiments.

[0128] Figure 8 For the high-availability recovery device of the database according to an embodiment of the application, for executing Figure 4 The device shown in the method for detecting a down server, the device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the above embodiments.

[0129] According to some embodiments of the application, a non-volatile computer storage medium of the method for high-availability recovery of a database is provided, and the computer executable instructions are stored on the non-volatile computer storage medium, and the computer executable instructions are set to execute the following when run by a processor: the method for detecting a down server described in the above embodiments, or the method for high-availability recovery of a database described in the above embodiments.

[0130] Each of the embodiments in the present application is described in a progressive manner, and the same and similar parts of each of the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the device, equipment and computer readable storage medium embodiments, since they are basically similar to the method embodiments, the description is simplified, and the relevant parts can be referred to the part of the method embodiment.

[0131] The device, equipment and computer readable storage medium provided by the embodiments of the present application are one-to-one corresponding to the method, and therefore, the device, equipment and computer readable storage medium also have the similar beneficial technical effects as the method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device, equipment and computer readable storage medium will not be described here.

[0132] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0133] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0134] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0135] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0136] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0137] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory. The memory is an example of computer-readable media.

[0138] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology for storing information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. In addition, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in that particular order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps.

[0139] Although the spirit and principles of the present application have been described with reference to several specific embodiments, it should be understood that the present application is not limited to the disclosed specific embodiments, and the division of aspects does not mean that the features in these aspects cannot be combined to benefit. This division is only for the convenience of expression. The present application is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

Claims

1. A method for detecting a down server, characterized by, The method is applied to a detection server, the detection server includes a plurality of cluster-type detection server nodes, the detection server is connected to a plurality of detection clients, the plurality of detection clients are respectively deployed on a plurality of servers in a server cluster, and the method comprises: Obtaining server state information of the plurality of servers in the server cluster, the server state information being collected by the detection client deployed on the server and uploaded to the detection server; When the server state information corresponding to a target server in the server cluster meets a preset abnormal condition, it is determined that the target server is suspected to be down; Any one or more detection server nodes of the detection server perform port state detection on a specified port of the target server suspected to be down, and determine whether the target server is down according to the result of the port state detection.

2. The method of claim 1, wherein, The determination that the target server is suspected to be down further comprises: When it is detected that the server state information corresponding to the target server meets the preset abnormal condition, an abnormal decision request is initiated to the plurality of detection server nodes in the detection server, and the abnormal decision request is used to determine whether the server state of the target server is abnormal; When more than a preset proportion of the detection server nodes in the detection server determine that the server state of the target server is abnormal, it is determined that the target server is suspected to be down.

3. The method of claim 1, wherein, The port state detection further comprises detecting the port state of the specified port on the target server through an ICMP protocol.

4. The method of claim 1, wherein, The specified port comprises an SSH port and / or a docker port.

5. The method of claim 1, wherein, The preset abnormal condition is that: Any one detection server node in the detection server receives server abnormal information of the target server; and / or, The detection server does not receive a heartbeat uploaded by the detection client deployed on the target server within more than a preset number of heartbeat cycles.

6. The method of claim 1, wherein, The port state detection on the specified port of the target server suspected to be down further comprises: Continuously detecting the specified port on the target server suspected to be down at a set frequency; When the port detection fails continuously more than a set number of times, it is determined that the target server is down; When the port state detection fails continuously less than a set number of times, it is determined that the target server is normal.

7. The method of claim 6, wherein, When the port detection fails continuously more than a set number of times, it further comprises: Querying a down event and / or a recovery event corresponding to the target server within a specified time range; In the case that there is no down event of the target server, and / or in the case that the number of down events and recovery events corresponding to the target server is consistent, a down event corresponding to the target server is generated.

8. The method of claim 6, wherein, When the port detection fails continuously less than a set number of times, it further comprises: Querying a down event and / or a recovery event corresponding to the target server within a specified time range; In a case where the number of the downtime events and the recovery events corresponding to the target server do not match, it is determined that the target server corresponding to the target server has recovered from downtime, and a recovery event corresponding to the target server is generated.

9. A device for detecting a downtime server, characterized in that, The apparatus is configured to perform the method of any one of claims 1-8, wherein the apparatus is a probe service end including a plurality of probe service end nodes, the probe service end is connected to a plurality of probe clients, and the plurality of probe clients are respectively deployed on a plurality of servers in a server cluster. The probe service end is configured to: Obtain server state information of the plurality of servers in the server cluster, the server state information being collected by the probe clients deployed on the servers and uploaded to the probe service end; When the server state information corresponding to a target server in the server cluster meets a preset abnormal condition, it is determined that the target server is suspected to be down; Perform port state detection on a specified port of the target server suspected to be down by any one of the probe service end nodes of the probe service end, and determine whether the target server is down according to a result of the port state detection.

10. A high-availability recovery method of a database, characterized by, The method comprises: Determining a down server according to the method for detecting a down server of any one of claims 1-8, the down server having a first database container deployed thereon; Finding a migration server meeting a reconstruction condition in a server cluster where the down server is located; Creating a second server container on the migration server, the second server container having a same container configuration as the first database container; Obtaining a database configuration file of the first database container, parsing to obtain database configuration information, persisting the database configuration information into the second server container, and performing database initialization in the first server container; Obtaining a backup image of the first database container, and performing database restoration in the second database container according to the backup image; Building a replication relationship of the second database container to a current primary database container to synchronize data, wherein the current primary database container and the first database container have a primary-backup relationship before the down server is down.

11. The method of claim 10, wherein, The method comprises: Determining the down server from a server cluster by using the method of any one of claims 1-8.

12. The method of claim 10, wherein, Determining a down server where a downtime event occurs, comprising: Periodically querying a database management platform to determine a server range; Querying server downtime events in the server range to determine the down server.

13. The method of claim 10, wherein, After determining the down server, further comprising: Querying a container type of the first database container deployed on the down server in the database management platform; Determining whether the first database container is allowed to be reconstructed according to the container type.

14. The method of claim 10, wherein, Finding a migration server meeting a reconstruction condition in a server cluster where the down server is located, further comprising: Determining the migration server meeting the reconstruction condition from the server cluster according to the dynamic index evaluation factor and / or the static index evaluation factor of each server; The static index evaluation factor includes one or more of the following: a business dimension index, a software dimension index, and a resource allocation rate. The dynamic index evaluation factor includes one or more of the following: a CPU usage rate, a CPU fluctuation rate, a CPU maximum value within a single day, a memory usage rate, a disk usage rate, a disk input / output value per second, a disk maximum delay, and a network card traffic.

15. The method of claim 14, wherein, Further comprising: Periodically acquiring the static evaluation factor of each server in the server cluster; When detecting that a server in the server cluster is down, acquiring the dynamic index evaluation factor of each server in the server cluster in real time.

16. The method of claim 14, wherein, Further comprising: According to the weighted sum result of one or more of the static index evaluation factors and one or more of the dynamic index evaluation factors, determining the migration server that meets the reconstruction condition from the server cluster.

17. The method of claim 10, wherein, Acquiring a database configuration file of the first database container, and parsing to obtain database configuration information, further comprising: Converting the parameters in the database configuration file obtained by querying into a plurality of key-value pair data; Filtering the plurality of key-value pair data according to a database full parameter template to obtain the database configuration information; Storing the key-value pair data as the database configuration information in a cache database REDIS. The filtering of the plurality of key-value pair data according to the database full parameter template comprises: when the database full parameter template does not contain a key in any key-value pair data, discarding the any key-value pair.

18. The method of claim 10, wherein, The container configuration includes one or more of an image name, a container name, a CPU, a memory, a file system, and an IP address of the container.

19. A high-availability recovery device for a database, characterized in that, The apparatus is configured to perform the method of any one of claims 10-18, and the apparatus comprises: A down server determination module configured to determine a down server according to the method of detecting a down server of any one of claims 1-8, the down server being deployed with a first database container; A migration server determination module configured to find a migration server that meets a reconstruction condition in a server cluster where the target server is located; A container creation module configured to create a second server container on the migration server, the second server container having the same container configuration as the first database container; A database initialization module configured to acquire a database configuration file of the first database container, parse to obtain database configuration information, persist the database configuration information into the second server container, and perform database initialization in the first server container; A database restoration module configured to acquire a backup image of the first database container, and perform database restoration in the second database container according to the backup image; A database synchronization module configured to build a replication relationship from the second database container to a current primary database container to synchronize data, wherein the current primary database container has a primary-backup relationship with the first database container before the target server is down.

20. An apparatus for high-availability recovery of a database, the apparatus comprising: Comprise: At least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8 or the method of any one of claims 10-18.

21. A computer readable storage medium storing a program, which, when executed by a multi-core processor, causes the multi-core processor to perform the method of any one of claims 1-8 or the method of any one of claims 10-18.

Images