Backup exception processing method, system and device, computer equipment and storage medium

Abstract

The invention relates to a backup exception processing method, system and device, computer equipment, a storage medium and a computer program product. The method comprises the steps that under the condition that a current backup node in a backup server detects that communication between the current backup node and a target backup node is interrupted, an arbitration request is initiated to an arbitration client group connected with the backup server, and the arbitration client group responds to the arbitration request and returns a node state obtaining command to each backup node included in the backup server, the current backup node obtains node state information based on the node state obtaining command and sends the node state information to the arbitration client side group, the arbitration client side group gathers the node state information sent by the backup nodes to obtain the node state information and return the node state information to the backup nodes, and the current backup node receives the node state gathering information and sends the node state gathering information to the arbitration client side group. And backup exception processing is carried out according to the node state summary information. By adopting the method, backup exception processing can be accurately carried out, and the reliability of data backup is improved.

Description

Technical Field

[0001] The present application relates to the technical field of fault handling, and in particular to a node backup exception handling method, system, device, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] When providing data protection services to users, data protection for key and core business systems is of vital importance. In some user data protection scenarios such as finance and manufacturing, insufficient or missed backups are not allowed. Failure to back up in time will result in loss of recoverable time points of the business system, and may also cause the database to stop because the storage space is full of logs. Therefore, users have put forward business continuity requirements for backup systems such as high availability and redundancy for data protection services. In order to achieve the above requirements, it is proposed to use two servers to form a mirror mode for high availability.

[0003] However, this arbitration method has low reliability, is prone to brain splits, and has the problem of low data backup reliability. Summary of the invention

[0004] Based on this, it is necessary to provide a backup exception handling method, system, device, computer equipment, computer-readable storage medium and computer program product for improving data backup reliability in response to the above technical problems.

[0005] In a first aspect, the present application provides a backup exception handling method, which is applied to any current backup node among a plurality of backup nodes included in a backup server, the backup server being connected to an arbitration client group, and comprising:

[0006] When the current backup node detects that the communication with the target backup node is interrupted, an arbitration request is initiated to the arbitration client group; the arbitration client group is used to respond to the arbitration request and return a node status acquisition command to each backup node included in the backup server; the target backup node is located in the multiple backup nodes included in the backup server;

[0007] Based on the node status acquisition command, the node status information of the current backup node is obtained, and the node status information of the current backup node is sent to the arbitration client group; the arbitration client group is used to summarize the node status information sent by each backup node, obtain the node status summary information and return it to each backup node included in the backup server;

[0008] Receive node status summary information and perform backup exception processing based on the node status summary information.

[0009] In one of the embodiments, when the current backup node is a master node among multiple backup nodes, backup exception processing is performed according to the node status summary information, including:

[0010] If the current backup node receives the node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node, the current backup node continues to serve as the master node;

[0011] If the current backup node receives the node status summary information, and the node status summary information indicates that the current backup node is a faulty backup node, the current backup node is downgraded to a backup node among the multiple backup nodes.

[0012] In one embodiment, the method further comprises:

[0013] When the current backup node does not receive the node status summary information, the current backup node continues to serve as the master node.

[0014] In one embodiment, when the current backup node is a backup node among multiple backup nodes, backup exception processing is performed according to the node status summary information, including:

[0015] If the current backup node receives the node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node, and the master node among the multiple backup nodes is a healthy backup node, the current backup node continues to be used as a backup node;

[0016] If the current backup node receives the node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node and the master node is a faulty backup node, the static configuration information of the current backup node is sent to the backup server; the backup server determines a new master node from multiple backup nodes based on the static configuration information of each backup node, generates upgrade indication information and returns it to the current backup node;

[0017] Based on the upgrade indication information, determine whether to upgrade the current backup node to a new primary node.

[0018] In a second aspect, the present application further provides a backup exception handling method, which is applied to an arbitration client group, the arbitration client group is connected to a backup server, and the backup server includes multiple backup nodes, including:

[0019] In response to the arbitration request initiated by the current backup node, a node status acquisition command is returned to each backup node included in the backup server; the arbitration request is generated and returned to the arbitration client group when the current backup node detects that the communication with the target backup node is interrupted; the current backup node is any backup node, and the target backup node is located among the multiple backup nodes included in the backup server;

[0020] Receive and summarize the node status information sent by each backup node to obtain node status summary information; the node status information is obtained by each backup node responding to the node status acquisition command;

[0021] The node status summary information is sent to each backup node; the node status summary information is used by the current backup node to perform backup exception processing based on the node status summary information.

[0022] In a third aspect, the present application further provides a backup exception handling system, the system comprising any current backup node among a plurality of backup nodes included in a backup server and an arbitration client group, the backup server being connected to the arbitration client group;

[0023] The current backup node is used to initiate an arbitration request to the arbitration client group when the current backup node detects that the communication with the target backup node is interrupted; the target backup node is located in the multiple backup nodes included in the backup server;

[0024] The arbitration client group is used to respond to the arbitration request and return a node status acquisition command to each backup node included in the backup server;

[0025] The current backup node is further used to receive a node status acquisition command, and obtain node status information of the current backup node based on the node status acquisition command, and send the node status information of the current backup node to the arbitration client group;

[0026] The arbitration client group is also used to aggregate the node status information sent by each backup node, obtain the node status summary information and return it to each backup node included in the backup server;

[0027] The current backup node is also used to receive node status summary information and perform backup exception processing according to the node status summary information.

[0028] In a fourth aspect, the present application further provides a backup exception handling device, which is applied to any current backup node among a plurality of backup nodes included in a backup server, the backup server being connected to an arbitration client group, and comprising:

[0029] The request initiation module is used to initiate an arbitration request to the arbitration client group when the current backup node detects that the communication with the target backup node is interrupted; the arbitration client group is used to respond to the arbitration request and return a node status acquisition command to each backup node included in the backup server; the target backup node is located in the multiple backup nodes included in the backup server;

[0030] An information acquisition module is used to acquire the node status information of the current backup node based on the node status acquisition command, and send the node status information of the current backup node to the arbitration client group; the arbitration client group is used to summarize the node status information sent by each backup node, obtain the node status summary information and return it to each backup node included in the backup server;

[0031] The exception handling module is used to receive node status summary information and perform backup exception handling according to the node status summary information.

[0032] In a fifth aspect, the present application further provides a backup exception handling device, which is applied to an arbitration client group, the arbitration client group is connected to a backup server, and the backup server includes multiple backup nodes, including:

[0033] The command issuing module is used to respond to the arbitration request initiated by the current backup node and return the node status acquisition command to each backup node included in the backup server; the arbitration request is generated and returned to the arbitration client group when the current backup node detects that the communication with the target backup node is interrupted; the current backup node is any backup node, and the target backup node is located in the multiple backup nodes included in the backup server;

[0034] The information aggregation module is used to receive and aggregate the node status information sent by each backup node to obtain node status aggregation information; the node status information is obtained by each backup node responding to the node status acquisition command;

[0035] The information sending module is used to send node status summary information to each backup node; the node status summary information is used by the current backup node to perform backup exception processing based on the node status summary information.

[0036] In a sixth aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0037] When the current backup node detects that the communication with the target backup node is interrupted, an arbitration request is initiated to the arbitration client group; the arbitration client group is used to respond to the arbitration request and return a node status acquisition command to each backup node included in the backup server; the target backup node is located in the multiple backup nodes included in the backup server;

[0038] Based on the node status acquisition command, the node status information of the current backup node is obtained, and the node status information of the current backup node is sent to the arbitration client group; the arbitration client group is used to summarize the node status information sent by each backup node, obtain the node status summary information and return it to each backup node included in the backup server;

[0039] Receive node status summary information and perform backup exception processing based on the node status summary information.

[0040] In a seventh aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0041] When the current backup node detects that the communication with the target backup node is interrupted, an arbitration request is initiated to the arbitration client group; the arbitration client group is used to respond to the arbitration request and return a node status acquisition command to each backup node included in the backup server; the target backup node is located in the multiple backup nodes included in the backup server;

[0042] Based on the node status acquisition command, the node status information of the current backup node is obtained, and the node status information of the current backup node is sent to the arbitration client group; the arbitration client group is used to summarize the node status information sent by each backup node, obtain the node status summary information and return it to each backup node included in the backup server;

[0043] Receive node status summary information and perform backup exception processing based on the node status summary information.

[0044] In an eighth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0045] When the current backup node detects that the communication with the target backup node is interrupted, an arbitration request is initiated to the arbitration client group; the arbitration client group is used to respond to the arbitration request and return a node status acquisition command to each backup node included in the backup server; the target backup node is located in the multiple backup nodes included in the backup server;

[0046] Based on the node status acquisition command, the node status information of the current backup node is obtained, and the node status information of the current backup node is sent to the arbitration client group; the arbitration client group is used to summarize the node status information sent by each backup node, obtain the node status summary information and return it to each backup node included in the backup server;

[0047] Receive node status summary information and perform backup exception processing based on the node status summary information.

[0048] The above-mentioned backup exception processing method, system, device, computer equipment, computer-readable storage medium and computer program product, when the current backup node in the backup server detects that the communication with the target backup node is interrupted, initiates an arbitration request to the arbitration client group connected to the backup server, the arbitration client group responds to the arbitration request, returns a node status acquisition command to each backup node included in the backup server, the current backup node obtains node status information based on the node status acquisition command, and sends the node status information to the arbitration client group, the arbitration client group summarizes the node status information sent by each backup node, obtains the node status information and returns it to the backup node, the current backup node receives the node status summary information, and performs backup exception processing according to the node status summary information. Each backup node in the backup server is used to realize data backup. The backup nodes can communicate with each other to ensure data reliability. In the case of communication interruption between backup nodes, an arbitration request is initiated to the arbitration client group to handle the backup exception, and the node status information of the backup node is sent under the instruction of the arbitration client group. The arbitration client group summarizes the node status information and returns the node status summary information to each backup node, so that each backup node knows the node status of the remaining backup nodes. The current backup node handles the backup exception according to the node status summary, which improves the stability and availability of the backup system, and thus improves the reliability of data backup. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0050] FIG1 is an application environment diagram of a backup exception handling method according to an embodiment;

[0051] FIG2 is a schematic diagram of a flow chart of a backup exception handling method in one embodiment;

[0052] FIG3 is a schematic diagram of a flow chart of a backup exception handling method in another embodiment;

[0053] FIG4 is a diagram of multiple network components in one embodiment;

[0054] FIG5 is a schematic diagram of a switching service process when a master node fails in one embodiment;

[0055] 6 is a schematic diagram of a switching service process when communication between a master node and a standby node fails in another embodiment;

[0056] FIG7 is a structural block diagram of a backup exception handling device in one embodiment;

[0057] FIG8 is a structural block diagram of a backup exception handling device in another embodiment;

[0058] FIG. 9 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] The backup exception handling method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the backup service end 102 communicates with the arbitration client group 104 through the network, and the backup service end includes multiple backup nodes, and the current backup node and the target backup node are any one of the multiple backup nodes. The data storage system can arbitrate the data that the client group 104 needs to process. The data storage system can be integrated on the arbitration client group 104, or it can be placed on the cloud or other network servers. When the current backup node detects that the communication with the target backup node is interrupted, it initiates an arbitration request to the arbitration client group 104, and the arbitration client group 104 responds to the arbitration request, and returns a node status acquisition command to each backup node included in the backup service end 102. The current backup node obtains node status information based on the node status acquisition command, and sends the node status information of the current backup node to the arbitration client group 104. The arbitration client group 104 summarizes the node status information sent by each backup node, obtains the node status summary information and returns it to each backup node included in the backup service end 102, and the current backup node receives the node status summary information, and performs backup exception handling according to the node status summary information. The backup server 102 and the arbitration client group 104 may be independent physical servers, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0061] In an exemplary embodiment, as shown in FIG2 , a backup exception handling method is provided, which is described by taking the method applied to the current backup node included in the backup server 102 in FIG1 as an example, and includes the following steps S201 to S203. Among them:

[0062] Step S201, when the current backup node detects that the communication with the target backup node is interrupted, it initiates an arbitration request to the arbitration client group; the arbitration client group is used to respond to the arbitration request and return a node status acquisition command to each backup node included in the backup server; the target backup node is located among the multiple backup nodes included in the backup server.

[0063] Among them, communication interruptions may include failure of the backup node, failure of the heartbeat network between the backup nodes, and failure of the network to which the backup node belongs. At this time, the states between the primary and backup nodes may conflict, so the backup request will initiate an arbitration request to the arbitration client group, waiting for relevant data to be returned to the backup node to determine the primary and backup states to which they should belong.

[0064] Optionally, when the current backup node in the backup server 102 detects that it has failed, the target backup node has failed, the heartbeat network between the current backup node and the target backup node has failed, and the network to which the backup node belongs has failed, an arbitration request is initiated to the arbitration client group 104, and the arbitration client group 104 responds to the arbitration request and returns a node status acquisition command to each backup node included in the backup server 102. When the current backup node does not detect that the communication with the target backup node is interrupted, each backup node included in the backup server is in the normal operation backup process, and the arbitration module is in a dormant state. Only when the relevant conditions are detected to be triggered, the arbitration module starts to work and initiates an arbitration request to the arbitration client group, so as to transmit arbitration related information to the arbitration client group regularly or in real time. By triggering arbitration by the above conditions, the operation cost of the backup server is reduced, and it is also ensured that the exception can be handled in time when the backup exception occurs.

[0065] Step S202, based on the node status acquisition command, obtain the node status information of the current backup node, and send the node status information of the current backup node to the arbitration client group; the arbitration client group is used to summarize the node status information sent by each backup node, obtain the node status summary information and return it to each backup node included in the backup server.

[0066] The node status information may include online status information of the backup node, backup service status of the backup node, original database status of the backup node, and metadata database replication status.

[0067] For example, the current backup node obtains the node status information of the current backup node based on the node status acquisition command, and sends the node status information of the current backup node to the arbitration client group 104. The arbitration client group 104 summarizes the node status information sent by each backup node, obtains the node status summary information and returns it to each backup node included in the backup server 102. The current backup node can obtain its own status information in a timely manner and send it to the arbitration client group, ensuring the real-time update of the status information of each backup node of the backup server 102, which is conducive to the arbitration client group to summarize accurate node status information and return it to each backup node, helping each backup node to know the status of other backup nodes, so as to facilitate timely abnormal processing.

[0068] Step S203: receiving node status summary information, and performing backup exception processing according to the node status summary information.

[0069] Optionally, when the current backup node is the master node or backup node among multiple backup nodes, the master-slave switching is performed according to the node status of each backup node represented by the node status summary information. By informing the current backup node of the summary information, the communication between the backup nodes that have been disconnected can be interoperable, making the operation of each backup node more accurate when performing exception processing, avoiding the situation of brain split caused by competing for the master node, thereby improving the reliability of data backup.

[0070] In the above backup exception handling method, when the current backup node in the backup server detects that the communication with the target backup node is interrupted, an arbitration request is initiated to the arbitration client group connected to the backup server, the arbitration client group responds to the arbitration request, and returns a node status acquisition command to each backup node included in the backup server, the current backup node obtains node status information based on the node status acquisition command, and sends the node status information to the arbitration client group, the arbitration client group summarizes the node status information sent by each backup node, obtains the node status information and returns it to the backup node, the current backup node receives the node status summary information, and performs backup exception handling according to the node status summary information. Each backup node in the backup server is used to realize data backup, and the backup nodes can communicate with each other to ensure the reliability of the data. When the communication between the backup nodes is interrupted, the backup exception is handled by initiating an arbitration request to the arbitration client group, and the node status information of the backup node is sent under the instruction of the arbitration client group, and the arbitration client group summarizes the node status information and returns the node status summary information to each backup node, so that each backup node knows the node status of the remaining backup nodes, and the current backup node performs backup exception handling according to the node status summary, which improves the stability and availability of the backup system, thereby improving the reliability of data backup.

[0071] In one embodiment, when the current backup node is a master node among multiple backup nodes, backup exception processing is performed according to the node status summary information, including:

[0072] If the current backup node receives the node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node, the current backup node continues to serve as the master node;

[0073] If the current backup node receives the node status summary information, and the node status summary information indicates that the current backup node is a faulty backup node, the current backup node is downgraded to a backup node among the multiple backup nodes.

[0074] Among them, the master node can be understood as a backup node responsible for managing backup tasks and coordinating other backup nodes to help improve the regularity of data backup; the backup node can be understood as an ordinary backup node other than the master node.

[0075] For example, in the case where the current backup node is the master node, if the current backup node receives node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node, the current backup node will continue to serve as the master node; in addition, if the current backup node receives node status summary information, and the node status summary information indicates that the current backup node is a faulty backup node, the current backup node will be downgraded to a backup node among multiple backup nodes. On the premise that the current backup node assumes the responsibilities of the master node, if the current backup node is still a healthy backup node, it will continue to assume this responsibility, and the remaining backup nodes cannot surpass the current backup node to become the new master node, unless the current backup node is a faulty backup node and downgrades itself to a backup node. The above-mentioned master-slave switching method has the following advantages:

[0076] 1. Continue to use the healthy current backup node as the master node to ensure the stability of the system under normal operation and avoid frequent master node switching causing service interruption or performance degradation.

[0077] 2. If the current backup node fails and receives the corresponding status summary information, it will be downgraded to a standby node, which can effectively prevent the failed node from continuing to assume the responsibilities of the master node, thereby protecting the reliability of the system and data security.

[0078] 3. Restrict other backup nodes from competing to become the master node when the current backup node is healthy, ensuring the orderliness of the system and the centralization of management, and avoiding the brain split problem caused by competing for the master node role that affects the business continuity of data backup.

[0079] In one of the embodiments, the method further includes: in the case that the current backup node does not receive the node status summary information, the current backup node continues to serve as the master node.

[0080] Optionally, when the current backup node serves as the master node among multiple backup nodes, if the current backup node does not receive the node status summary information, no operation is performed and the current backup node continues to serve as the master node. In the absence of new status information, maintaining the role of the current master node can reduce the management complexity of the system, avoid frequent adjustments to node roles due to status uncertainty, and ensure the stability of system operation. Choosing not to perform an operation when new status summary information is not received helps prevent unnecessary switching due to misjudgment or status information delays, and reduces potential service interruption risks.

[0081] In an exemplary embodiment, when the current backup node is a backup node among multiple backup nodes, backup exception processing is performed according to the node status summary information, including:

[0082] If the current backup node receives the node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node, and the master node among the multiple backup nodes is a healthy backup node, the current backup node continues to be used as a backup node;

[0083] If the current backup node receives the node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node and the master node is a faulty backup node, the static configuration information of the current backup node is sent to the backup server; the backup server determines a new master node from multiple backup nodes based on the static configuration information of each backup node, generates upgrade indication information and returns it to the current backup node;

[0084] Based on the upgrade indication information, determine whether to upgrade the current backup node to a new primary node.

[0085] The static configuration information may be understood as setting information that is fixed in a system or network node, including network configuration, node identification, hardware configuration information, and the like.

[0086] Optionally, in the case where the current backup node is a backup node among multiple backup nodes, if the current backup node receives node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node, and the primary node among the multiple backup nodes is a healthy backup node, the current backup node continues to be the backup node; in addition, if the current backup node receives node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node, and the primary node is a faulty backup node, the static configuration information of the current backup node is sent to the backup server 102, and the backup server 102 obtains the priority of each backup node based on the static configuration information of each backup node, determines the backup node corresponding to the highest priority as the new primary node, generates upgrade indication information and returns it to the current backup node. If the upgrade indication information indicates an upgrade, the current backup node is upgraded to a new primary node, and if the upgrade indication information does not indicate an upgrade, the current backup node continues to be the backup node.

[0087] The above method has the following advantages:

[0088] 1. When the master node is healthy, the current backup node will not be promoted to ensure system stability and consistency and avoid service interruptions caused by frequent status changes.

[0089] 2. When the master node fails and the current backup node is healthy, the static configuration information is sent to the backup server in a timely manner. The backup server selects a new master node based on the static configuration information, thereby improving the performance and efficiency of the backup system.

[0090] 3. By generating upgrade instruction information, a feedback mechanism is established, allowing each backup node to make judgments based on actual conditions when adjusting its role, thereby reducing the risk of erroneous operations.

[0091] In an exemplary embodiment, as shown in FIG3 , a backup exception handling method is provided, which is described by taking the method applied to the current backup node included in the arbitration client group 104 in FIG1 as an example, and includes the following steps S301 to S303. Among them:

[0092] Step S301, responding to the arbitration request initiated by the current backup node, returning a node status acquisition command to each backup node included in the backup server; the arbitration request is generated and returned to the arbitration client group when the current backup node detects that the communication with the target backup node is interrupted; the current backup node is any backup node, and the target backup node is located among the multiple backup nodes included in the backup server.

[0093] For example, when any current backup node in the backup server 102 detects that the communication with the target backup node is interrupted, an arbitration request is generated and returned to the arbitration client group 104. The arbitration client group responds to the arbitration request and returns a node status acquisition command to each backup node included in the backup server 102. Before receiving the arbitration request, the arbitration client group does not play an arbitration role, but only runs the planned backup, recovery and other tasks. There is no need to initiate a node status acquisition command to obtain the node status information of each backup node. The sleep-triggered operation mode reduces the operation cost of the arbitration client group.

[0094] Step S302, receiving and summarizing the node status information sent by each backup node to obtain node status summary information; the node status information is obtained by each backup node in response to a node status acquisition command.

[0095] Step S303, sending node status summary information to each backup node; the node status summary information is used by the current backup node to perform backup exception processing based on the node status summary information.

[0096] Optionally, each backup node responds to the node status acquisition command to obtain its own node status information, and sends the node status information to the arbitration client group 104. The arbitration client group 104 receives and summarizes the node status information sent by each backup node to obtain the node status summary information. Then, the node status summary information is sent to each backup node, and the current backup node performs backup exception processing based on the node status summary information. The above method achieves the following technical effects:

[0097] 1. Each backup node actively reports its own status, allowing the system to understand the health status of each node in real time, improving the timeliness and accuracy of monitoring.

[0098] 2. The arbitration client group summarizes the status of each node to form a centralized management perspective, optimizes node scheduling and resource allocation, and improves overall operational efficiency.

[0099] 3. By sharing information between backup nodes, each node can make decisions based on global status information, thereby enhancing the collaboration capabilities between nodes in the system.

[0100] In one of the embodiments, a backup exception handling system is provided, the system comprising any current backup node among a plurality of backup nodes included in a backup server and an arbitration client group, the backup server being connected to the arbitration client group;

[0101] The current backup node is used to initiate an arbitration request to the arbitration client group when the current backup node detects that the communication with the target backup node is interrupted; the target backup node is located among the multiple backup nodes included in the backup server; the arbitration client group is used to respond to the arbitration request and return a node status acquisition command to each backup node included in the backup server; the current backup node is also used to receive the node status acquisition command, and obtain the node status information of the current backup node based on the node status acquisition command, and send the node status information of the current backup node to the arbitration client group; the arbitration client group is also used to summarize the node status information sent by each backup node, obtain the node status summary information and return it to each backup node included in the backup server; the current backup node is also used to receive the node status summary information and perform backup exception processing according to the node status summary information.

[0102] By way of example, the backup exception handling system includes any current backup node among a plurality of backup nodes included in the backup server and an arbitration client group, the backup server being connected to the arbitration client group;

[0103] When the current backup node detects that the communication with the target backup node is interrupted, the current backup node initiates an arbitration request to the arbitration client group. The target backup node is located among the multiple backup nodes included in the backup server. The arbitration client group responds to the arbitration request and returns a node status acquisition command to each backup node included in the backup server. The current backup node receives the node status acquisition command, obtains the node status information of the current backup node based on the node status acquisition command, and sends the node status information of the current backup node to the arbitration client group. The arbitration client group summarizes the node status information sent by each backup node, obtains the node status summary information and returns it to each backup node included in the backup server. The current backup node receives the node status summary information.

[0104] In the case that the current backup node is the master node, if the current backup node receives node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node, or the current backup node does not receive the node status summary information, the current backup node continues to serve as the master node; in addition, if the current backup node receives node status summary information, and the node status summary information indicates that the current backup node is a faulty backup node, the current backup node is downgraded to a backup node among multiple backup nodes.

[0105] In the case where the current backup node is a backup node among multiple backup nodes, if the current backup node receives node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node, and the primary node among the multiple backup nodes is a healthy backup node, the current backup node continues to be the backup node; in addition, if the current backup node receives node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node, and the primary node is a faulty backup node, the static configuration information of the current backup node is sent to the backup server, and the backup server obtains the priority of each backup node based on the static configuration information of each backup node, determines the backup node corresponding to the highest priority as the new primary node, generates upgrade indication information and returns it to the current backup node. If the upgrade indication information indicates an upgrade, the current backup node is upgraded to a new primary node, and if the upgrade indication information does not indicate an upgrade, the current backup node continues to be the backup node.

[0106] The current backup node can promptly detect the interruption of communication with the target backup node and initiate an arbitration request. The arbitration client group responds to the arbitration request to summarize the node status information of each backup node and obtain the node status summary information, which provides a global perspective for the current backup node, realizes the reliable adjustment of the backup role of the current backup node, ensures the business continuity of data backup, and improves the reliability and availability of the entire backup system.

[0107] In an exemplary embodiment, a specific implementation of a backup exception handling method is provided:

[0108] 1. Component description:

[0109] As shown in Figure 4, the backup management architecture consists of four components: Other Agent (other clients), Agent Group (client group), Master (backup server), Storage (storage service), Master_A (master node), and Master_B (backup node).

[0110] The backup server contains the master node and the backup node. The master node manages all nodes of Other Agent, Agent Group, and Storage. Due to different business network areas, in actual application scenarios, the nodes of Agent Group are connected to different networks, but the backup server is configured with multiple network cards so that the client (Agent) can connect to it. The following is a detailed description:

[0111] Backup server: It consists of backup management services and metadata databases, and provides backup system user and user group management, backup, recovery, remote replication and other job configuration and scheduling management, backup console global configuration management, and recording of resource information, backup set information and storage media correspondence.

[0112] Storage service end: It consists of storage management services and storage media. It provides interfaces for writing and reading backup set data, automatically clears expired or invalid data according to configuration, and provides management functions for storage media such as disks, tapes, optical disks, and objects.

[0113] Backup client: A backup proxy host installed on a business host, used to obtain jobs, read backup data, and transfer data to the storage service end.

[0114] Client group (i.e. arbitration client group): Mark multiple clients as arbitration nodes to form a client group, which provides arbitration function for high availability of the backup server.

[0115] Heartbeat line: When the backup server becomes highly available, the heartbeat network connection between the two server nodes is used to detect the link of the node online status.

[0116] 2. Operation principle:

[0117] 1. Clients marked as arbitration are added to the client group and are always managed by the backup server. When a client goes offline, it is automatically removed from the group.

[0118] 2. The client information of the client group, including the client host name, address, connection port, backup proxy type and other information, is recorded in the metadata database of the backup server. And in the high-availability cluster nodes, the client group information recorded by the cluster nodes is kept consistent through the synchronization of the metadata database.

[0119] 3. The arbitration information includes: online status information of the backup server node, backup service status in the backup server node, metadata database status in the backup server node, and metadata database replication status.

[0120] 4. When no node failure occurs in the backup server cluster, the clients in the client group only run the planned backup, recovery and other tasks, and there is no need to transmit high-availability arbitration-related information to the backup server regularly or in real time.

[0121] 5. When the master node of the backup server cluster fails, the arbitration mechanism is triggered. The backup node of the cluster reads the client group list from the metadata database and sends arbitration request information to all clients in the group.

[0122] 6. The standby node that issues the arbitration request needs to wait for the arbitration information of all clients to be returned until the timeout, and then the information can be aggregated and merged.

[0123] 7. The standby node determines whether it can be switched based on the summary information:

[0124] a). All detection items of the master node are normal, all detection items of the standby node are normal, and the metadata database replication delay is lower than the threshold, so switching is not allowed.

[0125] b). The primary node cannot be accessed, all detection items of the standby node are normal, and the metadata database replication delay is lower than the threshold, so switching is allowed.

[0126] c). The primary node backup service fails, the standby node detection items are all normal, and the metadata database replication delay is lower than the threshold, so switching is allowed.

[0127] d). The primary node metadata database fails, the standby node detection items are all normal, and the metadata database replication delay is lower than the threshold, allowing switching.

[0128] e). The primary node detection items are abnormal or inaccessible, the backup service of the standby node fails, and the metadata database replication delay is lower than the threshold, so switching is not allowed.

[0129] f). The primary node detection items are abnormal or inaccessible, the standby node metadata database fails, and the metadata database replication delay is higher than the threshold, so switching is not allowed.

[0130] g). The primary node detection items are abnormal or inaccessible, the standby node detection items are all normal, and the metadata database replication delay is higher than the threshold, so switching is not allowed.

[0131] 3. Example description:

[0132] 1. As shown in Figure 5, the switching service process when Master_A fails:

[0133] (1) The master node Master_A fails, and Master_B finds that it cannot connect to Master_A, or receives a failure notification from Master_A.

[0134] (2) Master_B sends an arbitration request to all nodes in the Agent Group.

[0135] (3) All nodes in the Agent Group (C1\2\…\n) send a check command to check the status of Master_A and Master_B (host status, service status, metadata database status, etc.).

[0136] (4) Master_A and Master_B return status information respectively (Master_A is faulty, Master_B is healthy).

[0137] (5) C1\2\…\nMerge the status information of Master_A and Master_B.

[0138] (6) Agent Group sends status information to Master_A and Master_B.

[0139] (7) If Master_A can receive the status information, it will downgrade to Standby. If it cannot receive the status information, it will skip it.

[0140] (8) Master_B receives consistent responses from all nodes in the Agent Group, indicating that Master_A is faulty and Master_B is healthy. Master_B is upgraded to Active.

[0141] 2. As shown in Figure 6, the switching business process 1 when the communication between Master_A and Master_B fails:

[0142] (1) The heartbeat network fails. Master_B finds that it cannot connect to Master_A, and Master_A finds that it cannot connect to Master_B.

[0143] (2) Master_A and Master_B send arbitration requests to all nodes in the Agent Group.

[0144] (3) C1\2\…\nSend the check command to check the status of Master_A and Master_B (host status, service status, metadata database status, etc.).

[0145] (4) Master_A and Master_B return status information respectively (Master_A is healthy, Master_B is healthy).

[0146] (5) C1\2\…\nMerge the status information of Master_A and Master_B.

[0147] (6) Agent Group sends status information to Master_A and Master_B.

[0148] (7) Master_A receives information that both Master_A and Master_B are healthy and remains in the Active (master) state.

[0149] (8) Master_B receives information that both Master_A and Master_B are healthy and remains in the Standby state. No split-brain occurs.

[0150] 3. As shown in Figure 6, the switching business process 2 when the communication between Master_A and Master_B fails:

[0151] (1) Network n fails. Master_B finds that it cannot connect to Master_A. At the same time, Master_A finds that it cannot connect to Master_B.

[0152] (2) Master_A and Master_B send arbitration requests to all nodes in the Agent Group.

[0153] (3) C1 on the intact network 1 sends a check command to check the status of Master_A and Master_B (host status, service status, metadata database status, etc.).

[0154] (4) Master_A and Master_B return status information respectively (Master_A is healthy, Master_B is healthy).

[0155] (5) C1 merges the status information of Master_A and Master_B.

[0156] (6) C1 sends status information to Master_A and Master_B.

[0157] (7) Master_A receives information that both Master_A and Master_B are healthy and remains in the Active (master) state.

[0158] (8) Master_B receives information that both Master_A and Master_B are healthy and remains in the Standby state. No split-brain occurs.

[0159] Compared with the prior art, this application has the following advantages:

[0160] 1. The method of this application has strong compatibility, can be adapted to a variety of backup application scenarios, and has strong solution stability.

[0161] 2. This application can ensure that when a distributed cluster component fails, the backup system is still available, ensuring high availability requirements for data storage.

[0162] 3. This application provides a technology for automatically detecting the host status to avoid the occurrence of brain split problems.

[0163] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0164] Based on the same inventive concept, the embodiment of the present application also provides a backup exception handling device for implementing the backup exception handling method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more backup exception handling device embodiments provided below can refer to the limitations of the backup exception handling method above, and will not be repeated here.

[0165] In an exemplary embodiment, as shown in FIG. 7 , a backup exception handling device is provided, which is applied to any current backup node among a plurality of backup nodes included in a backup server, the backup server is connected to an arbitration client group, and includes: a request initiating module 701, an information obtaining module 702 and an exception handling module 703, wherein:

[0166] The request initiation module 701 is used to initiate an arbitration request to the arbitration client group when the current backup node detects that the communication with the target backup node is interrupted; the arbitration client group is used to respond to the arbitration request and return a node status acquisition command to each backup node included in the backup server; the target backup node is located in the multiple backup nodes included in the backup server;

[0167] The information acquisition module 702 is used to acquire the node status information of the current backup node based on the node status acquisition command, and send the node status information of the current backup node to the arbitration client group; the arbitration client group is used to summarize the node status information sent by each backup node, obtain the node status summary information and return it to each backup node included in the backup server;

[0168] The exception handling module 703 is used to receive the node status summary information and perform backup exception handling according to the node status summary information.

[0169] In one embodiment, when the current backup node is a master node among multiple backup nodes, the exception handling module 703 is also used to, if the current backup node receives node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node, continue to use the current backup node as the master node; if the current backup node receives node status summary information, and the node status summary information indicates that the current backup node is a faulty backup node, downgrade the current backup node to a backup node among multiple backup nodes.

[0170] In one of the embodiments, the exception handling module 703 is further configured to continue to use the current backup node as the master node when the current backup node does not receive the node status summary information.

[0171] In an exemplary embodiment, when the current backup node is a backup node among multiple backup nodes, the exception handling module 703 further includes:

[0172] The first processing submodule is used to continue to use the current backup node as a backup node if the current backup node receives node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node, and the master node among the multiple backup nodes is a healthy backup node.

[0173] The second processing submodule is used to send the static configuration information of the current backup node to the backup server if the current backup node receives the node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node and the master node is a faulty backup node; the backup server determines a new master node from multiple backup nodes based on the static configuration information of each backup node, generates upgrade indication information and returns it to the current backup node; based on the upgrade indication information, determines whether to upgrade the current backup node to a new master node.

[0174] In an exemplary embodiment, as shown in FIG8 , a backup exception handling device is provided, which is applied to an arbitration client group, the arbitration client group is connected to a backup server, the backup server includes multiple backup nodes, including: a command issuing module 801, an information aggregation module 802 and an information sending module, wherein:

[0175] The command issuing module is used to respond to the arbitration request initiated by the current backup node and return the node status acquisition command to each backup node included in the backup server; the arbitration request is generated and returned to the arbitration client group when the current backup node detects that the communication with the target backup node is interrupted; the current backup node is any backup node, and the target backup node is located in the multiple backup nodes included in the backup server;

[0176] The information aggregation module is used to receive and aggregate the node status information sent by each backup node to obtain node status aggregation information; the node status information is obtained by each backup node responding to the node status acquisition command;

[0177] The information sending module is used to send node status summary information to each backup node; the node status summary information is used by the current backup node to perform backup exception processing based on the node status summary information.

[0178] Each module in the above-mentioned backup exception handling device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0179] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG9. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store node status information and node status summary information data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a backup exception handling method is implemented.

[0180] Those skilled in the art will understand that the structure shown in FIG. 9 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0181] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the backup exception handling method of the above embodiment is implemented.

[0182] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the backup exception handling method of the above embodiment is implemented.

[0183] In one embodiment, a computer program product is provided, including a computer program, which implements the backup exception handling method of the above embodiment when executed by a processor.

[0184] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0185] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0186] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0187] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A backup exception handling method, characterized in that: Applied to any current backup node among a plurality of backup nodes included in a backup server, the backup server being connected to an arbitration client group, the method comprising: In the case where the current backup node detects that the communication with the target backup node is interrupted, an arbitration request is initiated to the arbitration client group; the arbitration client group is used to respond to the arbitration request and return a node status acquisition command to each backup node included in the backup server; the target backup node is located in the multiple backup nodes included in the backup server; Based on the node status acquisition command, the node status information of the current backup node is acquired, and the node status information of the current backup node is sent to the arbitration client group; the arbitration client group is used to aggregate the node status information sent by each of the backup nodes, obtain the node status aggregate information and return it to each of the backup nodes included in the backup server; The node status summary information is received, and backup exception processing is performed according to the node status summary information.

2. The method according to claim 1, characterized in that In the case where the current backup node is a master node among the multiple backup nodes, performing backup exception processing according to the node status summary information includes: If the current backup node receives the node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node, the current backup node continues to serve as the master node; If the current backup node receives the node status summary information, and the node status summary information indicates that the current backup node is a faulty backup node, the current backup node is downgraded to a backup node among the multiple backup nodes.

3. The method according to claim 2, characterized in that The method further comprises: In the case that the current backup node does not receive the node status summary information, the current backup node continues to serve as the master node.

4. The method according to claim 1, characterized in that: In the case where the current backup node is a backup node among the multiple backup nodes, performing backup exception processing according to the node status summary information includes: If the current backup node receives the node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node, and the master node among the multiple backup nodes is a healthy backup node, the current backup node continues to be used as a backup node; If the current backup node receives the node status summary information, and the node status summary information indicates that the current backup node is a healthy backup node and the master node is a faulty backup node, the static configuration information of the current backup node is sent to the backup server; the backup server determines a new master node from the multiple backup nodes based on the static configuration information of each backup node, generates upgrade indication information and returns it to the current backup node; Based on the upgrade indication information, determine whether to upgrade the current backup node to a new master node.

5. A backup exception handling method, characterized in that: Applied to an arbitration client group, the arbitration client group is connected to a backup server, the backup server includes a plurality of backup nodes, the method includes: In response to an arbitration request initiated by the current backup node, a node status acquisition command is returned to each backup node included in the backup server; the arbitration request is generated and returned to the arbitration client group when the current backup node detects that the communication with the target backup node is interrupted; the current backup node is any one of the backup nodes, and the target backup node is located among the multiple backup nodes included in the backup server; Receiving and summarizing the node status information sent by each backup node to obtain node status summary information; the node status information is obtained by each backup node in response to the node status acquisition command; The node status summary information is sent to each of the backup nodes; the node status summary information is used by the current backup node to perform backup exception processing based on the node status summary information.

6. A backup exception handling system, characterized in that: The system includes any current backup node among a plurality of backup nodes included in a backup server and an arbitration client group, wherein the backup server is connected to the arbitration client group; The current backup node is configured to initiate an arbitration request to the arbitration client group when the current backup node detects that the communication with the target backup node is interrupted; The target backup node is located in a plurality of backup nodes included in the backup server; The arbitration client group is used to respond to the arbitration request and return a node status acquisition command to each backup node included in the backup server; The current backup node is further configured to receive the node status acquisition command, acquire node status information of the current backup node based on the node status acquisition command, and send the node status information of the current backup node to the arbitration client group; The arbitration client group is further used to summarize the node status information sent by each backup node, obtain node status summary information and return it to each backup node included in the backup server; The current backup node is further used to receive the node status summary information and perform backup exception processing according to the node status summary information.

7. A backup exception handling device, characterized in that: Applied to any current backup node among a plurality of backup nodes included in a backup server, the backup server being connected to an arbitration client group, the device comprising: a request initiation module, configured to initiate an arbitration request to the arbitration client group when the current backup node detects that the communication with the target backup node is interrupted; the arbitration client group is configured to respond to the arbitration request and return a node status acquisition command to each backup node included in the backup server; the target backup node is located among the multiple backup nodes included in the backup server; An information acquisition module, configured to acquire the node status information of the current backup node based on the node status acquisition command, and send the node status information of the current backup node to the arbitration client group; the arbitration client group is configured to aggregate the node status information sent by each of the backup nodes, obtain the node status aggregate information and return it to each of the backup nodes included in the backup server; The exception handling module is used to receive the node status summary information and perform backup exception handling according to the node status summary information.

8. A backup exception handling device, characterized in that: Applied to an arbitration client group, the arbitration client group is connected to a backup server, the backup server includes a plurality of backup nodes, and the device includes: A command issuing module, used for responding to the arbitration request initiated by the current backup node, and returning a node status acquisition command to each backup node included in the backup server; the arbitration request is generated and returned to the arbitration client group when the current backup node detects that the communication with the target backup node is interrupted; the current backup node is any one of the backup nodes, and the target backup node is located in the multiple backup nodes included in the backup server; An information aggregation module, used to receive and aggregate the node status information sent by each backup node to obtain node status aggregation information; the node status information is obtained by each backup node in response to the node status acquisition command; The information sending module is used to send the node status summary information to each backup node; the node status summary information is used by the current backup node to perform backup exception processing based on the node status summary information.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images