Fault isolation method and device for symmetric active-active disaster recovery system

Abstract

The invention relates to the technical field of disaster recovery, and discloses a fault isolation method and device for a symmetric active-active disaster recovery system, and the method comprises the steps: transmitting a first operation instruction to a second station, and determining that the response of the second station is overtime under the condition of detecting that response information is not received within a first threshold time; detecting whether the process of processing the operation instruction is in a healthy state; sending an offline instruction to the second station to instruct the second station to be offline under the condition that it is determined that the station is in the healthy state; and under the condition that indication information of successful offline is received, the working mode of the device is switched to a single-end operation working mode. Therefore, the data synchronization between the first station and the second station is prevented from being influenced by the fault problem of the second station, the technical effect of fault isolation is further achieved, and the data security and service stability of the first station are guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of disaster recovery, and in particular to a fault isolation method, device, computer equipment and storage medium of a symmetric active-active disaster recovery system. Background Art

[0002] At present, during the data synchronization process of a symmetric active-active system, a site may be affected by software defects or IO stuck faults, which may affect the timely writing of data, prolonging the service response time and preventing data from being synchronized in time, resulting in the two sites providing services independently and inconsistent data between the two sites, which poses the risk of business data damage or loss. Therefore, how to prevent a site failure from spreading to a system failure is an urgent problem to be solved. Summary of the invention

[0003] In view of this, the present invention provides a fault isolation method, apparatus, computer equipment and storage medium for a symmetric active-active disaster recovery system to solve the problem that site faults cannot be isolated in the related art.

[0004] In a first aspect, the present invention provides a fault isolation method for a symmetric active-active disaster recovery system. The method is applied to an application scenario of a symmetric active-active disaster recovery system. The method is performed by a first site in the symmetric active-active disaster recovery system. The first site is any site in the symmetric active-active disaster recovery system. The method includes:

[0005] Sending a first operation instruction to a second site, where the second site is a site other than the first site in the symmetric active-active disaster recovery system, and the first operation instruction is used to instruct the second site to synchronize data;

[0006] In the case where it is detected that the response information corresponding to the first operation instruction sent by the second station is not received within the first threshold time, determining that the response of the second station has timed out;

[0007] Check whether the process of processing operation instructions is in a healthy state;

[0008] When it is determined that the process of processing the operation instruction by itself is in a healthy state, sending an offline instruction to the second site to instruct the second site to go offline;

[0009] When receiving the indication information that the second site is successfully offlined fed back by the second site, the operating mode of the site is switched to the single-ended operation mode.

[0010] The present invention provides a fault isolation method for a symmetric active-active disaster recovery system, which has the following advantages:

[0011] First, the first operation instruction is sent to the second site through the first site, and the corresponding response information is waited for the second site to recover. If the response information of the second site for the above-mentioned first operation instruction is not received within the first threshold time, the first site can determine that the second site response has timed out. Then the first site needs to determine whether the process of processing the operation instruction by itself is normal, that is, whether it is in the above-mentioned healthy state. After determining that it is in a healthy state, the first site determines that it has the ability to run on a single end. Then send an offline instruction to the second site, instruct the above-mentioned second site to go offline, and wait for the second site to respond to the offline instruction. After receiving the indication information fed back by the second site to determine that the second site is successfully offline, the first site can switch its own working mode to a single-end operation mode. Through the above scheme, when the first site finds that the second site processes the first operation instruction, it verifies its own health state to a limited extent, and when it determines that it is in a healthy state, it instructs the second site to go offline, and after determining that the second site is successfully offline, it switches its own working mode to a single-end operation mode, thereby avoiding the fault problem of the second site affecting the data synchronization between the first site and the second site, thereby achieving the technical effect of fault isolation, and ensuring the data security and business stability of the first site.

[0012] In an optional implementation, when it is detected that the response information corresponding to the first operation instruction sent by the second station is not received within the first threshold time, determining that the second station response times out specifically includes:

[0013] Input the first operation instruction and the corresponding sending time into a waiting sequence to wait for the response information corresponding to the first operation instruction;

[0014] Based on the sending time, determining the waiting time for the first operation instruction to enter the waiting sequence;

[0015] Comparing the waiting time with the first threshold time;

[0016] When the waiting time is greater than or equal to the first threshold time, it is determined that the response information corresponding to the first operation instruction sent by the second station is not received within the first threshold time.

[0017] Specifically, in the steps of the method, the first operation instruction and the corresponding sending time are input into the waiting sequence, and the response information corresponding to the first operation instruction is waited. Then, the waiting time of the first operation instruction entering the waiting sequence is determined according to the sending time. By comparing the waiting time and the first threshold time, it is determined that the response information corresponding to the first operation instruction sent by the second station is not received within the first threshold time. By setting the waiting sequence, unified monitoring of multiple first operation instructions can be achieved by monitoring one waiting sequence, so that the first station can make a judgment that the response of the second station has timed out.

[0018] In an optional implementation, detecting whether the process of processing the operation instruction itself is in a healthy state specifically includes:

[0019] Acquire self-operation instruction processing information, the self-operation instruction processing information including: response time of responding to the second site, delay time of self-processing operation instruction, and self-volume status;

[0020] Check whether the above reaction time and the above stall time have timed out or the volume status itself is abnormal;

[0021] When it is determined that the above reaction time and the above stalling time are not timed out and there is no abnormality in the state of the self-volume, it is determined that the process of the above-mentioned self-processing operation instructions is in a healthy state.

[0022] Specifically, by obtaining the response time of responding to the second site, the freeze time of its own processing operation instructions, and a series of its own operation instruction processing information of its own volume status, it is respectively detected whether the above reaction time and the above freeze event have timed out or whether its own volume status is abnormal, thereby determining its own health status; by detecting the timeout of the above reaction time and the above freeze time or the abnormality of its own volume status, it is determined whether it can run single-ended.

[0023] In an optional implementation, the above method further includes:

[0024] When it is detected that no indication information indicating that the second site is offline successfully fed back by the second site within a second threshold time is received, sending an arbitration request to a third-party arbitration server;

[0025] The arbitration request includes: its own health status information and priority arbitration information; the health status information indicates that the function of processing the operation instruction internally is in a healthy state, and the priority arbitration information is used to indicate the priority of the first site switching to the single-ended operation working mode when both the first site and the second site are in a healthy state;

[0026] When receiving the arbitration notice sent by the third-party arbitration server, the time starts;

[0027] When a switching instruction is received within a third threshold time, the operating mode of the device is switched to the single-ended operating mode;

[0028] The arbitration notification is a notification indicating that the second site is offline, determined by the third-party arbitration server according to the arbitration request sent by the first site and the health status information and priority arbitration information corresponding to the second site obtained from the second site.

[0029] Specifically, after determining that the indication information of the offline success of the above-mentioned second site fed back by the above-mentioned second site has not been received within the second threshold time, an arbitration request is sent to the third-party arbitration server. The health status information and priority arbitration information are sent to the third-party arbitration server to indicate that the own status supports single-end operation, and the priority of switching to the single-end operation working mode is provided. After receiving the arbitration notification sent by the third-party arbitration server, the timing is started to compare the above-mentioned third threshold time. If the switching instruction sent by the third-party server is received within the third threshold time, the working mode of the self is switched to the single-end operation working mode. Through the above scheme, when the indication information of the offline success fed back by the second site is not received within the timeout, the third-party arbitration server is introduced to determine whether to use the first site as a site in the single-end operation working mode, and the synchronization between the two sites is cut off, so that the link failure problem between the two sites can be responded to, avoiding the problem of data synchronization caused by the link failure between the sites of the active-active disaster recovery system.

[0030] In an optional implementation, switching the operating mode to the single-ended operation mode specifically includes:

[0031] If no switching instruction is received within the third threshold time, the operating mode of the device is forcibly switched to the single-ended operating mode.

[0032] Specifically, if no switching instruction is received within the third threshold time, it means that the second site has timed out of offline operation, and there is no need to continue waiting, and the first site can be directly forced to switch its working mode to the single-end operation working mode. Through the above scheme, when the second site times out of offline operation, the first site can directly force the working mode to switch to the single-end operation working mode to avoid the offline process failure of the second site delaying the first site from continuing to work in the single-end operation working mode, thereby reducing the fault handling time of the disaster recovery system.

[0033] In an optional implementation, the above method further includes:

[0034] In the case of receiving the second operation instruction sent by the second site, feeding back response information corresponding to the second operation instruction to the second site and recording the response processing time, wherein the second operation instruction is used to instruct the first site to synchronize data;

[0035] The response processing time is the time interval between receiving the second operation instruction sent by the second site and feeding back the response information corresponding to the second operation instruction to the second site;

[0036] When receiving the offline instruction sent by the second site, comparing the response processing time with a fourth threshold time;

[0037] If it is determined that the response processing time is longer than the fourth threshold time, switching the working mode to an offline mode;

[0038] When the working mode of the station is the offline mode, feeding back to the second station an indication that the station is offline successfully;

[0039] Alternatively, when it is determined that the response processing time is not greater than the fourth threshold time, priority arbitration information is sent to a third-party arbitration server, where the priority arbitration information is used to indicate the priority of the first site switching to the single-ended operation working mode when both the first site and the second site are in a healthy state;

[0040] Upon receiving the arbitration failure notification sent by the third-party arbitration server, switching its own working mode to the offline mode, and detecting the offline duration of the switching process of its own working mode to the offline mode, wherein the arbitration failure notification is a notification sent by the third-party arbitration server when it is determined that the arbitration of the first site has failed based on the priority arbitration information sent by the first site and the priority arbitration information corresponding to the second site obtained from the second site;

[0041] Determine whether the offline duration exceeds the preset arbitration process duration;

[0042] If the arbitration process is detected to be in offline mode within the duration of the arbitration process, a completion notification of offline is sent to the third-party arbitration server;

[0043] Alternatively, when the offline duration exceeds the arbitration process duration, the working mode is switched to a forced offline mode.

[0044] Specifically, after receiving the second operation instruction sent by the second site, the corresponding response information is fed back to the site, and the response processing time is recorded. Then, when the offline instruction sent by the second site is received, the response processing time recorded above is detected whether it exceeds the fourth threshold time. If it is determined that the response processing time is overtime, it is switched to offline mode, and the indication information of its own offline success is fed back to the second site. If it is determined that the response processing time is not overtime, the priority arbitration information is sent to the third-party arbitration server for arbitration by the third arbitration server. When the arbitration failure notification sent by the third-party arbitration server is received later, it is switched to offline mode and it is detected whether the process of switching to offline mode is timed out. If it is not timed out, the offline completion notification is sent to the third-party arbitration server, and if it is timed out, it is forced to go offline. Through the above scheme, when the offline instruction sent by the second site is received and it is determined that the response processing time is not overtime, the third-party arbitration server is introduced to determine whether to use the first site as a site in offline mode, and the synchronization between the two sites is cut off, so that the link failure problem between the two sites can be responded to, avoiding the problem of data synchronization caused by the link failure between the sites of the active-active disaster recovery system. In the case of the first site being offline for a timeout, the first site is directly forced to go offline, which can avoid the offline process failure of the first site delaying the second site from continuing to work in a single-end operation mode, thereby reducing the fault handling time of the disaster recovery system.

[0045] In an optional implementation, the above method further includes:

[0046] When it is detected that no indication information indicating that the second site is offline successfully fed back by the second site within a second threshold time is received, sending an arbitration request to a third-party arbitration server;

[0047] The arbitration request includes: its own health status information and priority arbitration information, the health status information indicates that the function of processing the operation instruction internally is in a healthy state, and the priority arbitration information is used to indicate the priority of the first site to switch to the single-ended operation working mode when both the first site and the second site are in a healthy state;

[0048] Upon receiving the arbitration failure notification sent by the third-party arbitration server, switching its own working mode to the offline mode;

[0049] Detecting the offline duration of the switching process of switching its own working mode to the above-mentioned offline mode, and determining whether the above-mentioned offline duration exceeds the preset arbitration process duration;

[0050] When the offline duration exceeds the arbitration duration, the work mode will be switched to forced offline mode.

[0051] When the arbitration process is completed within the duration of the arbitration process, the arbitration server sends an offline completion notification.

[0052] Specifically, by detecting that the indication information of offline success feedback from the second site has timed out, a third-party arbitration server is requested to intervene, and its own health status information and priority arbitration information are sent to the third-party arbitration server to indicate that its own state supports single-end operation, and provides its own priority for switching to single-end operation mode. After receiving the arbitration failure notification sent by the third-party arbitration server, its own working mode is switched to offline mode, and the offline duration of the switching process is detected, and whether it has timed out is determined based on the preset arbitration process duration. If it times out, its own working mode is switched to forced offline mode. If it does not time out, a completion offline notification is sent to the above-mentioned third-party arbitration server. Through the above scheme, in the case of timeout without receiving the indication information of offline success feedback from the second site, a third-party arbitration server is introduced to determine whether to use the first site or the second site as a site in single-end operation mode, and the synchronization between the two sites is cut off, so that a response can be made to the link failure problem between the two sites, avoiding data synchronization problems caused by the link failure between the sites in the active-active disaster recovery system. At the same time, when the first site is offline for a timeout, the first site is directly forced to go offline, which can avoid the offline process failure of the first site delaying the second site from continuing to work in a single-end operation mode, thereby reducing the fault handling time of the disaster recovery system.

[0053] In a second aspect, the present invention provides a fault isolation device for a symmetric active-active disaster recovery system. The device is applied to an application scenario of a symmetric active-active disaster recovery system. The device is used in a first site in the symmetric active-active disaster recovery system. The first site is any site in the symmetric active-active disaster recovery system, including:

[0054] A first sending module is used to send a first operation instruction to a second site, where the second site is a site other than the first site in the symmetric active-active disaster recovery system, and the first operation instruction is used to instruct the second site to synchronize data;

[0055] A first detection module, configured to determine that a response of the second station has timed out when it is detected that a response message corresponding to the first operation instruction sent by the second station is not received within a first threshold time;

[0056] The second detection module is used to detect whether the process of processing the operation instruction itself is in a healthy state;

[0057] A second sending module is used to send an offline instruction to the second site to instruct the second site to go offline when it is determined that the process of processing the operation instruction by itself is in a healthy state;

[0058] The first switching module is configured to switch its own working mode to a single-ended operation working mode upon receiving indication information that the second site is successfully offline fed back by the second site.

[0059] The present invention provides a fault isolation device for a symmetric active-active disaster recovery system, which has the following advantages:

[0060] First, the first operation instruction is sent to the second site through the first site, and the corresponding response information is waited for the second site to recover. If the response information of the second site for the above-mentioned first operation instruction is not received within the first threshold time, the first site can determine that the second site response has timed out. Then the first site needs to determine whether the process of processing the operation instruction by itself is normal, that is, whether it is in the above-mentioned healthy state. After determining that it is in a healthy state, the first site determines that it has the ability to run on a single end. Then send an offline instruction to the second site, instruct the above-mentioned second site to go offline, and wait for the second site to respond to the offline instruction. After receiving the indication information fed back by the second site to determine that the second site is successfully offline, the first site can switch its own working mode to a single-end operation mode. Through the above scheme, when the first site finds that the second site processes the first operation instruction, it verifies its own health state to a limited extent, and when it determines that it is in a healthy state, it instructs the second site to go offline, and after determining that the second site is successfully offline, it switches its own working mode to a single-end operation mode, thereby avoiding the fault problem of the second site affecting the data synchronization between the first site and the second site, thereby achieving the technical effect of fault isolation, and ensuring the data security and business stability of the first site.

[0061] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the fault isolation method of the symmetric active-active disaster recovery system of the first aspect or any corresponding embodiment thereof.

[0062] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the fault isolation method of a symmetric active-active disaster recovery system of the first aspect or any corresponding embodiment thereof.

[0063] In a fifth aspect, the present invention provides a computer program product, including computer instructions, which are used to enable a computer to execute the fault isolation method of a symmetric active-active disaster recovery system of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 It is a schematic diagram of a symmetric active-active disaster recovery system in the related technology;

[0066] Figure 2 is a flow chart of a fault isolation method for a symmetric active-active disaster recovery system according to an embodiment of the present invention;

[0067] Figure 3 is a schematic diagram of a fault isolation device of a symmetric active-active disaster recovery system according to an embodiment of the present invention;

[0068] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0070] Figure 1 Schematic diagram of a symmetric active-active disaster recovery system in related technologies, such as Figure 1 As shown, it includes: a first site and a second site for mirrored IO communication, and both sites can receive write requests for data outside the system. Any of the sites may affect the timely writing of data due to software defects or IO stuck faults. For example, if one of the sites receives a write request and needs to complete the write operation on two sites, it will be affected by the other site due to software vulnerabilities or IO stuck or high response latency, and the link between the two sites is in a high latency state, which prolongs the service response time and the data cannot be synchronized in time, resulting in the two sites providing services separately and inconsistent data between the two sites, and there is a risk of damage or loss of service data.

[0071] According to an embodiment of the present invention, an embodiment of a fault isolation method for a symmetric active-active disaster recovery system is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0072] In this embodiment, a fault isolation method for a symmetric active-active disaster recovery system is provided, which can be performed by a first site in the above-mentioned symmetric active-active disaster recovery system. The above-mentioned first site is any site in the above-mentioned symmetric active-active disaster recovery system. Figure 2 is a flow chart of a fault isolation method of a symmetric active-active disaster recovery system according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0073] Step S201: Send a first operation instruction to a second site.

[0074] The second site is a site other than the first site in the symmetric active-active disaster recovery system, and the first operation instruction is used to instruct the second site to synchronize data.

[0075] The first operation instruction may be a write operation instruction for data synchronization between two sites in a symmetric active-active system, and may be used to instruct the second site to write the target data to be synchronized into a corresponding storage location in the second site.

[0076] Step S202: When it is detected that the response information corresponding to the first operation instruction sent by the second station is not received within the first threshold time, it is determined that the response of the second station has timed out.

[0077] The first threshold time may be set by a timer. The first station synchronizes data with the second station through the first operation instruction, and needs to detect whether the response information corresponding to the first operation instruction sent by the second station is received after the timer expires. If not, it is determined that the response of the second station has timed out.

[0078] Step S203, detecting whether the process of processing the operation instruction itself is in a healthy state.

[0079] After determining that the second site has timed out its response, the first site prepares to switch to the single-ended operation mode. Before switching, it needs to ensure that there is no fault in the process of processing the operation instruction itself, that is, it is in a healthy state. The process of processing the operation instruction itself can be the process of processing the operation instruction outside the system itself.

[0080] Step S204: When it is determined that the process of processing the operation instruction is in a healthy state, an offline instruction is sent to the second site to instruct the second site to go offline.

[0081] After the first site is determined to be in a healthy state, it is ready to switch to the single-ended operation mode. Before that, it is necessary to ensure that the second site will not continue to process the operation instructions input from outside the system, so the second site needs to be offline first. The above offline instruction can be to instruct the second site to switch the working state to the offline state.

[0082] Step S205: upon receiving the indication information that the second site is successfully offlined fed back by the second site, switching its own working mode to a single-ended operation working mode.

[0083] The single-ended operation working mode may terminate data synchronization with the second site and continue to process operation instructions outside the system, thereby avoiding the two sites providing services separately and data inconsistency between the two sites, and reducing the risk of business data damage or loss.

[0084] The present invention provides a fault isolation method for a symmetric active-active disaster recovery system, which has the following advantages:

[0085] First, the first operation instruction is sent to the second site through the first site, and the corresponding response information is waited for the second site to recover. If the response information of the second site for the above-mentioned first operation instruction is not received within the first threshold time, the first site can determine that the second site response has timed out. Then the first site needs to determine whether the process of processing the operation instruction by itself is normal, that is, whether it is in the above-mentioned healthy state. After determining that it is in a healthy state, the first site determines that it has the ability to run on a single end. Then send an offline instruction to the second site, instruct the above-mentioned second site to go offline, and wait for the second site to respond to the offline instruction. After receiving the indication information fed back by the second site to determine that the second site is successfully offline, the first site can switch its own working mode to a single-end operation mode. Through the above scheme, when the first site finds that the second site processes the first operation instruction, it verifies its own health state to a limited extent, and when it determines that it is in a healthy state, it instructs the second site to go offline, and after determining that the second site is successfully offline, it switches its own working mode to a single-end operation mode, thereby avoiding the fault problem of the second site affecting the data synchronization between the first site and the second site, thereby achieving the technical effect of fault isolation, and ensuring the data security and business stability of the first site.

[0086] In an optional implementation manner, in order to further accurately determine whether the second station responds to a timeout, the step S202 specifically includes:

[0087] The first operation instruction and the corresponding sending time are input into a waiting sequence to wait for the response information corresponding to the first operation instruction.

[0088] In order to determine whether the second station has timed out, it is first necessary to determine the sending time corresponding to the first operation instruction. The first operation instruction waiting queue may be a queue in which a plurality of first operation instructions are arranged according to the input time.

[0089] Based on the sending time, the waiting time for the first operation instruction to enter the waiting sequence is determined.

[0090] The waiting time is compared with the first threshold time.

[0091] The first threshold time may be set by a timer. Preferably, the waiting time of the first operation instruction at the head of the queue may be compared first, and when the waiting time of the first operation instruction at the head of the queue has not timed out, the comparison of the waiting time of other first operation instructions in the queue is stopped.

[0092] When the waiting time is greater than or equal to the first threshold time, it is determined that the response information corresponding to the first operation instruction sent by the second station is not received within the first threshold time.

[0093] Specifically, in the steps of the method, the first operation instruction and the corresponding sending time are input into the waiting sequence, and the response information corresponding to the first operation instruction is waited. Then, the waiting time of the first operation instruction entering the waiting sequence is determined according to the sending time. By comparing the waiting time and the first threshold time, it is determined that the response information corresponding to the first operation instruction sent by the second station is not received within the first threshold time. By setting the waiting sequence, unified monitoring of multiple first operation instructions can be achieved by monitoring one waiting sequence, so that the first station can make a judgment that the response of the second station has timed out.

[0094] In an optional implementation, in order to further determine whether the process of the first station detecting its own processing of the operation instruction is in a healthy state, the above step S203 specifically includes:

[0095] Get the processing information of its own operation instructions.

[0096] The above-mentioned self-operation instruction processing information includes: the response time of responding to the above-mentioned second site, the delay time of the self-processing operation instruction, and the self-volume status.

[0097] By obtaining the above-mentioned self-operation instruction processing information, it is convenient to determine the corresponding indicators generated by the first site's own processing of the operation instruction process, providing a reference for subsequent judgment. The reaction time of the above-mentioned second site can be to start the timer when receiving the operation instruction of the above-mentioned second site, and obtain the reaction time after feeding back the response to the above-mentioned second site. The above-mentioned stall time of the self-processing operation instruction can be the length of time when the stall occurs in the processing process of reading and writing data at the self-site. The above-mentioned self-volume status can be the abnormal volume status information obtained by the first site.

[0098] Check whether the above reaction time and the above stall time have timed out or the volume status itself is abnormal.

[0099] When it is determined that the above reaction time and the above stalling time are not timed out and there is no abnormality in the state of the self-volume, it is determined that the process of the above-mentioned self-processing operation instructions is in a healthy state.

[0100] By comparing and checking whether the above reaction time and stall time have timed out, and whether the volume status itself is abnormal, the process of processing operation instructions can be quickly and accurately judged.

[0101] Specifically, by obtaining the response time of responding to the second site, the freeze time of its own processing operation instructions, and a series of its own operation instruction processing information of its own volume status, it is respectively detected whether the above reaction time and the above freeze event have timed out or whether its own volume status is abnormal, thereby determining its own health status; by detecting the timeout of the above reaction time and the above freeze time or the abnormality of its own volume status, it is determined whether it can run single-ended.

[0102] In an optional implementation, the first site does not receive the indication information of the second site being offline successfully fed back by the second site. At this time, both sites may be in a healthy state, but the network link connection between the sites fails, and a third-party arbitration server needs to be introduced for arbitration. The above method also includes:

[0103] When it is detected that no indication information indicating that the second site is offline successfully fed back by the second site is received within the second threshold time, an arbitration request is sent to the third-party arbitration server.

[0104] The third-party arbitration server can be a server that is independent of the active-active disaster recovery system and has established communication relationships with both the first site and the second site. In the above steps, the first site finds that the second site's response has timed out, so the first site needs to send an arbitration request to the third-party arbitration server to introduce the third-party arbitration server for arbitration.

[0105] The arbitration request includes: its own health status information and priority arbitration information. The above-mentioned health status information indicates that the function of processing the operation instruction internally is in a healthy state, and the above-mentioned priority arbitration information is used to indicate the priority of the first site to switch to the single-ended operation working mode when both the first site and the second site are in a healthy state.

[0106] Since both sites are in a healthy state, but the network link connection between the sites fails, the third-party arbitration server needs to determine how to arbitrate based on the status of the two sites, so it needs to obtain its own health status information and priority arbitration information. The above-mentioned own health status information can be the specific response time of the second site, the time of processing the operation instruction, the status of its own volume, or the status information that the first site is in a healthy state. The above-mentioned priority arbitration information can be the specific host connection status and traffic pressure information of the first site.

[0107] When the arbitration notice sent by the third-party arbitration server is received, the time starts.

[0108] The arbitration notification is a notification that the third-party arbitration server determines that the first site can be used as a single-end operation mode based on its own health status information and priority arbitration information. However, at this time, the first site cannot be switched directly and needs to wait for the second site to go offline.

[0109] When a switching instruction is received within the third threshold time, its own working mode is switched to the above-mentioned single-ended operation working mode.

[0110] The third threshold time is set to allow offline time for the second site. After the second site goes offline within the third threshold time, the third-party arbitration server notifies the first site through a switching instruction that its working mode can be switched to the above-mentioned single-end operation working mode.

[0111] The arbitration notification is a notification indicating that the second site is offline, determined by the third-party arbitration server according to the arbitration request sent by the first site and the health status information and priority arbitration information corresponding to the second site obtained from the second site.

[0112] Specifically, after determining that the indication information of the offline success of the above-mentioned second site fed back by the above-mentioned second site has not been received within the second threshold time, an arbitration request is sent to the third-party arbitration server. The health status information and priority arbitration information are sent to the third-party arbitration server to indicate that the own status supports single-end operation, and the priority of switching to the single-end operation working mode is provided. After receiving the arbitration notification sent by the third-party arbitration server, the timing is started to compare the above-mentioned third threshold time. If the switching instruction sent by the third-party server is received within the third threshold time, the working mode of the self is switched to the single-end operation working mode. Through the above scheme, when the indication information of the offline success fed back by the second site is not received within the timeout, the third-party arbitration server is introduced to determine whether to use the first site as a site in the single-end operation working mode, and the synchronization between the two sites is cut off, so that the link failure problem between the two sites can be responded to, avoiding the problem of data synchronization caused by the link failure between the sites of the active-active disaster recovery system.

[0113] Preferably, the IO communication, working state switching interaction, and arbitration between sites are all processed by independent task modules, which can be processed separately by independent devices or isolated by firewalls or network management. Thus, the software vulnerabilities for processing different tasks can be further isolated within the site.

[0114] In an optional implementation, during the arbitration process, a timeout may occur without receiving a switching instruction. Therefore, upon receiving an arbitration notification sent by the third-party arbitration server, the working mode of the arbitration server is switched to the single-ended operation working mode, specifically including:

[0115] If no switching instruction is received within the third threshold time, the operating mode of the device is forcibly switched to the single-ended operating mode.

[0116] The failure to receive a switching command within the third threshold time may be due to a fault in the second site that prevents it from going offline smoothly. In order not to delay the continued operation of the first site, the first site no longer waits for a switching command and directly forces its own working mode to switch to the single-ended working mode.

[0117] Specifically, if no switching instruction is received within the third threshold time, it means that the second site has timed out of offline operation, and there is no need to continue waiting, and the first site can be directly forced to switch its working mode to the single-end operation working mode. Through the above scheme, when the second site times out of offline operation, the first site can directly force the working mode to switch to the single-end operation working mode to avoid the offline process failure of the second site delaying the first site from continuing to work in the single-end operation working mode, thereby reducing the fault handling time of the disaster recovery system.

[0118] In an optional implementation, considering that the present solution is applied to a symmetric active-active disaster recovery system, the above-mentioned fault situation may occur in any of the two sites. In order to handle the possible fault situation of the first site, the above-mentioned method further includes:

[0119] When receiving the second operation instruction sent by the second site, the response information corresponding to the second operation instruction is fed back to the second site, and the response processing time is recorded. The second operation instruction is used to instruct the first site to synchronize data.

[0120] The response processing time is used to subsequently determine whether the first site has timed out in processing the second operation instruction, that is, whether the processing of the synchronization data has timed out.

[0121] The response processing time length is the time interval between receiving the second operation instruction sent by the second site and feeding back the response information corresponding to the second operation instruction to the second site.

[0122] When the offline instruction sent by the second site is received, the response processing time is compared with a fourth threshold time.

[0123] The second station instructs the first station to go offline through an offline instruction, and the first station needs to monitor whether it has a fault, that is, whether it should go offline. Therefore, it is necessary to compare the above response processing time with the fourth threshold time.

[0124] When it is determined that the response processing time is longer than the fourth threshold time, the operating mode is switched to the offline mode.

[0125] By comparing the response processing time and the fourth threshold time, the first station determines that its own response has timed out, and thus can go offline.

[0126] When its own working mode is the offline mode, it feeds back the indication information of its own offline success to the second site.

[0127] After the first site goes offline, that is, after the working mode is switched to the offline mode, it is necessary to notify the second site of the indication information that the first site goes offline successfully, so that the second site can start single-end operation.

[0128] Alternatively, when it is determined that the response processing time is not greater than the fourth threshold time, priority arbitration information is sent to a third-party arbitration server, and the priority arbitration information is used to indicate the priority of the first site to switch to a single-ended operation mode when both the first site and the second site are in a healthy state.

[0129] That is to say, the first site determines that there is no fault in itself by comparing the above response processing time with the fourth threshold time. It may be that the network link connection between the sites fails, and it is necessary to introduce a third-party arbitration server for arbitration by sending priority arbitration information to the third-party arbitration server.

[0130] When receiving the arbitration failure notification sent by the third-party arbitration server, the operating mode is switched to the offline mode, and the offline duration of the switching process of the operating mode to the offline mode is detected.

[0131] The arbitration failure notification is a notification sent by the third-party arbitration server when it is determined that the arbitration of the first site has failed based on the priority arbitration information sent by the first site and the priority arbitration information corresponding to the second site obtained from the second site.

[0132] After the third-party arbitration server determines that the first site arbitration fails, it instructs the first site to go offline. The first site needs to go offline within the specified time, that is, within the preset arbitration process duration, so it needs to detect its own offline duration. The offline mode can be that the status of the volume in the site is set to offline.

[0133] Determine whether the offline duration exceeds the preset arbitration process duration.

[0134] When the arbitration process is completed within the duration of the arbitration process, the arbitration server sends an offline completion notification.

[0135] That is to say, if the first site is successfully offline within the specified time, it is necessary to send an offline completion notification to the third-party arbitration server so that it can instruct the second site to switch the working mode.

[0136] Alternatively, when the offline duration exceeds the arbitration process duration, the working mode is switched to a forced offline mode.

[0137] However, it is also possible that the first site fails to go offline within the specified time. In this case, it needs to be forced offline to avoid data inconsistency between sites caused by continuing to take over the business. The forced offline mode can trigger path failure through the communication protocol and set a mark in its own system to ensure that all responses to read and write requests fail.

[0138] Specifically, after receiving the second operation instruction sent by the second site, the corresponding response information is fed back to the site, and the response processing time is recorded. Then, when the offline instruction sent by the second site is received, the response processing time recorded above is detected whether it exceeds the fourth threshold time. If it is determined that the response processing time is overtime, it is switched to offline mode, and the indication information of its own offline success is fed back to the second site. If it is determined that the response processing time is not overtime, the priority arbitration information is sent to the third-party arbitration server for arbitration by the third arbitration server. When the arbitration failure notification sent by the third-party arbitration server is received later, it is switched to offline mode and it is detected whether the process of switching to offline mode is timed out. If it is not timed out, the offline completion notification is sent to the third-party arbitration server, and if it is timed out, it is forced to go offline. Through the above scheme, when the offline instruction sent by the second site is received and it is determined that the response processing time is not overtime, the third-party arbitration server is introduced to determine whether to use the first site as a site in offline mode, and the synchronization between the two sites is cut off, so that the link failure problem between the two sites can be responded to, avoiding the problem of data synchronization caused by the link failure between the sites of the active-active disaster recovery system. In the case of the first site being offline for a timeout, the first site is directly forced to go offline, which can avoid the offline process failure of the first site delaying the second site from continuing to work in a single-end operation mode, thereby reducing the fault handling time of the symmetric active-active disaster recovery system.

[0139] In an optional implementation, considering that the solution is applied to a symmetric active-active disaster recovery system, any one of the two sites may initiate arbitration and receive an arbitration failure notification. To handle this situation, the method further includes:

[0140] When it is detected that no indication information indicating that the second site is offline successfully fed back by the second site is received within the second threshold time, an arbitration request is sent to the third-party arbitration server.

[0141] Similar to the above implementation, the first site also finds that the response of the second site has timed out, and introduces a third-party arbitration server to perform arbitration.

[0142] The above-mentioned arbitration request includes: its own health status information and priority arbitration information. The above-mentioned health status information indicates that the function of processing operation instructions internally is in a healthy state. The above-mentioned priority arbitration information is used to indicate the priority of the above-mentioned first site to switch to the single-ended operation working mode when both the above-mentioned first site and the above-mentioned second site are in a healthy state.

[0143] When receiving the arbitration failure notification sent by the third-party arbitration server, the operating mode of the device is switched to the offline mode.

[0144] That is to say, after the first site initiates an arbitration request and receives an arbitration failure notification sent by the third-party arbitration server, it also needs to switch to the offline mode.

[0145] Detect the offline duration of the switching process of switching its own working mode to the above-mentioned offline mode, and determine whether the above-mentioned offline duration exceeds the preset arbitration process duration.

[0146] When the offline duration exceeds the arbitration process duration, the working mode is switched to the forced offline mode.

[0147] If the first site does not go offline within the specified time, it needs to be forced offline to avoid data inconsistency between sites caused by continuing to take over the business. The forced offline mode can trigger path failure through the communication protocol and set a mark in its own system to ensure that all responses fail after receiving read and write requests.

[0148] When the arbitration process is completed within the duration of the arbitration process, the arbitration server sends an offline completion notification.

[0149] If the first site is successfully offline within the specified time, a completion notification of offline needs to be sent to the third-party arbitration server so that it can instruct the second site to switch the working mode.

[0150] Specifically, by detecting that the indication information of offline success feedback from the second site has timed out, a third-party arbitration server is requested to intervene, and its own health status information and priority arbitration information are sent to the third-party arbitration server to indicate that its own state supports single-end operation, and provides its own priority for switching to single-end operation mode. After receiving the arbitration failure notification sent by the third-party arbitration server, its own working mode is switched to offline mode, and the offline duration of the switching process is detected, and whether it has timed out is determined based on the preset arbitration process duration. If it times out, its own working mode is switched to forced offline mode. If it does not time out, a completion offline notification is sent to the above-mentioned third-party arbitration server. Through the above scheme, in the case of timeout without receiving the indication information of offline success feedback from the second site, a third-party arbitration server is introduced to determine whether to use the first site or the second site as a site in single-end operation mode, and the synchronization between the two sites is cut off, so that a response can be made to the link failure problem between the two sites, avoiding data synchronization problems caused by the link failure between the sites in the active-active disaster recovery system. At the same time, when the first site is offline for a timeout, the first site is directly forced to go offline, which can avoid the offline process failure of the first site delaying the second site from continuing to work in a single-end operation mode, thereby reducing the fault handling time of the disaster recovery system.

[0151] In an optional implementation manner, in order to ensure the security and confidentiality of data transmission between sites or between a site and a user terminal in a symmetric active-active disaster recovery system, the method further includes:

[0152] During data transmission, a dynamic encryption algorithm is used to encrypt the transmitted data. The above encryption algorithm is automatically adjusted according to the real-time network environment and data sensitivity.

[0153] The data sensitivity may be determined according to the storage location of the transmitted data. The real-time network environment may be the network bandwidth utilization.

[0154] Perform authentication and permission checks every time data is accessed to ensure that only authorized users can access sensitive data.

[0155] Prevent sensitive data leakage by setting access permissions for sensitive data.

[0156] Generate data access logs regularly and automatically trigger alerts when unusual access attempts occur, logging the event and notifying system administrators.

[0157] In other words, the access logs of unauthorized users need to be recorded and notified to the system administrator.

[0158] Specifically, the data transmission between the two sites in the above disaster recovery system is dynamically encrypted according to the network environment and data sensitivity. When data is accessed or backed up between sites, the identity and authority of the accessed site are verified. At the same time, data access logs are generated regularly to record and warn of abnormal access time. This achieves the technical effect of improving the system's data protection capabilities and ensuring information security.

[0159] In this embodiment, a fault isolation device for a symmetric active-active disaster recovery system is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0160] This embodiment provides a fault isolation device for a symmetric active-active disaster recovery system. The device is applied to an application scenario of a symmetric active-active disaster recovery system. The device is used in a first site in the symmetric active-active disaster recovery system. The first site is any site in the symmetric active-active disaster recovery system. Figure 3 is a schematic diagram of a fault isolation device of a symmetric active-active disaster recovery system according to an embodiment of the present invention. Figure 3 As shown, including:

[0161] The first sending module 301 is used to send a first operation instruction to a second site, where the second site is a site other than the first site in the symmetric active-active disaster recovery system, and the first operation instruction is used to instruct the second site to synchronize data.

[0162] The first detection module 302 is used to determine that the response of the second station has timed out when it is detected that the response information corresponding to the first operation instruction sent by the second station is not received within a first threshold time.

[0163] The second detection module 303 is used to detect whether the process of processing the operation instruction itself is in a healthy state.

[0164] The second sending module 304 is used to send an offline instruction to the second site to instruct the second site to go offline when it is determined that the process of processing the operation instruction itself is in a healthy state.

[0165] The first switching module 305 is configured to switch its own working mode to a single-ended operation working mode upon receiving the indication information that the second site is successfully offline fed back by the second site.

[0166] In an optional implementation, the first detection module 302 includes:

[0167] The waiting unit is used to input the first operation instruction and the corresponding sending time into a waiting sequence to wait for response information corresponding to the first operation instruction.

[0168] A duration unit is used to determine the waiting duration for the first operation instruction to enter the waiting sequence based on the sending time.

[0169] A comparing unit is used to compare the waiting time with the first threshold time.

[0170] The first determining unit is configured to determine, when the waiting time is greater than or equal to the first threshold time, that no response information corresponding to the first operation instruction sent by the second station is received within the first threshold time.

[0171] In an optional implementation, the second detection module 303 includes:

[0172] The acquisition unit is used to acquire its own operation instruction processing information, wherein the self-operation instruction processing information includes: the response time of responding to the second site, the delay time of its own processing operation instruction, and its own volume status.

[0173] The detection unit is used to detect whether the reaction time and the stall time have timed out or the state of the volume itself is abnormal.

[0174] The second determining unit is used to determine that the process of processing the operation instruction itself is in a healthy state when it is determined that both the reaction time and the stall time have not timed out and there is no abnormality in the state of the own volume.

[0175] In an optional embodiment, the above device further includes:

[0176] The first arbitration management module is configured to send an arbitration request to a third-party arbitration server when detecting that no indication information indicating that the second site is offline successfully fed back by the second site is received within a second threshold time.

[0177] The arbitration request includes: its own health status information and priority arbitration information. The own health status information indicates that the function of processing operation instructions within itself is in a healthy state, and the priority arbitration information is used to indicate the priority of the first site switching to the single-ended operation working mode when both the first site and the second site are in a healthy state.

[0178] The timing module is used to start timing when receiving the arbitration notification sent by the third-party arbitration server.

[0179] The second switching module is used to switch its own working mode to the single-ended operation working mode when receiving a switching instruction within a third threshold time.

[0180] The arbitration notification is a notification indicating that the second site is offline, determined by the third-party arbitration server according to the arbitration request sent by the first site, and the health status information and priority arbitration information corresponding to the second site obtained from the second site.

[0181] In an optional implementation, the second switching module is further configured to forcibly switch its own working mode to the single-ended operation working mode if no switching instruction is received within the third threshold time.

[0182] In an optional embodiment, the above device further includes:

[0183] A response module is used to feed back response information corresponding to the second operation instruction to the second site and record the response processing time when receiving the second operation instruction sent by the second site, and the second operation instruction is used to instruct the first site to synchronize data.

[0184] The response processing duration is the time interval between receiving the second operation instruction sent by the second site and feeding back response information corresponding to the second operation instruction to the second site.

[0185] A comparison module is used to compare the response processing time with a fourth threshold time when receiving an offline instruction sent by the second site.

[0186] The third switching module is used to switch its own working mode to the offline mode when it is determined that the response processing time is greater than the fourth threshold time.

[0187] The feedback module is used to feed back indication information of its own offline success to the second site when its own working mode is the offline mode.

[0188] The third sending module is configured to send the priority arbitration information to the third-party arbitration server when it is determined that the response processing time is not greater than the fourth threshold time.

[0189] The priority arbitration information is used to indicate the priority of the first station switching to a single-ended operation mode when both the first station and the second station are in a healthy state.

[0190] a fourth switching module, configured to, upon receiving an arbitration failure notification sent by the third-party arbitration server, switch its own working mode to the offline mode, and detect an offline duration of the switching process of switching its own working mode to the offline mode, wherein the arbitration failure notification is a notification sent by the third-party arbitration server when it is determined that the arbitration of the first site has failed based on the priority arbitration information sent by the first site and the priority arbitration information corresponding to the second site obtained from the second site.

[0191] The first judgment module is used to judge whether the offline duration exceeds a preset arbitration process duration.

[0192] When the arbitration process is detected to be in offline mode within the arbitration process duration, an offline completion notification is sent to the third-party arbitration server.

[0193] The fifth switching module is used to switch its own working mode to a forced offline mode when the offline duration exceeds the arbitration process duration.

[0194] In an optional embodiment, the above device further includes:

[0195] The second arbitration management module is configured to send an arbitration request to a third-party arbitration server when detecting that no indication information indicating that the second site is offline successfully fed back by the second site is received within a second threshold time.

[0196] The arbitration request includes: its own health status information and priority arbitration information, the own health status information indicates that the function of processing operation instructions internally is in a healthy state, and the priority arbitration information is used to indicate the priority of the first site switching to a single-ended operation mode when both the first site and the second site are in a healthy state.

[0197] The sixth switching module is configured to switch its own working mode to the offline mode upon receiving the arbitration failure notification sent by the third-party arbitration server.

[0198] The second judgment module is used to detect the offline duration of the switching process of its own working mode to the offline mode, and judge whether the offline duration exceeds the preset arbitration process duration.

[0199] The seventh switching module is used to switch its own working mode to a forced offline mode when the offline duration exceeds the arbitration process duration.

[0200] The fourth sending module is used to send an offline completion notification to the third-party arbitration server when it is detected that the arbitration server is in offline mode within the arbitration process duration.

[0201] In an optional embodiment, the above device further includes:

[0202] The encryption module is used to encrypt the transmitted data using a dynamic encryption algorithm during the data transmission process. The encryption algorithm is automatically adjusted according to the real-time network environment and data sensitivity.

[0203] The access module is used to perform identity authentication and permission checks every time data is accessed to ensure that only authorized users can access sensitive data.

[0204] The alarm module is used to generate data access logs regularly and automatically trigger alarms when abnormal access attempts occur, record the event and notify the system administrator.

[0205] The present invention provides a fault isolation device for a symmetric active-active disaster recovery system, which has the following advantages:

[0206] First, the first operation instruction is sent to the second site through the first site, and the corresponding response information is waited for the second site to recover. If the response information of the second site for the above-mentioned first operation instruction is not received within the first threshold time, the first site can determine that the second site response has timed out. Then the first site needs to determine whether the process of processing the operation instruction by itself is normal, that is, whether it is in the above-mentioned healthy state. After determining that it is in a healthy state, the first site determines that it has the ability to run on a single end. Then send an offline instruction to the second site, instruct the above-mentioned second site to go offline, and wait for the second site to respond to the offline instruction. After receiving the indication information fed back by the second site to determine that the second site is successfully offline, the first site can switch its own working mode to a single-end operation mode. Through the above scheme, when the first site finds that the second site processes the first operation instruction, it verifies its own health state to a limited extent, and when it determines that it is in a healthy state, it instructs the second site to go offline, and after determining that the second site is successfully offline, it switches its own working mode to a single-end operation mode, thereby avoiding the fault problem of the second site affecting the data synchronization between the first site and the second site, thereby achieving the technical effect of fault isolation, and ensuring the data security and business stability of the first site.

[0207] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0208] The fault isolation device of the symmetric active-active disaster recovery system in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0209] The embodiment of the present invention also provides a computer device having the above Figure 3 The fault isolation device of the symmetric active-active disaster recovery system shown.

[0210] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0211] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include an integrated circuit. The integrated circuit may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0212] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0213] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0214] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0215] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0216] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0217] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A fault isolation method for a symmetric active-active disaster recovery system, characterized in that: The method is applied in a symmetric active-active system application scenario. The method is performed by a first site in the symmetric active-active disaster recovery system. The first site is any site in the symmetric active-active disaster recovery system. The method includes: Sending a first operation instruction to a second site, where the second site is a site other than the first site in the symmetric active-active disaster recovery system, and the first operation instruction is used to instruct the second site to synchronize data; In the case where it is detected that the response information corresponding to the first operation instruction sent by the second station is not received within the first threshold time, determining that the second station response times out; Check whether the process of processing operation instructions is in a healthy state; When it is determined that the process of processing the operation instruction by itself is in a healthy state, sending an offline instruction to the second site to instruct the second site to go offline; When receiving the indication information that the second site is offline successfully fed back by the second site, the working mode thereof is switched to the single-ended operation working mode.

2. The method according to claim 1, characterized in that In a case where it is detected that the response information corresponding to the first operation instruction sent by the second station is not received within a first threshold time, determining that the response of the second station is timed out includes: Inputting the first operation instruction and the corresponding sending time into a waiting sequence to wait for a response information corresponding to the first operation instruction; Based on the sending time, determining a waiting time for the first operation instruction to enter the waiting sequence; Comparing the waiting time with the first threshold time; When the waiting time is greater than or equal to the first threshold time, it is determined that response information corresponding to the first operation instruction sent by the second station is not received within the first threshold time.

3. The method according to claim 2, characterized in that Check whether the process of processing operation instructions is in a healthy state, including: Acquire the self-operation instruction processing information, wherein the self-operation instruction processing information includes: the response time of responding to the second site, the delay time of the self-processing operation instruction, and the self-volume status; Detect whether the reaction time and the stall time have timed out or the volume status itself is abnormal; When it is determined that the reaction time and the stall time are not timed out and there is no abnormality in the state of the self-volume, it is determined that the process of processing the operation instruction itself is in a healthy state.

4. The method according to any one of claims 1 to 3, characterized in that: Also includes: When it is detected that no indication information indicating that the second site is offline successfully fed back by the second site within a second threshold time is received, sending an arbitration request to a third-party arbitration server; The arbitration request includes: its own health status information and priority arbitration information; the own health status information indicates that the function of processing operation instructions within itself is in a healthy state, and the priority arbitration information is used to indicate the priority of the first site switching to the single-ended operation working mode when both the first site and the second site are in a healthy state; When receiving the arbitration notice sent by the third-party arbitration server, starting the timer; When a switching instruction is received within a third threshold time, the operating mode of the device is switched to the single-ended operating mode; The arbitration notification is a notification indicating that the second site is offline, determined by the third-party arbitration server according to the arbitration request sent by the first site, and the health status information and priority arbitration information corresponding to the second site obtained from the second site.

5. The method according to claim 4, characterized in that Switching its own working mode to the single-ended operation mode includes: If no switching instruction is received within the third threshold time, the operating mode of the device is forcibly switched to the single-ended operating mode.

6. The method according to any one of claims 1 to 3, characterized in that: Also includes: When receiving a second operation instruction sent by the second site, feeding back response information corresponding to the second operation instruction to the second site and recording a response processing time, wherein the second operation instruction is used to instruct the first site to synchronize data; The response processing time is the time interval between receiving the second operation instruction sent by the second site and feeding back the response information corresponding to the second operation instruction to the second site; When receiving the offline instruction sent by the second site, comparing the response processing time with a fourth threshold time; If it is determined that the response processing time is longer than the fourth threshold time, switching the working mode to an offline mode; When the working mode of the station is the offline mode, feeding back to the second station an indication that the station is offline successfully; Alternatively, when it is determined that the response processing time is not greater than the fourth threshold time, priority arbitration information is sent to a third-party arbitration server, where the priority arbitration information is used to indicate the priority of the first site switching to a single-ended operation working mode when both the first site and the second site are in a healthy state; Upon receiving the arbitration failure notification sent by the third-party arbitration server, switching its own working mode to the offline mode, and detecting the offline duration of the switching process of its own working mode to the offline mode, wherein the arbitration failure notification is a notification sent by the third-party arbitration server when it is determined that the arbitration of the first site has failed based on the priority arbitration information sent by the first site and the priority arbitration information corresponding to the second site obtained from the second site; Determining whether the offline duration exceeds a preset arbitration process duration; When detecting that the arbitration server is in offline mode within the arbitration process duration, sending an offline completion notification to the third-party arbitration server; Alternatively, when the offline duration exceeds the arbitration process duration, the working mode thereof is switched to a forced offline mode.

7. The method according to any one of claims 1 to 3, characterized in that: Also includes: When it is detected that no indication information indicating that the second site is offline successfully fed back by the second site within a second threshold time is received, sending an arbitration request to a third-party arbitration server; The arbitration request includes: its own health status information and priority arbitration information, wherein the own health status information indicates that the function of processing the operation instruction internally is in a healthy state, and the priority arbitration information is used to indicate the priority of the first station switching to the single-ended operation working mode when both the first station and the second station are in a healthy state; Upon receiving an arbitration failure notification sent by the third-party arbitration server, switching its own working mode to an offline mode; Detecting the offline duration of the switching process of switching its own working mode to the offline mode, and determining whether the offline duration exceeds the preset arbitration process duration; When the offline duration exceeds the arbitration process duration, the working mode is switched to a forced offline mode; When the arbitration process is detected to be in offline mode within the arbitration process duration, an offline completion notification is sent to the third-party arbitration server.

8. A fault isolation device for a symmetric active-active disaster recovery system, characterized in that: The device is applied in a symmetric active-active system application scenario. The device is used in a first site in the symmetric active-active disaster recovery system. The first site is any site in the symmetric active-active disaster recovery system, and includes: A first sending module is used to send a first operation instruction to a second site, where the second site is a site other than the first site in the symmetric active-active disaster recovery system, and the first operation instruction is used to instruct the second site to synchronize data; A first detection module, configured to determine that a response of the second station has timed out when detecting that a response message corresponding to the first operation instruction sent by the second station is not received within a first threshold time; The second detection module is used to detect whether the process of processing the operation instruction itself is in a healthy state; A second sending module is used to send an offline instruction to the second site to instruct the second site to go offline when it is determined that the process of processing the operation instruction by itself is in a healthy state; The first switching module is configured to switch its own working mode to a single-ended operation working mode upon receiving indication information that the second station is offline successfully fed back by the second station.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the fault isolation method of the symmetric active-active disaster recovery system described in any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the fault isolation method of the symmetric active-active disaster recovery system according to any one of claims 1 to 7.

Images