Automatic disaster recovery backup switching method for voting simulation
By adopting the automatic disaster recovery switching method based on proxy simulation app in the system disaster recovery system, the business impact problem caused by manual switching is solved, and fast and reliable automatic disaster recovery switching is achieved, which reduces user downtime and improves the System reliability and fault analysis efficiency.
Patent Information
- Application Number
- CN202410980285.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-03
- Publication Date
- 2024-11-12
AI Technical Summary
In the system disaster recovery system, manually switching the production environment to the disaster recovery environment will cause short-term business impacts, and it is difficult for the existing technology to quickly realize automated switching, affecting database stability and recovery time goals.
Using the automatic disaster recovery switching method based on proxy simulation app, by deploying multiple proxy nodes and simulation apps, simulating user network conditions, calling check_health interface for read and write operations, collecting and analyzing error information, and automatically switching traffic when more than 70 % of the agents switch to the disaster recovery cluster when reporting errors and manually recover at low peaks.
It realizes fast and reliable automatic disaster recovery switching, reduces user downtime, improves system reliability, and reduces mutual interference through independent collection and switching procedures, helping to quickly analyze the causes of cluster failures.
Smart Images

Figure CN118945041A_ABST
Abstract
Description
[0001] This proposal is a division of the application number 2023100539497 entitled "Automatic disaster recovery switching method based on proxy simulation app voting" Technical Field
[0002] The present invention relates to the technical field of server side, and more specifically, to an automatic disaster recovery switching method for simulated voting. Background Art
[0003] When designing a system disaster recovery system, you need to consider the RTO indicator (Recovery Time Objective) to minimize the downtime of the production environment. In actual applications, database stability is the main consideration for the continuous provision of services in the production environment. When the production database is damaged, you need to manually switch the application route of the production environment to the disaster recovery environment. In the disaster recovery system, this manual switch will cause a certain business impact in a short period of time. Summary of the invention
[0004] The present invention overcomes the deficiencies of the prior art and provides an automatic disaster recovery switching method for simulated voting.
[0005] The technical solution of the present invention is as follows: An automatic disaster recovery switching method based on proxy simulation app voting specifically includes the following steps: 101) Steps for deploying proxy nodes: deploy multiple proxy nodes across the country, and deploy multiple simulation apps on each node for voting debugging; 102) Preprocessing step: The proxy program of the simulated app is started, and the voting information of each cluster is obtained from each node; 103) Voting steps: When simulating app voting, the check_health interface is called to simulate the business on the actual server to perform read and write operations; when an error occurs, in addition to voting, the simulation app will also upload the error results for cause analysis; Each node first collects and obtains the configuration of all clusters, then loops by cluster, checks each cluster in turn, and obtains an address for access after taking the remainder of the current number of addresses for each cluster. The inspection result is reported to the collection node. If the cluster inspection is good, wait appropriately according to the overall agreed time. 104) Switching steps: Based on the voting results, if there is a bad result in the voting of a cluster and it exceeds the threshold, the disaster recovery switching procedure automatically switches the traffic.
[0006] Furthermore, there are at least 8 server locations across the country, mainly in public cloud services in Guangzhou, Shanghai and Beijing.
[0007] Furthermore, after the cluster configuration information is read in step 102), the check starts, and all clusters are checked once every certain period of time, and each cluster calls the check_health interface in turn according to the set order.
[0008] Furthermore, each certain time interval is within 15 seconds.
[0009] Furthermore, the voting results of the simulated app are stored in the database of the collection node. The switching program periodically checks the voting results. If more than 70% of the agents in a cluster report errors, it is considered that there is a problem with this cluster and the backend service needs to be switched to the disaster recovery cluster on the other end.
[0010] Furthermore, the switching procedure includes an error switching function. When there are fewer than 3 reporting agents, only an alarm is issued without switching. The expected number of agents is n, and n is greater than 5. When the actual number of reports is insufficient, it is determined that the network is not good and a manual disaster recovery switching suggestion is issued.
[0011] Furthermore, the periodic check is performed once every 5 seconds.
[0012] Furthermore, it also includes 105) recovery step: after the switching action of the disaster recovery switching program occurs, a switching flag is set and relevant personnel are notified, and manual switching back is performed during off-peak hours.
[0013] Compared with the prior art, the present invention has the following advantages: the present invention simulates the actual network situation of users, and multiple agents ensure the reliability of voting. The switching speed is much faster than manual, and the user downtime is reduced as much as possible. The collection program and the switching program are independent and will not interfere with each other, thereby improving the reliability of the system. The reason for the proxy node uploading error helps to quickly analyze the cause of the cluster failure. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 It is a schematic diagram of the present invention. DETAILED DESCRIPTION
[0015] The embodiments of the present invention are described in detail below, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and cannot be used as a limitation of the present invention.
[0016] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.
[0017] The numbers of the steps mentioned in each embodiment are only for the convenience of description and have no substantial sequence relationship. Different steps in each specific embodiment can be combined in different sequences to achieve the purpose of the invention.
[0018] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.
[0019] like Figure 1 As shown, an automatic disaster recovery switching method based on proxy simulation app voting specifically includes the following steps: 101) Steps for deploying proxy nodes: deploy multiple proxy nodes across the country, and deploy multiple simulation apps on each node for voting debugging; among them, there are at least 8 server points across the country, mainly distributed on public cloud services in Guangzhou, Shanghai and Beijing.
[0020] 102) Preprocessing step: The proxy program of the simulated app is started, and the voting information of each cluster is obtained from each node; after reading the cluster configuration information, the check is started, and all clusters are checked once every certain interval, and each cluster calls the check_health interface in the set order. Each certain interval is generally within 15 seconds.
[0021] 103) Voting steps: When simulating app voting, the check_health interface is called to simulate the business on the actual server to perform read and write operations; when an error occurs, in addition to voting, the simulation app will also upload the error results for cause analysis; The voting results of the simulated app are stored in the database of the collection node. The switching program regularly checks the voting results. If more than 70% of the agents in a cluster report errors, it is considered that there is a problem with this cluster and the backend service needs to be switched to the disaster recovery cluster on the other end. The switching program includes an error switching function. When there are fewer than 3 reporting agents, only an alarm is issued without switching. The expected number of agents is n, and n is greater than 5. When the actual number of reports is insufficient, it is determined that the network is not good and a manual disaster recovery switching suggestion is issued. Regular checks are performed every 5 seconds, or less, to speed up the time to query problems and reduce business impact.
[0022] Each node first collects and obtains the configuration of all clusters, then loops by cluster, checks each cluster in turn, and obtains an address for access after taking the remainder of the current number of addresses for each cluster. The inspection result is reported to the collection node. If the cluster inspection is good, wait appropriately according to the overall agreed time. 104) Switching steps: According to the voting results, if there is a bad result in the voting of a cluster and it exceeds the threshold, the disaster recovery switching program automatically switches the traffic. The collection program and the switching program are independent of each other and will not interfere with each other, thereby improving system reliability.
[0023] It also includes 105) recovery steps: after the disaster recovery switching program switching action occurs, set the switching flag and notify relevant personnel. Generally, it is manually switched back during off-peak hours. Naturally, it can also be manually switched during peak hours according to actual needs. It depends on the actual situation.
[0024] The specific method called by the proxy node is ClusterAgent, which implements the judgment and detection of the cluster. First, obtain the configuration of all clusters from the collection node, loop by cluster, and check each cluster in turn. For each cluster, multiple addresses are obtained by taking the remainder of the current number of times to access an address, and the inspection result is reported to the collection node. If there is an error, the error log will also be reported. If all clusters are good, wait appropriately according to the overall agreed time.
[0025] class ClusterAgent(ClusterHealthBase): def chk_all(self): self.set_dict_c_url() i=0 # Total number of runs max1 = 15 # Overall running time, seconds sleep1 = 0.01 # sleep for a number of seconds after a cluster check is completed exit = False while (exit != True): count_ko = 0 # Number of unhealthy clusters start = time.time() for (c_code, urls) in self.dict_c_url.items(): try: chk_ok = self.chk_cluster(c_code, urls, i) self.report(c_code, chk_ok) if chk_ok == False: count_ko += 1 logger.info(c_code+'='+str(chk_ok)+',count_ko='+str(count_ko)) time.sleep(sleep1) except SystemExit: logger.info('exit') exit = True end = time.time() i += 1 # All cluster checks are good, wait for the agreed minimum time logger.info('wait,max1='+str(max1)+' seconds') while end-start <max1: end = time.time() time.sleep(1) Among them, the collection node obtains the configuration of all clusters, and obtains json similar to {"Cluster A": ["https: / / domain name 1: port 1 / slh / checkCluster.do?kind=checkHealth&serverApplicationName=stsServer", "http: / / ip2: port 2 / slh / checkCluster.do?kind=checkHealth&serverApplicationName=stsServer"],"Cluster B": ["https: / / domain name 1: port 1 / slh / check.do?svc=checkHealth&db=true&redis=true&thread=true", "https: / / ip2: port 2 / slh / check.do?svc=checkHealth&db=true&redis=true&thread=true", "https: / / ip3: port 3 / slh / check.do?svc=checkHealth&db=true&redis=true&thread=true"],...}. Each cluster has multiple access addresses to prevent false alarms of single line failures.
[0026] serverApplicationName=stsServer means that dubbo calls need to be checked; db=true means that database read and write need to be checked; redis=true means that cache read and write need to be checked; thread=true means that threads need to be checked for excess.
[0027] The switching program first obtains a list of all cluster codes that need to be checked from the collection node database, then checks each cluster in turn, and then checks whether this cluster needs to start disaster recovery. If more than 70% of the agents in a cluster report errors, it is considered that this cluster has problems and needs to switch the backend service to the disaster recovery cluster on the other end. In order to prevent accidental switching, the switching program will do some checks. For example, if there are fewer than 3 reporting agents, only an alarm will be issued without switching. For example, if the expected number of agents is 8, but the actual number of reports is insufficient, it means that the network is not good and cannot be reported. Manual switching is recommended. Then, if a cluster starts disaster recovery, all clusters will start a new round of checks, and the already started clusters will be automatically skipped. The results of the disaster recovery will be notified to the corresponding group for subsequent manual processing. The specific method used is ClusterGuard: class ClusterGuard(ClusterHealthBase): def chk_all(self, mod_result): # self.set_dict_c_url() mod = 3 max1 = 10 dao = DeployDAO() exit = False while (exit != True): start119 = False list_agent_err = [] start = time.time() codes = dao.get_cluster119_need_chk_codes(mod, mod_result) for c_code in codes: try: start119 = self.chk_cluster(c_code, dao, list_agent_err) time.sleep(0.1) if start119: # If a cluster starts disaster recovery, exit immediately break except SystemExit: print('exit') exit = True self.ding_all(list_agent_err) time.sleep(3) In summary, this solution simulates the actual network conditions of users, and multiple proxies ensure the reliability of voting. The switching speed is much faster than manual, which minimizes the user downtime. The collection program and the switching program are independent and will not interfere with each other, which improves the reliability of the system. The reason for the proxy node uploading error helps to quickly analyze the cause of the cluster failure.
[0028] The above is only a preferred embodiment of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and modifications without departing from the concept of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.
Claims
1. An automatic disaster recovery switching method for simulated voting, characterized in that: The specific steps include: 101) Steps for deploying proxy nodes: deploy multiple proxy nodes across the country, and deploy multiple simulation apps on each node for voting debugging; 102) Preprocessing step: The proxy program of the simulated app is started, and the voting information of each cluster is obtained from each node; After reading the cluster configuration information, the check starts and all clusters are checked once every certain period of time. Each cluster calls the check_health interface in the set order. 103) Voting steps: When simulating app voting, the check_health interface is called to simulate the business on the actual server to perform read and write operations; when an error occurs, in addition to voting, the simulation app will also upload the error results for cause analysis; Each node first collects and obtains the configuration of all clusters, then loops by cluster, checks each cluster in turn, and obtains an address for access after taking the remainder of the current number of addresses for each cluster. The inspection result is reported to the collection node. If the cluster inspection is good, wait appropriately according to the overall agreed time. 104) Switching steps: According to the voting results, if there is a bad result in the voting of a cluster and it exceeds the threshold, the disaster recovery switching program automatically switches the traffic; 105) Recovery steps: After the disaster recovery switching program switching action occurs, set the switching flag and notify relevant personnel, and manually switch back during off-peak hours; The voting results of the simulated app are stored in the database of the collection node. The switching program periodically checks the voting results. If more than 70% of the agents in a cluster report errors, it is considered that there is a problem with the cluster and the backend service needs to be switched to the disaster recovery cluster on the other end. The switching procedure includes an error switching function. When there are fewer than 3 alarm agents, only an alarm will be issued without switching. The expected number of agents is n, and n is greater than 5. When the actual number of reports is insufficient, it is determined that the network is not good and a manual disaster recovery switching suggestion is issued.
2. The automatic disaster recovery switching method for simulated voting according to claim 1 is characterized in that: There are at least 8 server locations across the country, mainly in public cloud services in Guangzhou, Shanghai and Beijing.
3. The automatic disaster recovery switching method for simulated voting according to claim 1 is characterized in that: Each interval is within 15 seconds.
4. The automatic disaster recovery switching method for simulated voting according to claim 1 is characterized in that: Check periodically every 5 seconds.
