A disaster recovery switching device for automated operation and maintenance system

Abstract

The present invention discloses a disaster recovery switching device for an automated operation and maintenance system. The database restore point of the device improves the recovery efficiency of data; it considers the preservation of data such as components and processes being edited, reduces the possibility of data loss, and improves the reliability of disaster recovery switching; and at the same time solves the problem of proxy connection efficiency, greatly improving the efficiency of disaster recovery switching. The custom-configured monitoring method of the present invention improves user experience, flexibility and scalability, and at the same time solves the drawback of the traditional method of relying on the network to determine whether a disaster has occurred in a data center, reduces the misjudgment caused by occasional network fluctuations, and improves the accuracy of disaster judgment.

Description

Technical Field

[0001] The present invention relates to the field of operation and maintenance systems, and in particular to a disaster recovery switching device for an automated operation and maintenance system. Background Art

[0002] With the rapid development of IT technology, most companies currently use IT technology to support the normal operation of business systems, and the architecture of the business system itself is becoming more and more complex, involving more and more operational links and related personnel, especially in the financial industry. In order to meet the operation and maintenance management needs of the financial industry, ensure the quality of operation and maintenance management, and reduce operation and maintenance costs, most financial industries have used automated operation and maintenance systems to assist in the internal operation and maintenance management of the enterprise. Therefore, when a disaster occurs, it will become particularly important to ensure the availability and continuity of the automated operation and maintenance system.

[0003] In this context, the current automated operation and maintenance system faces the following problems when responding to disasters:

[0004] 1. Manual operation is cumbersome and costly: Due to the complexity of the system architecture, the operation and maintenance personnel need to be very familiar with the system architecture, otherwise they cannot respond to disasters quickly and efficiently. In addition, the operation and maintenance personnel need to perform a series of operations when performing disaster switching, which consumes a lot of human resources and costs. In addition, manual judgment and disaster recovery switching operations are prone to errors, which affects the stability of the system.

[0005] 2. Disaster recovery switching takes a long time: On the one hand, the complexity of the system and the tediousness of manual operation lead to a long time for disaster recovery switching. On the other hand, the automated operation and maintenance system has a large amount of data. When disaster recovery occurs, data recovery takes a long time. Furthermore, the automated operation and maintenance system relies on the agent to execute various business needs. When a disaster occurs, the connection processing of the agent takes a long time. All of the above reasons have greatly reduced the availability and continuity of the automated operation and maintenance system.

[0006] 3. Single switching strategy: Currently, the disaster recovery switching operation of the automated operation and maintenance system can only be performed when a disaster actually occurs. The user subjectively judges and manually executes which backup data center to switch to in order to ensure the normal operation of the automated operation and maintenance system. This single switching judgment strategy can only be determined by human subjectivity, and cannot form an automatic and objective judgment, thus losing objectivity.

[0007] 4. Poor data consistency: When a disaster occurs while a user is performing editing operations on components or processes, the unsaved data being edited will be lost, causing the user to re-edit it, resulting in poor data consistency and business continuity.

[0008] Under the above background, there are currently some disaster recovery monitoring and switching methods, but none of them can solve all the above problems at the same time. Therefore, a disaster recovery switching device for an automated operation and maintenance system is needed to solve the problems mentioned above. Summary of the invention

[0009] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a disaster recovery switching device for an automated operation and maintenance system.

[0010] The purpose of the present invention is achieved through the following technical solutions:

[0011] A disaster recovery switching device for an automated operation and maintenance system comprises a data acquisition and analysis module, a disaster recovery scheme selection module, a disaster recovery switching execution module, a disaster recovery switching verification module, and a disaster recovery switching data recovery module; wherein:

[0012] The data collection and analysis module is used to obtain the environmental data of the automated operation and maintenance system and determine whether a disaster has occurred in the current data center based on preset judgment rules;

[0013] The disaster recovery plan selection module is used to determine which backup data center will be switched to when a disaster occurs. When the automatic switching task is configured, the backup data center is obtained from the automated operation and maintenance system configuration. Otherwise, the user determines the backup data center by clicking.

[0014] The disaster recovery switching execution module is used to execute disaster recovery switching, switch the data center from the primary data center to the backup data center, and perform a series of switching operations, including database state change, database restore point setting, and firewall setting;

[0015] The disaster recovery switch verification module is used to execute the verification process after the disaster recovery switch, check whether the process, components, tasks are normal, and whether the connection of the agent is normal, confirm that the disaster recovery switch is executed correctly, and the backup data center is currently available;

[0016] The disaster recovery switching data recovery module is used to determine whether the original main data center has returned to normal based on the data from the data collection and analysis module. If it has returned to normal, it will automatically or manually switch back to the production data center.

[0017] In the disaster recovery plan selection module, the automatic switching task is that the user configures the disaster recovery connection relationship of each data center in advance, that is, when a disaster occurs in the current data center, which backup data center will be used to continue operation.

[0018] The specific working process of the disaster recovery solution selection module is as follows:

[0019] Step 1: The automated operation and maintenance system monitors several indicators of the production environment in real time and calculates the status value of the production data center. If the status value of the production data center is greater than a preset value, the current production data center is judged to be in an abnormal state and the process goes to step 2.

[0020] Step 2: Determine whether the automatic operation and maintenance system is configured with an automatic switching task. If so, the system automatically obtains the disaster recovery connection relationship of the current data center and performs a disaster recovery switching operation, so that the automatic operation and maintenance system switches to the backup data center. Otherwise, the user subjectively selects which backup data center to switch to and clicks the switch button for that data center.

[0021] When actually performing disaster recovery switching, the time point when the disaster occurred is first set as the database restore point. The setting method is that the management system of the automated operation and maintenance system first cancels the recovery operation of the managed database currently in progress, then creates a recovery point and specifies the recovery database, and finally disconnects the connection between the primary database and the backup database to improve the efficiency of data recovery after disaster recovery. Data recovery only starts from the restore point, and the execution data within the disaster time period is incrementally restored to the production database, greatly shortening the data recovery time;

[0022] Then the management system of the automated operation and maintenance system changes the "Quick Connection" setting of the server to the enabled state, which is used by the agent to quickly connect to the server and ensure that the business can be executed normally using the agent; then the firewall configuration of the operating system and server is turned on to intercept all requests sent to the production data center and return the interception information to the sender of the request; then the recovery operation of the currently ongoing managed database is canceled and the standby data center is activated, so that the database of the standby data center becomes readable and writable, making preliminary preparations for running the business in the standby data center in the future, and writing the operating data to the standby database.

[0023] In step 1, the state value Q of the production data center is calculated as follows:

[0024] If the system administrator has no special requirements for the five monitoring indicators of memory usage, network latency, CPU usage, thread usage, and number of online servers, the default algorithm of the automated operation and maintenance system is used to determine the status of the production data center:

[0025] Q=0.18x+0.3y+0.15z+0.12i+0.25j;

[0026] Among them, the memory ratio is x, the network delay is y, the CPU ratio is z, the number of threads is i, and the number of online servers is j;

[0027] If the system administrator has special requirements for the five monitoring indicators of memory share, network latency, CPU share, thread share, and number of online servers, the administrator can customize the indicator critical value, indicator occurrence number, and indicator weight to calculate the status value Q of the new production data center.

[0028] The agent of the automated operation and maintenance system is started in a "quick connection" manner, that is, first ensuring that the agent can connect to the server quickly and stably, and then processing the connection check of the agent in an asynchronous manner.

[0029] Assuming that the database of the automated operation and maintenance system is created at time point t0, then when no disaster occurs, the data is synchronized from the primary data center to the backup data center in real time, and the data of the two data centers remain synchronized; when a disaster occurs at time point t2, the primary data center no longer synchronizes data to the backup data center, and time point t2 is used as the database restore point; during the disaster period from t2 to t6, the automated operation and maintenance system has switched to the backup data center, and all generated operation data is in the backup data center; when the disaster is recovered normally at time point t6, the operation data in the time period from t2 to t6 is restored to the primary data center, and from then on, the data continues to be synchronized from the primary data center to the backup data center in real time.

[0030] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0031] 1. The present invention uses database restore points to improve data recovery efficiency.

[0032] 2. The present invention takes into account the preservation of data such as components and processes being edited, reduces the possibility of data loss, and improves the reliability of disaster recovery switching.

[0033] 3. The present invention solves the problem of proxy connection efficiency and greatly improves the efficiency of disaster recovery switching.

[0034] 4. The custom-configured monitoring method of the present invention improves user experience, flexibility and scalability, while solving the drawbacks of the traditional method of relying on the network to determine whether a disaster has occurred in the data center, reducing misjudgments caused by occasional network fluctuations and improving the accuracy of disaster judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is an architectural diagram of the automated operation and maintenance system described in the present invention.

[0036] Figure 2 This is a multi-data center structure diagram of the automated operation and maintenance system of the present invention.

[0037] Figure 3It is a structural schematic diagram of the disaster recovery switching device for the automated operation and maintenance system according to the present invention.

[0038] Figure 4 This is a schematic diagram of restoring data to a primary data center according to the present invention.

[0039] Figure 5 This is a flow chart of the disaster recovery switching described in the present invention. DETAILED DESCRIPTION

[0040] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0041] 1. Introduction to the disaster recovery architecture of the automated operation and maintenance system.

[0042] like Figure 1 , first introduce the architecture of the automated operation and maintenance system (such as Figure 1 As shown in the figure, it consists of three parts: server, control, and agent. The server is used to provide automated operation and maintenance services and related functions. It is deployed in clusters and uses redis clusters to improve read and write capabilities. The database type supports multiple types such as mysql, oracle, and Gauss. This article takes oracle as an example and adopts the mode of real-time synchronization between the main database and the standby database. The control connects to the server and is the user's operation page. You can edit components, processes, tasks, etc. in the control console. The agent connects to the server and is mainly responsible for executing tasks and processes. Only when the agent is stably connected to the server can the normal execution of tasks and processes be guaranteed.

[0043] like Figure 2 , the automated operation and maintenance system can deploy a production data center (i.e., the main data center) and multiple backup data centers, and the production data center and each backup data center can be connected to each other for disaster recovery. Under normal circumstances, the automated operation and maintenance system runs in the production data center, and other backup data centers only serve as backup operating environments. Data is synchronized from the production data center to each backup data center in real time.

[0044] 2. Disaster event judgment model.

[0045] The present invention establishes a disaster event judgment model. Through the automated operation and maintenance system, the five indicators of memory share, network delay, CPU share, thread share, and number of online servers of the production data center are obtained and calculated every 5 seconds. Combined with the number of indicator occurrences and indicator weights, it is calculated whether the current production data center is in an abnormal state. If it is in an abnormal state, a switching operation is performed; otherwise, the status quo is maintained.

[0046] Implementation case 1: If the system administrator has no special requirements for the five monitoring indicators, the system's own algorithm is used by default to determine the status of the production data center. The specific algorithm is shown below.

[0047] For memory usage, when the percentage is in the range of [0,60%] and the number of occurrences is less than or equal to 5 times, it is judged as a normal alarm level; when the percentage is in the range of [0,60%] and the number of occurrences is greater than 5 times, it is judged as a general alarm level; when the percentage is in the range of (60%,80%] and the number of occurrences is less than or equal to 5 times, it is judged as a general alarm level; when the percentage is in the range of (60%,80%] and the number of occurrences is greater than 5 times, it is judged as a minor alarm level; when the percentage is in the range of (80%,100%] and the number of occurrences is less than or equal to 5 times, it is judged as a minor alarm level; when the percentage is in the range of (80%,100%] and the number of occurrences is greater than 5 times, it is judged as a serious alarm level.

[0048] Table 1 Memory usage alarm levels

[0049]

[0050] For network delay, when the network delay is in the range of [0ms, 50ms] and the number of occurrences is less than or equal to 10 times, it is judged as a normal alarm level; when the network delay is in the range of [0ms, 50ms] and the number of occurrences is greater than 10 times, it is judged as a general alarm level; when the network delay is in the range of (50ms, 100ms] and the number of occurrences is less than or equal to 10 times, it is judged as a general alarm level; when the network delay is in the range of (50ms, 100ms] and the number of occurrences is greater than 10 times, it is judged as a minor alarm level; when the network delay is in the range of (100ms, +∞) and the number of occurrences is less than or equal to 10 times, it is judged as a minor alarm level; when the network delay is in the range of (100ms, +∞) and the number of occurrences is greater than 10 times, it is judged as a serious alarm level.

[0051] Table 2 Network delay alarm levels

[0052]

[0053] For the CPU ratio, when the ratio is in the range of [0,60%] and the number of occurrences is less than or equal to 8 times, it is judged as a normal alarm level; when the ratio is in the range of [0,60%] and the number of occurrences is greater than 8 times, it is judged as a general alarm level; when the ratio is in the range of (60%,80%] and the number of occurrences is less than or equal to 8 times, it is judged as a general alarm level; when the ratio is in the range of (60%,80%] and the number of occurrences is greater than 8 times, it is judged as a minor alarm level; when the ratio is in the range of (80%,100%] and the number of occurrences is less than or equal to 8 times, it is judged as a minor alarm level; when the ratio is in the range of (80%,100%] and the number of occurrences is greater than 8 times, it is judged as a serious alarm level.

[0054] Table 3 CPU usage alarm levels

[0055]

[0056] For the thread number ratio, when the ratio is in the range of [0,60%] and the number of occurrences is less than or equal to 8 times, it is judged as a normal alarm level; when the ratio is in the range of [0,60%] and the number of occurrences is greater than 8 times, it is judged as a general alarm level; when the ratio is in the range of (60%,80%] and the number of occurrences is less than or equal to 8 times, it is judged as a general alarm level; when the ratio is in the range of (60%,80%] and the number of occurrences is greater than 8 times, it is judged as a minor alarm level; when the ratio is in the range of (80%,100%] and the number of occurrences is less than or equal to 8 times, it is judged as a minor alarm level; when the ratio is in the range of (80%,100%] and the number of occurrences is greater than 8 times, it is judged as a serious alarm level.

[0057] Table 4 Thread ratio alarm level

[0058]

[0059] For the number of servers online, when the number of online servers is not zero and the number of occurrences is greater than or equal to 2 times, it is judged as a normal alarm level; when the number of online servers is not zero and the number of occurrences is less than 2 times, it is judged as a general alarm level; when the number of online servers is zero and the number of occurrences is greater than or equal to 2 times, it is judged as a serious alarm level; when the number of online servers is zero and the number of occurrences is less than 2 times, it is judged as a minor alarm level.

[0060] Table 5 Alarm levels for online server numbers

[0061]

[0062] Assume that the memory ratio is x, the network latency is y, the CPU ratio is z, the thread ratio is i, the number of online servers is j, and the weights of these indicators are 0.18, 0.3, 0.15, 0.12, and 0.25 respectively, and the sum of the weights is 1. Then the state model Q of the production data center is established as:

[0063] Q=0.18x+0.3y+0.15z+0.12i+0.25j;

[0064] The automated operation and maintenance system obtains and calculates the data of five indicators of the production data center every 5 seconds: memory share, network latency, CPU share, thread share, and online server number. It also determines the alarm level of the event based on the number of occurrences of the indicators. If the alarm level of the event is severe, the variable corresponding to the indicator is 4; if the alarm level of the event is minor, the variable corresponding to the indicator is 3; if the alarm level of the event is general, the variable corresponding to the indicator is 2; if the alarm level of the event is normal, the variable corresponding to the indicator is 1. Then, substitute it into Q to calculate the status value of the production data center. If the status value Q is greater than 3.6, it is determined that the production data center is in an abnormal state and a disaster recovery switch operation is required. For example: According to the rules in the previous article, the alarm level of memory usage is severe, the alarm level of network delay is minor, the alarm level of CPU usage is general, the alarm level of thread number usage is normal, and the alarm level of online server number is normal. Then Q=0.18*4+0.3*3+0.15*2+0.12*1+0.25*1=2.29. If the value is less than 3.6, the status of the data center is normal and no disaster recovery switching is required.

[0065] Implementation case 2: If the system administrator has special requirements for the five monitoring indicators, the administrator can customize the indicator critical value, indicator occurrence frequency and indicator weight. Assume that the information configured by the system administrator is shown in the following table.

[0066] Table 6 Custom indicator configuration table

[0067]

[0068] Based on the indicator critical values ​​and indicator occurrence times customized by the system administrator, the rules for the automated operation and maintenance system to determine the event alarm level are as follows.

[0069] For memory usage, when the percentage is in the range of [0,70%] and the number of occurrences is less than or equal to 5 times, it is judged as a normal alarm level; when the percentage is in the range of [0,70%] and the number of occurrences is greater than 5 times, it is judged as a general alarm level; when the percentage is in the range of (70%,100%] and the number of occurrences is less than or equal to 5 times, it is judged as a minor alarm level; when the percentage is in the range of (70%,100%] and the number of occurrences is greater than 5 times, it is judged as a severe alarm level.

[0070] Table 7 Memory usage alarm levels

[0071]

[0072] For network delay, when the network delay is in the range of [0ms, 50ms] and the number of occurrences is less than or equal to 10 times, it is judged as a normal alarm level; when the network delay is in the range of [0ms, 50ms] and the number of occurrences is greater than 10 times, it is judged as a general alarm level; when the network delay is in the range of (50ms, +∞) and the number of occurrences is less than or equal to 10 times, it is judged as a minor alarm level; when the network delay is in the range of (50ms, +∞) and the number of occurrences is greater than 10 times, it is judged as a severe alarm level.

[0073] Table 8 Network delay alarm levels

[0074]

[0075] For the CPU ratio, when the ratio is in the range of [0,65%] and the number of occurrences is less than or equal to 5 times, it is judged as a normal alarm level; when the ratio is in the range of [0,65%] and the number of occurrences is greater than 5 times, it is judged as a general alarm level; when the ratio is in the range of (65%,100%] and the number of occurrences is less than or equal to 5 times, it is judged as a minor alarm level; when the ratio is in the range of (65%,100%] and the number of occurrences is greater than 5 times, it is judged as a severe alarm level.

[0076] Table 9 CPU usage alarm levels

[0077]

[0078] For the thread number ratio, when the ratio is in the range of [0,68%] and the number of occurrences is less than or equal to 5 times, it is judged as a normal alarm level; when the ratio is in the range of [0,68%] and the number of occurrences is greater than 5 times, it is judged as a general alarm level; when the ratio is in the range of (68%,100%] and the number of occurrences is less than or equal to 5 times, it is judged as a minor alarm level; when the ratio is in the range of (68%,100%] and the number of occurrences is greater than 5 times, it is judged as a serious alarm level.

[0079] Table 10 Thread ratio alarm level

[0080]

[0081] For the number of servers online, when the number of online servers is not zero and the number of occurrences is greater than or equal to 2 times, it is judged as a normal alarm level; when the number of online servers is not zero and the number of occurrences is less than 2 times, it is judged as a general alarm level; when the number of online servers is zero and the number of occurrences is greater than or equal to 2 times, it is judged as a serious alarm level; when the number of online servers is zero and the number of occurrences is less than 2 times, it is judged as a minor alarm level.

[0082] Table 11 Alarm levels for online server count

[0083]

[0084] Combined with the five indicator weights customized by the system administrator, the production data center status model Q is:

[0085] Q=0.2x+0.3y+0.1z+0.1i+0.3j;

[0086] The automated operation and maintenance system obtains and calculates the status value of the production data center every 5 seconds. The calculation method is the same as before and will not be repeated. If the status value Q is greater than 3.6, it is determined that the production data center is in an abnormal state and a disaster recovery switching operation is required.

[0087] This judgment mode combines the number of indicator occurrences and weights to clarify the administrator's priority and focus on indicators, solving the drawbacks of the traditional method of relying on the network to judge whether a disaster has occurred in the data center, improving the accuracy of disaster judgment, and optimizing the decision-making process. At the same time, the custom configuration monitoring method has the advantages of improving user experience, flexibility, and scalability.

[0088] 3. Disaster Switching Model

[0089] like Figure 3, a disaster recovery switching device for an automated operation and maintenance system, is suitable for disaster recovery switching condition monitoring and switching of primary and standby data centers of an automated operation and maintenance system. The device specifically includes: a data acquisition and analysis module, a disaster recovery plan selection module, a disaster recovery switching execution module, a disaster recovery switching verification module, and a disaster recovery switching data recovery module.

[0090] Data collection and analysis module: used to obtain environmental data of the automated operation and maintenance system, and determine whether a disaster has occurred in the current data center based on judgment rules.

[0091] Disaster recovery plan selection module: used to determine which backup data center to switch to when a disaster occurs. When the automatic switching task is configured, the backup data center is obtained from the system configuration, otherwise the user determines the backup data center by clicking.

[0092] Disaster recovery switching execution module: used to execute disaster recovery switching, switch the data center from the primary data center to the backup data center, and perform a series of switching operations, including database status change, database restore point setting, firewall setting, etc.

[0093] Disaster recovery switch verification module: used to execute the verification process after disaster recovery switch, check whether the process, components, tasks, etc. are normal, and whether the connection of the agent is normal, confirm that the disaster recovery switch is executed correctly, and the backup data center is currently available.

[0094] Disaster recovery switching data recovery module: used to determine whether the original main data center has returned to normal based on the data from the "data collection and analysis module". If it has returned to normal, it will automatically or manually switch back to the production data center.

[0095] When a disaster occurs, a series of operations such as switching data centers need to be performed to ensure that the processes and tasks of the automated operation and maintenance system can run normally in the backup data center. This article provides two switching methods, namely manual switching and automatic task switching. Automatic task switching is when the user configures the disaster recovery connection relationship of each data center in advance, that is, when a disaster occurs in the current data center, which backup data center will be used to continue running. Manual switching is when a disaster actually occurs, the user subjectively determines which backup data center to switch to, and clicks the switch button for that data center. Figure 5 , the specific switching steps are as follows:

[0096] Step 1: The management system monitors the five indicators of the production environment in real time: memory usage, network latency, CPU usage, thread usage, and number of online servers. The status value Q of the production data center is calculated using the algorithm described above. If Q is greater than 3.6, the current production data center is judged to be in an abnormal state.

[0097] Step 2: When the current environment is determined to be abnormal based on the monitoring data in step 1, determine whether an automatic switching task is configured. If an automatic switching task is configured, the system automatically obtains the disaster recovery connection relationship of the current data center and performs the disaster recovery switching operation, so that the automated operation and maintenance system switches to the backup data center. Otherwise, the user subjectively determines which backup data center to switch to and clicks the switch button for that data center.

[0098] When actually performing disaster recovery switching, the time point when the disaster occurred is first set as the database restore point. The setting method is that the management system first cancels the recovery operation of the currently ongoing managed database, then creates a recovery point and specifies the recovery database, and finally disconnects the connection between the primary database and the backup database. The operation aims to improve the efficiency of data recovery after disaster recovery. Data recovery only starts from the restore point, and the execution data within the disaster time period is incrementally restored to the production database, greatly shortening the data recovery time.

[0099] Then the management system changes the "Quick Connection" setting of the server to the enabled state, so that the agent can quickly connect to the server and ensure that the business can be executed normally using the agent. Then open the firewall configuration of the operating system and server, intercept all requests sent to the production data center, and return the interception information to the sender of the request. Then cancel the recovery operation of the currently ongoing managed database, activate the standby data center, make the database of the standby data center readable and writable, make preliminary preparations for running the business in the standby data center in the future, and write the running data to the standby database.

[0100] In this embodiment, the issue of saving the data of components, processes, etc. being edited is taken into consideration. For data such as components, processes, and indicators, even if the user does not click the Save button, the server will save the data as a temporary db file every 5 minutes. The file will be synchronized with the database to each backup data center. When the user clicks Save, the temporary file is automatically cleared to release storage space. If a disaster occurs while the user is editing, after the disaster recovery switch is completed, the backup data center will automatically read the data of the temporary db file to the backup data center for use, reducing the possibility of data loss and improving the reliability of the disaster recovery switch.

[0101] In this embodiment, the problem of agent connection efficiency is solved. Since the agent is the key to task execution, the backup data center needs to handle the connection problems of tens of thousands of agents. In traditional disaster recovery switching, batches of agent connections will cause the server to be unable to respond or even restart. In this method, the agent is started using the "quick connection" method, that is, first ensure that the agent can quickly and stably connect to the server, and then process the agent connection check in an asynchronous manner. After using this method, the connection rate of 20,000 agents has been shortened from the original 15 to 20 minutes to the current 1 to 2 minutes, greatly improving the efficiency of disaster recovery switching. In addition, this solution also solves the problem of console freezes caused by a large number of agent connections, and enables the console to operate normally by configuring the system to retain the number of threads.

[0102] Normal agent startup requires checking the agent's indicators, tasks and other information. Only when the check is passed will the agent connect to the server. This checking process is very time-consuming. The logic of turning on "Quick Connection" is to connect to the server first, and then start checking when the server is free.

[0103] Step 3: After the automated operation and maintenance system has switched to the backup data center, it is necessary to run the corresponding verification process to check whether key components, processes, tasks, etc. exist, and whether the agent is online. If the verification process is executed normally, it indicates that the disaster recovery switch is successful. Otherwise, it indicates that the disaster recovery switch has failed, and a notification is sent to the corresponding administrator.

[0104] Step 4: The management system still monitors in real time whether the production environment has returned to normal. When the abnormal judgment conditions are not met, it is determined that the production environment has returned to normal, and the system's switching method is obtained. If the switching method is a manual switching task, the administrator user clicks on the production data center to switch, otherwise the system automatically obtains the pre-configured disaster recovery switching plan. When actually performing disaster recovery, first activate the production data center so that the database of the production data center becomes readable and writable. Then turn off the system's firewall settings so that all requests can be sent to the production data center. Next, starting from the restore point, all data generated after the restore point time is restored to the production database. After the data recovery is completed, clear the database restore point.

[0105] like Figure 4In this embodiment, the problem of data recovery efficiency is solved by using database restore points. Assuming that the database is created at time point t0, the data is synchronized from the primary data center to the backup data center in real time when no disaster occurs, and the data of the two data centers remain synchronized. When a disaster occurs at time point t2, the primary data center no longer synchronizes data to the backup data center, and uses time point t2 as the database restore point. During the disaster period from t2 to t6, the automated operation and maintenance system has switched to the backup data center, and all generated operation data is in the backup data center. When the disaster is recovered normally at time point t6, the operation data in the time period from t2 to t6 is restored to the primary data center, and from then on the data remains synchronized from the primary data center to the backup data center in real time. In this way, the efficiency of data recovery to the primary data center is greatly improved.

[0106] In this embodiment, the problems of proxy connection efficiency and data recovery efficiency are solved. If the automatic switching task is configured, the system can automatically determine the situation of the environment and implement automatic switching, providing users with a smooth, seamless and natural experience.

[0107] Step 5: After the automated operation and maintenance system has switched to the production data center, the corresponding verification process needs to be run to check whether key components, processes, tasks, etc. exist, and whether the agent is online. If the verification process is executed normally, it indicates that the disaster recovery switch is successful. Otherwise, it indicates that the disaster recovery switch has failed, and a notification is sent to the corresponding administrator.

[0108] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A disaster recovery switching device for an automated operation and maintenance system, characterized in that: It includes data collection and analysis module, disaster recovery plan selection module, disaster recovery switch execution module, disaster recovery switch verification module, and disaster recovery switch data recovery module; among which: The data collection and analysis module is used to obtain the environmental data of the automated operation and maintenance system and determine whether a disaster has occurred in the current data center based on preset judgment rules; The disaster recovery plan selection module is used to determine which backup data center will be switched to when a disaster occurs. When the automatic switching task is configured, the backup data center is obtained from the automated operation and maintenance system configuration. Otherwise, the user determines the backup data center by clicking. The specific working process of the disaster recovery solution selection module is as follows: Step 1: The automated operation and maintenance system monitors several indicators of the production environment in real time and calculates the status value of the production data center. If the status value of the production data center is greater than a preset value, the current production data center is judged to be in an abnormal state and the process goes to step 2. Step 2: Determine whether the automatic operation and maintenance system is configured with an automatic switching task. If so, the system automatically obtains the disaster recovery connection relationship of the current data center and performs a disaster recovery switching operation, so that the automatic operation and maintenance system switches to the backup data center. Otherwise, the user subjectively selects which backup data center to switch to and clicks the switch button for that data center. When actually performing disaster recovery switching, the time point when the disaster occurred is first set as the database restore point. The setting method is that the management system of the automated operation and maintenance system first cancels the recovery operation of the managed database currently in progress, then creates a recovery point and specifies the recovery database, and finally disconnects the connection between the primary database and the backup database to improve the efficiency of data recovery after disaster recovery. Data recovery only starts from the restore point, and the execution data within the disaster time period is incrementally restored to the production database, greatly shortening the data recovery time; Then the management system of the automated operation and maintenance system changes the "Quick Connection" setting of the server to the enabled state, so that the agent can quickly connect to the server and ensure that the business can be executed normally using the agent; then the firewall configuration of the operating system and the server is turned on to intercept all requests sent to the production data center and return the interception information to the sender of the request; then the recovery operation of the currently ongoing managed database is canceled and the standby data center is activated, so that the database of the standby data center becomes readable and writable, making preliminary preparations for running the business in the standby data center in the future, and writing the running data into the standby database; In step 1, the state value Q of the production data center is calculated as follows: If the system administrator has no special requirements for the five monitoring indicators of memory usage, network latency, CPU usage, thread usage, and number of online servers, the default algorithm of the automated operation and maintenance system is used to determine the status of the production data center: Q=0.18x+0.3y+0.15z+0.12i+0.25j; Among them, the memory ratio is x, the network delay is y, the CPU ratio is z, the number of threads is i, and the number of online servers is j; If the system administrator has special requirements for the five monitoring indicators, namely, memory percentage, network latency, CPU percentage, thread percentage, and number of online servers, the administrator can customize the indicator critical value, indicator occurrence times, and indicator weight to calculate the status value Q of the new production data center. The disaster recovery switching execution module is used to execute disaster recovery switching, switch the data center from the primary data center to the backup data center, and perform a series of switching operations, including database state change, database restore point setting, and firewall setting; The disaster recovery switch verification module is used to execute the verification process after the disaster recovery switch, check whether the process, components, tasks are normal, and whether the connection of the agent is normal, confirm that the disaster recovery switch is executed correctly, and the backup data center is currently available; The disaster recovery switching data recovery module is used to determine whether the original main data center has returned to normal based on the data from the data collection and analysis module. If it has returned to normal, it will automatically or manually switch back to the production data center.

2. The disaster recovery switching device for an automated operation and maintenance system according to claim 1, characterized in that: In the disaster recovery plan selection module, the automatic switching task is that the user configures the disaster recovery connection relationship of each data center in advance, that is, when a disaster occurs in the current data center, which backup data center will be used to continue operation.

3. The disaster recovery switching device for an automated operation and maintenance system according to claim 1, characterized in that: The agent of the automated operation and maintenance system is started using a "quick connection" method, which first ensures that the agent can connect to the server quickly and stably, and then processes the connection check of the agent in an asynchronous manner.

4. The disaster recovery switching device for an automated operation and maintenance system according to claim 1, characterized in that: Assuming that the database of the automated operation and maintenance system is created at time point t0, then when no disaster occurs, the data is synchronized from the primary data center to the backup data center in real time, and the data of the two data centers remain synchronized; when a disaster occurs at time point t2, the primary data center no longer synchronizes data to the backup data center, and time point t2 is used as the database restore point; during the disaster period from t2 to t6, the automated operation and maintenance system has switched to the backup data center, and all generated operation data is in the backup data center; when the disaster is recovered normally at time point t6, the operation data in the time period from t2 to t6 is restored to the primary data center, and from then on, the data continues to be synchronized from the primary data center to the backup data center in real time.

Images