A method for implementing MySQL high availability based on DNS and MGR

By combining the distributed consensus protocol of DNS and MGR, a transparent and highly reliable MySQL high availability solution is designed, which solves the problems of split-brain and data loss in MySQL databases under complex failure scenarios, realizes high availability and transparent application system switching, simplifies the architecture and reduces costs.

CN114546427BActive Publication Date: 2024-11-22JIANGSU SUNING BANK CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210154698.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-21
Publication Date
2024-11-22
Estimated Expiration
2042-02-21

AI Technical Summary

Technical Problem

In existing technologies, high availability solutions for MySQL databases suffer from problems such as split-brain, data loss, complex architecture, and high cost when facing complex failure scenarios, and they fail to effectively integrate DNS and MGR technologies.

Method used

By combining the distributed consistency protocols of DNS and MGR, a transparent and highly reliable method for implementing MySQL high availability is designed. Probes are used for database cluster detection and failover, and DNS API or remote commands are used for domain name updates to ensure that the application system transparently switches to the new master node.

Benefits of technology

It achieves high availability of the database under complex failure scenarios, maintains transparency of the application system, simplifies the architecture, reduces costs, is suitable for various DNS infrastructures, requires no additional proxy layer, and ensures data integrity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114546427B_ABST
    Figure CN114546427B_ABST
Patent Text Reader

Abstract

The present invention discloses a MySQL high availability implementation method based on DNS and MGR. The method includes installing and configuring a database cluster online, initializing the configuration of a probe, and then detecting the database cluster in operation by the probe. The database cluster architecture, transaction accumulation, and domain name IP obtained by the detection are used to determine whether the database cluster is running normally. After an abnormality occurs, the probe is combined with MGR to complete the database switching, so that the application is automatically connected to the switched database cluster. The present invention is designed in combination with the overall architecture of the application system itself, DNS, MGR, and the characteristics of the probe, so its architecture is simple. At the same time, the probe is based on an interface design and is not limited to any language. When actually used, it can be implemented in combination with a language familiar to the user for easy maintenance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular to a method for implementing MySQL high availability based on DNS and MGR. Background Art

[0002] As a data storage device for information systems, databases mainly provide data storage and access functions, and are often the core of information systems. With the rapid iteration and development of database technology, various database vendors currently provide more functions and features, including database high availability technology.

[0003] Database high availability technology refers to a technology that uses the characteristics of the database software to provide a technology that can continue to provide services after some nodes fail due to server, operating system or database software failures. As a technology widely used in various industries, MySQL database has successively provided asynchronous master-slave, semi-synchronous master-slave and the latest MGR distributed technology, and its technology and solutions are becoming more and more perfect and mature.

[0004] MGR refers to MySQL Group Replication, a synchronization technology provided by MySQL database based on the Paxos distributed protocol since version 5.7. This technology is used to check and analyze conflicts in the transaction logs of the master and slave synchronization, thereby ensuring the consistency of database services and data in extreme scenarios.

[0005] Domain Name Service (DNS) is a service that provides a mapping relationship between domain names and IP addresses. Its main purpose is to enable application systems to access real services by configuring domain name servers and accessing domain names with business meanings without having to worry about IP addresses, thus achieving unified application configuration and reducing the difficulty of maintenance. DNS has gradually become a standard configuration for information system construction in various industries. Currently, the industry has provided complete solutions, including the commercial version ZDNS, open source versions bind, bind-dlz, etc.

[0006] Based on the above technologies, since the database and DNS belong to different technical fields, they have not been effectively integrated. Although the database provides perfect synchronization solutions such as master-slave synchronization and MGR, the application cannot recognize it after the switch is completed, and the application needs to implement related functions independently. The common solutions in the industry and their advantages and disadvantages are analyzed as follows:

[0007] 1. VIP-based switching solution. This solution detects database nodes through a separate control server. When the master node detection fails, it starts the failover and promotes the slave node to the master. However, since nodes other than the database are used for detection, when failover is triggered due to abnormal network isolation, server pseudo-death, etc., it may cause the database to split brain and cause data loss. At the same time, due to its particularity, the control node often adopts a single node or master-slave architecture. When faced with complex failure scenarios, it may fail itself, resulting in the inability to perform the switching operation. In addition, this solution is mostly implemented by a third-party organization, and its core switching logic is strongly related to the database version. There is a possibility that the high availability solution will fail due to incompatibility with the actual database version used, especially the current MySQL8 version.

[0008] 2. Intermediate proxy solution. This solution establishes a proxy layer between the database and the application, identifies the database master node through the proxy layer, and then provides a protocol similar to the database for application connection. After the database master node fails, the intermediate proxy identifies the new node and rebuilds the application connection to the new database node. However, because an additional intermediate proxy layer needs to be introduced, any database request needs to pass through the intermediate proxy layer, which will inevitably lead to more complex overall application architecture, more request paths, and longer time consumption. At the same time, application proxies are often achieved through third-party organizations through the simulation of common MySQL operations to achieve compatibility, but there will inevitably be operations that the intermediate proxy cannot recognize, which will require additional compatibility modifications; in addition, the intermediate proxy layer itself may have high availability issues, performance bottlenecks and other problems.

[0009] 3. DNS health check solution. This solution uses the DNS server to regularly check the database. When it is found that the original master node fails and causes the status to change, the IP of the corresponding domain name is updated to the IP of the new master node. However, this solution only provides how to connect to the database after the database is switched. It does not provide a complete solution, including how to perform a database failover, how to prevent data loss, and how the application can connect to the new master node based on the updated domain name information. At the same time, it also relies on specific DNS products to implement active detection based on the database, and its general promotion value is insufficient. Summary of the invention

[0010] The purpose of the present invention is to provide a MySQL high availability implementation method based on DNS and MGR in view of the deficiencies in the prior art.

[0011] To achieve the above object, the present invention provides a MySQL high availability implementation method based on DNS and MGR, comprising:

[0012] Step 1: Install and configure the database cluster online;

[0013] Step 2: Initialize the probe and then detect the running database cluster;

[0014] Step 3: Determine whether the database cluster is running normally based on the database cluster architecture, transaction accumulation, and domain name IP obtained through detection, including:

[0015] If the database cluster master node IP is equal to the domain name IP, and the database cluster node is equal to the configuration node, it means that the cluster is running normally;

[0016] If the database cluster master node IP is equal to the domain name IP, but the database cluster node is not equal to the configuration node, it means that there is an abnormal operation of the slave node, triggering an alarm;

[0017] If the database cluster master node IP is not equal to the domain name IP, and the local node IP is equal to the database cluster master node IP, the failover process is triggered;

[0018] If the database cluster master node IP is not equal to the domain name IP, and the local node IP is not equal to the database cluster master node IP, an alarm is triggered.

[0019] Furthermore, the failover process includes:

[0020] A new master node is selected through a distributed consensus protocol. The database instance of the original master node is terminated or enters offline mode, and the detection interval is adjusted.

[0021] Analyze the database operation status and confirm whether there are any uncompleted transactions on the new primary node;

[0022] After the accumulated transaction application is completed, if the node is the new primary node, the probe initiates the DNS switching process;

[0023] After the DNS switch is successful, if there is a DNS cache, perform a DNS cache refresh operation;

[0024] Verify the domain name IP, new master node IP and PING result. Successful verification indicates that the failover is complete.

[0025] The detection interval is restored to the configured interval and detection is continued.

[0026] Furthermore, the DNS switching process includes a consistency check between the domain name and the IP before the update, the IP of the updated domain name, and the consistency check between the domain name and the IP after the update. For commercial DNS devices, the API interface provided by them is used for inspection and update. For open source DNS devices, remote commands or SQL requests are used for update. Finally, if the verification is successful, the DNS switching is successful. Otherwise, it indicates an abnormality and will exit.

[0027] Furthermore, the step 1 specifically includes:

[0028] Step 1.1: Apply for and configure resources;

[0029] Step 1.2: Deploy database software;

[0030] Step 1.3, optimize parameter configuration;

[0031] Step 1.4: Initialize and start the database;

[0032] Step 1.5: Configure users and permissions;

[0033] Step 1.6, configure the MGR plug-in and parameters;

[0034] Step 1.7, start MGR operation;

[0035] Step 1.8: Configure and start each database node probe. At this point, the configuration of the database node is completed, and the configuration of all database nodes is gradually completed.

[0036] Step 1.9: The application system connects to the database cluster based on the configured database domain name, user name, password and other information.

[0037] Furthermore, the step 2 specifically includes:

[0038] Step 2.1, read the fixed path configuration file, confirm whether the configuration file format is illegal and whether the parameter value is reasonable;

[0039] Step 2.2, parse and obtain database login information, login database query variables, cluster structure and transaction accumulation;

[0040] Step 2.3, verify whether the cluster structure matches the information in the configuration file. If the node set of the configuration information is equal to or contains the database real-time cluster architecture, the verification returns success, otherwise it returns failure;

[0041] Step 2.4: Perform a ping operation based on the domain name to obtain the IP address corresponding to the domain name;

[0042] Step 2.5: Verify the DNS update interface, print the initialization information, and verify the initialization configuration.

[0043] Furthermore, after the failover process is completed, the fault is repaired, including:

[0044] Analyze the root cause of the database cluster failover, and determine whether the operating system, database, probe and other components need to be optimized based on the analysis results;

[0045] Check the operating system running status, logs and other data to confirm whether it is necessary to perform operating system-level repair operations. If the failover is caused by an operating system defect, restart the operating system;

[0046] Check the database error log to confirm whether the database instance needs to be repaired. If a database instance or process exception triggers a failover, restart the database.

[0047] Confirm whether the key parameters of the database of the failed node are normal, including relevant parameters such as offline mode and read-only configuration;

[0048] Restart MGR. After the operation is started, data recovery will be automatically performed and the failed node will be rejoined to the database cluster.

[0049] Execute the probe startup operation. If the log is normal and the initialization is successful, it means that the fault is repaired successfully.

[0050] Beneficial effects: The present invention is applicable to applications using MySQL database and has the following advantages:

[0051] 1. Completely transparent to applications and has a broad application base. This database high-availability solution retains the native MySQL standard from application coding, driver, and SQL parsing execution, and is completely transparent to the application system. When the application system migrates from other high-availability solutions to use this solution, it only needs to make necessary standardized transformations before it can be put into production.

[0052] 2. It is completely based on the distributed consistency protocol, safe and reliable. The solution is designed based on the distributed consistency protocol provided by MGR. All operations that affect the database and cluster structure, such as cluster transaction verification and view switching, are negotiated through consistency. Operations are performed only after more than half of the members vote in favor. At the same time, the probe is based on the consistency switching results and further performs operations such as view updates. Therefore, the entire solution is safe and reliable under any circumstances.

[0053] 3. Based on common infrastructure, production and cost are controllable. Since the solution does not change the overall architecture of the system, it will maintain the basic architecture of common application services + databases, and will not introduce other intermediate proxy services. At the same time, DNS, as a basic resource of the data center, can meet the needs of this solution regardless of whether it uses commercial DNS software (such as ZDNS) or open source DNS (such as bind, bind-dlz). At the same time, during the fault switching process, the probe actively switches to DNS based on the database consistency switching result, and can be customized according to the target DNS (can be API interface, remote command and SQL request), without the need to customize DNS to control costs and controllability.

[0054] 4. Simple architecture, easy to configure and maintain. In summary, this high-availability solution is designed based on the overall architecture of the application system, DNS, MGR and the characteristics of the probe, so its architecture is simple. At the same time, the probe is designed based on an interface and is not limited to any language. In actual use, it can be implemented in a language that the user is familiar with for easy maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flowchart for installing and configuring the database cluster online;

[0056] Figure 2 This is a flow chart of initializing the probe and detecting the database cluster;

[0057] Figure 3 It is a schematic diagram of the failover process;

[0058] Figure 4 This is a flowchart of repairing a fault;

[0059] Figure 5 This is the architecture diagram of the application system directly accessing DNS;

[0060] Figure 6 This is the architecture diagram of the application system accessing the DNS cache. DETAILED DESCRIPTION

[0061] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. The embodiments are implemented based on the technical solution of the present invention. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0062] like Figures 1 to 4 As shown, an embodiment of the present invention provides a MySQL high availability implementation method based on DNS and MGR, including:

[0063] Step 1: Install and configure the database cluster. For details, see Figure 1 , step 1 includes:

[0064] Step 1.1: Apply for and configure resources.

[0065] During the system launch process, the person in charge of the application system applies for resources such as servers and domain names through existing standardized processes (such as ITSM). After the application is submitted and approved, the relevant responsible team allocates and configures the resources. For example, the system management department allocates the database server and configures modules such as IP addresses, security policies, and monitoring policies; the network management department allocates domain names based on the allocated IPs and configures related views and permissions.

[0066] Step 1.2: Deploy the database software.

[0067] This step is performed by the database management department to install and deploy the database software, including the installation of operating system dependency packages, operating system user configuration, operating system kernel parameter optimization, database directory creation, database software installation, and directory permission modification.

[0068] Step 1.3: Optimize parameter configuration.

[0069] Specifically, according to the server configuration (such as the number of CPU cores, total memory size), calculate the relevant database parameter values. For example, set innodb_thread_concurrency to the number of server CPUs, innodb_buffer_pool_size to 70% of the total server memory size, slave_parallel_workers to the number of server cores, and calculate the unique server-id. As well as other key parameter settings, see Table 1 for details:

[0070]

[0071]

[0072] Table 1

[0073] Step 1.4: Initialize and start the database.

[0074] Use the deployed database software, combined with optimized parameter files and directory configurations, to initialize the database. This mainly includes generating system tablespaces, generating redo log files, generating double write cache files, creating undo tablespaces, creating temporary sort files, and generating temporary passwords. After initialization is complete, execute the command to start the database.

[0075] Step 1.5: Configure users and permissions.

[0076] After the database is initialized, it is not convenient to use a random password for unified management. At the same time, the relevant passwords are recorded in the database log, which poses a security risk. It is also necessary to establish relevant users according to application and operation and maintenance requirements. The user list is shown in Table 2:

[0077]

[0078] Table 2

[0079] It should also be noted that users who follow the principle of minimizing permissions should be assigned table or database permissions based on actual needs. In particular, CONNECTION_ADMIN and SUPER permissions should not be granted.

[0080] Step 1.6: Configure the MGR plug-in and parameters.

[0081] To configure the database to support the complete MGR (MySQL Group Replication) function, you need to install two plug-ins, group_replication and mysql_clone. group_replication is used to implement the core functions of MGR, and mysql_clone is used to synchronize the full data during fault repair. In addition, you need to set the MGR related parameters. The standard settings are shown in Table 3:

[0082]

[0083] Table 3

[0084] Step 1.7: Start the MGR operation.

[0085] After completing the MGR plug-in and parameter configuration, restart the database instance to make the parameters effective. After the restart is complete, set the group_replication_recovery channel to use, and then start group replication. When the first node of the database cluster starts group replication, it should be set to boot mode, and after the startup is complete, turn off the boot mode, and then start group replication for other nodes.

[0086] Step 1.8: Configure and start each database node probe. At this point, the configuration of the database node is completed, and the configuration of all database nodes is gradually completed.

[0087] When configuring and starting each database node probe, complete the probe configuration file according to the applied domain name address and database cluster configuration, execute the probe startup operation, and observe that the log is normal and the initialization is successful.

[0088] Step 1.9: The application system connects to the database cluster based on the configured database domain name, user name, password and other information.

[0089] In addition, it should be noted that, first, the MGR configuration modification should be carried out in accordance with the above-mentioned standard configuration to ensure that the switch operation is completed within the expected time range after the failure occurs. Secondly, the permissions of the application database user should be allocated on demand using the principle of minimum permissions to ensure that the original master node can disconnect the existing database connection after being expelled from the cluster.

[0090] Step 2: Initialize the probe and then detect the running database cluster. For details, see Figure 2 , step 2 specifically includes:

[0091] Step 2.1: Read the fixed path configuration file to confirm whether the configuration file format is illegal and whether the parameter values ​​are reasonable.

[0092] Step 2.2: parse and obtain database login information, database login query variables, cluster structure and accumulation status.

[0093] Step 2.3: Verify whether the cluster structure matches the information in the configuration file. If the node set of the configuration information is equal to or contains the database real-time cluster architecture, the verification returns success, otherwise it returns failure.

[0094] Step 2.4: Perform a ping operation based on the domain name to obtain the IP address corresponding to the domain name.

[0095] Step 2.5: Verify the DNS update interface, print the initialization information, and verify the initialization configuration.

[0096] Step 3: Determine whether the database cluster is running normally based on the database cluster architecture, transaction accumulation, and domain name IP obtained through detection, including:

[0097] If the database cluster master node IP is equal to the domain name IP, and the database cluster node is equal to the configuration node, it means that the cluster is running normally.

[0098] If the database cluster master node IP is equal to the domain name IP, but the database cluster node is not equal to the configuration node, it means that there is a slave node running abnormally, triggering an alarm.

[0099] If the database cluster master node IP is not equal to the domain name IP, and the local node IP is equal to the database cluster master node IP, the failover process is triggered.

[0100] If the database cluster master node IP is not equal to the domain name IP, and the local node IP is not equal to the database cluster master node IP, an alarm is triggered.

[0101] See also Figure 3 , the above failover process specifically includes:

[0102] A new master node is selected through a distributed consensus protocol, the original master node database instance is terminated or enters offline mode, and the detection interval is adjusted. After the original master node database instance is terminated or enters offline mode, all sessions or connections running on the original master node will be interrupted. The above probe interval is preferably adjusted to 1 second.

[0103] Analyze the database operation status and confirm whether there are any unapplied transactions on the new primary node.

[0104] After the accumulated transactions are applied, if this node is the new primary node, the probe initiates the DNS switching process.

[0105] After the DNS switch is successful, if there is a DNS cache, perform a DNS cache refresh operation.

[0106] Verify the domain name IP, new master node IP and PING result. A successful verification indicates that the failover is complete.

[0107] The probe interval is restored to the configured interval and the probe is continued.

[0108] The above DNS switching process includes consistency check of domain name and IP before update, IP of updated domain name, and consistency check of domain name and IP after update. For commercial DNS equipment, the API interface provided by it is used for inspection and update. For open source DNS equipment, remote commands or SQL requests are used for update. Finally, if the verification is successful, the DNS switching is successful. Otherwise, it means an abnormality occurs and the system will exit.

[0109] After the failover process is complete, you can also use the following methods to repair the fault, see Figure 4 , including:

[0110] Analyze the root cause of the failover of the database cluster and determine whether the operating system, database, probe and other components need to be optimized based on the analysis results.

[0111] Check the operating system running status, logs and other data to confirm whether it is necessary to perform operating system-level repair operations. If the failover is caused by an operating system defect, restart the operating system.

[0112] Check the database error log to confirm whether you need to perform a repair operation on the database instance. If the database instance or process triggers a failover, restart the database.

[0113] Confirm whether the key parameters of the database of the failed node are normal, including offline mode and read-only configuration related parameters. For details, see Table 4:

[0114] Parameter name Parameter Value offline_mode off super_read_only on read_only on

[0115] Table 4

[0116] Restart MGR. After the operation is started, data recovery will be automatically performed and the failed node will be added back to the database cluster.

[0117] Execute the probe startup operation. If the log is normal and the initialization is successful, it means that the fault is repaired successfully.

[0118] See also Figure 5 and Figure 6, the present invention can be implemented based on the application system directly accessing DNS or accessing DNS cache. When directly accessing DNS, after a fault occurs, MGR identifies the fault and switches; the probe accesses DNS to provide API to initiate domain name update and verification in combination with the MGR switching result; the application system obtains the IP after the domain name is updated through DNS when reconnecting because of the fault of the original master node of the database cluster, so as to connect to the new master node of the database cluster. When implemented by accessing DNS cache, after a fault occurs, MGR identifies the fault and switches; the probe accesses DNS to provide API to initiate domain name update and verification in combination with the MGR switching result, and refreshes the updated information to the cache in real time; the application system obtains the IP after the domain name is updated through DNS cache when reconnecting because of the fault of the original master node of the database cluster, so as to connect to the new master node of the database cluster.

[0119] The above is only a preferred embodiment of the present invention. It should be noted that for ordinary technicians in this technical field, other parts not specifically described belong to the prior art or common knowledge. Several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A method for implementing MySQL high availability based on DNS and MGR, characterized in that: include: Step 1: Install and configure the database cluster online; Step 2: Initialize the probe and then detect the running database cluster; Step 3: Determine whether the database cluster is running normally based on the database cluster architecture, transaction accumulation, and domain name IP obtained through detection, including: If the database cluster master node IP is equal to the domain name IP, and the database cluster node is equal to the configuration node, it means that the cluster is running normally; If the database cluster master node IP is equal to the domain name IP, but the database cluster node is not equal to the configuration node, it means that there is an abnormal operation of the slave node, triggering an alarm; If the database cluster master node IP is not equal to the domain name IP, and the local node IP is equal to the database cluster master node IP, the failover process is triggered; If the database cluster master node IP is not equal to the domain name IP, and the local node IP is not equal to the database cluster master node IP, an alarm is triggered; Step 1 specifically includes: Step 1.1: Apply for and configure resources; Step 1.2: Deploy database software; Step 1.3, optimize parameter configuration; Step 1.4: Initialize and start the database; Step 1.5: Configure users and permissions; Step 1.6, configure the MGR plug-in and parameters; Step 1.7, start MGR operation; Step 1.8: Configure and start each database node probe. At this point, the configuration of the database node is completed, and the configuration of all database nodes is gradually completed. Step 1.9, the application system connects to the database cluster according to the configured database domain name, user name and password; The MGR configuration modification must be performed according to the preset specification configuration to ensure that the switch operation is completed within the expected time range after the failure occurs; the permissions of the application database user should be allocated on demand based on the principle of minimization of permissions to ensure that the original master node can disconnect the existing database connection after being expelled from the cluster; The failover process includes: A new master node is selected through a distributed consistency protocol, the original master node database instance is terminated or enters offline mode, and the probe interval is adjusted; Analyze the database operation status and confirm whether there are any uncompleted transactions on the new primary node; After the accumulated transaction application is completed, if the node is the new primary node, the probe initiates the DNS switching process; After the DNS switch is successful, if there is a DNS cache, perform a DNS cache refresh operation; Verify the domain name IP, new master node IP and PING result. Successful verification indicates that the failover is complete. Restore the probe interval to the configured interval and continue to perform detection; The DNS switching process includes consistency check of the domain name and IP before updating, updating the IP of the domain name, and consistency check of the domain name and IP after updating. For commercial DNS devices, the API interface provided by them is used for inspection and update. For open source DNS devices, remote commands or SQL requests are used for update. If the final verification is successful, the DNS switching is successful. Otherwise, it means an abnormality has occurred and the system will exit.

2. According to the method for implementing MySQL high availability based on DNS and MGR according to claim 1, it is characterized in that: The step 2 specifically includes: Step 2.1, read the fixed path configuration file, confirm whether the configuration file format is illegal and whether the parameter value is reasonable; Step 2.2, parse and obtain database login information, login database query variables, cluster structure and transaction accumulation; Step 2.3, verify whether the cluster structure matches the information in the configuration file. If the node set of the configuration information is equal to or contains the database real-time cluster architecture, the verification returns success, otherwise it returns failure; Step 2.4: Perform a ping operation based on the domain name to obtain the IP address corresponding to the domain name; Step 2.5: Verify the DNS update interface, print the initialization information, and verify the initialization configuration.

3. The method for implementing MySQL high availability based on DNS and MGR according to claim 1, characterized in that: After the failover process is completed, the fault is repaired, including: Analyze the root cause of the database cluster failover, and determine whether the operating system, database, and probe need to be optimized based on the analysis results; Check the operating system running status and logs to confirm whether it is necessary to perform operating system-level repair operations. If the failover is caused by an operating system defect, restart the operating system; Check the database error log to confirm whether the database instance needs to be repaired. If a database instance or process exception triggers a failover, restart the database. Confirm whether the key parameters of the database of the failed node are normal, including parameters related to offline mode and read-only configuration; Restart MGR. After the operation is started, data recovery will be automatically performed and the failed node will be rejoined to the database cluster. Execute the probe startup operation. If the log is normal and the initialization is successful, it means that the fault is repaired successfully.