Data fault-tolerant synchronization method and device for switching main and standby cluster nodes based on Elasticsearch

Abstract

The invention relates to the field of big data, and provides a data fault-tolerant synchronization method and device for switching main and standby cluster nodes based on Elasticsearch. The disaster recovery method aims at solving the disaster recovery problem during MySQL and Elastics search data synchronization in the prior art. According to the main scheme, the disaster recovery method comprises the steps that a MySQL database and two distributed Elasticsearch clusters are configured; the method comprises the following steps of: configuring a synchronization tool from Elasticsearch to MySQL (Structured Query Language), and synchronizing MySQL data to two Elasticsearch clusters by using Logstash; configuring a main-standby mode, defining a main cluster and a backup cluster, keeping data consistency by using a synchronization tool, and automatically triggering main-standby switching when the health state of the main cluster is reduced; monitoring and failover are carried out, an alarm threshold value and a monitoring interval are set, automatic failover logic is defined, and it is ensured that automatic main and standby switching and health state detection are achieved when the main cluster goes wrong.

Description

Technical Field

[0001] The present invention relates to the fields of big data, distributed systems, data synchronization, high availability and data fault tolerance, and data storage and indexing, and provides a data fault-tolerant synchronization method and device based on Elasticsearch master-slave cluster node switching. Background Art

[0002] Existing technical cases: MySQL Binlog+Logstash Using MySQL's binary log (binlog) and Logstash together, data synchronization between the MySQL database and the Elasticsearch cluster can be achieved. MySQL's binlog records all changes in the database, including inserts, updates, and deletes. By using Logstash's binlog input plug-in, the data change stream in the binlog is passed to the Elasticsearch output plug-in in real time to achieve data synchronization. Alibaba Canal, Alibaba's open source Canal is an incremental data subscription and publishing tool based on the MySQL database. It can capture data changes in the MySQL database and pass them to other data storage or computing platforms, including Elasticsearch, in real time. By configuring Canal, data changes in the MySQL database are synchronized to the Elasticsearch cluster in real time to achieve real-time indexing and searching of data. However, in the existing technical cases, data fault tolerance is not fully considered, resulting in irreversible data once an ES cluster collectively collapses. Therefore, based on this, a method for a distributed ES cluster data synchronization system is proposed. Summary of the invention

[0003] This technical proposal aims to solve the disaster recovery problem of MySQL and Elasticsearch data synchronization in the existing technology. Specifically, in the field of big data and distributed systems, data synchronization and fault tolerance are key issues. Although existing technical solutions such as MySQL Binlog+Logstash and Alibaba Canal can achieve data synchronization between MySQL and Elasticsearch, they are insufficient in data fault tolerance. When an Elasticsearch cluster fails, these solutions cannot quickly switch to the backup cluster, which may cause data loss.

[0004] In order to achieve the above purpose, the present invention adopts the following technical means:

[0005] The present invention provides a data fault-tolerant synchronization method for master-slave cluster node switching based on Elasticsearch, comprising the following steps:

[0006] Step 1: Configure a MySQL database and two distributed Elasticsearch clusters;

[0007] Step 2: Configure the Elasticsearch to MySQL synchronization tool and use Logstash to synchronize MySQL data to the two Elasticsearch clusters;

[0008] Step 3: Configure the master-slave mode, define a master cluster and a backup cluster, use the synchronization tool to maintain data consistency, and automatically trigger the master-slave switch when the health status of the master cluster decreases;

[0009] Step 4: Monitoring and failover, set alarm thresholds and monitoring intervals, and define automated failover logic to ensure automated active / standby switching and health status detection when problems occur in the active cluster.

[0010] In the above technical solution, step 1 specifically includes the following steps:

[0011] Step 1.1: Install and configure the MySQL database to obtain a correctly configured MySQL database instance;

[0012] Step 1.2: Install and configure two Elasticsearch clusters to obtain two correctly configured Elasticsearch cluster instances, where the Elasticsearch cluster is referred to as the ES cluster;

[0013] Step 1.3: Deploy necessary components on a server that can access two Elasticsearch clusters. The deployment environment is ready.

[0014] Step 1.4: Create the required database and table in the MySQL database. The database structure is ready to store the data to be synchronized to Elasticsearch.

[0015] In the above technical solution, step 2 specifically includes the following steps:

[0016] Step 2.1: Select and configure Logstash as the synchronization tool and prepare for data synchronization;

[0017] Step 2.2: Read data from MySQL using Logstash.

[0018] Step 2.3: Forward the read data to the two Elasticsearch clusters, and the data in MySQL is synchronized to the two Elasticsearch clusters.

[0019] In the above technical solution, step 3 specifically includes the following steps:

[0020] Step 3.1: Define the primary and backup clusters: Define one of the Elasticsearch clusters as the primary cluster, responsible for processing write operations and index updates; define the other as a backup cluster, used for data synchronization and taking over services when the primary cluster fails;

[0021] Step 3.2: Configure the synchronization tool: Logstash sends data to the Elasticsearch nodes in the active and standby clusters at the same time to keep the data synchronized;

[0022] Step 3.3: Configure the active / standby switch script:

[0023] Step 3.3.1: Initialize the script parameters, including the URLs of the primary and backup clusters, and establish connections with the primary and backup clusters through the client connected to the Elasticsearch cluster;

[0024] Step 3.3.2: Implement the health check function: Develop a health check function to obtain cluster health information by calling the Elasticsearch API and determine the health status of the main cluster;

[0025] Step 3.3.3: Configure the automated trigger mechanism: If the health status of the primary cluster is detected to have deteriorated, trigger the primary-backup switchover logic, including promoting the backup node to the new primary node and redirecting write traffic to the new primary node.

[0026] Step 4: Monitoring and failover, set alarm thresholds and monitoring intervals, and define automated failover logic to ensure automated active / standby switching and health status detection when problems occur in the active cluster.

[0027] In the above technical solution, step 4 specifically includes the following steps:

[0028] Step 4.1, implement the monitoring system: configure the monitoring system to regularly check the health status of the active cluster, including the running status of the active and standby clusters, and the active and standby cluster health status monitoring items;

[0029] Step 4.2: Set alarm thresholds and monitoring intervals: Set alarm thresholds and monitoring intervals to periodically detect health problems of the main cluster.

[0030] Step 4.3, detect and trigger failover:

[0031] If the primary cluster is detected to be unavailable, the fault detection function is used to verify the unavailable status of the primary cluster. If the primary cluster is detected to be unavailable, the failover logic is triggered;

[0032] Step 4.4: Execute the failover logic:

[0033] Call the node promotion function to promote the node in the backup cluster to the new master node.

[0034] Mark the original primary node as a backup node.

[0035] Modify the configuration of the synchronization tool Logstash through automated configuration to redirect data traffic to the new master node.

[0036] Step 4.4, start the synchronization tool and restore data:

[0037] Start or restart the synchronization tool Logstash to synchronize the missing data in the backup cluster and maintain data consistency and integrity.

[0038] Step 4.5: Restore the original master node:

[0039] Once the health status of the original master node is restored, it is synchronized back to the new master-slave cluster and rejoined to the cluster.

[0040] Step 4.6, complete the failover:

[0041] The monitoring system confirms the health of the new primary cluster, updates the status of the monitoring system, and logs information about the failover event.

[0042] The present invention also provides a data fault-tolerant synchronization device for switching between master and standby cluster nodes based on Elasticsearch, comprising the following modules:

[0043] Configure modules, configure MySQL database and two distributed Elasticsearch clusters;

[0044] Synchronization tool module, configure the synchronization tool from Elasticsearch to MySQL, and use Logstash to synchronize MySQL data to two Elasticsearch clusters;

[0045] The master-slave mode configuration module defines a master cluster and a backup cluster, uses synchronization tools to maintain data consistency, and automatically triggers master-slave switching when the health status of the master cluster decreases;

[0046] The monitoring and failover module sets alarm thresholds and monitoring intervals, and defines automated failover logic to ensure automated active / standby switching and health status detection when problems occur in the active cluster.

[0047] In the above device, the configuration module specifically includes the following steps:

[0048] Step 1.1: Install and configure the MySQL database to obtain a correctly configured MySQL database instance;

[0049] Step 1.2: Install and configure two Elasticsearch clusters to obtain two correctly configured Elasticsearch cluster instances, where the Elasticsearch cluster is referred to as the ES cluster;

[0050] Step 1.3: Deploy necessary components on a server that can access two Elasticsearch clusters. The deployment environment is ready.

[0051] Step 1.4: Create the required database and table in the MySQL database. The database structure is ready to store the data to be synchronized to Elasticsearch.

[0052] In the above device, the synchronization tool module specifically includes the following steps:

[0053] Step 2.1: Select and configure Logstash as the synchronization tool and prepare for data synchronization;

[0054] Step 2.2: Read data from MySQL using Logstash;

[0055] Step 2.3: Forward the read data to the two Elasticsearch clusters, and the data in MySQL is synchronized to the two Elasticsearch clusters.

[0056] In the above device, the active-standby mode configuration module specifically includes the following steps:

[0057] Step 3.1: Define the primary and backup clusters: Define one of the Elasticsearch clusters as the primary cluster, responsible for processing write operations and index updates; define the other as a backup cluster, used for data synchronization and taking over services when the primary cluster fails;

[0058] Step 3.2: Configure the synchronization tool: Logstash sends data to the Elasticsearch nodes in the active and standby clusters at the same time to keep the data synchronized;

[0059] Step 3.3: Configure the active / standby switch script:

[0060] Step 3.3.1, initialize the script parameters, including the URLs of the primary and backup clusters, and establish connections with the primary and backup clusters through the client connected to the Elasticsearch cluster;

[0061] Step 3.3.2, implement the health check function: Develop a health check function to obtain cluster health information by calling the Elasticsearch API and determine the health status of the main cluster;

[0062] Step 3.3.3, configure the automated trigger mechanism: If it is detected that the health status of the primary cluster has deteriorated, trigger the primary-backup switchover logic, including promoting the backup node to the new primary node and redirecting the write traffic to the new primary node.

[0063] In the above device, the monitoring and failover module specifically includes the following steps:

[0064] Step 4.1: Implement monitoring system:

[0065] Configure the monitoring system to regularly check the health status of the active cluster, including the running status of the active and standby clusters and the monitoring items of the active and standby cluster health status;

[0066] Step 4.2: Set the alarm threshold and monitoring interval:

[0067] Set alarm thresholds and monitoring intervals to periodically detect health issues in the primary cluster;

[0068] Step 4.3, detect and trigger failover:

[0069] If the primary cluster is detected to be unavailable, the fault detection function is used to verify the unavailable status of the primary cluster. If the primary cluster is detected to be unavailable, the failover logic is triggered;

[0070] Step 4.4: Execute the failover logic:

[0071] Call the node promotion function to promote the node in the backup cluster to the new master node and mark the original master node as a backup node;

[0072] Modify the configuration of the synchronization tool Logstash through automated configuration to redirect data traffic to the new master node;

[0073] Step 4.4, start the synchronization tool and restore data:

[0074] Start or restart the synchronization tool Logstash to synchronize the missing data in the backup cluster and maintain data consistency and integrity.

[0075] Step 4.5: Restore the original master node:

[0076] Once the health status of the original master node is restored, synchronize it back to the new master-slave cluster and rejoin the cluster;

[0077] Step 4.6, complete the failover:

[0078] The monitoring system confirms the health of the new primary cluster, updates the status of the monitoring system, and logs information about the failover event.

[0079] Because the present invention adopts the above technical means, it has the following beneficial effects:

[0080] 1. High availability: This device uses two Elasticsearch clusters for data synchronization to provide higher availability. If one cluster fails or is unavailable, the other cluster can still continue to provide services. Using the distributed management of step three, compared with the previous single point, this architecture can reduce the risk of single point failure and improve the stability and reliability of the system.

[0081] 2. Load balancing: This device can achieve load balancing by distributing data in two Elasticsearch clusters. Using the distributed switching method in step 3, data writing and reading can be shared between the two clusters, reducing the pressure on a single cluster and improving overall performance and responsiveness. This can better cope with the needs of high concurrent loads and large-scale data processing.

[0082] 3. Data redundancy and disaster recovery: The device can provide data redundancy and disaster recovery functions by synchronizing data between two Elasticsearch clusters. Data exists in both clusters at the same time. If one cluster fails, the other cluster still has a complete copy of the data. By using the disaster recovery processing methods of steps 3 and 4, it can quickly switch to the backup cluster and continue to provide services, which helps to ensure the security and persistence of data.

[0083] 4. Elastic expansion: By connecting two Elasticsearch clusters in step 2, better elastic scalability can be achieved. When you need to expand data processing capabilities or storage capacity, you can expand the system by adding more nodes or clusters. This scalability enables the system to adapt to growing data demands and load pressure.

[0084] 5. Data isolation and partitioning: This device connects two es clusters in step 2 and distributes data in two independent Elasticsearch clusters to achieve data isolation and partitioning. This is very useful for scenarios where data needs to be kept logically and physically separated. For example, sensitive data is stored in one cluster and non-sensitive data is stored in another cluster to meet security and compliance requirements.

[0085] 6. Automated fault detection and transfer: This device uses the monitoring system in step 4 to achieve automated health status detection, which can quickly detect and trigger a failover when a failure occurs in the main cluster, improving the system's response speed compared to manual intervention. By using a shorter monitoring interval and an automated trigger mechanism in step 4, real-time fault detection and transfer are achieved, improving the high availability of the system and reducing the impact of failures on the business. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 It is a schematic diagram of the framework of the present invention. DETAILED DESCRIPTION

[0087] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions made to the present invention should all be included in the scope of the claims of the present invention.

[0088] In addition, in order to better illustrate the present invention, numerous specific details are given in the following specific embodiments. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.

[0089] The purpose of the present invention is to solve the problem of ensuring that when the first cluster fails or the health level decreases, it quickly switches to the backup cluster, thereby improving the disaster recovery capability and availability of the entire system. At the same time, by synchronizing data to Elasticsearch in real time, it is also guaranteed that the latest data can be obtained when the front-end query is made, and the query performance and real-time technical problems are improved. The device can achieve load balancing by distributing data in two Elasticsearch clusters. By adopting distributed switching means, data writing and reading can be shared between the two clusters, reducing the pressure of a single cluster and improving the overall performance and responsiveness. This can better cope with the needs of high concurrent loads and large-scale data processing. The device realizes automated health status detection through a monitoring system, and can quickly detect and trigger failover when a main cluster fails, which improves the response speed of the system compared to manual intervention. Through a shorter monitoring interval and an automated triggering mechanism, real-time fault detection and transfer are achieved, the high availability of the system is improved, and the impact of failures on the business is reduced.

[0090] Embodiment 1:

[0091] This embodiment provides a data fault-tolerant synchronization method based on the master-slave cluster node switching of Elasticsearch. The system implements the following steps:

[0092] Step 1: Configure a MySQL database and two distributed Elasticsearch clusters.

[0093] Specifically, this step includes:

[0094] Install and configure the MySQL database to obtain a correctly configured MySQL database instance;

[0095] Install and configure two Elasticsearch clusters to obtain two correctly configured Elasticsearch cluster instances, where the Elasticsearch cluster is referred to as the ES cluster;

[0096] Deploy necessary components on a server that can access two Elasticsearch clusters to prepare the deployment environment.

[0097] Create the required database and table in the MySQL database to prepare the database structure for storing the data to be synchronized to Elasticsearch.

[0098] Step 2: Configure the Elasticsearch to MySQL synchronization tool and use Logstash to synchronize MySQL data to the two Elasticsearch clusters.

[0099] Specifically, this step includes:

[0100] Select and configure Logstash as the synchronization tool;

[0101] Use Logstash to read data from MySQL;

[0102] The read data is forwarded to two Elasticsearch clusters to synchronize the data in MySQL to the two Elasticsearch clusters.

[0103] Step 3: Configure the master-slave mode, define a master cluster and a backup cluster, use the synchronization tool to maintain data consistency, and automatically trigger the master-slave switch when the health status of the master cluster deteriorates.

[0104] Specifically, this step includes:

[0105] Define the primary and backup clusters: define one of the Elasticsearch clusters as the primary cluster, which is responsible for processing write operations and index updates; define the other as the backup cluster, which is used for data synchronization and takes over services when the primary cluster fails;

[0106] Configuration synchronization tool: Use Logstash to send data to the Elasticsearch nodes in the active and standby clusters simultaneously to keep data synchronized;

[0107] Configure the active / standby switchover script, including initializing script parameters, implementing health check functions, and configuring an automated trigger mechanism.

[0108] Step 4: Monitoring and failover, set alarm thresholds and monitoring intervals, and define automated failover logic to ensure automated active / standby switching and health status detection when problems occur in the active cluster.

[0109] Specifically, this step includes:

[0110] Implement a monitoring system and configure it to regularly check the health of the main cluster;

[0111] Set alarm thresholds and monitoring intervals;

[0112] Detect and trigger failover. If the primary cluster is detected to be unavailable, the failover logic is triggered.

[0113] Execute the failover logic, including promoting the backup node to the new primary node, modifying the synchronization tool configuration, starting the synchronization tool and restoring data;

[0114] Restore the original master node and rejoin it to the cluster;

[0115] Completes the failover, updates the status of the monitoring system, and logs information about the failover event.

[0116] Through the above steps, this embodiment provides a data fault-tolerant synchronization system based on the master-slave cluster node switching of Elasticsearch and MySQL, which realizes the automatic switching of the master-slave cluster and data fault-tolerant synchronization, and improves the availability and stability of the system. At the same time, the system can be widely used in scenarios that require processing large amounts of data and high availability, such as finance, e-commerce, logistics and other fields.

[0117] Embodiment 2: This embodiment describes a data fault-tolerant synchronization method based on Elasticsearch master-slave cluster node switching. The method includes the following steps:

[0118] Step 1: Configure the environment

[0119] First, install and configure the MySQL database to obtain a correctly configured MySQL database instance. Next, install and configure two Elasticsearch clusters to obtain two correctly configured Elasticsearch cluster instances, where the Elasticsearch cluster is referred to as the ES cluster. Then, deploy the necessary components on the server that can access the two Elasticsearch clusters to complete the preparation of the deployment environment. Finally, create the required databases and tables in the MySQL database to store the data to be synchronized to Elasticsearch.

[0120] Step 2: Data Synchronization

[0121] Select and configure Logstash as the synchronization tool to synchronize data from MySQL to the Elasticsearch cluster. Use Logstash to read data from MySQL and forward the read data to the two Elasticsearch clusters to achieve real-time data synchronization.

[0122] Step 3: Configure the active / standby mode

[0123] In the master-slave mode, one of the Elasticsearch clusters is defined as the master cluster, which is responsible for handling write operations and index updates. The other Elasticsearch cluster is defined as the backup cluster, which is used for data synchronization and taking over services when the master cluster fails. Configure Logstash to send data to the Elasticsearch nodes in the master and backup clusters at the same time to keep data synchronized. In addition, configure the master-slave switching script, including initializing script parameters, implementing health check functions, and configuring automated triggering mechanisms. When it is detected that the health status of the master cluster has deteriorated, the master-slave switching logic will be triggered, including promoting the backup node to the new master node and redirecting write traffic to the new master node.

[0124] Step 4: Monitoring and Failover

[0125] Configure the monitoring system to regularly check the health status of the primary cluster, including the running status and health status monitoring items of the primary and standby clusters. Set alarm thresholds and monitoring intervals to periodically detect health problems of the primary cluster. If the primary cluster is detected to be unavailable, use the fault detection function to verify the unavailable state of the primary cluster and trigger the failover logic. Executing the failover logic includes calling the node promotion function to promote the node in the backup cluster to the new primary node, marking the original primary node as a backup node, and modifying the configuration of the synchronization tool Logstash through automated configuration to redirect data traffic to the new primary node. Start or restart Logstash so that the missing data in the backup cluster is synchronized back and the data consistency and integrity are maintained. Once the health status of the original primary node is restored, synchronize it back to the new primary and standby clusters and rejoin the cluster. Finally, the monitoring system confirms the health status of the new primary cluster, updates the status of the monitoring system, and records relevant information about the failover event.

[0126] Through the method of this embodiment, data fault-tolerant synchronization based on the master-slave cluster node switching of Elasticsearch and MySQL can be achieved, thereby improving data reliability and system availability. When a problem occurs in the master cluster, the master-slave switching can be automatically triggered, and rapid data recovery and consistency guarantee can be achieved. The method can be widely used in distributed systems that require high availability and data fault tolerance.

[0127] The present invention has the following characteristics

[0128] 1. Real-time: By monitoring MySQL binlog, this device can obtain data changes in real time and pass them to Elasticsearch to achieve fast data synchronization.

[0129] 2. Simplified configuration: This device uses the binlog input plug-in of Logstash and the Elasticsearch output plug-in to simplify configuration and integration and quickly start data synchronization.

[0130] 3. Scalability: Logstash is a highly scalable data processing engine that can add filters and converters as needed to process data and meet different synchronization requirements.

[0131] 4. Real-time data synchronization: This device supports real-time data synchronization. It can capture data changes in the MySQL database and transmit them to the Elasticsearch cluster in real time. This ensures that changes made in the MySQL database can be reflected in Elasticsearch in a timely manner, so that the indexing and search of data remain up to date.

[0132] 5. Flexible configuration options: This device provides flexible configuration options that can be customized according to specific needs. Configure data filters, converters, and mapping rules to meet specific data synchronization requirements. This flexibility makes it easier to adapt to different data structures and business needs.

[0133] 6. Incremental and full synchronization support: This device supports incremental and full synchronization. Incremental synchronization means only the changed data is transmitted, while full synchronization means the entire data set is transmitted. This flexibility makes data synchronization more efficient and controllable in different scenarios.

[0134] 7. The present invention can also configure data conversion and mapping according to requirements: Preferably, those skilled in the art can perform data conversion and mapping according to requirements to meet the data format differences between different systems. Define the mapping rules of specific data fields to convert the data in the MySQL database into the format required by the Elasticsearch cluster. Ensure that the data can be correctly converted and synchronized between the two systems.

[0135] 7.1 Perform data conversion and mapping

[0136] Data format conversion: Convert the data types in MySQL to data types suitable for Elasticsearch.

[0137] Data cleaning: Clean and process the data as needed, such as removing unnecessary spaces or special characters.

[0138] Data renaming: Rename MySQL columns to Elasticsearch fields.

[0139] Data reorganization: Merge or split data to meet the needs of Elasticsearch indexing.

[0140] 7.2 The corresponding mapping relationship between MySQL and Elasticsearch table structures is as follows:

[0141] There is a MySQL table and a corresponding Elasticsearch index with the following fields:

[0142] MySQL table (relational):

[0143] Column Name Data Types id INT name VARCHAR(50) description TEXT created_at DATETIME category VARCHAR(20) price DECIMAL(10,2)

[0144] Elasticsearch index (document type):

[0145] Each document corresponds to a row of data in the MySQL table.

[0146]

[0147] The above shows a mapping relationship between MySQL tables and Elasticsearch indexes.

[0148] 8. Scalability and high performance: This device is designed to handle large-scale data volumes and high concurrent loads. It has good scalability and can increase the size of nodes or clusters when needed to meet the needs of data synchronization. At the same time, it is also optimized for performance to ensure high efficiency and low latency of the data synchronization process.

[0149] It should be noted that the above embodiments are only exemplary and do not limit the scope of protection of the present invention. In practical applications, the embodiments can be modified and adjusted according to specific needs as long as they comply with the principles and spirit of the present invention.

Claims

1. A data fault-tolerant synchronization method based on Elasticsearch master-slave cluster node switching, characterized in that: The following steps are involved: Step 1. Configure the MySQL database and two distributed Elasticsearch clusters; Step 2: Configure the Elasticsearch to MySQL synchronization tool and use Logstash to synchronize MySQL data to the two Elasticsearch clusters. Step 3: Configure the master-slave mode, define a master cluster and a backup cluster, use the synchronization tool to maintain data consistency, and automatically trigger the master-slave switch when the health status of the master cluster decreases; Step 4: Monitoring and failover, set alarm thresholds and monitoring intervals, and define automated failover logic to ensure automated active / standby switching and health status detection when problems occur in the active cluster.

2. According to claim 1, a data fault-tolerant synchronization method based on Elasticsearch master-slave cluster node switching is characterized in that: Step 1 specifically includes the following steps: Step 1.1: Install and configure the MySQL database to obtain a correctly configured MySQL database instance; Step 1.2: Install and configure two Elasticsearch clusters to obtain two correctly configured Elasticsearch cluster instances, where the Elasticsearch cluster is referred to as the ES cluster; Step 1.3: Deploy necessary components on a server that can access two Elasticsearch clusters. The deployment environment is ready. Step 1.4: Create the required database and table in the MySQL database. The database structure is ready to store the data to be synchronized to Elasticsearch.

3. According to a data fault-tolerant synchronization method based on Elasticsearch master-slave cluster node switching according to claim 1, it is characterized in that: Step 2 specifically includes the following steps: Step 2.1: Select and configure Logstash as the synchronization tool and prepare for data synchronization; Step 2.2: Read data from MySQL using Logstash; Step 2.3: Forward the read data to the two Elasticsearch clusters, and the data in MySQL is synchronized to the two Elasticsearch clusters.

4. According to a data fault-tolerant synchronization method based on Elasticsearch master-slave cluster node switching according to claim 1, it is characterized in that: Step 3 specifically includes the following steps: Step 3.1: Define the primary and backup clusters: Define one of the Elasticsearch clusters as the primary cluster, responsible for processing write operations and index updates; define the other as a backup cluster, used for data synchronization and taking over services when the primary cluster fails; Step 3.2: Configure the synchronization tool: Logstash sends data to the Elasticsearch nodes in the active and standby clusters at the same time to keep the data synchronized; Step 3.3: Configure the active / standby switch script: Step 3.3.1, initialize the script parameters, including the URLs of the primary and backup clusters, and establish connections with the primary and backup clusters through the client connected to the Elasticsearch cluster; Step 3.3.2, implement the health check function: Develop a health check function to obtain cluster health information by calling the Elasticsearch API and determine the health status of the main cluster; Step 3.3.3, configure the automated trigger mechanism: If it is detected that the health status of the primary cluster has deteriorated, trigger the primary-backup switchover logic, including promoting the backup node to the new primary node and redirecting the write traffic to the new primary node.

5. According to a data fault-tolerant synchronization method based on Elasticsearch master-slave cluster node switching according to claim 1, it is characterized in that: Step 4 specifically includes the following steps: Step 4.1: Implement monitoring system: Configure the monitoring system to regularly check the health status of the active cluster, including the running status of the active and standby clusters and the monitoring items of the active and standby cluster health status; Step 4.2: Set the alarm threshold and monitoring interval: Set alarm thresholds and monitoring intervals to periodically detect health issues in the primary cluster; Step 4.3, detect and trigger failover: If the primary cluster is detected to be unavailable, the fault detection function is used to verify the unavailable status of the primary cluster. If the primary cluster is detected to be unavailable, the failover logic is triggered; Step 4.4: Execute the failover logic: Call the node promotion function to promote the node in the backup cluster to the new master node and mark the original master node as a backup node; Modify the configuration of the synchronization tool Logstash through automated configuration to redirect data traffic to the new master node; Step 4.4, start the synchronization tool and restore data: Start or restart the synchronization tool Logstash to synchronize the missing data in the backup cluster and maintain data consistency and integrity. Step 4.5: Restore the original master node: Once the health status of the original master node is restored, synchronize it back to the new master-slave cluster and rejoin the cluster; Step 4.6, complete the failover: The monitoring system confirms the health of the new primary cluster, updates the status of the monitoring system, and logs information about the failover event.

6. A data fault-tolerant synchronization device for switching between master and standby cluster nodes based on Elasticsearch, characterized in that: Includes the following modules: Configure modules, configure MySQL database and two distributed Elasticsearch clusters; Synchronization tool module, configure the synchronization tool from Elasticsearch to MySQL, and use Logstash to synchronize MySQL data to two Elasticsearch clusters; The master-slave mode configuration module defines a master cluster and a backup cluster, uses synchronization tools to maintain data consistency, and automatically triggers master-slave switching when the health status of the master cluster decreases; The monitoring and failover module sets alarm thresholds and monitoring intervals, and defines automated failover logic to ensure automated active / standby switching and health status detection when problems occur in the active cluster.

7. A data fault-tolerant synchronization device for switching between master and standby cluster nodes based on Elasticsearch according to claim 6, characterized in that: The configuration module specifically includes the following steps: Step 1.1: Install and configure the MySQL database to obtain a correctly configured MySQL database instance; Step 1.2: Install and configure two Elasticsearch clusters to obtain two correctly configured Elasticsearch cluster instances, where the Elasticsearch cluster is referred to as the ES cluster; Step 1.3: Deploy necessary components on a server that can access two Elasticsearch clusters. The deployment environment is ready. Step 1.4: Create the required database and table in the MySQL database. The database structure is ready to store the data to be synchronized to Elasticsearch.

8. The data fault-tolerant synchronization device for switching between master and standby cluster nodes based on Elasticsearch according to claim 6, characterized in that: The synchronization tool module specifically includes the following steps: Step 2.1: Select and configure Logstash as the synchronization tool and prepare for data synchronization; Step 2.2: Read data from MySQL using Logstash; Step 2.3: Forward the read data to the two Elasticsearch clusters, and the data in MySQL is synchronized to the two Elasticsearch clusters.

9. The data fault-tolerant synchronization device for switching between master and standby cluster nodes based on Elasticsearch according to claim 1, characterized in that: The active / standby mode configuration module specifically includes the following steps: Step 3.1: Define the primary and backup clusters: Define one of the Elasticsearch clusters as the primary cluster, responsible for processing write operations and index updates; define the other as a backup cluster, used for data synchronization and taking over services when the primary cluster fails; Step 3.2: Configure the synchronization tool: Logstash sends data to the Elasticsearch nodes in the active and standby clusters at the same time to keep the data synchronized; Step 3.3: Configure the active / standby switch script: Step 3.3.1, initialize the script parameters, including the URLs of the primary and backup clusters, and establish connections with the primary and backup clusters through the client connected to the Elasticsearch cluster; Step 3.3.2, implement the health check function: Develop a health check function to obtain cluster health information by calling the Elasticsearch API and determine the health status of the main cluster; Step 3.3.3, configure the automated trigger mechanism: If it is detected that the health status of the primary cluster has deteriorated, trigger the primary-backup switchover logic, including promoting the backup node to the new primary node and redirecting the write traffic to the new primary node.

10. The data fault-tolerant synchronization device for switching between master and standby cluster nodes based on Elasticsearch according to claim 6, characterized in that: The monitoring and failover module specifically includes the following steps: Step 4.1: Implement monitoring system: Configure the monitoring system to regularly check the health status of the active cluster, including the running status of the active and standby clusters and the monitoring items of the active and standby cluster health status; Step 4.2: Set the alarm threshold and monitoring interval: Set alarm thresholds and monitoring intervals to periodically detect health issues in the primary cluster; Step 4.3, detect and trigger failover: If the primary cluster is detected to be unavailable, the fault detection function is used to verify the unavailable status of the primary cluster. If the primary cluster is detected to be unavailable, the failover logic is triggered; Step 4.4: Execute the failover logic: Call the node promotion function to promote the node in the backup cluster to the new master node and mark the original master node as a backup node; Modify the configuration of the synchronization tool Logstash through automated configuration to redirect data traffic to the new master node; Step 4.4, start the synchronization tool and restore data: Start or restart the synchronization tool Logstash to synchronize the missing data in the backup cluster and maintain data consistency and integrity. Step 4.5: Restore the original master node: Once the health status of the original master node is restored, synchronize it back to the new master-slave cluster and rejoin the cluster; Step 4.6, complete the failover: The monitoring system confirms the health of the new primary cluster, updates the status of the monitoring system, and logs information about the failover event.

Images