A method and device for realizing strong consistency synchronization of an Elasticsearch cluster

Abstract

The application discloses a method and device for realizing strong consistency synchronization of an Elasticsearch cluster, and the method comprises the following steps: writing data into a master shard of a master index to which a document belongs; after the data is written successfully, sending a write request to a node where a master shard of a slave index to which the document belongs is located, regarding the master shard of the slave index as a remote copy of the master shard of the master index, writing data into the remote copy, and realizing real-time data synchronization; the master shard of the master index sends a segment file which has been landed on a disk to the remote copy, and analyzes storage information of actual data in the segment file; the master shard of the master index sends a translog log which has not been landed on a disk to the remote copy, so that the remote copy is restored to a data state consistent with the master shard, and historical data recovery is realized. The method and device copy indexes in a remote cluster to a local cluster, ensure strong consistency of master-slave data, provide high availability and fast disaster recovery capability, and realize uninterrupted inter-cluster index migration.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to a method and apparatus for achieving strong consistency synchronization of an Elasticsearch cluster. Background Technology

[0002] Native Elasticsearch provides high availability to ensure the reliability of data within a cluster. However, this high availability is insufficient for users with higher reliability requirements. Native Elasticsearch has some limitations, such as the inability to quickly recover from data center-level failures and the high cost of migrating data between clusters, requiring the use of external tools to perform multiple complex operations.

[0003] Currently, enterprises often address high availability requirements by simultaneously writing data to different clusters. However, this method is highly dependent, does not support historical data synchronization, and cannot guarantee strong consistency between master and slave index data. Elasticsearch introduced inter-cluster data replication in version 6.7.0, but this feature is not free and only guarantees eventual consistency between master and slave index data. Many enterprises' core business requires strong consistency between master and slave index data. Therefore, this feature is still insufficient to meet enterprise needs. Summary of the Invention

[0004] To address the above issues, this invention provides a method and apparatus for achieving strong consistency synchronization of Elasticsearch clusters. The aim is to replicate indexes from a remote cluster to a local cluster, ensuring strong consistency between master and slave data, thereby providing high availability and rapid disaster recovery capabilities, while also enabling uninterrupted index migration between clusters.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In one embodiment of the present invention, a method for achieving strong consistency synchronization of an Elasticsearch cluster is proposed, the method comprising:

[0007] The data is written to the primary shard of the primary index to which the document belongs. After the data is successfully written, the write request is sent to the node where the primary shard of the secondary index to which the document belongs is located. At the same time, the primary shard of the secondary index is regarded as a remote copy of the primary shard of the primary index, and the data is written to the remote copy to achieve real-time data synchronization.

[0008] The primary shard of the primary index sends the segment files that have been written to disk to the remote replica and parses the storage information of the actual data in the segment files; the primary shard of the primary index sends the translog logs that have not been written to disk to the remote replica, thereby restoring the remote replica to the same data state as the primary shard, realizing historical data recovery.

[0009] Furthermore, the association between master and slave indexes is defined in the Elasticsearch cluster state to establish links between Elasticsearch clusters, and updates are made according to Elasticsearch cluster state change events, thereby triggering real-time data synchronization.

[0010] Furthermore, periodically check the connection status between Elasticsearch clusters and the data differences between master and slave indexes; when the data differences between master and slave indexes are too large or the connection is abnormal, the master cluster will actively disconnect the connection between Elasticsearch clusters and notify the slave index to perform the difference data recovery operation.

[0011] Furthermore, historical data recovery from the secondary index is triggered via RPC calls from the primary cluster, while only the primary shard data of the secondary index needs to be recovered.

[0012] Once the historical data recovery is complete, the primary shard of the index will be added to the remote replica group, and write requests for the primary index will begin to be received.

[0013] In one embodiment of the present invention, a device for implementing strong consistency synchronization of an Elasticsearch cluster is also proposed, the device comprising:

[0014] The Elasticsearch write module is used to write data to the primary shard of the primary index to which the document belongs. After the data is successfully written, the write request is sent to the node where the primary shard of the secondary index to which the document belongs is located. At the same time, the primary shard of the secondary index is regarded as a remote replica of the primary shard of the primary index, and the data is written to the remote replica to achieve real-time data synchronization.

[0015] The Elasticsearch secondary shard recovery module is used to send the segment files that have been persisted to disk to the remote replicas from the primary shard of the primary index, and to parse the storage information of the actual data in the segment files; the primary shard of the primary index sends the translog logs that have not been persisted to disk to the remote replicas, thereby restoring the remote replicas to the same data state as the primary shard, and realizing the recovery of historical data.

[0016] Furthermore, the association between master and slave indexes is defined in the Elasticsearch cluster state to establish links between Elasticsearch clusters, and updates are made according to Elasticsearch cluster state change events, thereby triggering real-time data synchronization.

[0017] Furthermore, periodically check the connection status between Elasticsearch clusters and the data differences between master and slave indexes; when the data differences between master and slave indexes are too large or the connection is abnormal, the master cluster will actively disconnect the connection between Elasticsearch clusters and notify the slave index to perform the difference data recovery operation.

[0018] Furthermore, historical data recovery from the secondary index is triggered via RPC calls from the primary cluster, while only the primary shard data of the secondary index needs to be recovered.

[0019] Once the historical data recovery is complete, the primary shard of the index will be added to the remote replica group, and write requests for the primary index will begin to be received.

[0020] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned method for implementing strong consistency synchronization of an Elasticsearch cluster.

[0021] In one embodiment of the present invention, a computer-readable storage medium is also proposed, which stores a computer program that implements a strong consistency synchronization method for an Elasticsearch cluster.

[0022] Beneficial effects:

[0023] 1. This invention enables consistent replication of master-slave index data across Elasticsearch clusters, ensuring a high degree of consistency between master and slave index data and meeting the high reliability requirements of core enterprise business.

[0024] 2. This invention draws on the replica recovery mechanism to achieve efficient historical data recovery, and adopts a master-write-slave-read mechanism to ensure the timeliness of real-time data synchronization.

[0025] 3. This invention performs regular data verification and automatic recovery, fully ensuring the reliability of master-slave index data.

[0026] 4. This invention supports seamless primary / standby switching and inter-cluster migration, providing continuous service capabilities and shortening business migration time.

[0027] 5. This invention has minimal intrusion into the source code, low development difficulty, strong versatility, and reduces research and development and application costs. Attached Figure Description

[0028] Figure 1 This is a flowchart of the Elasticsearch write process of this invention;

[0029] Figure 2 This is a flowchart of the Elasticsearch secondary shard recovery process in this invention;

[0030] Figure 3 This is the core conceptual diagram of how the present invention achieves strong consistency synchronization of Elasticsearch clusters;

[0031] Figure 4 This is a schematic diagram of the process for implementing a strongly consistent synchronization method for an Elasticsearch cluster according to the present invention;

[0032] Figure 5 This is a flowchart illustrating the process of building inter-cluster links in Elasticsearch according to an embodiment of the present invention.

[0033] Figure 6 This is a flowchart of a historical data recovery process according to an embodiment of the present invention;

[0034] Figure 7 This is a flowchart of real-time data synchronization according to an embodiment of the present invention;

[0035] Figure 8 This is a flowchart of the master-slave index data quality verification process according to an embodiment of the present invention;

[0036] Figure 9 This is a schematic diagram of the structure of the Elasticsearch cluster strong consistency synchronization device of the present invention;

[0037] Figure 10 This is a schematic diagram of the computer device structure of the present invention. Detailed Implementation

[0038] The principles and spirit of the present invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0039] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0040] According to an embodiment of the present invention, a method for achieving strong consistency synchronization of Elasticsearch clusters is proposed. This method aims to replicate indexes from a remote cluster to a local cluster, ensuring strong consistency between master and slave data, thereby providing high availability and rapid disaster recovery capabilities, while also enabling uninterrupted inter-cluster index migration. The following two aspects need to be considered during the replication process:

[0041] 1. Real-time data synchronization and historical data synchronization:

[0042] (1) Real-time data synchronization refers to copying the latest data from the remote cluster to the local cluster in real time to ensure that the data between the two clusters remains synchronized.

[0043] (2) Historical data synchronization refers to copying historical data from a remote cluster to a local cluster to ensure that the local cluster has a complete copy of the data.

[0044] 2. Real-time data synchronization relies on Elasticsearch's write mechanism, and data synchronization also relies on Elasticsearch's replica recovery mechanism.

[0045] (1) During real-time data synchronization, this invention relies on the Elasticsearch writing mechanism to ensure that data can be synchronized to the local cluster in a timely manner after being written to the remote cluster.

[0046] (2) During the historical data synchronization process, this invention relies on Elasticsearch’s replica recovery mechanism to ensure that data can be recovered in the event of failure or damage.

[0047] The solution and implementation details of this invention are based on the Elasticsearch write and recovery process. The following is a brief overview of the two key processes of Elasticsearch.

[0048] 1. Basic writing process

[0049] The Elasticsearch write process follows these steps: First, data is written to the primary shard (P) of the primary index. After the primary shard (P) is written, the request is simultaneously forwarded to the replica (R). After the replica (R) completes its processing, it returns the write result to the primary shard. Note: In this invention, Si represents the specific shard of the Elasticsearch primary index, P represents the primary shard, and R represents the replica. The specific process is as follows: Figure 1As shown: When the client initiates a write request, the Master node NODE1 determines that the document belongs to shard 0 of the primary index based on the write request ID, and routes the write request to the NODE3 node where the primary shard P0 of shard 0 is located. After the primary shard P0 of the NODE3 node successfully writes the document, it sends the write request to the replica R0. When the replica R0 completes the write, it returns the write result to the client.

[0050] 2. Instance Recovery Process

[0051] To ensure data replica consistency, replicas need to be restored to a state consistent with the primary shard to provide normal external services. Elasticsearch replica recovery is performed at the shard level, including primary shard recovery and replica recovery processes. Because the Elasticsearch replica recovery process is extremely complex, and this invention only involves the data recovery process related to replica recovery, only a brief introduction to the replica recovery process is provided.

[0052] The goal of replica recovery is to restore local data to a state consistent with the primary shard. The main process consists of two phases:

[0053] Phase 1: In this phase, where the primary shard sends segment files to the replicas, the primary shard sends the segment files, which have already been persisted to disk, to the replicas and parses the segment files containing the actual data. This segment file contains the storage information for the actual data. The segment file is the smallest unit of physical storage and querying in Elasticsearch; it contains all document data, index information, document IDs, etc., belonging to that segment file.

[0054] Phase Two: In this phase, where the primary shard sends translog logs to the replicas, the primary shard sends data that has not yet been persisted to disk, similar to MySQL's WAL log, i.e., the translog log. This log records data operations that have not yet been persisted. In Elasticsearch, the translog log is an implementation of the Write-Ahead Log (WAL) used to record the most recent data changes that have not yet been persisted, for data recovery and to ensure data consistency.

[0055] The process is complete after completing the two stages described above. Through these steps, the replica achieves a data state consistent with the primary shard, ensuring data consistency and availability.

[0056] like Figure 2 As shown, the specific copy restoration process is as follows:

[0057] (1) The replica recovery is triggered by the cluster state change event, and then the INIT initialization checks the local replica index data status.

[0058] (2) After initialization, the recovery process begins, and INDEX obtains the index file from the primary shard.

[0059] (3) The primary shard sends the persisted segment file to the replica. The replica writes the received segment file to its local storage.

[0060] (4) After writing to the local machine, before receiving the translog operation log, VERIFY_INDEX needs to verify the index integrity by comparing the file lists of the primary shard and the replica shard to check for any missing files. If any files are missing, the primary shard will be requested to resend the missing files.

[0061] (5) The replica sharding engine will be started during the VERIFY_INDEX verification phase. Changes to the operation log can only be applied after the engine is started.

[0062] (6) At the same time, the replica shard is added to the replica group of the primary shard. Only after being added to the replica group will the primary shard send the data changes in the operation log to the replica shard.

[0063] If not joined to a replica group, the primary shard will not replicate the operation log to that shard.

[0064] (7) TRANSLOG retrieves operation logs from the primary shard.

[0065] The primary shard sends non-persistent translog operation logs to the replicas. The replicas read the logs, replay the operations, and apply them to their local storage.

[0066] (8) Final processing

[0067] The replica shard is set to an available state and can provide normal service. The replica shard is then added to the normal replica group of the primary shard.

[0068] 3. Design Concept

[0069] The core concept of this invention is to treat the shard corresponding to the primary index as a remote copy of the shard corresponding to the primary index. For example... Figure 3 As shown, the primary shard S0(P) of the primary index is regarded as a remote copy of the primary shard S0(P).

[0070] The remote replica in this invention is an extension of the Elasticsearch data replication model. It uses the primary shard of the primary index to store the relevant metadata of the remote replica and borrows from Microsoft's PacificA algorithm in its implementation. This design aligns with the Elasticsearch data replication model, fully reusing Elasticsearch replication logic, thereby reducing development difficulty and minimizing intrusion into the open-source Elasticsearch core. The PacificA algorithm is a distributed, strongly consistent replication algorithm proposed by Microsoft in 2013. The Elasticsearch data replication model refers to the design method or pattern of Elasticsearch replicas. The core concepts of the Elasticsearch data replication model are primary shards and replicas. The primary shard is the original replica of each index shard, responsible for handling all write operations and query requests; replicas are copies of the primary shard, used to provide redundancy and load balancing; replicas can be distributed across different nodes, thereby achieving high data availability and fault tolerance.

[0071] Table 1 below lists some of the core terms of the algorithm and their correspondence with the Elasticsearch data replication model.

[0072] Table 1

[0073]

[0074] This invention includes the following:

[0075] (1) Building inter-cluster links: Define inter-cluster links in Elasticsearch using custom metadata, including master index, slave index, and slave cluster information. Update the metadata of inter-cluster links to the Elasticsearch cluster state using the API provided by Elasticsearch, and update it according to Elasticsearch cluster state change events, thereby triggering real-time data synchronization.

[0076] (2) Historical Data Recovery: Recovery of replicas of the secondary index is triggered via RPC calls from the primary cluster. Only the primary shard of the secondary index is recovered to expedite the recovery process. During the migration, the primary shard is locked to prevent data migration. Once recovery is complete, the primary shard is added to the remote replica group from the secondary index.

[0077] (3) Real-time data synchronization: After historical data synchronization is completed, incremental data is synchronized to the slave index after a successful primary shard write, and the data is forwarded to replicas and remote replicas. This approach ensures strong data consistency. Strong consistency is guaranteed by synchronizing and forwarding data to the slave index after a successful primary shard write.

[0078] (4) Data quality verification: Periodically check the status of links between Elasticsearch clusters. If a problem is found, automatically disconnect the links between clusters and notify the data nodes to perform a recovery operation. The master node checks the metadata, while the data nodes determine whether the primary and secondary shards need to be recovered.

[0079] This invention first involves defining the relationship between master and slave indexes in the Elasticsearch cluster state to establish an inter-cluster link, which is the foundation for the entire invention's data synchronization. Once the inter-cluster link is established, historical data recovery is required. This is done by replicating the data from the master index to the slave index at the shard level, ensuring that the data state of the slave index is consistent with the master index, thus laying the foundation for subsequent real-time data synchronization. After historical data recovery is complete, a real-time data synchronization mechanism is initiated, synchronously forwarding write requests from the master index to the slave index to achieve strong data consistency between the master and slave indexes. Simultaneously, data quality checks are performed periodically to check for data differences between the master and slave indexes. If a problem is found, the inter-cluster link is disconnected and a new historical data recovery is triggered to ensure data reliability. Through the organic coordination of these steps, this invention achieves strong consistency data replication between different Elasticsearch clusters, applicable to purposes such as high availability and downtime-free migration between Elasticsearch clusters.

[0080] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0081] To provide a clearer explanation of the above method for achieving strong consistency synchronization in an Elasticsearch cluster, a specific embodiment will be used for illustration below. However, it is worth noting that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation of the present invention.

[0082] Example:

[0083] This invention relates to a method for implementing data replication capabilities between Elasticsearch clusters. To achieve this functionality, firstly, the index template or index that needs to be replicated between Elasticsearch clusters needs to be identified, i.e., establishing the Elasticsearch cluster link according to this invention. Secondly, the slave index in this invention acts as a remote replica and needs to be restored to a data state consistent with the master index to provide normal service, i.e., historical data recovery. After achieving consistency between the slave and master indexes, it is necessary to determine how to write new data to the slave index when new data is added to the master index, i.e., real-time data synchronization. After completing the above steps, the slave index will be able to provide normal service. To ensure data reliability, data quality verification of the master and slave indexes is also required.

[0084] like Figure 4 As shown, the design of this invention includes the following four aspects:

[0085] 1. Establishing inter-cluster links

[0086] An Elasticsearch cluster is a system driven by its cluster state. The core of this invention is to establish links between Elasticsearch clusters by changing their states and applying the new Elasticsearch cluster state configuration to the corresponding nodes. Typically, enterprises use Elasticsearch internally using index templates (a set of indexes with the same prefix). Therefore, the design of links between Elasticsearch clusters needs to support both template-level and index-level propagation between Elasticsearch clusters.

[0087] This invention enables the propagation of cluster metadata between Elasticsearch clusters by customizing the `metaData` field in the Elasticsearch cluster status. Links between Elasticsearch clusters use a unified naming convention to distinguish between templates and indexes. Key information includes:

[0088] Links between template clusters:

[0089]

[0090] Inter-cluster links in the index:

[0091]

[0092] "replica_cluster": "ClusterA", / / from cluster name

[0093] "replication_state": true / / Inter-cluster link status

[0094] }

[0095] }

[0096] Elasticsearch clusters provide an Application Programming Interface (API) for storing and managing inter-cluster link information. This invention adds access to and update functionality for the `metaData` field to the API to synchronize inter-cluster link metadata to the cluster state configuration of each cluster. When the cluster state configuration changes, the system detects this event and initiates the data synchronization process, such as... Figure 5 As shown:

[0097] When designing a data synchronization mechanism for inter-cluster links, the following details need to be considered:

[0098] (1) Considering the development difficulty and the intrusiveness to the source code, the primary and secondary indexes use a unified index name and index ID. When creating a secondary index, the system propagates the index ID corresponding to the primary index to the secondary cluster. This avoids the secondary index generating a new, inconsistent index ID on its own during creation.

[0099] (2) After an index is deleted, the Elasticsearch system temporarily retains the data of the deleted index in the shard responsible for reclamation. To prevent the failure of rebuilding the index due to this, the system of this invention will scan the shard responsible for reclamation before creating the secondary index and delete the index whose index ID is the same as the primary index, thus solving the problem of the secondary index being unable to be rebuilt after deletion.

[0100] (3) When a new inter-cluster link is established, the system synchronizes the mapping definition of the primary index to the secondary cluster. In the secondary cluster, it creates a secondary index corresponding to the primary index based on the propagated mapping definition. This ensures that the primary and secondary indexes of each cluster in the inter-cluster link always maintain consistency in their mapping structure.

[0101] 2. Historical data recovery

[0102] The historical data recovery solution borrows from Elasticsearch's replica recovery mechanism in its design. Specifically, data recovery across cluster links is also performed at the shard level, requiring segment and translog replication. Conditions for historical data recovery:

[0103] (1) When creating a new inter-cluster link, historical data needs to be restored based on the main index.

[0104] (2) When the write of sharded data from the index fails, the main index timer task will rebuild the inter-cluster link of the present invention.

[0105] like Figure 6As shown, the historical data recovery process for a slave index as a remote replica is basically the same as that of Elasticsearch, with the main differences being the data recovery triggering condition in step 1 and the method of adding the remote replica group in step 6. Since Elasticsearch replica recovery is driven by cluster state change events, while slave index recovery is inter-cluster, it relies on RPC calls from the master cluster to trigger historical data recovery for the slave cluster. Simultaneously, the slave index only needs to recover its own primary shard data, while the replica recovery of the slave index depends on the Elasticsearch replica mechanism of the slave cluster. After the first phase (phase 1) of Elasticsearch replication ends, the replica engine starts and adds the recovered replica to the primary shard replica group, beginning to receive write requests from the primary shard. The recovery of the slave index is similar; the primary shard of the slave index acts as a remote replica of the primary shard corresponding to the primary index. After the first phase (phase 1) of the primary shard of the primary index ends, the slave index engine starts and adds the corresponding primary shard of the slave index to the remote replica group, beginning to receive write requests. After the historical data recovery of the slave index is complete, the slave index can normally receive write requests from the primary index, thus greatly reducing the historical data recovery time for inter-cluster links in this invention.

[0106] The implementation of remote replicas borrows from Microsoft's PacificA algorithm. It adds a remote prepared list named `preparedRemoteCopies` to the `ReplicationGroup` class of the primary index to track remote replicas. When the primary shard needs to synchronize a write request to a remote replica, it first adds the remote replica node information to the `preparedRemoteCopies` list, then calls the `syncWithRemoteCopies` method of `ReplicationGroup` to synchronously transmit the write request to the remote replica over the network. Once the remote replica data synchronization is complete, it is moved from the `preparedRemoteCopies` list to the official `remoteCopies` list. Afterward, write requests from the primary shard can be replicated to both the local and remote replicas simultaneously.

[0107] 3. Real-time data synchronization

[0108] The real-time data synchronization of this invention refers to the process of immediately replicating incremental data to the slave index after historical data synchronization is completed. According to the Elasticsearch write principle, write requests are first sent to the primary shard, and then the write operation is synchronously forwarded to the replicas. This invention adopts a technical solution that immediately replicates and forwards data to local and remote replicas after a successful write to the primary shard to achieve real-time data synchronization. This solution ensures strongly consistent data synchronization.

[0109] like Figure 7 As shown, the real-time data synchronization strategy of this invention is similar to the historical data recovery mechanism, both employing the technique of forwarding write requests to remote replicas. Specifically, when the primary shard receives a write request, it writes the data to its local replica while simultaneously forwarding the write request to the remote replica. By calling the `syncWithRemoteCopies` method of the replication group, the write request can be sent to the remote replica node over the network. After receiving the write request, the remote replica node writes the data to the shard file. Once the remote replica has successfully written the data, it sends an acknowledgment message to the primary shard node. It is important to note the handling strategy when a remote replica write fails. In Elasticsearch, the handling of a replica write failure involves removing the faulty replica from the synchronization replica group and re-executing the data recovery process. The remote replica write failure handling strategy of this invention is similar to that of Elasticsearch: the faulty remote replica is removed from the primary shard's remote replica group. The primary index shard then no longer forwards write requests to that secondary index, instead relying on the secondary index's periodic check mechanism to re-trigger the data recovery process.

[0110] 4. Master-slave index data quality verification

[0111] Data quality verification is a crucial step in ensuring the reliability of indexed data. It periodically compares the metadata information in the cluster status with the actual operational status of inter-cluster links, and takes appropriate action on the links based on the verification results. When the data differences between the master and slave indexes are too large or the links are abnormal, the master cluster will proactively disconnect the inter-cluster links and notify the slave index to perform data recovery operations. In an Elasticsearch cluster, the master node is responsible for managing cluster metadata; therefore, if... Figure 8 As shown, in the scheme of this invention, the master node of the master cluster mainly completes the generation of inter-cluster link metadata and verification of the existence of the secondary index, etc.; the data node of the master cluster is responsible for data storage, and its main function is to determine whether the master-slave shards need to be restored, including comparing the total amount of master-slave index data, verifying remote replica metadata, verifying the recovery of differential data, and checking the health of index links, etc.

[0112] The above four steps enable native data replication from one Elasticsearch cluster to another. Combined with a master-slave failover strategy, high availability between clusters can be achieved while ensuring strong data consistency. For the goal of non-downtime index migration between clusters, this invention synchronizes data to the destination cluster, waits for the existing data to be restored, and then performs a master-slave failover. For the goal of reliable version upgrades, this invention replicates the data of the version to be upgraded to a standby cluster, enabling rapid cluster switching in the event of an upgrade failure.

[0113] Based on the same inventive concept, this invention also proposes a device for implementing strong consistency synchronization of Elasticsearch clusters. The implementation of this device can refer to the implementation of the methods described above, and repeated details will not be repeated. The term "module" used below can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0114] Figure 9 This is a schematic diagram of the structure of the Elasticsearch cluster strong consistency synchronization device of the present invention. Figure 9 As shown, the device includes:

[0115] Elasticsearch write module 101 is used to write data to the primary shard of the primary index to which the document belongs. After the data is successfully written, the write request is sent to the node where the primary shard of the secondary index to which the document belongs is located. At the same time, the primary shard of the secondary index is regarded as a remote replica of the primary shard of the primary index, and the data is written to the remote replica to achieve real-time data synchronization.

[0116] The Elasticsearch secondary shard recovery module 102 is used to send the segment files that have been persisted to disk to the remote replicas from the primary shard of the primary index, and to parse the storage information of the actual data in the segment files; the primary shard of the primary index sends the translog logs that have not been persisted to disk to the remote replicas, thereby restoring the remote replicas to the same data state as the primary shard, and realizing the recovery of historical data.

[0117] Define the relationship between master and slave indexes in the Elasticsearch cluster state to establish links between Elasticsearch clusters, and update them according to Elasticsearch cluster state change events to trigger real-time data synchronization.

[0118] Regularly check the connection status between Elasticsearch clusters and the data differences between master and slave indexes; when the data difference between master and slave indexes is too large or the connection is abnormal, the master cluster will actively disconnect the connection between Elasticsearch clusters and notify the slave index to perform the difference data recovery operation.

[0119] Historical data recovery from the secondary index is triggered by RPC calls from the primary cluster, and only the primary shard data of the secondary index needs to be recovered.

[0120] Once the historical data recovery is complete, the primary shard of the index will be added to the remote replica group, and write requests for the primary index will begin to be received.

[0121] It should be noted that although several modules for implementing a strong consistency synchronization mechanism for an Elasticsearch cluster are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in a single module. Conversely, the features and functions of a single module described above can be further divided and embodied by multiple modules.

[0122] Based on the aforementioned inventive concept, such as Figure 10 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored on the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, it implements the aforementioned method for implementing strong consistency synchronization of Elasticsearch clusters.

[0123] Based on the aforementioned inventive concept, the present invention also proposes a computer-readable storage medium storing a computer program that executes the aforementioned method for implementing strong consistency synchronization of an Elasticsearch cluster.

[0124] The method and apparatus for achieving strong consistency synchronization of Elasticsearch clusters proposed in this invention have the following highlights:

[0125] 1. It implements strong consistency replication of index data across Elasticsearch clusters, ensuring high consistency between master and slave index data, and meeting the high reliability requirements of core business operations.

[0126] 2. By drawing on the replica recovery mechanism, efficient historical data recovery was achieved, and the master-write-slave-read mechanism ensured the timeliness of real-time data synchronization.

[0127] 3. Regular data verification and automatic recovery are performed to fully ensure the reliability of master-slave index data.

[0128] 4. Supports seamless primary / standby switching and inter-cluster migration, providing continuous service capabilities and shortening business migration time.

[0129] 5. It has minimal intrusion into the source code, low development difficulty, strong versatility, and reduces R&D and application costs.

[0130] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0131] Regarding the limitation of the scope of protection of this invention, those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solution of this invention are still within the scope of protection of this invention.

Claims

1. A method for achieving strong consistency synchronization in an Elasticsearch cluster, characterized in that, The method includes: The data is written to the primary shard of the primary index to which the document belongs. After the data is successfully written, the write request is sent to the node where the primary shard of the secondary index to which the document belongs is located. At the same time, the primary shard of the secondary index is regarded as a remote copy of the primary shard of the primary index, and the data is written to the remote copy to achieve real-time data synchronization. The primary shard of the primary index sends the segment files that have been persisted to disk to the remote replica and parses the storage information of the actual data in the segment files. This storage information includes all document data, index information, and document IDs belonging to the segment file. The primary shard of the primary index sends the translog logs that have not been persisted to disk to the remote replica, thereby restoring the remote replica to a data state consistent with that of the primary shard, thus realizing historical data recovery. The specific dungeon recovery process is as follows: (1) The replica recovery is triggered by the cluster state change event, and then the INIT initialization checks the local replica index data status; (2) After initialization, the recovery process begins, and INDEX obtains the index file from the primary shard; (3) The primary shard sends the persisted segment file to the replica; the replica writes the received segment file to its local storage; (4) After writing to the local machine, before receiving the translog operation log, VERIFY_INDEX needs to verify the integrity of the index, compare the file lists of the primary shard and the replica shard, and check if there are any missing files; if there are any missing files, it will request the primary shard to resend the missing files. (5) The replica sharding engine will be started during the VERIFY_INDEX verification phase. Changes to the operation log can only be applied after the engine is started. (6) At the same time, add the replica shard to the replica group of the primary shard. Only after it is added to the replica group will the primary shard send the data changes in the operation log to the replica shard. If not joined to a replica group, the primary shard will not replicate the operation log to that shard; (7) TRANSLOG retrieves operation logs from the primary shard. The primary shard sends the non-persistent translog operation logs to the replicas; the replicas read the logs, replay the operations, and apply them to their local storage. (8) Final processing Set the replica shard to an available state so it can provide normal service, and add the replica shard to the normal replica group of the primary shard.

2. The method for achieving strong consistency synchronization of an Elasticsearch cluster according to claim 1, characterized in that, Define the relationship between master and slave indexes in the Elasticsearch cluster state to establish links between Elasticsearch clusters, and update them according to Elasticsearch cluster state change events to trigger real-time data synchronization.

3. The method for achieving strong consistency synchronization of an Elasticsearch cluster according to claim 1, characterized in that, Regularly check the connection status between Elasticsearch clusters and the data differences between master and slave indexes; when the data difference between master and slave indexes is too large or the connection is abnormal, the master cluster will actively disconnect the connection between Elasticsearch clusters and notify the slave index to perform the difference data recovery operation.

4. The method for achieving strong consistency synchronization in an Elasticsearch cluster according to claim 1, characterized in that, The recovery of historical data from the secondary index is triggered by an RPC call to the primary cluster, and only the primary shard data of the secondary index needs to be recovered. Once the historical data recovery is complete, the primary shard of the index will be added to the remote replica group, and write requests for the primary index will begin to be received.

5. A device for implementing strong consistency synchronization in an Elasticsearch cluster, characterized in that, The device includes: The Elasticsearch write module is used to write data to the primary shard of the primary index to which the document belongs. After the data is successfully written, the write request is sent to the node where the primary shard of the secondary index to which the document belongs is located. At the same time, the primary shard of the secondary index is regarded as a remote replica of the primary shard of the primary index, and the data is written to the remote replica to achieve real-time data synchronization. The Elasticsearch secondary shard recovery module is used to send the segment files that have been persisted to disk from the primary shard of the primary index to the remote replica, and to parse the storage information of the actual data in the segment files. This storage information includes all document data, index information, and document IDs belonging to the segment file. The primary shard of the primary index sends the translog logs that have not been persisted to disk to the remote replica, thereby restoring the remote replica to a data state consistent with the primary shard, realizing historical data recovery. The specific dungeon recovery process is as follows: (1) The replica recovery is triggered by the cluster state change event, and then the INIT initialization checks the local replica index data status; (2) After initialization, the recovery process begins, and INDEX obtains the index file from the primary shard; (3) The primary shard sends the persisted segment file to the replica; the replica writes the received segment file to its local storage; (4) After writing to the local machine, before receiving the translog operation log, VERIFY_INDEX needs to verify the integrity of the index, compare the file lists of the primary shard and the replica shard, and check if there are any missing files; if there are any missing files, it will request the primary shard to resend the missing files. (5) The replica sharding engine will be started during the VERIFY_INDEX verification phase. Changes to the operation log can only be applied after the engine is started. (6) At the same time, add the replica shard to the replica group of the primary shard. Only after it is added to the replica group will the primary shard send the data changes in the operation log to the replica shard. If not joined to a replica group, the primary shard will not replicate the operation log to that shard; (7) TRANSLOG retrieves operation logs from the primary shard. The primary shard sends the non-persistent translog operation logs to the replicas; the replicas read the logs, replay the operations, and apply them to their local storage. (8) Final processing Set the replica shard to an available state so it can provide normal service, and add the replica shard to the normal replica group of the primary shard.

6. The device for implementing strong consistency synchronization of an Elasticsearch cluster according to claim 5, characterized in that, Define the relationship between master and slave indexes in the Elasticsearch cluster state to establish links between Elasticsearch clusters, and update them according to Elasticsearch cluster state change events to trigger real-time data synchronization.

7. The device for implementing strong consistency synchronization of an Elasticsearch cluster according to claim 5, characterized in that, Regularly check the connection status between Elasticsearch clusters and the data differences between master and slave indexes; when the data difference between master and slave indexes is too large or the connection is abnormal, the master cluster will actively disconnect the connection between Elasticsearch clusters and notify the slave index to perform the difference data recovery operation.

8. The device for implementing strong consistency synchronization of an Elasticsearch cluster according to claim 5, characterized in that, The recovery of historical data from the secondary index is triggered by an RPC call to the primary cluster, and only the primary shard data of the secondary index needs to be recovered. Once the historical data recovery is complete, the primary shard of the index will be added to the remote replica group, and write requests for the primary index will begin to be received.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the method according to any one of claims 1-4.

Images