Implementation method and device for smooth upgrade of ES cluster

Abstract

The invention discloses an ES cluster smooth upgrade realization method and device, and the method comprises the steps: building a new cluster of a target version; requests of old cluster data are agented and forwarded by using ES access proxy middleware as an entry point; the proxy middleware is accessed through the ES, a snapshot creation request of the old cluster is initiated, a snapshot file is generated, and meanwhile, the old cluster returns an ID of the snapshot request; when a snapshot creation request is initiated, the ES access proxy middleware generates a transaction log according to newly added, updated and deleted requests; after synchronizing the snapshot file to a new cluster, based on the snapshot request ID, positioning, reading and analyzing a corresponding transaction log, generating addition, deletion and modification operations, and writing into the new cluster; and when the new and old cluster data are consistent, smoothly switching to the new cluster through the ES access proxy middleware. According to the method and the device, smooth upgrading of the ES cluster version is realized through offline initialization and real-time synchronization of data and introduction of proxy forwarding and transaction log synchronization components.

Description

Technical Field

[0001] The present invention relates to the field of Elasticsearch cluster version upgrade, and in particular to a method and device for implementing smooth upgrade of an ES cluster. Background Art

[0002] When upgrading the Elasticsearch cluster version, the following problems may occur:

[0003] 1. The Elasticsearch service needs to be stopped, which has a great impact on the business:

[0004] (1) ES service suspension means that businesses cannot access data, resulting in business interruption and service unavailability.

[0005] (2) Stopping ES services requires prior notification and planning to minimize the impact on the business.

[0006] (3) Long downtime will cause business delays and degraded user experience.

[0007] (4) Business persistence and high availability need to be guaranteed during downtime.

[0008] 2. There are certain risks in performing ES version upgrade operations on the same node:

[0009] (1) ES version upgrades involve changes in data structure and indexes, which may cause data corruption or unavailability.

[0010] (2) An ES version upgrade failure may cause the index to be unreadable or unwritable, affecting the normal operation of the business.

[0011] (3) ES version upgrades are usually irreversible, and rolling back to the previous version may require a lot of time and resources.

[0012] (4) Data loss or inconsistency during ES version upgrades may make data recovery difficult.

[0013] 3. It is difficult to support upgrades across multiple major versions:

[0014] (1) Upgrading across multiple major versions directly in an ES cluster is complex and difficult.

[0015] (2) Cross-version upgrades may require dealing with compatibility issues and data structure changes between multiple versions.

[0016] (3) Each version upgrade requires detailed planning and testing to ensure the stability of the upgrade process and the integrity of the data.

[0017] (4) Cross-version upgrades may require adjustments to indexing and querying methods to adapt to the functions and syntax of the new version. Summary of the invention

[0018] In order to solve the above problems existing in the upgrade of Elasticsearch cluster version, the present invention provides a method and device for realizing smooth upgrade of ES cluster, which realizes smooth upgrade of ES cluster version through offline initialization and real-time synchronization of data, as well as the introduction of proxy forwarding and transaction log synchronization components.

[0019] To achieve the above object, the present invention adopts the following technical solutions:

[0020] In one embodiment of the present invention, a method for implementing a smooth upgrade of an ES cluster is proposed, the method comprising:

[0021] Create a new ES cluster whose version is the target version to be upgraded and is used to synchronize the data of the cluster to be upgraded.

[0022] Build ES access proxy middleware as the only entry point to access the old and new ES clusters; all ES access requests for adding, updating, deleting, querying, and creating snapshots of the old ES cluster data are proxied and forwarded through the ES access proxy middleware as the entry point;

[0023] Through the ES access proxy middleware, initiate a snapshot request for the old ES cluster and generate a snapshot file. When the snapshot request for the old ES cluster is initiated, the old ES cluster will return a unique ID as the ID of the snapshot request.

[0024] When a snapshot creation request is initiated for the old ES cluster, the ES access proxy middleware generates a transaction log for the new, updated, and deleted ES access requests and sends it to the Kafka topic;

[0025] Synchronize the snapshot file of the old ES cluster to the new ES cluster. After the snapshot file synchronization is completed, locate the Kakfa topic where the transaction log is stored based on the snapshot request ID; read and parse the transaction log corresponding to the Kakfa topic, generate add, delete and modify operations, and synchronize them to the new ES cluster to achieve data playback and recovery;

[0026] After the data of the new ES cluster and the old ES cluster are consistent, smoothly switch to the new ES cluster through the ES access proxy middleware.

[0027] Further, use the following steps to create a new ES cluster of the target version to be upgraded:

[0028] Determine the target version to be upgraded, and make corresponding plans and preparations to ensure that the new cluster can meet the performance and reliability requirements after the upgrade;

[0029] According to the upgrade plan, determine the upgrade span of each version and ensure that it is eventually upgraded to the latest version;

[0030] Download and install the iterative version of the ES software package on the nodes of the new cluster according to the official documentation or guide;

[0031] Perform necessary configurations on the new cluster, including node name, cluster name, network binding, memory settings, and storage paths, to ensure that the configuration of the new cluster is consistent with that of the cluster to be upgraded;

[0032] Start the nodes of the new cluster one by one to ensure that the nodes start normally and join the new cluster; monitor the node startup log and the status of the new cluster to ensure that the new cluster runs normally.

[0033] Furthermore, the following steps are used to parse the ES access request, identify the type of request, and perform corresponding processing operations according to the type:

[0034] Identify the source and format of the request based on the request header information and request path;

[0035] Design different request parser modules for different request sources and formats to parse request data packets and extract key information, including operation type, operation target, and specific data content;

[0036] Convert request data packets of different formats into a unified internal representation, which is a standardized structure including operation type, operation target and specific data content;

[0037] After parsing and abstraction, if the request involves a data change operation, a corresponding transaction log is generated, which records the detailed information of the operation for auditing, rollback, or replay operations.

[0038] In one embodiment of the present invention, a device for implementing a smooth upgrade of an ES cluster is also proposed, the device comprising:

[0039] The ES new cluster module is used to establish a new ES cluster. The version of the new ES cluster is the target version to be upgraded and is used to synchronize the data of the cluster to be upgraded.

[0040] ES access proxy middleware module is used to build ES access proxy middleware as the only entry point for accessing the old and new ES clusters. All ES access requests for adding, updating, deleting, querying and creating snapshots of the old ES cluster data are proxied and forwarded through the ES access proxy middleware as the entry point. Through the ES access proxy middleware, a snapshot creation request for the old ES cluster is initiated and a snapshot file is generated. When a snapshot creation request for the old ES cluster is initiated, the old ES cluster will return a unique ID as the ID of the snapshot request. When a snapshot creation request for the old ES cluster is initiated, the ES access proxy middleware will generate a transaction log for the new, updated and deleted ES access requests and send it to the Kafka topic. The snapshot file of the old ES cluster is synchronized to the new ES cluster. After the snapshot file synchronization is completed, the topic of kakfa where the transaction log is stored is located based on the ID of the snapshot request. The transaction log corresponding to the kakfa topic is read and parsed, and the operations of adding, deleting and modifying are generated and synchronized to the new ES cluster to achieve data playback and recovery. After waiting for the data of the new ES cluster and the old ES cluster to be consistent, the ES access proxy middleware is used to smoothly switch to the new ES cluster.

[0041] The ES old cluster module is used to initiate a snapshot creation request for the ES old cluster. The ES old cluster will return a unique ID as the ID of the snapshot request.

[0042] Further, use the following steps to create a new ES cluster of the target version to be upgraded:

[0043] Determine the target version to be upgraded, and make corresponding plans and preparations to ensure that the new cluster can meet the performance and reliability requirements after the upgrade;

[0044] According to the upgrade plan, determine the upgrade span of each version and ensure that it is eventually upgraded to the latest version;

[0045] Download and install the iterative version of the ES software package on the nodes of the new cluster according to the official documentation or guide;

[0046] Perform necessary configurations on the new cluster, including node name, cluster name, network binding, memory settings, and storage paths, to ensure that the configuration of the new cluster is consistent with that of the cluster to be upgraded;

[0047] Start the nodes of the new cluster one by one to ensure that the nodes start normally and join the new cluster; monitor the node startup log and the status of the new cluster to ensure that the new cluster runs normally.

[0048] Furthermore, the following steps are used to parse the ES access request, identify the type of request, and perform corresponding processing operations according to the type:

[0049] Identify the source and format of the request based on the request header information and request path;

[0050] Design different request parser modules for different request sources and formats to parse request data packets and extract key information, including operation type, operation target, and specific data content;

[0051] Convert request data packets of different formats into a unified internal representation, which is a standardized structure including operation type, operation target and specific data content;

[0052] After parsing and abstraction, if the request involves a data change operation, a corresponding transaction log is generated, which records the detailed information of the operation for auditing, rollback, or replay operations.

[0053] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the aforementioned method for smoothly upgrading the ES cluster when executing the computer program.

[0054] In one embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program for implementing a method for smoothly upgrading an ES cluster.

[0055] Beneficial effects:

[0056] 1. The initialization of new cluster data of the present invention adopts a gradual conversion method according to the span of version iteration, avoiding one-time large-scale data conversion and reducing the pressure on system resources. This controllable data conversion method improves the stability and reliability of data conversion.

[0057] 2. The present invention can receive read and write requests from the client through the newly added ES access proxy middleware, and can read and write any new and old clusters. This provides flexible data access capabilities, allowing seamless switching to the new cluster during the upgrade process while ensuring business continuity.

[0058] 3. The present invention separates the upgrade process from the normal operation of the cluster to be upgraded by establishing a new ES cluster of the target version. After the data of the new and old clusters are consistent, a smooth switch is performed to the new cluster to achieve the version upgrade. This method can minimize the impact and risk on the system and improve the overall stability and availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a flow chart of a method for implementing a smooth upgrade of an ES cluster according to an embodiment of the present invention;

[0060] Figure 2It is a schematic diagram of the structure of an implementation device for smooth upgrade of an ES cluster according to an embodiment of the present invention;

[0061] Figure 3 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0062] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0063] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a device, apparatus, equipment, method or computer program product. Therefore, the present disclosure may be implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0064] According to the implementation of the present invention, a method for implementing smooth upgrade of ES cluster is proposed. Through offline initialization and real-time synchronization of data, as well as the introduction of proxy forwarding and transaction log synchronization components, smooth upgrade of ES cluster version is achieved. During the upgrade process, the business will not be interrupted and the consistency of data is guaranteed. It can cope with the challenges brought by ES cluster upgrade, reduce the impact on business, and reduce the upgrade risk.

[0065] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0066] Figure 1 FIG. 1 is a flow chart of a method for implementing a smooth upgrade of an ES cluster according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0067] 1. Establish a new ES cluster whose version is the target version to be upgraded and is used to synchronize the data of the cluster to be upgraded. The new ES cluster can be deployed on the same hardware infrastructure or created in a new hardware environment to ensure performance and stability after the upgrade.

[0068] 2. Build ES access proxy middleware as the only entry point to access the old and new ES clusters; all ES access requests such as adding, updating, deleting, querying, and creating snapshots of the old ES cluster data are proxied and forwarded through the ES access proxy middleware as the entry point;

[0069] 3. Initiate a snapshot request for the old ES cluster through the ES access proxy middleware and generate a snapshot file. When initiating a snapshot request for the old ES cluster, the old ES cluster will return a unique ID as the ID of the snapshot request.

[0070] 4. When a snapshot creation request is initiated for the old ES cluster, the ES access proxy middleware will generate a transaction log (transaction log generator) for the newly added, updated, and deleted ES access requests, and send it to the Kafka topic; based on this transaction log, the data can be replayed and the data of the old ES cluster can be synchronized in real time in the new ES cluster. The naming rule of Kafka's topic is: fixed topic prefix + snapshot request ID, the topic prefix and snapshot request ID are connected with an underscore, etc. For example, if the fixed topic prefix is ​​sync and the snapshot request ID is 123, the topic name is: sync_123. When using this rule to name the topic, after the snapshot file is synchronized to the new ES cluster, the transaction log that needs to be synchronized can be quickly located based on the snapshot file ID.

[0071] 5. Synchronize the snapshot file of the old ES cluster to the new ES cluster. After the snapshot file synchronization is completed, locate the topic of Kakfa where the transaction log is stored based on the ID of the snapshot request; read and parse the transaction log corresponding to the Kakfa topic, generate the add, delete and modify operations, and synchronize them to the new ES cluster to achieve data playback and recovery;

[0072] 6. After the data of the new ES cluster and the old ES cluster are consistent, smoothly switch to the new ES cluster through the ES access proxy middleware.

[0073] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0074] In order to explain the implementation method of the smooth upgrade of the ES cluster more clearly, a specific embodiment is described below. However, it should be noted that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.

[0075] Example:

[0076] 1. Create a new ES cluster of the target version to synchronize the data of the cluster to be upgraded

[0077] (1) Determine the target version: Determine the target ES version to be upgraded, and make corresponding plans and preparations to ensure that the new cluster can meet the performance and reliability requirements after the upgrade.

[0078] (2) Determine the span of version iterations: According to the upgrade plan, determine the upgrade span of each version, such as upgrading from ES version X to version Y, and from ES version Y to version Z, and ensure that the latest ES version is finally upgraded.

[0079] (3) Install the target version of ES: According to official documents or guidelines, download and install the iterative version of the ES software package to the nodes of the new cluster, such as downloading and installing version X -> version Y -> version Z.

[0080] (4) Configure the new cluster: Perform necessary configuration on the new cluster, including node name, cluster name, network binding, memory settings, storage path, etc. Ensure that the configuration of the new cluster is consistent with the cluster to be upgraded.

[0081] (5) Start the new cluster node: Start the nodes of the new cluster one by one to ensure that the nodes start normally and join the cluster. Monitor the node startup log and cluster status to ensure that the new cluster runs normally.

[0082] 2. Build ES access proxy middleware

[0083] ES access proxy middleware is the only entry point for accessing ES. All ES access requests such as adding, updating, deleting, querying, and creating snapshots of ES data are proxied and forwarded through the ES proxy middleware as the entry point.

[0084] The ES access proxy implements two functions:

[0085] (1) The access request to ES is transparently transmitted to the host of the proxy ES cluster.

[0086] (2) Parse the data packets of the ES access request, identify the type of request, and perform corresponding processing operations according to the type. The types of requests mainly include data addition, query, deletion, modification, and snapshot generation requests. If the request type involves operations related to data changes, such as data addition, deletion, and modification, a transaction log is generated, and each log entry contains the operation type, operation target, timestamp, and specific operation content.

[0087] However, there are many ways to access ES, such as accessing through the http interface, using the big data framework elasticsearch-hadoop, or accessing through the Elasticsearch Java API. These different access methods are connected to the ES access proxy middleware, and the data formats are different. How to parse and obtain the added, deleted, and modified information from these request data packets in different formats is a very challenging task. In order to solve this problem, a unified request parsing mechanism is designed and implemented. The mechanism includes the following key steps:

[0088] (1) Request format identification: First, identify the source and format of the request based on the request header information and request path. For example, by checking the Content-Type and URL path of the HTTP request, you can determine whether the request is issued through the HTTP interface or through other means (such as elasticsearch-hadoop or Elasticsearch's Java API). Content-Type is an HTTP header field that specifies the media type of the resource. It tells the browser or server what format the data sent or received is in, such as text, pictures, videos, etc.

[0089] (2) Request parser module: Different request parser modules are designed for different request sources and formats. These modules can parse request data packets and extract key information, such as operation type (add, delete, modify), operation target (index, document ID) and specific data content.

[0090] (3) Unified abstract module: To simplify subsequent processing, request data packets of different formats are converted into a unified internal representation. This representation is a standardized structure that includes the operation type, operation target, and specific data content.

[0091] (4) Transaction log generation: After parsing and abstraction, if the request involves a data change operation, a corresponding transaction log is generated. The transaction log records the detailed information of the operation and can be used for auditing, rolling back, or replaying operations.

[0092] The following are some examples of transaction logs, corresponding to the addition, deletion, and modification of data. Each log entry contains the operation type, operation target, timestamp, and specific operation content.

[0093]

[0094]

[0095]

[0096] Log Field Explanation

[0097] timestamp: The timestamp when the operation occurred.

[0098] operation: operation type, including create (addition), delete (deletion) and update (modification).

[0099] index: The name of the index involved in the operation.

[0100] document_id: The document ID involved in the operation.

[0101] content: In new and modified operations, it includes the content of the document.

[0102] For new operations, it contains the newly added document content.

[0103] For modification operations, contains the modified document content.

[0104] original_content: During modification operations, includes the original document content.

[0105] updated_content: In the modification operation, it contains the modified document content.

[0106] 3. Create a snapshot file

[0107] Use the Elasticsearch Snapshot (ES snapshot) API to create an ES snapshot and generate a snapshot file. Generating an ES snapshot returns a unique snapshot request ID.

[0108] 4. Transaction log generation

[0109] When the ES access proxy middleware initiates a snapshot creation request from ES, it generates transaction logs for new, updated, and deleted ES access requests, and sends the transaction logs to a new Kafka topic. The naming rule for Kafka topics is: fixed topic prefix + snapshot request ID, with the topic prefix and snapshot request connected by an underscore. For example, if the fixed topic prefix is ​​sync and the snapshot request ID is 123, the topic name is: sync_123. When using this rule to name topics, after the snapshot file is synchronized to the new cluster, the transaction log that needs to be synchronized can be quickly located based on the snapshot file ID.

[0110] Based on this transaction log, you can replay the data and synchronize the data of the old cluster with the new cluster in real time.

[0111] 5. Snapshot file synchronization

[0112] (1) Create a snapshot repository in the new cluster and specify the path to store snapshot files.

[0113] The following is an example of an API request to create a snapshot repository:

[0114] PUT / _snapshot / <repository_name>

[0115] {

[0116] "type":"<repository_type> ",

[0117] "settings":{

[0118] "location":"<path_to_snapshot_directory> "

[0119] }

[0120] }

[0121] <repository_name> : Specify the name of the snapshot repository

[0122] <repository_type> : Specify the repository type, such as "fs", "s3", etc. Here, "fs" is used to represent the local file system repository

[0123] <path_to_snapshot_directory> : Specify the directory path in the new cluster where the snapshots will be stored

[0124] (2) Synchronize the snapshot files of the old cluster to the directory storing snapshots of the new cluster.

[0125] (3) Restore the snapshot to the new cluster

[0126] Use the ES snapshot recovery API to restore the data of the snapshot file of the old ES cluster to the new cluster. The relevant API calls are as follows:

[0127] POST

[0128] / _snapshot / <repository_name> / <snapshot_name> / _restore

[0129] {

[0130] "indices":"<index_pattern> ",

[0131] "ignore_unavailable":true,

[0132] "include_global_state":false

[0133] }

[0134] <repository_name> : Specify the name of the snapshot repository

[0135] <snapshot_name> : Specifies the name of the snapshot to restore

[0136] <index_pattern> : Specifies the target index pattern to be restored, which can be a single index name, a wildcard pattern, or a pattern for all indexes

[0137] 6. Transaction log data synchronization

[0138] According to the snapshot request ID, locate the Kakfa topic where the transaction log is stored. Read the transaction log data of the Kafka topic, parse the transaction log data, generate data add, delete, and modify operations, write to the new ES cluster, and implement data playback and recovery.

[0139] 7. Wait for the data of the new and old clusters to remain consistent and switch to the new cluster smoothly

[0140] (1) Ensure data synchronization is complete

[0141] a. Monitor data synchronization status: Use monitoring tools or query the ES cluster status to ensure that the data in the new cluster is fully synchronized with the old cluster.

[0142] b. Verify data consistency: Select some key indexes or documents for comparison to ensure that the data in the new cluster is consistent with the old cluster.

[0143] (2) Switching read and write traffic

[0144] Update proxy server configuration: Update the proxy server configuration file to forward only read and write requests to the new cluster.

[0145] Based on the same inventive concept, the present invention also proposes a device for implementing a smooth upgrade of an ES cluster. The implementation of the device can refer to the implementation of the above method, and the repeated parts will not be repeated. The term "module" used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0146] Figure 2 FIG. 1 is a schematic diagram of a device structure for implementing a smooth upgrade of an ES cluster according to an embodiment of the present invention. Figure 2As shown, the device comprises:

[0147] ES new cluster module 101 is used to establish a new ES cluster whose version is the target version to be upgraded and to synchronize the data of the cluster to be upgraded;

[0148] Use the following steps to create a new ES cluster of the target version to be upgraded:

[0149] Determine the target version to be upgraded, and make corresponding plans and preparations to ensure that the new cluster can meet the performance and reliability requirements after the upgrade;

[0150] According to the upgrade plan, determine the upgrade span of each version and ensure that it is eventually upgraded to the latest version;

[0151] Download and install the iterative version of the ES software package on the nodes of the new cluster according to the official documentation or guide;

[0152] Perform necessary configurations on the new cluster, including node name, cluster name, network binding, memory settings, and storage paths, to ensure that the configuration of the new cluster is consistent with that of the cluster to be upgraded;

[0153] Start the nodes of the new cluster one by one to ensure that the nodes start normally and join the new cluster; monitor the node startup log and the status of the new cluster to ensure that the new cluster runs normally.

[0154] ES access proxy middleware module 102 is used to build ES access proxy middleware as the only entry point for accessing the old and new ES clusters; all kinds of ES access requests for adding, updating, deleting, querying and creating snapshots of the old ES cluster data are proxied and forwarded through the ES access proxy middleware as the entry point; through the ES access proxy middleware, a snapshot creation request for the old ES cluster is initiated, and a snapshot file is generated; when a snapshot creation request for the old ES cluster is initiated, the old ES cluster will return a unique ID as the ID of the snapshot request; when a snapshot creation request for the old ES cluster is initiated, the ES access proxy middleware will generate a transaction log for the ES access requests for adding, updating and deleting, and send it to the Kafka topic; synchronize the snapshot file of the old ES cluster to the new ES cluster, and after the snapshot file synchronization is completed, locate the Kakfa topic where the transaction log is stored based on the snapshot request ID; read and parse the transaction log corresponding to the Kakfa topic, generate add, delete and modify operations, synchronize to the new ES cluster, and realize data playback and recovery; after waiting for the data of the new ES cluster and the old ES cluster to be consistent, smoothly switch to the new ES cluster through the ES access proxy middleware;

[0155] Use the following steps to parse ES access requests, identify the request type, and perform corresponding processing operations based on the type:

[0156] Identify the source and format of the request based on the request header information and request path;

[0157] Design different request parser modules for different request sources and formats to parse request data packets and extract key information, including operation type, operation target, and specific data content;

[0158] Convert request data packets of different formats into a unified internal representation, which is a standardized structure including operation type, operation target and specific data content;

[0159] After parsing and abstraction, if the request involves a data change operation, a corresponding transaction log is generated, which records the detailed information of the operation for auditing, rollback, or replay operations.

[0160] The ES old cluster module 103 is used for initiating a request to create a snapshot of the ES old cluster. The ES old cluster will return a unique ID as the ID of the snapshot request.

[0161] It should be noted that although several modules of the implementation device for smooth upgrade of ES cluster are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for concretization.

[0162] Based on the above invention concept, Figure 3 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220, and the processor 220 implements the aforementioned method for smoothly upgrading the ES cluster when executing the computer program 230.

[0163] Based on the aforementioned inventive concept, the present invention further proposes a computer-readable storage medium, which stores a computer program for executing the aforementioned ES cluster smooth upgrade implementation method.

[0164] The method and device for implementing the smooth upgrade of the ES cluster proposed in the present invention have the following highlights:

[0165] 1. The initialization of new cluster data adopts the method of gradual conversion according to the span of version iteration, avoiding one-time large-scale data conversion and reducing the pressure on system resources. This controllable data conversion method improves the stability and reliability of data conversion.

[0166] 2. Through the newly added ES access proxy middleware, you can receive read and write requests from the client, and can read and write any new or old clusters. This provides flexible data access capabilities, allowing seamless switching to new clusters during upgrades while ensuring business continuity.

[0167] 3. By establishing a new ES cluster of the target version, the upgrade process is separated from the normal operation of the cluster to be upgraded. After the data of the new and old clusters are consistent, a smooth switch to the new cluster is performed to achieve the version upgrade. This method can minimize the impact and risk on the system and improve the overall stability and availability.

[0168] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the attached claims.

[0169] Regarding the limitation of the protection scope of the present invention, those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A method for implementing a smooth upgrade of an ES cluster, characterized in that: The method includes: Create a new ES cluster whose version is the target version to be upgraded and used to synchronize the data of the cluster to be upgraded; Build ES access proxy middleware as the only entry point to access the old and new ES clusters; all ES access requests for adding, updating, deleting, querying, and creating snapshots of the old ES cluster data are proxied and forwarded through the ES access proxy middleware as the entry point; Through the ES access proxy middleware, initiate a snapshot request for the old ES cluster and generate a snapshot file. When the snapshot request for the old ES cluster is initiated, the old ES cluster will return a unique ID as the ID of the snapshot request. When a snapshot creation request is initiated for the old ES cluster, the ES access proxy middleware generates a transaction log for the new, updated, and deleted ES access requests and sends it to the Kafka topic; Synchronize the snapshot file of the old ES cluster to the new ES cluster. After the snapshot file synchronization is completed, locate the Kakfa topic where the transaction log is stored based on the snapshot request ID; read and parse the transaction log corresponding to the Kakfa topic, generate add, delete and modify operations, and synchronize them to the new ES cluster to achieve data playback and recovery; After the data of the new ES cluster and the old ES cluster are consistent, smoothly switch to the new ES cluster through the ES access proxy middleware.

2. The method for implementing a smooth upgrade of an ES cluster according to claim 1, characterized in that: Use the following steps to create a new ES cluster of the target version to be upgraded: Determine the target version to be upgraded, and make corresponding plans and preparations to ensure that the new cluster can meet the performance and reliability requirements after the upgrade; According to the upgrade plan, determine the upgrade span of each version and ensure that it is eventually upgraded to the latest version; Download and install the iterative version of the ES software package on the nodes of the new cluster according to the official documentation or guide; Perform necessary configurations on the new cluster, including node name, cluster name, network binding, memory settings, and storage paths, to ensure that the configuration of the new cluster is consistent with that of the cluster to be upgraded; Start the nodes of the new cluster one by one to ensure that the nodes start normally and join the new cluster; monitor the node startup log and the status of the new cluster to ensure that the new cluster runs normally.

3. The method for implementing a smooth upgrade of an ES cluster according to claim 1, characterized in that: Use the following steps to parse ES access requests, identify the request type, and perform corresponding processing operations based on the type: Identify the source and format of the request based on the request header information and request path; Design different request parser modules for different request sources and formats to parse request data packets and extract key information, including operation type, operation target, and specific data content; Convert request data packets of different formats into a unified internal representation, which is a standardized structure including operation type, operation target and specific data content; After parsing and abstraction, if the request involves a data change operation, a corresponding transaction log is generated, which records the detailed information of the operation for auditing, rollback, or replay operations.

4. A device for implementing a smooth upgrade of an ES cluster, characterized in that: The device includes: The ES new cluster module is used to establish a new ES cluster. The version of the new ES cluster is the target version to be upgraded and is used to synchronize the data of the cluster to be upgraded. ES access proxy middleware module is used to build ES access proxy middleware as the only entry point for accessing the old and new ES clusters. All ES access requests for adding, updating, deleting, querying and creating snapshots of the old ES cluster data are proxied and forwarded through the ES access proxy middleware as the entry point. Through the ES access proxy middleware, a snapshot creation request for the old ES cluster is initiated and a snapshot file is generated. When a snapshot creation request for the old ES cluster is initiated, the old ES cluster will return a unique ID as the ID of the snapshot request. When a snapshot creation request for the old ES cluster is initiated, the ES access proxy middleware will generate a transaction log for the new, updated and deleted ES access requests and send it to the Kafka topic. The snapshot file of the old ES cluster is synchronized to the new ES cluster. After the snapshot file synchronization is completed, the topic of kakfa where the transaction log is stored is located based on the ID of the snapshot request. The transaction log corresponding to the kakfa topic is read and parsed, and the operations of adding, deleting and modifying are generated and synchronized to the new ES cluster to achieve data playback and recovery. After waiting for the data of the new ES cluster and the old ES cluster to be consistent, the ES access proxy middleware is used to smoothly switch to the new ES cluster. The ES old cluster module is used to initiate a snapshot creation request for the ES old cluster. The ES old cluster will return a unique ID as the ID of the snapshot request.

5. The device for implementing smooth upgrade of an ES cluster according to claim 4, characterized in that: Use the following steps to create a new ES cluster of the target version to be upgraded: Determine the target version to be upgraded, and make corresponding plans and preparations to ensure that the new cluster can meet the performance and reliability requirements after the upgrade; According to the upgrade plan, determine the upgrade span of each version and ensure that it is eventually upgraded to the latest version; Download and install the iterative version of the ES software package on the nodes of the new cluster according to the official documentation or guide; Perform necessary configurations on the new cluster, including node name, cluster name, network binding, memory settings, and storage paths, to ensure that the configuration of the new cluster is consistent with that of the cluster to be upgraded; Start the nodes of the new cluster one by one to ensure that the nodes start normally and join the new cluster; monitor the node startup log and the status of the new cluster to ensure that the new cluster runs normally.

6. The device for implementing smooth upgrade of an ES cluster according to claim 4, characterized in that: Use the following steps to parse ES access requests, identify the request type, and perform corresponding processing operations based on the type: Identify the source and format of the request based on the request header information and request path; Design different request parser modules for different request sources and formats to parse request data packets and extract key information, including operation type, operation target, and specific data content; Convert request data packets of different formats into a unified internal representation, which is a standardized structure including operation type, operation target and specific data content; After parsing and abstraction, if the request involves a data change operation, a corresponding transaction log is generated, which records the detailed information of the operation for auditing, rollback, or replay operations.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 3.

Images