Elastic Search inter-cluster data migration method, device, and system

Abstract

The present invention provides a method, device and system for data migration between Elastic Search clusters, which can be used in the field of artificial intelligence technology. The method includes: writing all indexes to be migrated in the source ES cluster into a preset migration queue; determining the number of parallel tasks and the migration speed of parallel migration according to the migration configuration file; starting the data migration process corresponding to the number of parallel tasks and transmitting the indexes to be migrated corresponding to the number of parallel tasks to the target ES cluster according to the migration speed. The present invention can realize batch migration of data of ES clusters of different versions and improve the data migration efficiency of ES clusters.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a method, device, and system for migrating data between Elastic Search clusters. Background Art

[0002] Elastic Search (ES) is a distributed, highly scalable, and highly real-time search and data analysis engine that can be used to store index data (an index is equivalent to a table in a traditional database), providing users with more flexible online data query services. When using ES, it is often necessary to migrate index data in the ES cluster due to reasons such as data backup, product upgrades, and cluster relocation. ES can be considered a database when used for query functions, but it does not have the multi-table association, consistency, and transactional features of general relational databases. When using the file transfer method of a general database for data migration, it is necessary to fix the data in the file through snapshots, and it does not support data migration between ES clusters of different versions. Therefore, it cannot solve the problem of data synchronization between the old and new ES clusters during product upgrades. Summary of the Invention

[0003] One object of the present invention is to provide a method for migrating data between ElasticSearch clusters, enabling batch migration of data between ES clusters of different versions and improving the efficiency of data migration in ES clusters. Another object of the present invention is to provide an apparatus for migrating data between ElasticSearch clusters. Another object of the present invention is to provide a system for migrating data between ElasticSearch clusters. Another object of the present invention is to provide a computer device. Another object of the present invention is to provide a readable medium.

[0004] To achieve the above objectives, the present invention discloses a method for migrating data between Elastic Search clusters, including:

[0005] Write all indices to be migrated in the source ES cluster to the preset migration queue;

[0006] Determine the number of parallel tasks and migration speed for parallel migration based on the migration profile;

[0007] The data migration process corresponding to the number of parallel tasks is started, and the indexes to be migrated corresponding to the number of parallel tasks are respectively transmitted to the target ES cluster according to the migration speed.

[0008] Preferably, further comprising:

[0009] Pre-write the IP address of the source ES cluster to the whitelist of the target ES cluster.

[0010] Preferably, the method further includes the step of backing up the index in the target ES cluster in advance:

[0011] Create a backup index corresponding to each target index in the target ES cluster;

[0012] The data in the target index is copied to the backup index.

[0013] Preferably, further comprising:

[0014] Writing the backup progress of copying the data in the target index to the backup index into a preset backup progress table;

[0015] When the backup progress in the backup progress table indicates that the backup is completed, the data to be migrated of the index to be migrated is migrated to the target ES cluster through a data migration process.

[0016] Preferably, the transferring of the to-be-migrated indexes corresponding to the number of parallel tasks to the target ES cluster specifically includes:

[0017] Acquire the data to be migrated of the to-be-migrated index corresponding to the number of parallel tasks;

[0018] Encrypting the data to be migrated according to preset encryption rules;

[0019] The encrypted data to be migrated is transferred to the target ES cluster.

[0020] Preferably, further comprising:

[0021] Determine throttling configuration information based on process configuration management files;

[0022] The number of parallel tasks and the migration speed are adjusted according to the throttling configuration information.

[0023] Preferably, further comprising:

[0024] Write the migration results of the index to be migrated to the target ES cluster into the preset result table;

[0025] Determine whether all the indexes to be migrated to the target ES cluster have been successfully migrated based on the migration results in the preset result table.

[0026] If not, determine to roll back or re-migrate the unmigrated indexes.

[0027] The present invention also discloses an Elastic Search inter-cluster data migration device, comprising:

[0028] The index writing module is used to write all the indices to be migrated in the source ES cluster into the preset migration queue;

[0029] A migration configuration module is used to determine the number of parallel tasks and the migration speed of parallel migration according to the migration configuration file;

[0030] The data migration module is used to start the data migration process corresponding to the number of parallel tasks and transmit the to-be-migrated indexes corresponding to the number of parallel tasks to the target ES cluster according to the migration speed.

[0031] The present invention also discloses an Elastic Search inter-cluster data migration system, including a source ES cluster, a target ES cluster, and the data migration device as described above.

[0032] The present invention also discloses a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0033] When the processor executes the program, the method described above is implemented.

[0034] The present invention also discloses a computer readable medium having a computer program stored thereon.

[0035] When the program is executed by a processor, the above-mentioned method is implemented.

[0036] The present invention sets a migration queue when migrating data between ES clusters, writes all to-be-migrated indexes in the source ES cluster into a preset migration queue, determines the number of parallel tasks and the migration speed for parallel migration based on a pre-configured migration configuration file, then establishes a corresponding number of data migration processes based on the number of parallel tasks, and transmits each to-be-migrated index corresponding to the number of parallel tasks to the target ES cluster via a data migration process based on the migration speed. The present invention provides a configurable, automatically scheduled ES cluster data migration solution, implements batch migration of data from different versions of ES clusters, and improves the efficiency of data migration between ES clusters. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A structural diagram showing a specific embodiment of the Elastic Search inter-cluster data migration system of the present invention;

[0039] Figure 2A flowchart showing a specific embodiment of the method for migrating data between Elastic Search clusters according to the present invention;

[0040] Figure 3 A flowchart illustrating a specific embodiment S020 of the method for migrating data between Elastic Search clusters according to the present invention is shown;

[0041] Figure 4 A flowchart illustrating backup progress monitoring in a specific embodiment of the Elastic Search inter-cluster data migration method of the present invention is shown;

[0042] Figure 5 A flowchart illustrating a specific embodiment S300 of the method for migrating data between Elastic Search clusters according to the present invention is shown;

[0043] Figure 6 A flowchart illustrating throttling configuration in a specific embodiment of the Elastic Search inter-cluster data migration method of the present invention;

[0044] Figure 7 A flowchart illustrating verification of migration results according to a specific embodiment of the method for migrating data between Elastic Search clusters of the present invention is shown;

[0045] Figure 8 A structural diagram showing a specific embodiment of the device for migrating data between Elastic Search clusters according to the present invention;

[0046] Figure 9 A structural diagram showing a specific embodiment of the Elastic Search inter-cluster data migration device of the present invention, including a pre-processing module;

[0047] Figure 10 A schematic diagram showing the structure of a computer device suitable for implementing an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] It should be noted that the method, device, and system for migrating data between Elastic Search clusters disclosed in this application can be used in the field of artificial intelligence technology, and can also be used in any field other than the field of artificial intelligence technology. The application field of the method, device, and system for migrating data between Elastic Search clusters disclosed in this application is not limited.

[0050] In order to facilitate the understanding of the technical solution provided by the present application, the relevant contents of the technical solution of the present application are first described below. The Elastic Search inter-cluster data migration method provided by the embodiment of the present invention sets the index to be migrated in a preset queue, determines the parallel migration parameters of the number of parallel tasks and the migration speed according to the migration configuration file configured by the user, establishes one or more data migration processes corresponding to the number of parallel tasks, and transmits the index to be migrated corresponding to the number of parallel tasks to the target database in parallel at the migration speed through the corresponding data migration process according to the parallel migration parameters, thereby solving the current gap in automatic data migration between ES clusters of different versions, and realizing the parallel migration of data between different ES clusters, thereby improving the speed and efficiency of data migration.

[0051] Figure 1 FIG is a schematic diagram of the structure of the Elastic Search inter-cluster data migration system provided by an embodiment of the present invention. Figure 1 As shown, the Elastic Search inter-cluster data migration system provided by the embodiment of the present invention includes a source ES cluster 1, a target ES cluster 2 and a data migration device 3.

[0052] The source ES cluster stores index data. Specifically, the source ES cluster contains a large number of indexes, each equivalent to a table storing index data. This allows the source ES cluster to function as a database, providing users with online data query services. When using an ES cluster, it's often necessary to migrate (copy) index data from the source ES cluster to the target ES cluster for reasons such as data backup, product upgrades, and cluster relocation.

[0053] Since the current ES cluster uses snapshots for data transmission and does not support data migration between ES clusters of different versions, it is impossible to achieve the purpose of migrating data from the source ES cluster to the target ES cluster during product upgrades and other operations. Therefore, the data migration device of this embodiment can write all the indexes to be migrated in the source ES cluster into a preset migration queue, determine the number of parallel tasks and the migration speed of parallel migration according to the migration configuration file, start the data migration process corresponding to the number of parallel tasks, and transfer the indexes to be migrated corresponding to the number of parallel tasks to the target ES cluster according to the migration speed, thereby realizing batch parallel migration of data from ES clusters of different versions and improving the data migration efficiency of the ES cluster.

[0054] It should be noted that the data migration device in the present invention can be a standalone device that communicates with the source ES cluster and the target ES cluster via a preset communication protocol to achieve data migration between Elastic Search clusters. Of course, the data migration device can also be integrated into the source ES cluster or the target ES cluster, and the present invention is not limited to this.

[0055] The following uses the Elastic Search inter-cluster data migration device as an example to illustrate the implementation process of the Elastic Search inter-cluster data migration method provided by the embodiment of the present invention. It is understood that the execution subject of the Elastic Search inter-cluster data migration method provided by the embodiment of the present invention includes but is not limited to the Elastic Search inter-cluster data migration device.

[0056] According to one aspect of the present invention, this embodiment discloses a method for migrating data between Elastic Search clusters. Figure 2 As shown, in this embodiment, the method includes:

[0057] S100: Write all indices to be migrated in the source ES cluster into the preset migration queue.

[0058] S200: Determine the number of parallel tasks and the migration speed of the parallel migration according to the migration configuration file.

[0059] S300: Start the data migration process corresponding to the number of parallel tasks and transmit the to-be-migrated indexes corresponding to the number of parallel tasks to the target ES cluster according to the migration speed.

[0060] It is understandable that, in a specific example, the preset migration queue can be a reindex queue, and the queue process can be called through the curl command to add all the to-be-migrated indexes that need to be migrated to the reindex queue. p ost command: POST / api / v1 / reindex / remote / queue / start starts the data migration process corresponding to the number of parallel tasks and migration speed according to the configured parameters, and migrates the indices to be migrated in the reindex queue to the target ES cluster in batches.

[0061] The present invention sets a migration queue when migrating data between ES clusters, writes all to-be-migrated indexes in the source ES cluster into a preset migration queue, determines the number of parallel tasks and the migration speed for parallel migration based on a pre-configured migration configuration file, then establishes a corresponding number of data migration processes based on the number of parallel tasks, and transmits each to-be-migrated index corresponding to the number of parallel tasks to the target ES cluster via a data migration process based on the migration speed. The present invention provides a configurable, automatically scheduled ES cluster data migration solution, implements batch migration of data from different versions of ES clusters, and improves the efficiency of data migration between ES clusters.

[0062] In a preferred embodiment, the method further comprises the step of backing up the index in the target ES cluster in advance:

[0063] S010: Write the IP address of the source ES cluster to the whitelist of the target ES cluster.

[0064] Specifically, in order to achieve the migration of the index data of the source ES cluster and write it into the target ES cluster, while avoiding the need to perform identity authentication or authorization and other operations each time the data is migrated, the problem of slow efficiency of ES cluster data migration is caused. In this preferred embodiment, the IP address of the source ES cluster can be written into the whitelist of the target ES cluster, so that the target ES cluster can directly perform operations such as data storage after receiving the data transmitted by the source ES cluster, without the need to authenticate the source of the migrated data or authorization, thereby improving the data migration speed and efficiency between ES clusters. For example, in a specific example, the source ES cluster includes multiple servers, and the IP addresses of multiple servers form an IP address list. The IP address list of the source ES cluster can be written into the reindex remote whitelist of the target ES cluster, so that the target ES cluster can receive the index data transmitted by the source ES cluster.

[0065] In a preferred embodiment, Figure 3 As shown, the method further includes step S020 of backing up the index in the target ES cluster in advance:

[0066] S021: Create a backup index corresponding to each target index in the target ES cluster.

[0067] S022: Copy the data in the target index to the backup index.

[0068] Specifically, when the index of the source ES cluster is migrated to the target ES cluster, the index data of the source ES cluster will directly overwrite the data in the corresponding target index in the target ES cluster to complete the data migration. The target index in the target ES cluster stores the original data, and the migration of the source ES cluster index will directly overwrite the original data in the target index, resulting in the loss of the original data in the target index. When the data migration fails and the data migration needs to be rolled back, the target index in the target ES cluster cannot be restored to its original state, which may cause the loss of key data in the target ES cluster. Based on this, in this preferred embodiment, before migrating the index in the source ES cluster to the target ES cluster, the target index can be backed up in the target ES cluster in advance to prevent the loss of index data in the target ES cluster when data loss occurs during data migration. For example, in a specific example, a corresponding backup index can be created for each target index for backup and data recovery in case of migration failure. For example, if the target index corresponding to the migration is named a_example_2020, a corresponding backup index named a_example_2020_bak is created. Further, a copy of the initial data in the target index is made within the target ES cluster, such as copying the index data of the a_example_2020 index to a_example_2020_bak, to achieve backup of the target index. If the migration of the to-be-migrated index data in the source ES cluster fails and the data migration needs to be rolled back, the index data in the backup index can be copied to the corresponding original target index to restore the initial data in the target index, such as copying the data of the a_example_2020_bak backup index back to a_example_2020.

[0069] In a preferred embodiment, Figure 4 As shown, the method further comprises:

[0070] S023: Writing the backup progress of copying the data in the target index to the backup index into a preset backup progress table.

[0071] S024: When the backup progress in the backup progress table indicates that the backup is completed, the data to be migrated of the index to be migrated is migrated to the target ES cluster through a data migration process.

[0072] Specifically, in this preferred embodiment, during the data backup process of the target index in the target ES cluster being copied to the backup index, the progress of the data backup is monitored. When the data copy (backup) process of the index data in the target index being copied to the backup index is completed, the data migration process from the source ES cluster to the target ES cluster is started again, so that the initial data of the target index in the target ES cluster is not lost, and the index data in the backup index can be restored to the target index to achieve the restoration of the target index data in the target ES cluster. For example, in a specific example, after establishing a corresponding backup index for each target index, the index data in the target index can be copied to the backup index by calling the REST API and utilizing the data transmission service provided by REST. Among them, REST will automatically record the backup progress of the data copy during the process of copying the index data to the backup index. The backup progress in REST can be obtained regularly, and the backup progress can be written into the preset backup schedule table. By analyzing the backup progress in the backup schedule table, it can be determined whether the target index has been backed up. That is, when the backup progress in the backup schedule table is backup completed, it is confirmed that the target index backup is complete, and the subsequent source ES cluster data migration process can be started, and the data to be migrated of the index to be migrated is migrated to the target ES cluster through the data migration process. In this specific example, you can use the curl command to regularly access the ES interface http: / / <target ES cluster any node IP>:9200 / _tasks? detailed=true&actions=*reindex&pretty to obtain the backup progress provided by REST and write the backup progress to the preset backup schedule.

[0073] In a preferred embodiment, Figure 5 As shown, the S300 transmits the to-be-migrated indexes corresponding to the number of parallel tasks to the target ES cluster respectively, specifically including:

[0074] S310: Acquire data to be migrated of indexes to be migrated corresponding to the number of parallel tasks.

[0075] S320: Encrypt the data to be migrated according to a preset encryption rule.

[0076] S330: Transmit the encrypted data to be migrated to the target ES cluster.

[0077] Specifically, the index data in the source ES cluster's index may contain sensitive user information. Transmitting this sensitive information over the internet could lead to user information leakage. Therefore, in this preferred embodiment, before migrating the index from the source ES database to the target ES cluster, the data to be migrated in the index to be migrated can be encrypted using pre-set encryption rules. This encrypted data is then transmitted to the target ES cluster, preventing sensitive user information from being leaked or stolen during transmission. In one specific example, a developer can pre-configure an encrypted field table and set encryption rules, such as the field names and encryption methods, for the encrypted index data. By analyzing the encrypted field table configured by the developer, the fields to be encrypted and the encryption methods can be determined. For example, a developer can pre-configure the encrypted field name "ID" in the encrypted field table and set the encryption method to "888888." Before data migration, each field in the index to be migrated is analyzed to determine whether the "ID" field exists. If so, the data in the "ID" field of the index to be migrated is modified to "888888" before being transmitted to the target ES cluster, preventing user ID information from being leaked.

[0078] In a preferred embodiment, Figure 6 As shown, the method further comprises:

[0079] S410: Determine throttling configuration information according to the process configuration management file.

[0080] S420: Adjust the number of parallel tasks and the migration speed according to the throttling configuration information.

[0081] Specifically, developers can also pre-set the process configuration management file and set the throttling configuration information in the process configuration management file. During the ES cluster data migration, the number of parallel tasks and the migration speed of the data migration can be dynamically adjusted by analyzing the throttling configuration information of the process configuration management file to meet different data migration methods in different situations or time periods. The number of parallel tasks and the migration speed can be adjusted according to the importance and timeliness of the migration. In a specific example, the throttling configuration information may include multiple time periods and the number of parallel tasks and the migration speed of data migration in each time period. Then, when the time reaches the start time of each time period, the number of parallel tasks and the migration speed corresponding to each time period can be written into the migration configuration file. For example, the number of parallel tasks and the process configuration management file currently set for the source ES cluster can be obtained through the get command. The specific instructions can be:

[0082] GET / api / v1 / reindex / remote / queue / task_pool,

[0083] GET / api / v1 / reindex / rethrottle / schedule.

[0084] Then, according to the process configuration management file, use the post command to update the number of parallel tasks, migration speed, and throttling configuration information of the ES cluster reindex task. The specific instructions can be:

[0085] POST / api / v1 / reindex / remote / queue / task_pool,

[0086] POST / api / v1 / reindex / rethrottle / request_per_second,

[0087] POST / api / v1 / reindex / rethrottle / schedule.

[0088] In a preferred embodiment, Figure 7 As shown, the method further comprises:

[0089] S510: Writing the migration result of the index to be migrated to the target ES cluster into a preset result table.

[0090] S520: Determine whether all to-be-migrated indexes transferred to the target ES cluster have been successfully migrated based on the migration results in the preset result table.

[0091] S530: If not, determine to roll back or re-migrate the unsuccessfully migrated index.

[0092] Specifically, to confirm whether the indexes in the source ES cluster have been successfully migrated to the target ES cluster, the migration results of the indexes to be migrated can be analyzed and written to a preset result table. The migration results of the indexes to be migrated can then be read from the preset result table to determine whether all indexes to be migrated have been successfully migrated. For example, in one specific example, the data transfer service provided by REST can be used to copy the data to be migrated from the index to the target index in the target ES cluster by calling a REST API. During the data copy process, REST automatically records the migration progress and results. The migration results from REST can be periodically retrieved and written to a preset result table. By analyzing the migration results in the preset result table, it can be determined whether all indexes to be migrated have been successfully migrated. Specifically, when the migration results for all indexes to be migrated in the preset result table indicate "Migration Completed," the migration of all indexes to be migrated is confirmed to be successful. Specifically, monitoring can be initiated using the POST command POST / api / v1 / monitor / start. Then, the REST service can be called using the interface call instructions to retrieve the migration results stored in REST and store them in the preset result table for real-time monitoring.

[0093] Finally, if not all indices to be migrated are successfully migrated, the unmigrated indices can be rolled back or re-migrated. When performing a rollback, if the target index in the target ES cluster has been backed up beforehand, if the corresponding indices to be migrated in the source ES cluster are not successfully migrated, the index data of the backup index can be restored to the target index, returning the target index to its original state. When re-migrating, the data migration process can be restarted in the source ES cluster to transfer the unsuccessfully migrated indices to the target ES cluster.

[0094] The present invention is further illustrated below through a specific example. Taking the parallel migration of three indexes: a_example_info_2020, b_example_info_2020, and c_example_info_2020 as an example, the entire migration process is briefly described:

[0095] 1. Initialization and backup:

[0096] (1) First, complete the whitelist configuration, create three backup indexes (a_example_info_2020_bak, b_example_info_2020_bak, c_example_info_2020_bak), and copy the index data.

[0097] (2) After starting index data replication, enable index replication progress monitoring and write the backup progress into the backup progress table.

[0098] (3) Configure the encrypted field table: Based on the actual situation, the b_example_info_2020 index to be migrated contains the id field, which needs to be encrypted to 888888. Therefore, the encrypted field table is configured as follows:

[0099] b_example_info_2020, id, 888888.

[0100] 2. Data Migration

[0101] (1) Write the three indexes that need to be migrated into the index queue table, call the queue process, and add the indexes in the index queue table to the migration queue of the ES cluster.

[0102] (2) Call the configuration management process and obtain the parallel task list and migration speed from the migration configuration file. According to actual needs, set the parallel migration mode and the number of parallel tasks to 3.

[0103] (3) Determine the throttling configuration information based on the process configuration management file. The time interval for each time period of the scheduled throttling configuration is 2 minutes. The number of parallel tasks and migration speed of the ES cluster reindex are updated every 2 minutes.

[0104] (4) After the target index backup is completed, the data migration process is started. According to the number of parallel tasks and throttling configuration information configured in steps (2) and (3), three reindex migration tasks are started in parallel, and the migration speed is checked and adjusted every 2 minutes for throttling.

[0105] (5) After starting the migration task, the encrypted field table is read and it is determined that the id field in the b_example_info_2020 index in the migrated data needs to be encrypted. Therefore, the data in the id field in the b_example_info_2020 index is encrypted to 888888 before being migrated to the target ES cluster.

[0106] (6) After starting the migration task, enable migration progress monitoring and check the migration status every 5 minutes.

[0107] 3. Result verification:

[0108] (1) After the migration of all three indexes is complete, start the migration result verification.

[0109] (2) Obtain the migration results for each index and write them into the result table.

[0110] (3) Read the result table to obtain the migration results.

[0111] (4) If all three indexes are successfully migrated, the migration is complete.

[0112] (5) If the migration contains unsuccessful indexes (such as b_example_info_2020 migration failure), choose to roll back or re-migrate based on the actual situation.

[0113] (a) If rollback is selected, data is restored from the backup index b_example_info_2020_bak to the target index b_example_info_2020.

[0114] (b) If re-migration is selected, the scheduling module rewrites b_example_info_2020 into the index queue table and starts a new round of data migration from step 2.

[0115] The present invention can realize the parallel migration of multiple index data, and schedule, monitor and flow control the migration tasks of index data. By modifying the configuration, the number of migration tasks and the migration speed can be monitored and adjusted in real time, and the function of current limiting and throttling can be provided, while the encryption function of the migrated data can be realized. In addition, the data migration of the present invention is a data migration in a non-overwriting manner, and its underlying implementation method is to scan the source index and write the target index in bulk. Through this invention, business personnel and data analysts can automatically and configurably perform data migration between ElasticSearch clusters according to needs, and can adjust and monitor the migration progress in real time through parameter configuration, thereby improving the convenience and extensiveness of ElasticSearch use.

[0116] Based on the same principle, this embodiment also discloses a data migration device between Elastic Search clusters. Figure 8 As shown, in this embodiment, the apparatus includes an index writing module 11 , a migration configuration module 12 and a data migration module 13 .

[0117] The index writing module 11 is used to write all the indexes to be migrated in the source ES cluster into a preset migration queue.

[0118] The migration configuration module 12 is used to determine the number of parallel tasks and the migration speed of the parallel migration according to the migration configuration file.

[0119] The data migration module 13 is configured to start the data migration process corresponding to the number of parallel tasks and transmit the to-be-migrated indexes corresponding to the number of parallel tasks to the target ES cluster according to the migration speed.

[0120] In a specific example, the preset migration queue can be a reindex queue. The queue process can be invoked using the curl command to add all pending indices to the reindex queue. The post command (POST / api / v1 / reindex / remote / queue / start) is then used to initiate the data migration process, migrating the indices in the reindex queue to the target ES cluster in batches, based on the configured number of parallel tasks and migration speed.

[0121] The present invention sets a migration queue when migrating data between ES clusters, writes all to-be-migrated indexes in the source ES cluster into a preset migration queue, determines the number of parallel tasks and the migration speed for parallel migration based on a pre-configured migration configuration file, then establishes a corresponding number of data migration processes based on the number of parallel tasks, and transmits each to-be-migrated index corresponding to the number of parallel tasks to the target ES cluster via a data migration process based on the migration speed. The present invention provides a configurable, automatically scheduled ES cluster data migration solution, implements batch migration of data from different versions of ES clusters, and improves the efficiency of data migration between ES clusters.

[0122] In a preferred embodiment, Figure 9 As shown, the device further includes a pre-processing module 10. The pre-processing module 10 is used to back up the index in the target ES cluster in advance, that is, the pre-processing module 10 is used to write the IP address of the source ES cluster into the whitelist of the target ES cluster.

[0123] Specifically, in order to achieve the migration of the index data of the source ES cluster and write it into the target ES cluster, while avoiding the need to perform identity authentication or authorization and other operations each time the data is migrated, the problem of slow efficiency of ES cluster data migration is caused. In this preferred embodiment, the IP address of the source ES cluster can be written into the whitelist of the target ES cluster, so that the target ES cluster can directly perform operations such as data storage after receiving the data transmitted by the source ES cluster, without the need to authenticate the source of the migrated data or authorization, thereby improving the data migration speed and efficiency between ES clusters. For example, in a specific example, the source ES cluster includes multiple servers, and the IP addresses of multiple servers form an IP address list. The IP address list of the source ES cluster can be written into the reindex remote whitelist of the target ES cluster, so that the target ES cluster can receive the index data transmitted by the source ES cluster.

[0124] In a preferred embodiment, the preprocessing module 10 is further used to back up the index in the target ES cluster in advance, that is, the preprocessing module 10 is used to establish a backup index corresponding to each target index in the target ES cluster, and copy the data in the target index to the backup index.

[0125] Specifically, when the index of the source ES cluster is migrated to the target ES cluster, the index data of the source ES cluster will directly overwrite the data in the corresponding target index in the target ES cluster to complete the data migration. The target index in the target ES cluster stores the original data, and the migration of the source ES cluster index will directly overwrite the original data in the target index, resulting in the loss of the original data in the target index. When the data migration fails and the data migration needs to be rolled back, the target index in the target ES cluster cannot be restored to its original state, which may cause the loss of key data in the target ES cluster. Based on this, in this preferred embodiment, before migrating the index in the source ES cluster to the target ES cluster, the target index can be backed up in the target ES cluster in advance to prevent the loss of index data in the target ES cluster when data loss occurs during data migration. For example, in a specific example, a corresponding backup index can be created for each target index for backup and data recovery in case of migration failure. For example, if the target index corresponding to the migration is named a_example_2020, a corresponding backup index named a_example_2020_bak is created. Further, a copy of the initial data in the target index is made within the target ES cluster, such as copying the index data of the a_example_2020 index to a_example_2020_bak, to achieve backup of the target index. If the migration of the to-be-migrated index data in the source ES cluster fails and the data migration needs to be rolled back, the index data in the backup index can be copied to the corresponding original target index to restore the initial data in the target index, such as copying the data of the a_example_2020_bak backup index back to a_example_2020.

[0126] In a preferred embodiment, the migration configuration module 12 is specifically configured to write the backup progress of copying the data in the target index to the backup index into a preset backup schedule. When the backup progress in the backup schedule is backup completed, the data to be migrated of the to-be-migrated index is migrated to the target ES cluster through the data migration process.

[0127] Specifically, in this preferred embodiment, during the data backup process of the target index in the target ES cluster being copied to the backup index, the progress of the data backup is monitored. When the data copy (backup) process of the index data in the target index being copied to the backup index is completed, the data migration process from the source ES cluster to the target ES cluster is started again, so that the initial data of the target index in the target ES cluster is not lost, and the index data in the backup index can be restored to the target index to achieve the restoration of the target index data in the target ES cluster. For example, in a specific example, after establishing a corresponding backup index for each target index, the index data in the target index can be copied to the backup index by calling the REST API and utilizing the data transmission service provided by REST. Among them, REST will automatically record the backup progress of the data copy during the process of copying the index data to the backup index. The backup progress in REST can be obtained regularly, and the backup progress can be written into the preset backup schedule table. By analyzing the backup progress in the backup schedule table, it can be determined whether the target index has been backed up. That is, when the backup progress in the backup schedule table is backup completed, it is confirmed that the target index backup is complete, and the subsequent source ES cluster data migration process can be started, and the data to be migrated of the index to be migrated is migrated to the target ES cluster through the data migration process. In this specific example, you can use the curl command to periodically access the ES interface http: / / <target ES cluster any node ip>:9200 / _tasks? detailed=true&actions=*reindex&prett y , obtain the backup progress provided by REST and write the backup progress to the preset backup progress table.

[0128] In a preferred embodiment, the data migration module 13 is specifically configured to obtain data to be migrated of a to-be-migrated index corresponding to the number of parallel tasks, encrypt the data to be migrated according to a preset encryption rule, and transmit the encrypted data to be migrated to the target ES cluster.

[0129] Specifically, the index data in the source ES cluster's index may contain sensitive user information. Transmitting this sensitive information over the internet could lead to user information leakage. Therefore, in this preferred embodiment, before migrating the index from the source ES database to the target ES cluster, the data to be migrated in the index to be migrated can be encrypted using pre-set encryption rules. This encrypted data is then transmitted to the target ES cluster, preventing sensitive user information from being leaked or stolen during transmission. In one specific example, a developer can pre-configure an encrypted field table and set encryption rules, such as the field names and encryption methods, for the encrypted index data. By analyzing the encrypted field table configured by the developer, the fields to be encrypted and the encryption methods can be determined. For example, a developer can pre-configure the encrypted field name "ID" in the encrypted field table and set the encryption method to "888888." Before data migration, each field in the index to be migrated is analyzed to determine whether the "ID" field exists. If so, the data in the "ID" field of the index to be migrated is modified to "888888" before being transmitted to the target ES cluster, preventing user ID information from being leaked.

[0130] In a preferred embodiment, the data migration module 13 is further configured to determine throttling configuration information according to the process configuration management file, and adjust the number of parallel tasks and the migration speed according to the throttling configuration information.

[0131] Specifically, developers can also pre-set the process configuration management file and set the throttling configuration information in the process configuration management file. During the ES cluster data migration, the number of parallel tasks and the migration speed of the data migration can be dynamically adjusted by analyzing the throttling configuration information of the process configuration management file to meet different data migration methods in different situations or time periods. The number of parallel tasks and the migration speed can be adjusted according to the importance and timeliness of the migration. In a specific example, the throttling configuration information may include multiple time periods and the number of parallel tasks and the migration speed of data migration in each time period. Then, when the time reaches the start time of each time period, the number of parallel tasks and the migration speed corresponding to each time period can be written into the migration configuration file. For example, the number of parallel tasks and the process configuration management file currently set for the source ES cluster can be obtained through the get command. The specific instructions can be:

[0132] GET / api / v1 / reindex / remote / queue / task_pool,

[0133] GET / api / v1 / reindex / rethrottle / schedule.

[0134] Then, according to the process configuration management file, use the post command to update the number of parallel tasks, migration speed, and throttling configuration information of the ES cluster reindex task. The specific instructions can be:

[0135] POST / api / v1 / reindex / remote / queue / task_pool,

[0136] POST / api / v1 / reindex / rethrottle / request_per_second,

[0137] POST / api / v1 / reindex / rethrottle / schedule.

[0138] In a preferred embodiment, the data migration module 13 is further configured to write the migration results of the to-be-migrated indexes to the target ES cluster into a preset result table. Based on the migration results in the preset result table, it is determined whether all to-be-migrated indexes transferred to the target ES cluster have been successfully migrated. If not, it is determined to roll back or re-migrate the unsuccessfully migrated to-be-migrated indexes.

[0139] Specifically, to confirm whether the indexes in the source ES cluster have been successfully migrated to the target ES cluster, the migration results of the indexes to be migrated can be analyzed and written to a preset result table. The migration results of the indexes to be migrated can then be read from the preset result table to determine whether all indexes to be migrated have been successfully migrated. For example, in one specific example, the data transfer service provided by REST can be used to copy the data to be migrated from the index to the target index in the target ES cluster by calling a REST API. During the data copy process, REST automatically records the migration progress and results. The migration results from REST can be periodically retrieved and written to a preset result table. By analyzing the migration results in the preset result table, it can be determined whether all indexes to be migrated have been successfully migrated. Specifically, when the migration results for all indexes to be migrated in the preset result table indicate "Migration Completed," the migration of all indexes to be migrated is confirmed to be successful. Specifically, monitoring can be initiated using the POST command POST / api / v1 / monitor / start. Then, the REST service can be called using the interface call instructions to retrieve the migration results stored in REST and store them in the preset result table for real-time monitoring.

[0140] Finally, if not all indices to be migrated are successfully migrated, the unmigrated indices can be rolled back or re-migrated. When performing a rollback, if the target index in the target ES cluster has been backed up beforehand, if the corresponding indices to be migrated in the source ES cluster are not successfully migrated, the index data of the backup index can be restored to the target index, returning the target index to its original state. When re-migrating, the data migration process can be restarted in the source ES cluster to transfer the unsuccessfully migrated indices to the target ES cluster.

[0141] Since the principle of solving the problem by this device is similar to that of the above method, the implementation of this device can refer to the implementation of the method and will not be repeated here.

[0142] Based on the same principle, this embodiment also discloses an Elastic Search cluster data migration system. Figure 1 As shown, in this embodiment, the system includes a source ES cluster, a target ES cluster, and a data migration device as described in this embodiment.

[0143] Since the principle of solving the problem by this system is similar to that of the above method, the implementation of this system can refer to the implementation of the method and will not be repeated here.

[0144] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer device. Specifically, the computer device may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0145] In a typical example, a computer device specifically includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method executed by the client as described above is implemented, or when the processor executes the program, the method executed by the server as described above is implemented.

[0146] Reference below Figure 10 , which shows a structural diagram of a computer device 600 suitable for implementing an embodiment of the present application.

[0147] like Figure 10As shown, computer device 600 includes a central processing unit (CPU) 601, which can perform various appropriate tasks and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. Various programs and data required for the operation of system 600 are also stored in RAM 603. CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0148] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including devices such as a hard disk; and a communication section 609 including a network interface card such as a LAN card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read therefrom can be installed in the storage section 608 as needed.

[0149] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program including program code for executing the methods illustrated in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication portion 609 and / or installed from removable media 611.

[0150] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0151] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0152] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0155] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0156] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0157] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0158] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0159] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for migrating data between Elastic Search clusters, characterized by: include: Write all indices to be migrated in the source ES cluster to the preset migration queue; Determine the number of parallel tasks and migration speed for parallel migration based on the migration profile; Start the data migration process corresponding to the number of parallel tasks and transfer the to-be-migrated indexes corresponding to the number of parallel tasks to the target ES cluster according to the migration speed; The method further comprises: Determine throttling configuration information based on process configuration management files; Adjusting the number of parallel tasks and the migration speed according to the throttling configuration information; wherein the throttling configuration information includes multiple time periods and the number of parallel tasks and the migration speed of data migration in each of the multiple time periods; specifically, when the time reaches the start time of each time period, writing the number of parallel tasks and the migration speed corresponding to each time period into the migration configuration file to adjust the number of parallel tasks and the migration speed; The method further includes backing up the index in the target ES cluster in advance: Create a backup index corresponding to each target index in the target ES cluster; Copying the data in the target index to the backup index; Among them, before migrating the index in the source ES cluster to the target ES cluster, a corresponding backup index is established for each target index for backup and data recovery in case of migration failure. Further, a copy of the initial data in the target index is made within the target ES cluster. When the migration of the index data to be migrated in the source ES cluster fails and the data migration needs to be rolled back, the index data in the backup index is copied to the corresponding target index to restore the initial data in the target index.

2. The method for migrating data between Elastic Search clusters according to claim 1, wherein: Further including; Pre-write the IP address of the source ES cluster to the whitelist of the target ES cluster.

3. The method for migrating data between Elastic Search clusters according to claim 1, wherein: Further including: Writing the backup progress of copying the data in the target index to the backup index into a preset backup progress table; When the backup progress in the backup progress table indicates that the backup is completed, the data to be migrated of the index to be migrated is migrated to the target ES cluster through a data migration process.

4. The method for migrating data between Elastic Search clusters according to claim 1, wherein: The step of transmitting the to-be-migrated indexes corresponding to the number of parallel tasks to the target ES cluster specifically includes: Acquire the data to be migrated of the to-be-migrated index corresponding to the number of parallel tasks; Encrypting the data to be migrated according to preset encryption rules; The encrypted data to be migrated is transferred to the target ES cluster.

5. The method for migrating data between Elastic Search clusters according to claim 1, wherein: Further including: Write the migration results of the index to be migrated to the target ES cluster into the preset result table; Determine whether all the indexes to be migrated to the target ES cluster have been successfully migrated based on the migration results in the preset result table. If not, determine to roll back or re-migrate the unmigrated indexes.

6. An Elastic Search inter-cluster data migration device, characterized in that: include: The index writing module is used to write all the indices to be migrated in the source ES cluster into the preset migration queue; A migration configuration module is used to determine the number of parallel tasks and the migration speed of parallel migration according to the migration configuration file; A data migration module is used to start the data migration process corresponding to the number of parallel tasks and transfer the to-be-migrated indexes corresponding to the number of parallel tasks to the target ES cluster according to the migration speed; The data migration module is also used for: Determine throttling configuration information based on process configuration management files; Adjusting the number of parallel tasks and the migration speed according to the throttling configuration information; wherein the throttling configuration information includes multiple time periods and the number of parallel tasks and the migration speed of data migration in each of the multiple time periods; when the time reaches the start time of each time period, writing the number of parallel tasks and the migration speed corresponding to each time period into the migration configuration file to adjust the number of parallel tasks and the migration speed; The device also includes a preprocessing module; the preprocessing module is further used to back up the index in the target ES cluster in advance, that is, the preprocessing module is used to establish a backup index corresponding to each target index in the target ES cluster, and copy the data in the target index to the backup index, wherein, before migrating the index in the source ES cluster to the target ES cluster, a corresponding backup index is established for each target index for backup and data recovery of failed migration, and further a copy of the initial data in the target index is made within the target ES cluster. When the migration of the index data to be migrated in the source ES cluster fails and the data migration needs to be rolled back, the index data in the backup index is copied to the corresponding target index to restore the initial data in the target index.

7. An Elastic Search inter-cluster data migration system, characterized by: It includes a source ES cluster, a target ES cluster and the data migration device as claimed in claim 6.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

9. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images