Index reconstruction method and system, electronic equipment and medium

Abstract

The invention relates to the technical field of databases, and mainly relates to an index reconstruction method and system, electronic equipment and a medium, and the method comprises the steps: building a target index according to an index reconstruction request task; receiving a data migration request, wherein the data migration request comprises an identifier of an original index, an identifier of a target index and an index alias corresponding to the original index; according to the identifier of the original index, monitoring the operation of the original index and recording an operation log; migrating the target data in the original index to a target index; under the condition that migration of the target data is completed, the operation log is replayed to the target index; under the condition that the number of the operation logs which are not replayed is smaller than a preset threshold value, blocking an index operation request of an original index; under the condition that all the operation logs are replayed, migrating the index alias to the target index; deleting the original index, and responding to the index operation request through the target index. By applying the scheme of the invention, on the basis of quickly reconstructing the index, the availability of the ES service in the reconstruction process is improved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and mainly to an index reconstruction method, system, electronic device and medium. Background Art

[0002] ES (Elasticsearch) is an open source, scalable, distributed, full-text search engine that can store and retrieve data in near real time and meet complex conditional search requirements. It has good scalability and can be expanded to hundreds of servers and process PB-level data. ES is developed in Java and uses Lucene as its core to implement indexing and search functions, but it hides the complexity of Lucene through simple Restful APIs and Java APIs, making full-text search simple.

[0003] ES organizes data into indexes. Before writing data, you need to create an index. When creating an index, you need to specify the number of index shards, index mapping attributes, attribute tokenizers, and other parameters. Then write the corresponding data into the index. When writing, the data is tokenized and parsed according to the index settings. ES indexes are designed to be immutable and only support a few incremental operations, such as adding mapping attributes and increasing the number of replicas. As business needs evolve, existing mapping attributes may need to be modified, or as the amount of data increases, the number of shards may need to be replanned.

[0004] In this regard, ES provides a reindex interface (index reconstruction interface) to meet the needs of index changes: 1) First create a new index; 2) Set the old index to read-only; 3) Use the reindex interface to migrate the data of the old index to the new index; 4) Migrate the index alias from the old index to the new index; 5) Delete the old index to complete the index replacement. This method has a more serious problem, that is, in order to ensure data consistency, the index is read-only during the reindex process, which will cause the ES service to be unavailable for writing. When the amount of index data is large, the write waiting time is very long, and the service availability is seriously affected.

[0005] Therefore, how to ensure the availability of ES services during index reconstruction is a difficult problem that needs to be solved. Summary of the invention

[0006] One of the purposes of the present application is to propose an index reconstruction method to solve or at least partially solve the problems in the related art. The second purpose of the present application is to provide an index reconstruction system, the third purpose of the present application is to provide an electronic device, and the fourth purpose of the present application is to provide a medium.

[0007] In order to achieve the above purpose, the technical solution of this application is as follows:

[0008] An index reconstruction method, the method comprising:

[0009] Create the target index according to the index rebuild request task;

[0010] Receive a data migration request, the data migration request including an identifier of an original index, an identifier of a target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index;

[0011] In response to the data migration request, monitoring the operation of the original index according to the identifier of the original index, and recording the operation log of the original index;

[0012] Migrate the target data in the original index to the target index; the target data is all the data stored in the original index before the operation of the original index is monitored;

[0013] When the target data migration is completed, replaying the operation log to the target index;

[0014] When the number of unreplayed operation logs is less than a preset threshold, blocking the index operation request of the original index;

[0015] When all the operation logs are replayed, the index alias is migrated to the target index;

[0016] The original index is deleted, and the index operation request is responded to through the target index.

[0017] Optionally, in response to the data migration request, monitoring the operation of the original index according to the identifier of the original index, and recording the operation log of the original index;

[0018] In response to the data migration request, a monitoring task for the original index is started according to the identifier of the original index; the monitoring task is pre-created according to the identifier of the original index through a hook function encapsulated in the index operation interface when the data migration request is detected;

[0019] The operation log of the original index is recorded in chronological order, and the operation log is transmitted to the master node.

[0020] Optionally, when the target data migration is completed, replaying the operation log to the target index includes:

[0021] Obtaining the operation log stored in the master node;

[0022] According to the operation log, the index operations performed by the original index during the migration of the target data are replayed to the target index one by one in chronological order.

[0023] Optionally, migrating the target data in the original index to the target index includes:

[0024] Determining a response time for the data migration request;

[0025] Determining the target data in the original index according to the response time;

[0026] According to the preset data volume, the target data is migrated to the target index.

[0027] Optionally, establishing a target index according to the index reconstruction request task includes:

[0028] Parse the index rebuilding request task to obtain configuration information of the target index; the configuration information includes the number of shards of the target index, the mapping attributes of the index, and the word segmenter of the attribute;

[0029] Call the index operation interface and create the target index according to the configuration information.

[0030] Optionally, when all the operation logs are replayed, migrating the index alias to the target index includes:

[0031] When all the operation logs are replayed, the corresponding relationship between the index alias and the original index is deleted;

[0032] Establishing a correspondence between the index alias and the target index so that the index alias routes the index operation request to the target index;

[0033] The deleting the original index and responding to the index operation request through the target index includes:

[0034] Deleting the original index;

[0035] Unblocking the index operation request;

[0036] Routing the index operation request to the target index according to the index alias;

[0037] In response to the index operation request, an index operation corresponding to the index operation request is performed through the target index.

[0038] Optionally, when the number of unreplayed operation logs is less than a preset threshold, blocking the index operation request of the original index includes:

[0039] Monitoring whether the number of unreplayed operation logs is less than the preset threshold;

[0040] When the number of unreplayed operation logs is less than a preset threshold, the newly added index operation request of the original index is stored in a waiting-for-response queue, and the original index is adjusted to a read-only state.

[0041] An index reconstruction system, the system comprising:

[0042] Create a new module to build the target index according to the index rebuild request;

[0043] A receiving module, configured to receive a data migration request, wherein the data migration request includes an identifier of an original index, an identifier of a target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index;

[0044] A monitoring module, configured to respond to the data migration request, monitor the operation of the original index according to the identifier of the original index, and record the operation log of the original index;

[0045] A data migration module, used to migrate the target data in the original index to the target index; the target data is all the data stored in the original index before the operation of the original index is monitored;

[0046] A replay module, used for replaying the operation log to the target index when the target data migration is completed;

[0047] A blocking module, used to block the index operation request of the original index when the remaining number of the unreplayed operation logs is less than a preset threshold;

[0048] An alias migration module, used for migrating the index alias to the target index when the operation log replay is completed;

[0049] The deletion module is used to delete the original index and respond to the index operation request through the target index.

[0050] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned index reconstruction method when executing the computer program.

[0051] A non-volatile storage medium stores a computer program, which implements the above-mentioned index reconstruction method when executed by a processor.

[0052] Beneficial effects of this application:

[0053] In the present application, a target index is established according to an index reconstruction request task; a data migration request is received, the data migration request including an identifier of an original index, an identifier of a target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index; in response to the data migration request, operations of the original index are monitored according to the identifier of the original index, and operation logs of the original index are recorded; the target data in the original index is migrated to the target index; the target data is all the data stored in the original index before the operations of the original index are monitored; when the migration of the target data is completed, the operation log is replayed to the target index; when the number of unreplayed operation logs is less than a preset threshold, the index operation request of the original index is blocked; when all the operation logs are replayed, the index alias is migrated to the target index; the original index is deleted, and the index operation request is responded to through the target index. During the target index data migration process, the original index can still perform new index operations to improve the availability of the ES service; by monitoring the index operations of the original index during the target index data migration process, after the target index data migration is completed, the new index operations of the original index are replayed to the target index until the new and old indexes reach a dynamic balance of data, and then the new and old indexes are switched through the index alias. In this way, data consistency can be guaranteed during the index reconstruction process, and the availability of the ES service is guaranteed, and the efficiency of ES index reconstruction is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 is a flow chart of an index reconstruction method described in an embodiment of the present application;

[0056] Figure 2 It is a partial logic diagram of the index reconstruction method described in one embodiment of the present application;

[0057] Figure 3 It is a schematic diagram of the framework of the index reconstruction system described in one embodiment of the present application.

[0058] Among them, 300-index reconstruction system, 301-new creation module, 302-receiving module, 303-monitoring module, 304-data migration module, 305-replay module, 306-blocking module, 307-alias migration module, 308-deletion module. DETAILED DESCRIPTION

[0059] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0060] This application proposes a solution for ES index reconstruction. During the index reconstruction process, the read and write requests of the original index are not restricted. Instead, the read and write requests are synchronized in the ES as log backups. After the original index data is migrated to the new index, the log is replayed for the new index until the new and old indexes reach a dynamic balance of data, and then the new and old indexes are switched through the index alias. This method can ensure data consistency during the reindex process and provide very good availability, improve the efficiency of ES index reconstruction, greatly simplify the user's workload, and reduce the complexity of the system.

[0061] Specifically, refer to Figure 1 , Figure 1 FIG. 1 is a flow chart of an index reconstruction method described in an embodiment of the present application. Figure 1 As shown, in one embodiment of the present application, the index reconstruction method can be performed according to the following steps:

[0062] Step S101: Create a target index according to the index reconstruction request task.

[0063] Specifically, when the user needs to adapt to business needs and make some modifications to the original index, the original index cannot be rebuilt quickly due to the functional limitations of the original ES. This application proposes that after receiving the index reconstruction request task, the target index is first established according to the index reconstruction task. Specifically, the original index corresponding to the target index is used as the basic template, and the target index is established according to the relevant parameters of the target index configured in the task.

[0064] In a feasible implementation manner, the above step S101, establishing a target index according to the index reconstruction request task, includes:

[0065] Parse the index rebuilding request task to obtain configuration information of the target index; the configuration information includes the number of shards of the target index, the mapping attributes of the index, and the word segmenter of the attribute;

[0066] Call the index operation interface and create the target index according to the configuration information.

[0067] Specifically, in the present application, after parsing the index reconstruction request task, the index reconstruction task can be divided into multiple subtasks: target index creation task, original index monitoring task, target index data migration task, operation log replay task, index alias migration task. In the target index creation task, the task includes the configuration information of the target index, and the configuration information includes parameters such as the number of shards of the target index, the mapping attributes of the index, and the attribute tokenizer. Call the index operation interface, such as the index creation interface, and execute the command to create a new index according to the configuration information to create the target index.

[0068] In this embodiment, by parsing the index reconstruction task, obtaining the configuration information of the target index, calling the index operation interface, and establishing the required target index according to the configuration information, the framework of the target index can be built first to facilitate the subsequent data migration operations.

[0069] Step S102, receiving a data migration request, wherein the data migration request includes an identifier of an original index, an identifier of a target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index.

[0070] Specifically, in this embodiment, the data migration request is used to migrate the data of the original index to the target index. In this application, the data migration request includes not only the identifier of the original index and the identifier of the target index, but also the index alias corresponding to the original index. The index alias can be regarded as a shortcut, pointing to a real index. At this time, the original index corresponds to the index alias, and the index alias routes the index operation request to the corresponding original index so that the original index responds to the index operation request to perform the corresponding operation. It can be understood that when the index corresponding to the subsequent index alias changes, the index operation request is routed to the changed index. The identifier of each index can be the name or number of the index, which is not limited here. In this embodiment, the creation and data migration process of the target index will not affect the index operation of the original index, and the index operation request routed to the original index by the index alias is still responded normally. Compared with the related art, the index alias is added to the data migration request to facilitate the subsequent switching of the original index and the target index, thereby improving the efficiency of index reconstruction.

[0071] Step S103 , in response to the data migration request, monitoring the operation of the original index according to the identifier of the original index, and recording the operation log of the original index.

[0072] Specifically, after detecting the data migration request, the index operation of the original index is monitored according to the identifier of the original index. For example, a monitoring event for the original index is started. Each time an index operation is performed, a corresponding operation log is generated and recorded and stored in chronological order.

[0073] Step S104, migrating the target data in the original index to the target index; the target data is all the data stored in the original index before the operation of the original index is monitored.

[0074] Specifically, after monitoring the operation of the original index, the target index data migration is started, and the target data in the original index is migrated to the target index. At this time, since the original index is still performing a new index operation, the data in the original index may be continuously updated. Therefore, the target data migrated in this data migration task is the data stored in the original index before the operation of the original index is monitored.

[0075] Step S105, when the target data migration is completed, replay the operation log to the target index.

[0076] Specifically, when the target data in the original index is migrated, the incremental data of the original index during the data migration process needs to be migrated to the target index. Then the operation log is replayed to the target index, so that the index operations monitored during this process are executed one by one in the target index.

[0077] Step S106: When the number of unreplayed operation logs is less than a preset threshold, the index operation request of the original index is blocked.

[0078] Specifically, in order to ensure the availability of the ES service as much as possible, the original index is still responding to new index operation requests during the process of replaying the operation logs to the target index. Therefore, new operation logs are also increasing. If the original index and the target index are switched after all the newly added operation logs have been replayed, it may take a long time to wait. Directly blocking the index operation request of the original index and ending the monitoring of the operation of the original index may result in a large number of operation logs and a long blocking time for the index operation request, which reduces the availability of the system.

[0079] Therefore, in this embodiment, when the number of unreplayed operation logs is less than a preset threshold, the index operation request of the original index is blocked, and the original index is no longer required to respond to new index operation requests; at the same time, the monitoring of the operation of the original index can be stopped, and no new operation logs are generated. The preset threshold can be set as needed. For example, the number of remaining unreplayed operation logs is monitored. When the number of remaining operation logs is 0 or less than a certain number, the index operation log recording task is stopped, the original index is temporarily blocked from performing index operations, and all log records are replayed.

[0080] Step S107: when all the operation logs are replayed, the index alias is migrated to the target index.

[0081] Specifically, after all operation logs are replayed, the index alias corresponding to the original index is migrated to the target index. The index corresponding to the migrated index alias is the target index, so subsequent index operations are completed by the target index.

[0082] Step S108: delete the original index and respond to the index operation request through the target index.

[0083] Specifically, after the migration of the index alias is completed, the original index is no longer needed and can be deleted. The index alias routes the index operation request to the target index and responds to the index operation request through the target index. In this way, the index reconstruction is completed.

[0084] In this embodiment, a target index is established according to an index reconstruction request task; a data migration request is received, the data migration request including an identifier of an original index, an identifier of a target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index; in response to the data migration request, operations of the original index are monitored according to the identifier of the original index, and an operation log of the original index is recorded; the target data in the original index is migrated to the target index; the target data is all the data stored in the original index before the operations of the original index are monitored; when the migration of the target data is completed, the operation log is replayed to the target index; when the number of unreplayed operation logs is less than a preset threshold, the index operation request of the original index is blocked; when all the operation logs are replayed, the index alias is migrated to the target index; the original index is deleted, and the index operation request is responded to through the target index. During the target index data migration process, the original index can still perform new index operations to improve system availability; by monitoring the index operations of the original index during the target index data migration process, after the target index data migration is completed, the new index operations of the original index are replayed to the target index until the new and old indexes reach a dynamic balance of data, and then the new and old indexes are switched through the index alias. In this way, data consistency and ES service availability can be guaranteed during the index reconstruction process, and the efficiency of ES index reconstruction can be improved.

[0085] In a feasible implementation manner, the above step S103, in response to the data migration request, monitoring the operation of the original index according to the identifier of the original index, and recording the operation log of the original index, may specifically include:

[0086] Step S1031, in response to the data migration request, starting a monitoring task for the original index according to the identifier of the original index; the monitoring task is pre-created according to the identifier of the original index through a hook function encapsulated in the index operation interface when the data migration request is detected;

[0087] Step S1032, record the operation log of the original index in chronological order, and transmit the operation log to the master node.

[0088] Specifically, after the index reconstruction task is parsed, the hook function encapsulated in the index operation interface is called first. According to the data migration request, a new listening task for the original index is created. When the data migration request begins to be responded to, the listening task is started, and the index operations performed on the original index are recorded one by one in chronological order. In this way, during the subsequent data migration process, the original index can still respond to the index operation request normally, but the operations it performs are recorded one by one, so as to synchronize the incremental index operations of the original index during the data migration process of the target index to the target index.

[0089] In this embodiment, by optimizing the index operation interface (encapsulating the hook function to monitor and record the operation of the original index), appropriate modifications are made inside ES to reduce the invasiveness of the ES overall software, reduce the optimization cost and optimization time. At this time, the operation log of the original index is transmitted to the master node for storage to maintain the stability of the data, so that when the log is replayed later, the operation log can be quickly obtained to avoid the log data being stored in fragments.

[0090] Continuing with the above implementation, in a feasible implementation, the above step S105, when the target data migration is completed, replaying the operation log to the target index, includes:

[0091] Obtaining the operation log stored in the master node;

[0092] According to the operation log, the index operations performed by the original index during the migration of the target data are replayed to the target index one by one in chronological order.

[0093] Specifically, following the application scenario of the distributed storage system in the above-mentioned implementation, after migrating the target data in the original index to the target index, the master node is requested to obtain the operation log of the original index, and these operation logs are replayed to the target index one by one in chronological order, and the index operations performed by the original index during the migration of the target data are executed in the target index in sequence.

[0094] In this embodiment, for the application scenario of the distributed storage system, the operation log of the original index is stored in the master node. After the target data migration is completed, the operation log of the original index stored in the master node is obtained, so that the operation log can be replayed to the target index to make the data of the original index consistent with that of the target index.

[0095] In a feasible implementation manner, migrating the target data in the original index to the target index includes:

[0096] Determine a response time to the data migration request; determine the target data in the original index according to the response time; and migrate the target data to the target index according to a preset data volume.

[0097] Specifically, when the data migration request is executed, the index operation of the original index is started, and the index operation after the data migration request starts to be executed has been recorded in the master node. During this data migration, the response time when the data migration request is responded to is first determined. It can be understood that at this response time, the monitoring task for the original index is started. According to the response time, the target data required for this migration can be determined (the data that already exists in the original index when the data migration task starts to be executed), and then the target data can be gradually migrated to the target index according to the preset data volume (that is, the preset batch size of the migration data). The batch size can be set according to historical experience or network bandwidth, or it can be a user-defined preset data volume.

[0098] In this embodiment, by determining the response time of the data migration request; determining the target data in the original index based on the response time; and migrating the target data to the target index according to the preset data volume, it is avoided that duplicate data is added to the target index during the replay of the operation log, reduces redundant data, and improves the speed of index reconstruction.

[0099] ES executes the reindex data migration task. The execution time of the task is related to the size of the index data. The larger the number of documents contained in the index, the longer the task execution time. At this time, the original index in the ES cluster can still be read and written normally. In a feasible implementation, when migrating data to the target index, in order to further improve the speed of data migration, the batch size of the migrated data is adjusted according to the current system performance (such as the commonly used number of thousands to tens of thousands of records). Too large a batch may cause memory overflow, and too small a batch may not fully utilize the network bandwidth and I / O performance. Therefore, when the data is migrated, the performance of the current system is tested according to the actual migration time of the batch size data, and the preset batch size is adjusted to increase the speed of data migration.

[0100] In a feasible implementation manner, when all the operation logs are replayed, migrating the index alias to the target index includes:

[0101] When all the operation logs are replayed, the corresponding relationship between the index alias and the original index is deleted; and the corresponding relationship between the index alias and the target index is established so that the index alias routes the index operation request to the target index;

[0102] The deleting the original index and responding to the index operation request through the target index includes:

[0103] The original index is deleted; the index operation request is unblocked; the index operation request is routed to the target index according to the index alias; and the index operation corresponding to the index operation request is executed through the target index in response to the index operation request.

[0104] Specifically, when all operation logs are replayed, the corresponding relationship between the index alias and the original index is deleted, and the corresponding relationship between the index alias and the target index is established, so that the index alias routes the original index operation request for the original index to the target index, and the target index performs the index operation. After the index alias is migrated to the target index, the original index is deleted to unblock the index operation request; at this time, the index operation request that has not been executed is routed to the target index by the index alias whose corresponding relationship has been modified; through the target index, respond to the new index operation request and perform the index operation corresponding to the index operation request.

[0105] In this embodiment, the switching between the original index and the target index is achieved by modifying the corresponding relationship of the index alias, and an index alias migration task is added to the index reconstruction task. There is no need to rename the target index or the original index, so that the index reconstruction is performed without the user's perception, thereby improving the user experience.

[0106] In a feasible implementation manner, when the number of unreplayed operation logs is less than a preset threshold, blocking the index operation request of the original index includes:

[0107] Monitoring whether the number of unreplayed operation logs is less than the preset threshold;

[0108] When the number of unreplayed operation logs is less than a preset threshold, the newly added index operation request of the original index is stored in a waiting-for-response queue, and the original index is adjusted to a read-only state.

[0109] Specifically, when the number of unreplayed operation logs is detected to be less than a preset threshold, the newly added index operation request of the original index is stored in the waiting response queue, and the original index is adjusted to a read-only state. In this way, the index operation requests in the waiting response queue can be responded to in sequence after the target index is converted, and at the same time, the original index is adjusted to a read-only state, and no new index operations are performed.

[0110] In this embodiment, by monitoring the number of unreplayed operation logs, the index operation request of the original index can be blocked in time to avoid the problem that the original index has been performing index operations and the generated operation logs cannot be replayed for a long time, thereby improving the efficiency of index reconstruction.

[0111] In one embodiment of the present application, referring to Figure 2 , Figure 2 FIG. 1 is a partial logic diagram of the index reconstruction method described in an embodiment of the present application. Figure 2 As shown in the figure, taking the reconstruction of the original index into the target index with increased number of shards as an example, the index reconstruction process is further explained:

[0112] In this scenario, the ES cluster is responsible for full-text retrieval of a large amount of IT hardware device performance log data. Its index name is device_performance_log, and its alias is devicePerformanceLog. Due to historical reasons, the number of shards is 2. As the business expands and the amount of data increases, the size of each shard of the index exceeds 100G, which has a great impact on the performance of the index. At this time, consider using the index reconstruction method to adjust the number of shards of the index shards and optimize its performance.

[0113] First, create the device_performance_log_new index and adjust its shard number number_of_shards to 8, which is 4 times the original number. Other index configurations remain unchanged. Then call the reindex interface. During the reindex execution, the original index is still used normally through the alias devicePerformanceLog. After the reindex is completed, the alias is migrated to the new index device_performance_log_new. The index is rebuilt without the user's awareness. The size of a single shard of the new index is reduced to 25G, and the performance returns to normal.

[0114] The execution process of index reconstruction is as follows: 1) trigger the index operation monitoring task for the original index device_performance_log. This task records the index operation as a log and sends it to the Master node. The Master node receives the log and records it in the device_performance_log_reindex_log log file; 2) migrate the original index data to the device_performance_log_new index; 3) the device_performance_log_new index reads the device_performance_log_reindex_log log file and replays its index operation log; 4) when the log replay is completed, close the index operation log monitoring task and temporarily block the index operation; 5) perform index alias migration. After the migration is completed, the index operation blockage is released, the old index is deleted, and the index reconstruction is completed.

[0115] Through this embodiment, the read and write requests of the original index are not restricted during the index reconstruction process, thereby improving the availability of the service; based on the upgrade and optimization of ES itself, there is no need to introduce third-party components, thereby ensuring the independence and portability of the service; based on the optimization of the original reindex interface (adding hooks), the user-side imperceptible and smooth migration is achieved, providing good ease of use; the original reindex function of ES is further expanded, and an alias migration function is added, thereby reducing the operation steps of index reconstruction, improving the efficiency of index reconstruction, and simplifying user operations.

[0116] Reference Figure 3 , Figure 3 FIG. 1 is a schematic diagram of the framework of the index reconstruction system described in one embodiment of the present application. Figure 3 As shown, based on the same or similar technical concept as the above embodiment, in one embodiment of the present application, an index reconstruction system is proposed, and the index reconstruction system 300 includes:

[0117] A new module 301 is created, for establishing a target index according to an index reconstruction request;

[0118] The receiving module 302 is used to receive a data migration request, wherein the data migration request includes an identifier of an original index, an identifier of a target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index;

[0119] A monitoring module 303 is used to respond to the data migration request, monitor the operation of the original index according to the identifier of the original index, and record the operation log of the original index;

[0120] A data migration module 304 is used to migrate the target data in the original index to the target index; the target data is all the data stored in the original index before the operation of the original index is monitored;

[0121] A replay module 305, configured to replay the operation log to the target index when the target data migration is completed;

[0122] The blocking module 306 is used to block the index operation request of the original index when the remaining number of the unreplayed operation logs is less than a preset threshold;

[0123] The alias migration module 307 is used to migrate the index alias to the target index when the operation log replay is completed;

[0124] The deletion module 308 is used to delete the original index and respond to the index operation request through the target index.

[0125] In this embodiment, a target index is established according to an index reconstruction request task; a data migration request is received, the data migration request including an identifier of an original index, an identifier of a target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index; in response to the data migration request, operations of the original index are monitored according to the identifier of the original index, and an operation log of the original index is recorded; the target data in the original index is migrated to the target index; the target data is all the data stored in the original index before the operations of the original index are monitored; when the migration of the target data is completed, the operation log is replayed to the target index; when the number of unreplayed operation logs is less than a preset threshold, the index operation request of the original index is blocked; when all the operation logs are replayed, the index alias is migrated to the target index; the original index is deleted, and the index operation request is responded to through the target index. During the target index data migration process, the original index can still perform new index operations to improve the availability of the ES service; by monitoring the index operations of the original index during the target index data migration process, after the target index data migration is completed, the new index operations of the original index are replayed to the target index until the new and old indexes reach a dynamic balance of data, and then the new and old indexes are switched through the index alias. In this way, data consistency can be guaranteed during the index reconstruction process, and the availability of the ES service is guaranteed, and the efficiency of ES index reconstruction is improved.

[0126] In a feasible implementation manner, the monitoring module includes:

[0127] A monitoring start module, used to start a monitoring task for the original index in response to the data migration request according to the identifier of the original index; the monitoring task is pre-created according to the identifier of the original index through a hook function encapsulated in the index operation interface when the data migration request is detected;

[0128] The transmission module is used to record the operation log of the original index in chronological order and transmit the operation log to the master node.

[0129] In a feasible implementation manner, the playback module includes:

[0130] A log acquisition module, used to acquire the operation log stored in the master node;

[0131] The operation execution module is used to replay the index operations performed by the original index during the migration of the target data to the target index one by one in chronological order according to the operation log.

[0132] In a feasible implementation manner, the data migration module includes:

[0133] A first determining module, configured to determine a response time for the data migration request;

[0134] A second determination module, configured to determine the target data in the original index according to the response time;

[0135] The batch migration module is used to migrate the target data to the target index according to a preset data volume.

[0136] In a feasible implementation manner, the newly created module includes:

[0137] A parsing module, used to parse the index rebuilding request task and obtain configuration information of the target index; the configuration information includes the number of shards of the target index, the mapping attributes of the index and the word separator of the attribute;

[0138] The interface calling module is used to call the index operation interface and create the target index according to the configuration information.

[0139] In the present application, the description of the system embodiment is relatively short, and specific details can be referred to the above method embodiment, which will not be repeated here.

[0140] In one embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned index reconstruction method when executing the computer program.

[0141] In one embodiment of the present application, a non-volatile storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the index reconstruction method as described in the above embodiment is implemented.

[0142] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0143] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0144] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0145] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0147] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0148] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0149] The above provides a detailed introduction to an index reconstruction method, system, electronic device and medium. This article uses specific examples to illustrate the principles and embodiments of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific embodiments and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An index reconstruction method, characterized in that: The method comprises: Create the target index according to the index rebuild request task; Receive a data migration request, the data migration request including an identifier of an original index, an identifier of a target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index; In response to the data migration request, monitoring the operation of the original index according to the identifier of the original index, and recording the operation log of the original index; Migrate the target data in the original index to the target index; the target data is all the data stored in the original index before the operation of the original index is monitored; When the target data migration is completed, replaying the operation log to the target index; When the number of unreplayed operation logs is less than a preset threshold, blocking the index operation request of the original index; When all the operation logs are replayed, the index alias is migrated to the target index; The original index is deleted, and the index operation request is responded to through the target index.

2. The method according to claim 1, characterized in that In response to the data migration request, monitoring the operation of the original index according to the identifier of the original index, and recording the operation log of the original index; In response to the data migration request, a monitoring task for the original index is started according to the identifier of the original index; the monitoring task is pre-created according to the identifier of the original index through a hook function encapsulated in the index operation interface when the data migration request is detected; The operation log of the original index is recorded in chronological order, and the operation log is transmitted to the master node.

3. The method according to claim 2, characterized in that When the target data migration is completed, replaying the operation log to the target index includes: Obtaining the operation log stored in the master node; According to the operation log, the index operations performed by the original index during the migration of the target data are replayed to the target index one by one in chronological order.

4. The method according to claim 1, characterized in that Migrating the target data in the original index to the target index includes: Determining a response time for the data migration request; Determining the target data in the original index according to the response time; According to the preset data volume, the target data is migrated to the target index.

5. The method according to claim 1, characterized in that The step of establishing a target index according to the index reconstruction request task includes: Parse the index rebuilding request task to obtain configuration information of the target index; the configuration information includes the number of shards of the target index, the mapping attributes of the index, and the word segmenter of the attribute; Call the index operation interface and create the target index according to the configuration information.

6. The method according to claim 1, characterized in that When all the operation logs are replayed, migrating the index alias to the target index includes: When all the operation logs are replayed, the corresponding relationship between the index alias and the original index is deleted; Establishing a correspondence between the index alias and the target index so that the index alias routes the index operation request to the target index; The deleting the original index and responding to the index operation request through the target index includes: Deleting the original index; Unblocking the index operation request; Routing the index operation request to the target index according to the index alias; In response to the index operation request, an index operation corresponding to the index operation request is performed through the target index.

7. The method according to claim 1, characterized in that When the number of the unreplayed operation logs is less than a preset threshold, blocking the index operation request of the original index includes: Monitoring whether the number of unreplayed operation logs is less than the preset threshold; When the number of unreplayed operation logs is less than a preset threshold, the newly added index operation request of the original index is stored in a waiting-for-response queue, and the original index is adjusted to a read-only state.

8. An index reconstruction system, characterized in that: The system comprises: Create a new module to build the target index according to the index rebuild request; A receiving module, configured to receive a data migration request, wherein the data migration request includes an identifier of an original index, an identifier of a target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index; A monitoring module, configured to respond to the data migration request, monitor the operation of the original index according to the identifier of the original index, and record the operation log of the original index; A data migration module, used to migrate the target data in the original index to the target index; the target data is all the data stored in the original index before the operation of the original index is monitored; A replay module, used for replaying the operation log to the target index when the target data migration is completed; A blocking module, used to block the index operation request of the original index when the remaining number of the unreplayed operation logs is less than a preset threshold; An alias migration module, used for migrating the index alias to the target index when the operation log replay is completed; The deletion module is used to delete the original index and respond to the index operation request through the target index.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the index reconstruction method according to any one of claims 1 to 7 is implemented.

10. A non-volatile storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the index reconstruction method as described in any one of claims 1 to 7 is implemented.

Images