Index reconstruction method, system, electronic device and medium
By monitoring and recording operation logs during the Elasticsearch index rebuild process, the ES service availability is maintained during the data migration process, solving the problem of service unavailability caused by index rebuilding and improving efficiency and consistency.
Patent Information
- Application Number
- CN202411303123.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2025-09-19
- Estimated Expiration
- 2044-09-18
AI Technical Summary
During the Elasticsearch index rebuild process, existing technologies cause ES service write unavailability, long write waiting time, and seriously affect service availability.
By monitoring the operations of the original index and recording the operation logs, the target data is migrated to the target index. After the operation log replay is completed, the index alias is switched to ensure that the original index can still perform operations during the data migration process until the new and old indexes reach a dynamic data balance.
This ensures data consistency and ES service availability during the index reconstruction process, improves index reconstruction efficiency, and simplifies user operations.
Smart Images
Figure CN119311686B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of database technology, and primarily to an index reconstruction method, system, electronic device, and medium. Background Art
[0002] ES (short for Elasticsearch) is an open-source, scalable, distributed, full-text search engine that stores and retrieves data in near real time, meeting complex search requirements. It offers excellent scalability, capable of scaling to hundreds of servers and processing petabytes of data. Developed in Java, ES uses Lucene as its core for indexing and search. However, it hides Lucene's complexity through simple RESTful and Java APIs, making full-text search simple.
[0003] ES organizes data into indexes. Before writing data, you must create an index. When creating an index, you specify parameters such as the number of shards, the index's mapping attributes, and the attribute's tokenizer. The corresponding data is then written to the index, and token parsing and other operations are performed based on the index settings. ES indexes are designed to be immutable and support only a limited number of incremental operations, such as adding mapping attributes or increasing the number of replicas. As business needs evolve, existing mapping attributes may need to be modified, or as data volume increases, the number of shards may need to be replanned.
[0004] ES provides a reindex API to meet index change requirements: 1) Create a new index; 2) Set the old index to read-only; 3) Use the reindex API to migrate the data from the old index to the new index; 4) Migrate the index alias from the old index to the new index; 5) Delete the old index to complete the index replacement. This method has a serious problem: to ensure data consistency, the index is read-only during the reindex process, which can cause ES service write unavailability. When the index data volume is large, write latency is very long, seriously affecting service availability.
[0005] Therefore, how to ensure the availability of ES services during index reconstruction is a difficult problem that needs to be solved. Summary of the Invention
[0006] One of the purposes of this application is to propose an index reconstruction method to solve or at least partially solve the problems in the related art. The second purpose of this application is to provide an index reconstruction system. The third purpose of this application is to provide an electronic device. The fourth purpose of this application is to provide a medium.
[0007] In order to achieve the above objectives, the technical solutions of this application are as follows:
[0008] An index reconstruction method, the method comprising:
[0009] Create the target index based on the index rebuild request task;
[0010] Receive a data migration request, the data migration request including an identifier of an original index, an identifier of a target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index;
[0011] In response to the data migration request, monitoring operations of the original index according to the identifier of the original index, and recording an operation log of the original index;
[0012] Migrating target data in the original index to the target index; the target data is all data stored in the original index before the operation of the original index is monitored;
[0013] When the target data migration is completed, replaying the operation log to the target index;
[0014] When the number of unreplayed operation logs is less than a preset threshold, blocking the index operation request of the original index;
[0015] When all the operation logs are replayed, the index alias is migrated to the target index;
[0016] The original index is deleted, and the index operation request is responded to through the target index.
[0017] Optionally, in response to the data migration request, monitoring operations of the original index according to the identifier of the original index, and recording an operation log of the original index;
[0018] In response to the data migration request, starting a monitoring task for the original index according to the identifier of the original index; the monitoring task is pre-created according to the identifier of the original index through a hook function encapsulated in the index operation interface when the data migration request is detected;
[0019] The operation log of the original index is recorded in chronological order and transmitted to the master node.
[0020] Optionally, when the target data migration is completed, replaying the operation log to the target index includes:
[0021] Obtaining the operation log stored in the master node;
[0022] According to the operation log, the index operations performed by the original index during the migration of the target data are replayed to the target index one by one in chronological order.
[0023] Optionally, migrating the target data in the original index to the target index includes:
[0024] determining a response time for the data migration request;
[0025] Determining the target data in the original index according to the response time;
[0026] Migrate the target data to the target index according to the preset data volume.
[0027] Optionally, establishing a target index according to the index reconstruction request task includes:
[0028] Parse the index rebuild request task to obtain configuration information of the target index; the configuration information includes the number of shards of the target index, the mapping attributes of the index, and the attribute word segmenter;
[0029] Call the index operation interface and create the target index according to the configuration information.
[0030] Optionally, when all the operation logs are replayed, migrating the index alias to the target index includes:
[0031] When all the operation logs are replayed, the corresponding relationship between the index alias and the original index is deleted;
[0032] Establishing a correspondence between the index alias and the target index so that the index alias routes the index operation request to the target index;
[0033] The deleting the original index and responding to the index operation request through the target index includes:
[0034] Deleting the original index;
[0035] Unblocking the index operation request;
[0036] Routing the index operation request to the target index according to the index alias;
[0037] In response to the index operation request, an index operation corresponding to the index operation request is performed using the target index.
[0038] Optionally, when the number of unreplayed operation logs is less than a preset threshold, blocking the index operation request of the original index includes:
[0039] Monitoring whether the number of unreplayed operation logs is less than the preset threshold;
[0040] When the number of unreplayed operation logs is less than a preset threshold, the newly added index operation request of the original index is stored in a waiting-for-response queue, and the original index is adjusted to a read-only state.
[0041] An index reconstruction system, comprising:
[0042] Create a new module to build the target index based on the index rebuild request;
[0043] A receiving module, configured to receive a data migration request, wherein the data migration request includes an identifier of an original index, an identifier of a target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index;
[0044] a monitoring module, configured to, in response to the data migration request, monitor operations of the original index according to the identifier of the original index, and record an operation log of the original index;
[0045] A data migration module, configured to migrate target data in the original index to the target index; the target data is all data stored in the original index before the operation of the original index is monitored;
[0046] A replay module, configured to replay the operation log to the target index when the target data migration is completed;
[0047] A blocking module, configured to block the index operation request of the original index when the remaining number of unreplayed operation logs is less than a preset threshold;
[0048] an alias migration module, configured to migrate the index alias to the target index when the operation log replay is completed;
[0049] The deletion module is used to delete the original index and respond to the index operation request through the target index.
[0050] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the index reconstruction method described above is implemented.
[0051] A non-volatile storage medium stores a computer program, which implements the above-mentioned index reconstruction method when executed by a processor.
[0052] Beneficial effects of this application:
[0053] In the present application, a target index is established according to an index reconstruction request task; a data migration request is received, the data migration request including the identifier of the original index, the identifier of the target index, and the index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index; in response to the data migration request, the operation of the original index is monitored according to the identifier of the original index, and the operation log of the original index is recorded; the target data in the original index is migrated to the target index; the target data is all the data stored in the original index before the operation of the original index is monitored; when the target data migration is completed, the operation log is replayed to the target index; when the number of unreplayed operation logs is less than a preset threshold, the index operation request of the original index is blocked; when all the operation logs are replayed, the index alias is migrated to the target index; the original index is deleted, and the index operation request is responded to through the target index. During the target index data migration process, the original index can still execute new index operations, improving the availability of the ES service. By monitoring the index operations of the original index during the target index data migration process, the new index operations of the original index are replayed to the target index after the target index data migration is completed. After the new and old indexes reach a dynamic data balance, the new and old indexes are switched using the index alias. This ensures data consistency during the index rebuild process, guarantees the availability of the ES service, and improves the efficiency of ES index rebuilds. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0055] Figure 1 is a flow chart of an index reconstruction method according to an embodiment of the present application;
[0056] Figure 2 This is a partial logic diagram of the index reconstruction method described in one embodiment of the present application;
[0057] Figure 3 It is a schematic diagram of the framework of the index reconstruction system described in one embodiment of the present application.
[0058] Among them, 300-index reconstruction system, 301-new creation module, 302-receiving module, 303-monitoring module, 304-data migration module, 305-replay module, 306-blocking module, 307-alias migration module, 308-deletion module. DETAILED DESCRIPTION
[0059] The following will describe the embodiments of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand the other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for the purpose of illustrating the present application and are not intended to limit the scope of protection of the present application.
[0060] This application proposes a solution for ES index reconstruction. During the index reconstruction process, read and write requests of the original index are not restricted. Instead, read and write requests are synchronously backed up in the ES log. After the original index data is migrated to the new index, the log is replayed for the new index until the new and old indexes reach a dynamic balance of data. Then, the old and new indexes are switched through the index alias. This method can ensure data consistency during the reindex process and provide excellent usability, improve the efficiency of ES index reconstruction, greatly simplify the user's workload, and reduce the complexity of the system.
[0061] Specifically, refer to Figure 1 , Figure 1 This is a flow chart of the index reconstruction method described in one embodiment of the present application. Figure 1 As shown, in one embodiment of the present application, the index reconstruction method can be performed according to the following steps:
[0062] Step S101: Create a target index according to the index reconstruction request task.
[0063] Specifically, when a user needs to modify an existing index to meet business needs, the original ES functionality limits the ability to quickly rebuild the original index. This application proposes that upon receiving an index rebuild request, a target index is first established based on the index rebuild task. Specifically, the target index is established using the original index corresponding to the target index as a base template, based on the target index parameters configured in the task.
[0064] In a feasible implementation, the above step S101, establishing a target index according to the index reconstruction request task, includes:
[0065] Parse the index rebuild request task to obtain configuration information of the target index; the configuration information includes the number of shards of the target index, the mapping attributes of the index, and the attribute word segmenter;
[0066] Call the index operation interface and create the target index according to the configuration information.
[0067] Specifically, in this application, after parsing the index reconstruction request task, the index reconstruction task can be divided into multiple subtasks: a target index creation task, an original index monitoring task, a target index data migration task, an operation log replay task, and an index alias migration task. In the target index creation task, the task includes the configuration information of the target index, which includes parameters such as the number of shards of the target index, the index mapping attributes, and the attribute tokenizer. Call the index operation interface, such as the index creation interface, and execute the command to create the new index based on the configuration information to create the target index.
[0068] In this embodiment, by parsing the index reconstruction task, obtaining the configuration information of the target index, calling the index operation interface, and establishing the required target index according to the configuration information, the framework of the target index can be built first to facilitate the subsequent data migration operations.
[0069] Step S102: receiving a data migration request, wherein the data migration request includes an identifier of an original index, an identifier of a target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index.
[0070] Specifically, in this embodiment, the data migration request is used to migrate the data of the original index to the target index. In this application, the data migration request includes not only the identifier of the original index and the identifier of the target index, but also the index alias corresponding to the original index. The index alias can be regarded as a shortcut, pointing to a real index. At this time, the original index corresponds to the index alias, and the index alias routes the index operation request to the corresponding original index, so that the original index responds to the index operation request and performs the corresponding operation. It can be understood that when the index corresponding to the subsequent index alias changes, the index operation request is routed to the changed index. The identifier of each index can be the name or number of the index, which is not limited here. In this embodiment, the creation and data migration process of the target index will not affect the index operation of the original index, and the index operation request routed to the original index by the index alias will still be responded to normally. Compared with the related art, the index alias is added to the data migration request to facilitate the subsequent switching between the original index and the target index, thereby improving the efficiency of index reconstruction.
[0071] Step S103 : In response to the data migration request, according to the identifier of the original index, monitoring the operation of the original index, and recording the operation log of the original index.
[0072] Specifically, after detecting a data migration request, the indexing operation of the original index is monitored according to the identifier of the original index. For example, a monitoring event for the original index is started. Each time an indexing operation is performed, a corresponding operation log is generated and recorded and stored in chronological order.
[0073] Step S104 , migrating the target data in the original index to the target index; the target data is all the data stored in the original index before the operation of the original index is monitored.
[0074] Specifically, after monitoring operations on the original index, data migration for the target index begins, migrating the target data from the original index to the target index. Since the original index is still performing new index operations, the data in the original index may be constantly updated. Therefore, the target data migrated in this data migration task is the data stored in the original index before the operations on the original index were monitored.
[0075] Step S105 : When the target data migration is completed, replay the operation log to the target index.
[0076] Specifically, after the target data in the original index is migrated, the incremental data of the original index during the data migration process needs to be migrated to the target index. The operation log is replayed to the target index, so that the index operations monitored during this process are executed one by one in the target index.
[0077] Step S106: When the number of unreplayed operation logs is less than a preset threshold, the index operation request of the original index is blocked.
[0078] Specifically, in order to ensure the availability of the ES service as much as possible, the original index is still responding to new index operation requests during the process of replaying the operation log to the target index. Therefore, new operation logs are also increasing. If the original index and the target index are switched after all the newly added operation logs are replayed, it may take a long time. Directly blocking the index operation request of the original index and ending the monitoring of the operation of the original index may result in a large number of operation logs and a long blocking time for the index operation request, which reduces the availability of the system.
[0079] Therefore, in this embodiment, if the number of unreplayed operation logs is less than a preset threshold, the index operation request of the original index is blocked, and the original index is no longer allowed to respond to new index operation requests. At the same time, monitoring of the original index's operations can be terminated, and no new operation logs are generated. This preset threshold can be set as needed. For example, the number of remaining unreplayed operation logs can be monitored. If the number of remaining operation logs is 0 or less than a certain number, the index operation log recording task is stopped, and the original index is temporarily blocked from performing index operations to complete the replay of all log records.
[0080] Step S107: When all the operation logs are replayed, the index alias is migrated to the target index.
[0081] Specifically, after all operation logs are replayed, the index alias corresponding to the original index is migrated to the target index. After the migration, the index corresponding to the index alias becomes the target index, so subsequent index operations are completed by the target index.
[0082] Step S108: Delete the original index and respond to the index operation request through the target index.
[0083] Specifically, after the index alias migration is complete, the original index is no longer needed and can be deleted. The index alias routes index operation requests to the target index, and responds to the index operation requests through the target index. In this way, the index reconstruction is completed.
[0084] In this embodiment, a target index is established according to an index reconstruction request task; a data migration request is received, the data migration request including an identifier of the original index, an identifier of the target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index; in response to the data migration request, the operation of the original index is monitored according to the identifier of the original index, and the operation log of the original index is recorded; the target data in the original index is migrated to the target index; the target data is all the data stored in the original index before the operation of the original index is monitored; when the target data migration is completed, the operation log is replayed to the target index; when the number of unreplayed operation logs is less than a preset threshold, the index operation request of the original index is blocked; when all the operation logs are replayed, the index alias is migrated to the target index; the original index is deleted, and the index operation request is responded to through the target index. During the target index data migration process, the original index can still execute new index operations, improving system availability. By monitoring the original index's index operations during the target index data migration process, the new index operations of the original index are replayed to the target index after the target index data migration is completed. After the new and old indexes reach a dynamic data balance, the new and old indexes are switched using the index alias. This ensures data consistency and ES service availability during the index rebuild process, improving the efficiency of ES index rebuilds.
[0085] In a feasible implementation, the above step S103, in response to the data migration request, monitoring the operation of the original index according to the identifier of the original index, and recording the operation log of the original index, may specifically include:
[0086] Step S1031: In response to the data migration request, a monitoring task for the original index is started according to the identifier of the original index; the monitoring task is pre-created according to the identifier of the original index through a hook function encapsulated in the index operation interface when the data migration request is detected;
[0087] Step S1032: record the operation log of the original index in chronological order, and transmit the operation log to the master node.
[0088] Specifically, after the index rebuild task is parsed, a hook function encapsulated within the index operation interface is called. Based on the data migration request, a new monitoring task is created for the original index. When the data migration request begins to be responded to, the monitoring task is activated, and the index operations performed on the original index are recorded in chronological order. This allows the original index to continue responding to index operation requests during subsequent data migrations, but its operations are recorded one by one, facilitating synchronization of incremental index operations performed by the original index during the target index's data migration process with the target index.
[0089] In this implementation, by optimizing the index operation interface (encapsulating hook functions to monitor and record operations on the original index), appropriate modifications are made within ES to reduce the invasiveness of the ES overall software, reducing optimization costs and time. Furthermore, the operation logs of the original index are transferred to the master node for storage to maintain data stability. This allows for quick access to the operation logs during subsequent log replay, preventing log data from being fragmented.
[0090] Continuing with the above embodiment, in a feasible embodiment, the above step S105, when the target data migration is completed, replaying the operation log to the target index, includes:
[0091] Obtaining the operation log stored in the master node;
[0092] According to the operation log, the index operations performed by the original index during the migration of the target data are replayed to the target index one by one in chronological order.
[0093] Specifically, following the application scenario of the distributed storage system in the above embodiment, after migrating the target data in the original index to the target index, the master node is requested to obtain the operation log of the original index, and these operation logs are replayed to the target index one by one in chronological order, and the index operations performed by the original index during the migration of the target data are executed in the target index in sequence.
[0094] In this embodiment, for the application scenario of the distributed storage system, the operation log of the original index is stored in the master node. After the target data migration is completed, the operation log of the original index stored in the master node is obtained, so that the operation log can be replayed to the target index to make the data of the original index consistent with that of the target index.
[0095] In a feasible implementation manner, migrating the target data in the original index to the target index includes:
[0096] Determine a response time to the data migration request; determine the target data in the original index based on the response time; and migrate the target data to the target index according to a preset data volume.
[0097] Specifically, when the data migration request is executed, the index operation of monitoring the original index is started, and the index operation after the data migration request starts to be executed has been recorded in the master node. During this data migration, the response time when the data migration request is responded to is first determined. It can be understood that at this response time, the monitoring task for the original index is started. According to the response time, the target data required for this migration can be determined (the data that already exists in the original index when the data migration task starts to be executed), and then the target data can be gradually migrated to the target index according to the preset data volume (that is, the preset batch size of the migration data). The batch size can be set according to historical experience or network bandwidth, or it can be a user-defined preset data volume.
[0098] In this embodiment, by determining the response time of the data migration request; determining the target data in the original index based on the response time; and migrating the target data to the target index according to the preset data volume, it is avoided that duplicate data is added to the target index during the replay of the operation log, reducing redundant data and improving the speed of index reconstruction.
[0099] ES executes the reindex data migration task. The execution time of the task is related to the size of the index data. The larger the number of documents contained in the index, the longer the task execution time. At this time, the original index in the ES cluster can still be read and written normally. In a feasible implementation method, when migrating data to the target index, in order to further improve the speed of data migration, the batch size of the migrated data is adjusted according to the current system performance (such as thousands to tens of thousands of records are commonly used). Too large a batch may cause memory overflow, and too small a batch may not fully utilize the network bandwidth and I / O performance. Therefore, when migrating data, the performance of the current system is tested based on the actual migration time of the batch size data, and the preset batch size is adjusted to increase the speed of data migration.
[0100] In a feasible implementation manner, when all the operation logs are replayed, migrating the index alias to the target index includes:
[0101] When all the operation logs are replayed, the corresponding relationship between the index alias and the original index is deleted; and the corresponding relationship between the index alias and the target index is established so that the index alias routes the index operation request to the target index;
[0102] The deleting the original index and responding to the index operation request through the target index includes:
[0103] Delete the original index; unblock the index operation request; route the index operation request to the target index according to the index alias; and execute the index operation corresponding to the index operation request in response to the index operation request through the target index.
[0104] Specifically, once all operation logs have been replayed, the correspondence between the index alias and the original index is deleted, and a correspondence between the index alias and the target index is established. This allows the index alias to route index operation requests originally intended for the original index to the target index, which then performs the index operation. After the index alias is migrated to the target index, the original index is deleted, unblocking index operation requests. Unexecuted index operation requests are now routed to the target index by the modified index alias. The target index then responds to new index operation requests and performs the corresponding index operation.
[0105] In this embodiment, the switching between the original index and the target index is achieved by modifying the correspondence between the index aliases, and an index alias migration task is added to the index reconstruction task. There is no need to rename the target index or the original index, so that the index reconstruction is performed without the user's perception, thereby improving the user experience.
[0106] In a feasible implementation manner, when the number of unreplayed operation logs is less than a preset threshold, blocking the index operation request of the original index includes:
[0107] Monitoring whether the number of unreplayed operation logs is less than the preset threshold;
[0108] When the number of unreplayed operation logs is less than a preset threshold, the newly added index operation request of the original index is stored in a waiting-for-response queue, and the original index is adjusted to a read-only state.
[0109] Specifically, if the number of unreplayed operation logs detected is less than a preset threshold, newly added index operation requests for the original index are stored in the pending response queue and the original index is adjusted to read-only status. This allows index operation requests in the pending response queue to be responded to sequentially after the target index is converted. Meanwhile, the original index is adjusted to read-only status, and no new index operations are executed.
[0110] In this embodiment, by monitoring the number of unreplayed operation logs, the index operation request of the original index can be blocked in time to avoid the problem that the original index has been performing index operations and the generated operation logs cannot be replayed for a long time, thereby improving the efficiency of index reconstruction.
[0111] In one embodiment of the present application, referring to Figure 2 , Figure 2 This is a partial logic diagram of the index reconstruction method described in one embodiment of the present application. Figure 2 As shown in the figure, the index reconstruction process is further explained by taking the reconstruction of the original index into the target index with increased number of shards as an example:
[0112] In this scenario, the ES cluster is responsible for full-text search of a large amount of IT hardware device performance log data. The index name is device_performance_log, with an alias of devicePerformanceLog. Due to historical reasons, the number of shards is limited to two. As the business expands and the amount of data increases, the size of each shard of this index exceeds 100GB, significantly impacting index performance. Consider rebuilding the index to adjust the number of shards and optimize performance.
[0113] First, create the device_performance_log_new index and adjust its shard count (number_of_shards) to 8, quadrupling the original number. Other index configurations remain unchanged. Then, call the reindex API. During the reindex process, the original index continues to be used normally using the alias devicePerformanceLog. After the reindex is complete, the alias is migrated to the new index device_performance_log_new. The index is rebuilt without user awareness. The size of each shard in the new index is reduced to 25GB, and performance returns to normal.
[0114] The execution process of index reconstruction is as follows: 1) Trigger the index operation monitoring task of the original index device_performance_log. This task records the index operation as a log and sends it to the Master node. The Master node receives the log and records it in the device_performance_log_reindex_log log file; 2) Migrate the original index data to the device_performance_log_new index; 3) The device_performance_log_new index reads the device_performance_log_reindex_log log file and replays its index operation log; 4) When the log replay is completed, close the index operation log monitoring task and temporarily block the index operation; 5) Perform index alias migration. After the migration is completed, the index operation blockage is released, the old index is deleted, and the index reconstruction is completed.
[0115] Through this embodiment, the read and write requests of the original index are not restricted during the index reconstruction process, thereby improving the availability of the service; based on the upgrade and optimization of ES itself, there is no need to introduce third-party components, thereby ensuring the independence and portability of the service; based on the optimization of the original reindex interface (adding hooks), the user-side imperceptible and smooth migration is achieved, providing good usability; the original ES reindex function is further expanded, and the alias migration function is added, which reduces the operation steps of index reconstruction, improves the efficiency of index reconstruction, and simplifies user operations.
[0116] Reference Figure 3 , Figure 3 Schematic diagram of the framework of the index reconstruction system in one embodiment of the present application. Figure 3 As shown, based on the same or similar technical concepts as the above embodiments, in one embodiment of the present application, an index reconstruction system is proposed. The index reconstruction system 300 includes:
[0117] A new module 301 is created, for creating a target index according to an index rebuild request;
[0118] Receiving module 302, configured to receive a data migration request, the data migration request including an identifier of an original index, an identifier of a target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index;
[0119] A monitoring module 303 is configured to monitor operations of the original index in response to the data migration request and record an operation log of the original index according to the identifier of the original index;
[0120] The data migration module 304 is configured to migrate target data in the original index to the target index; the target data is all data stored in the original index before the operation of the original index is monitored;
[0121] A replay module 305 is configured to replay the operation log to the target index when the target data migration is completed;
[0122] The blocking module 306 is configured to block the index operation request of the original index when the remaining number of the unreplayed operation logs is less than a preset threshold;
[0123] The alias migration module 307 is used to migrate the index alias to the target index when the operation log replay is completed;
[0124] The deletion module 308 is configured to delete the original index and respond to the index operation request through the target index.
[0125] In this embodiment, a target index is established according to an index reconstruction request task; a data migration request is received, the data migration request including an identifier of the original index, an identifier of the target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index; in response to the data migration request, the operation of the original index is monitored according to the identifier of the original index, and the operation log of the original index is recorded; the target data in the original index is migrated to the target index; the target data is all the data stored in the original index before the operation of the original index is monitored; when the target data migration is completed, the operation log is replayed to the target index; when the number of unreplayed operation logs is less than a preset threshold, the index operation request of the original index is blocked; when all the operation logs are replayed, the index alias is migrated to the target index; the original index is deleted, and the index operation request is responded to through the target index. During the target index data migration process, the original index can still execute new index operations, improving the availability of the ES service. By monitoring the index operations of the original index during the target index data migration process, the new index operations of the original index are replayed to the target index after the target index data migration is completed. After the new and old indexes reach a dynamic data balance, the new and old indexes are switched using the index alias. This ensures data consistency during the index rebuild process, guarantees the availability of the ES service, and improves the efficiency of ES index rebuilds.
[0126] In a feasible implementation manner, the monitoring module includes:
[0127] a monitoring startup module, configured to, in response to the data migration request, start a monitoring task for the original index according to the identifier of the original index; the monitoring task is pre-created according to the identifier of the original index through a hook function encapsulated in the index operation interface when the data migration request is detected;
[0128] The transmission module is used to record the operation log of the original index in chronological order and transmit the operation log to the master node.
[0129] In a feasible implementation manner, the playback module includes:
[0130] A log acquisition module, configured to acquire the operation log stored in the master node;
[0131] The operation execution module is used to replay the index operations performed by the original index during the migration of the target data to the target index one by one in chronological order according to the operation log.
[0132] In a feasible implementation, the data migration module includes:
[0133] a first determining module, configured to determine a response time for the data migration request;
[0134] A second determining module, configured to determine the target data in the original index according to the response time;
[0135] The batch migration module is used to migrate the target data to the target index according to a preset data volume.
[0136] In a feasible implementation, the newly created module includes:
[0137] A parsing module, configured to parse the index rebuild request task and obtain configuration information of the target index; the configuration information includes the number of shards of the target index, the mapping attributes of the index, and the attribute word segmenter;
[0138] The interface calling module is used to call the index operation interface and create the target index according to the configuration information.
[0139] In this application, the description of the system embodiment is relatively short. For details, please refer to the above method embodiment, which will not be repeated here.
[0140] In one embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned index reconstruction method when executing the computer program.
[0141] In one embodiment of the present application, a non-volatile storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the index reconstruction method as described in the above embodiment is implemented.
[0142] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0143] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0144] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0145] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.
[0147] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0148] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.
[0149] The above is a detailed introduction to the provided index reconstruction method, system, electronic device and medium. Specific examples are used in this article to illustrate the principles and embodiments of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific embodiments and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. An index reconstruction method, characterized in that: The method comprises: Create the target index based on the index rebuild request task; Receive a data migration request, the data migration request including an identifier of an original index, an identifier of a target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index; In response to the data migration request, monitoring operations of the original index according to the identifier of the original index, and recording an operation log of the original index; Migrating target data in the original index to the target index; the target data is all data stored in the original index before the operation of the original index is monitored; When the target data migration is completed, replaying the operation log to the target index; When the number of unreplayed operation logs is less than a preset threshold, blocking the index operation request of the original index; When all the operation logs are replayed, the index alias is migrated to the target index; The original index is deleted, and the index operation request is responded to through the target index.
2. The method according to claim 1, characterized in that In response to the data migration request, monitoring operations of the original index according to the identifier of the original index, and recording an operation log of the original index; In response to the data migration request, starting a monitoring task for the original index according to the identifier of the original index; the monitoring task is pre-created according to the identifier of the original index through a hook function encapsulated in the index operation interface when the data migration request is detected; The operation log of the original index is recorded in chronological order and transmitted to the master node.
3. The method according to claim 2, characterized in that When the target data migration is completed, replaying the operation log to the target index includes: Obtaining the operation log stored in the master node; According to the operation log, the index operations performed by the original index during the migration of the target data are replayed to the target index one by one in chronological order.
4. The method according to claim 1, wherein Migrating the target data in the original index to the target index includes: determining a response time for the data migration request; Determining the target data in the original index according to the response time; Migrate the target data to the target index according to the preset data volume.
5. The method according to claim 1, wherein The step of establishing a target index according to the index reconstruction request task includes: Parse the index rebuild request task to obtain configuration information of the target index; the configuration information includes the number of shards of the target index, the mapping attributes of the index, and the attribute word segmenter; Call the index operation interface and create the target index according to the configuration information.
6. The method according to claim 1, characterized in that When all the operation logs are replayed, migrating the index alias to the target index includes: When all the operation logs are replayed, the corresponding relationship between the index alias and the original index is deleted; Establishing a correspondence between the index alias and the target index so that the index alias routes the index operation request to the target index; The deleting the original index and responding to the index operation request through the target index includes: Deleting the original index; Unblocking the index operation request; Routing the index operation request to the target index according to the index alias; In response to the index operation request, an index operation corresponding to the index operation request is performed using the target index.
7. The method according to claim 1, characterized in that When the number of unreplayed operation logs is less than a preset threshold, blocking the index operation request of the original index includes: Monitoring whether the number of unreplayed operation logs is less than the preset threshold; When the number of unreplayed operation logs is less than a preset threshold, the newly added index operation request of the original index is stored in a waiting-for-response queue, and the original index is adjusted to a read-only state.
8. An index reconstruction system, characterized in that: The system comprises: Create a new module to build the target index based on the index rebuild request; A receiving module, configured to receive a data migration request, wherein the data migration request includes an identifier of an original index, an identifier of a target index, and an index alias corresponding to the original index; the index alias is used to route the index operation request to the corresponding index; a monitoring module, configured to, in response to the data migration request, monitor operations of the original index according to the identifier of the original index, and record an operation log of the original index; A data migration module, configured to migrate target data in the original index to the target index; the target data is all data stored in the original index before the operation of the original index is monitored; A replay module, configured to replay the operation log to the target index when the target data migration is completed; A blocking module, configured to block the index operation request of the original index when the remaining number of unreplayed operation logs is less than a preset threshold; an alias migration module, configured to migrate the index alias to the target index when the operation log replay is completed; The deletion module is used to delete the original index and respond to the index operation request through the target index.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the index reconstruction method according to any one of claims 1 to 7 is implemented.
10. A non-volatile storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the index reconstruction method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Index reconstruction method and device, equipment, computer readable storage medium and product
CN112835980A
Technique for improving operations log indexing
US20220253243A1