Index capacity expansion method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to the technical field of data storage, in particular to an index expansion method and device, electronic equipment, a storage medium and a program product.The index expansion method comprises the steps that whether index expansion is triggered or not is determined based on resource indexes of an elastic retrieval ES cluster; when the index expansion is triggered, automatically creating a new index based on a preset configuration rule; writing all newly generated data documents into an old index and a new index of the ES cluster at the same time; obtaining cold and hot scores of all historical data documents in the old index; based on the cold and hot scores of all the historical data documents, determining migration priorities of all the historical data documents; and writing each historical data document into a new index based on the migration priority of each historical data document. According to the method and the device, stable read-write performance of the ES cluster is guaranteed while capacity expansion is carried out, the influence of capacity expansion on business services is avoided, and the data storage and retrieval efficiency of the new index can be optimized through hierarchical migration of cold and hot data.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data storage, and particularly relates to an index capacity expansion method and device, electronic equipment, storage medium and program product. BACKGROUND

[0002] In the field of big data storage, as a popular full-text search engine, Elasticsearch (ES) is widely used for mass data storage and fast retrieval. The data storage of ES is in units of indexes, and the number of index shards is fixed when the index is created.

[0003] With the rapid development of business, the capacity of a single index is easy to reach the upper limit, and the traditional Elasticsearch cannot be dynamically expanded, which affects the performance of the system, such as slow writing speed and low query efficiency, and thus cannot meet the needs of large-scale data storage and fast retrieval. SUMMARY

[0004] In view of the above technical problems, the present disclosure provides an index capacity expansion method and device, electronic equipment, storage medium and program product.

[0005] In a first aspect, the present disclosure provides an index capacity expansion method based on Elasticsearch, comprising: determining whether to trigger index capacity expansion based on resource indicators of an Elasticsearch (ES) cluster; automatically creating a new index based on a preset configuration rule when the index capacity expansion is triggered; writing all newly generated data documents to an old index and a new index of the ES cluster at the same time; obtaining cold and hot scores of all historical data documents in the old index; determining migration priorities of the historical data documents based on the cold and hot scores of all the historical data documents; writing the historical data documents to the new index based on the migration priorities of the historical data documents.

[0006] In addition, according to the method of the first aspect of the present disclosure, the resource indicators include disk usage and CPU load.

[0007] In addition, according to the method of the first aspect of the present disclosure, determining whether to trigger index capacity expansion based on resource indicators of an Elasticsearch (ES) cluster comprises: determining an average load of the resource indicators based on the resource indicators of the ES cluster; triggering the index capacity expansion when the average load exceeds a trigger threshold.

[0008] In addition, according to the method of the first aspect of the present disclosure, the trigger threshold is calculated by the following formula:

[0009] wherein, is a trigger threshold, S is a current load of the resource indicator, is a load prediction increment.

[0010] Further, according to the method of the first aspect of the present disclosure, the configuration rule comprises: a number of shard groups, a routing key, and a weight of each shard group, wherein each shard group contains a plurality of shards.

[0011] Further, according to the method of the first aspect of the present disclosure, the weight of the shard group is dynamically updated based on CPU utilization, disk I / O load, and hotness score of the shard group.

[0012] Further, according to the method of the first aspect of the present disclosure, the hotness score of the shard group is determined based on access frequency and data document size of each data document in the shard group.

[0013] Further, according to the method of the first aspect of the present disclosure, the hotness score of all historical data documents in the old index is obtained, comprising: obtaining the access frequency of each historical data document in the old index; obtaining the hotness score of all historical data documents based on the access frequency and data size of each historical data document in the old index.

[0014] Further, according to the method of the first aspect of the present disclosure, each historical data document is written into the new index based on the migration priority of each historical data document, comprising: writing each historical data document into the new index in descending order of migration priority based on the migration priority of each historical data document.

[0015] Further, according to the method of the first aspect of the present disclosure, the method further comprises: during the process of writing the historical data document into the new index, writing each historical data document into a target shard in the new index based on the routing key and the weight of each shard group.

[0016] Further, according to the method of the first aspect of the present disclosure, the method further comprises: before performing the write operation, determining whether each historical data document and newly generated data document has an updated version in the new index based on the optimistic lock mechanism.

[0017] Further, according to the method of the first aspect of the present disclosure, the method further comprises: when the migration progress of the historical data document exceeds a first preset threshold and the new index health check passes, switching 5% of the read traffic data in the ES cluster to the new index.

[0018] Further, according to the method of the first aspect of the present disclosure, the method further comprises: After 5% of the read traffic data in the ES cluster is switched to the new index, the query delay and error rate of the new index are monitored, and if everything is normal, the read traffic data switched to the new index is gradually increased at a preset ratio. After each increase, the query delay and error rate of the new index are monitored for a preset time, and after everything is normal, the increase operation is performed again.

[0019] In addition, according to the method of the first aspect of the present disclosure, when the migration progress of the historical data documents exceeds the second preset threshold, all read traffic data in the ES cluster is switched to the new index.

[0020] In addition, according to the method of the first aspect of the present disclosure, the method further comprises: In the process of writing the historical data documents into the new index, the proxy server in the ES cluster performing the writing of the historical data documents into the new index is determined based on a distributed lock.

[0021] In addition, according to the method of the first aspect of the present disclosure, the method further comprises: In the process of writing each historical data document and newly generated data document into the new index, the writing state of each historical data document and newly generated data document is recorded through a task table; When the writing of a certain historical data document or newly generated data document abnormally terminates, Based on the writing state of each historical data document or newly generated data document recorded in the task table, the breakpoint of the writing is determined; Based on the breakpoint, the writing of the historical data document or newly generated data document is restarted.

[0022] The second aspect of the present disclosure provides an index expansion device based on elastic search, comprising: A first determination unit configured to determine whether to trigger index expansion based on resource indicators of an elastic search ES cluster; A creation unit configured to automatically create a new index based on a preset configuration rule when index expansion is triggered; A first writing unit configured to simultaneously write all newly generated data documents to an old index and a new index of the ES cluster; An acquisition unit configured to acquire the hot and cold scores of all historical data documents in the old index; A second determination unit configured to determine the migration priority of each historical data document based on the hot and cold scores of all historical data documents; A second writing unit configured to write each historical data document to the new index based on the migration priority of each historical data document.

[0023] In addition, according to the device of the first aspect of the present disclosure, the resource indicators include disk usage and CPU load.

[0024] Furthermore, according to the apparatus of the first aspect of this disclosure, the first determining unit is further configured as follows: Based on the resource metrics of the ES cluster, determine the average load of the resource metrics. When the average load exceeds the trigger threshold, index expansion is triggered.

[0025] Furthermore, according to the apparatus of the first aspect of this disclosure, the trigger threshold is calculated by the following formula:

[0026] in, The threshold for triggering the event is S, where S is the current load of the resource metric. Predict the incremental load.

[0027] Furthermore, according to the apparatus of the first aspect of this disclosure, the configuration rules include: the number of shard groups, the routing key, and the weight of each shard group, wherein each shard group contains a plurality of shards.

[0028] Furthermore, according to the apparatus of the first aspect of this disclosure, the weight of the shard group is determined based on the CPU utilization, disk I / O load, and hot / cold rating of the shard group.

[0029] Furthermore, according to the apparatus of the first aspect of this disclosure, the hot / cold rating of the shard group is determined based on the access frequency and data document size of each data document in the shard group.

[0030] Thirdly, this disclosure provides an electronic device, including: a memory for storing computer-readable instructions; and a processor for executing the computer-readable instructions, causing the electronic device to perform the method as described in any embodiment of the first aspect.

[0031] Fourthly, this disclosure provides a non-transitory computer-readable storage medium for storing computer-readable instructions that, when executed by a processor, cause the processor to perform the method as described in any embodiment of the first aspect.

[0032] Fifthly, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any embodiment of the first aspect.

[0033] This disclosure provides an index expansion method, apparatus, electronic device, storage medium, and program product. By monitoring the resource indicators of the Elasticsearch (ES) cluster, this disclosure determines whether index expansion is needed. When index expansion is triggered, a new index is automatically created according to preset configuration rules. By writing new data to both the old and new indexes simultaneously and determining the migration method of historical data documents based on hot / cold data scoring, this disclosure can ensure the complete retention and normal writing of new data during the expansion process, and can also quickly migrate historical data documents from the old index to the new index based on the hot / cold data scoring. This ensures the stability of the ES cluster's read and write performance during expansion, avoids the impact of expansion on business services, and optimizes the data storage and retrieval efficiency of the new index through hierarchical migration of hot and cold data, reducing manual intervention.

[0034] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0035] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0036] Figure 1 A flowchart illustrating an index expansion method based on elastic retrieval provided in this embodiment of the disclosure; Figure 2 A flowchart illustrating another index expansion method based on elastic retrieval provided in this disclosure embodiment; Figure 3 This is a schematic diagram of the architecture of an index expansion system provided in an embodiment of the present disclosure; Figure 4 A structural block diagram of an index expansion device based on elastic retrieval provided in an embodiment of this disclosure; Figure 5 A hardware block diagram of an electronic device provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of a computer-readable storage medium provided in an embodiment of this disclosure. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0038] In the field of big data storage, Elasticsearch, as a popular full-text search engine, is widely used for storing and quickly retrieving massive amounts of data. Its data storage is based on indexes, and the number of index shards is fixed when an index is created. With rapid business growth, the capacity of a single index can easily reach its limit, and traditional Elasticsearch cannot dynamically scale, impacting system performance such as slower write speeds and reduced query efficiency, thus failing to meet the demands of large-scale data storage and fast retrieval. Existing scaling solutions typically employ the following methods: 1. Reindexing: Using Elasticsearch's `_reindex` API, migrate all data from the old index to the new index. The new index requires pre-defined shard numbers and routing rules. If the sharding strategy is adjusted (e.g., adding shards), a custom script is needed to handle data distribution. During the migration, writes must be paused or a dual-write mechanism must be introduced to ensure data consistency.

[0039] 2. Alias ​​Switching: Use Elasticsearch's alias feature to bind logical aliases to physical indexes. Create a new index during scaling, and after migration is complete, switch the alias to point to the new index via the aliases API.

[0040] 3. Hot-Warm Architecture: Using Elasticsearch's ILM strategy, data is divided into hot data and cold data according to dimensions such as access frequency and time. Hot data is stored in high-performance hot storage media to meet the needs of fast querying; cold data is migrated to low-cost cold storage media to save storage costs.

[0041] The aforementioned existing capacity expansion solutions have the following technical problems: 1. High operational complexity: Traditional index expansion solutions (such as rebuilding indexes and switching aliases) rely on manual operations, such as manually creating new indexes, configuring dual-write logic, and switching traffic. The operation steps are cumbersome and prone to data inconsistency or service interruption due to errors. Solutions such as hot / cold data separation require pre-planning of hardware resources, and the operations team needs to frequently adjust data distribution strategies, significantly increasing maintenance costs.

[0042] 2. Low resource utilization: Static sharding design leads to rigid resource allocation: If the initial number of shards is too small, performance bottlenecks are likely to occur as data grows; if too many shards are pre-allocated, long-term idle shards will occupy a large amount of memory and file handles, resulting in resource waste; and full data migration (such as reindex) requires a large amount of I / O and computing resources, which may affect the stability of online services and increase the risk of downtime.

[0043] 3. Uneven data distribution: Traditional hash algorithms (such as modulo or consistent hashing) cannot dynamically perceive the shard load status, resulting in data skew: High-frequency access data is concentrated in a small number of shards, forming "hot shards", which causes performance bottlenecks (such as a shard with a CPU utilization of 90%, while other shards have a utilization of less than 20%). After expansion, the load difference between the old and new shards is significant, and additional manual intervention is still required to adjust the data distribution.

[0044] To address the aforementioned technical issues, this disclosure provides an index expansion method based on elastic retrieval. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating an index expansion method based on flexible retrieval provided in an embodiment of this disclosure. Figure 1 As shown, the method includes the following steps: Step 101: Based on the resource metrics of the Elastic Search ES cluster, determine whether to trigger index expansion.

[0045] In one embodiment of this disclosure, resource metrics include disk utilization and CPU load. In another embodiment of this disclosure, determining whether to trigger index resizing based on Elasticsearch cluster resource metrics includes: Based on the resource metrics of the ES cluster, determine the average load of the resource metrics. When the average load exceeds the trigger threshold, index expansion is triggered.

[0046] In one embodiment of this disclosure, the trigger threshold is a pre-set critical value used to determine whether the Elasticsearch (ES) cluster needs index expansion. It serves as a warning value for the ES cluster's resource load, quantifying whether the cluster's resource pressure has reached a level requiring expansion. By monitoring ES cluster resource metrics (such as disk utilization and CPU load) in real-time or periodically, a sliding window algorithm (window size T = 10 minutes) is used to calculate the average load of these resource metrics. The average load is calculated as follows:

[0047] When the average load of any resource metric exceeds the trigger threshold, the index expansion task is initiated, or the index expansion task can be manually triggered by the administrator through the interface.

[0048] In one embodiment of this disclosure, the trigger threshold is calculated using the following formula:

[0049] in, The threshold for triggering the event is S, where S is the current load of the resource metric. Predict the incremental load.

[0050] The acquisition of load prediction increments typically relies on time-series prediction algorithms.

[0051] First, by continuously collecting time-series data of historical load metrics (such as CPU and disk I / O), a data foundation for the prediction algorithm is established. Then, based on the actual business situation, the prediction algorithm or machine learning is used to analyze the historical load data, thereby making a quantitative prediction of load growth in the short term and deriving the corresponding expected load increment. For example, the calculation process of the predicted load increment is as follows: Assuming that it is currently 9:00 AM, the prediction model will predict the expected load increment for the next 10 minutes (9:00 AM-9:10 AM) based on the load data of the same time period (8:50 AM-9:00 AM) in the past 7 days and the recent overall growth trend.

[0052] Step 102: When index expansion is triggered, a new index is automatically created based on the preset configuration rules.

[0053] In one embodiment of this disclosure, the configuration rules include the number of shard groups, routing keys, weights of each shard group, and hot / cold data migration strategies. Each shard group contains several shards. For example, an administrator sets basic rules through a configuration interface or API, such as: number of shard groups 3, routing key = user ID. Based on the above configuration, physical resources are automatically created or allocated. For example, several physical shards are allocated to each shard group (Group 0, 1, 2), and their network access addresses are recorded. The aforementioned shard group weights are used to intelligently allocate data to the optimal shard group, solving the static allocation problem of traditional hash modulo methods. The shard group weights are dynamically updated based on the shard group's CPU utilization, disk I / O load, and hot / cold data rating.

[0054] Specifically, during the index expansion process, the weight P of each shard group is calculated based on its real-time load (CPU, disk I / O) and hot / cold rating. The formula is as follows:

[0055] CPU g DiskIO is used to calculate the CPU utilization of the fragment group (normalized to 0~1). g ColdHotScore represents the disk I / O load of the shard group (normalized to 0~1). g A hot / cold score is assigned to each data document group (calculated based on the access frequency of data documents within the group). For each data document, a hot / cold score is calculated based on its access frequency (e.g., access count in the last 24 hours).

[0056] The hot / cold rating of a shard group is a weighted average of the hot / cold ratings of all the data documents it contains.

[0057] The weight parameters (α, β, γ) mentioned above are determined by the administrator based on business experience. For example, in high-concurrency write scenarios, more attention may be paid to CPU and disk I / O, and α=0.5, β=0.4, γ=0.1 can be set; in scenarios with more reads and fewer writes, and clear distinction between hot and cold data, more attention may be paid to the effect of hot and cold data separation, and α=0.3, β=0.3, γ=0.4 can be set.

[0058] During the index expansion method, optimization can be performed based on actual results. For example, A / B testing can be conducted, using two sets of different parameter configurations during periods of low traffic to compare and observe which set of parameters results in lower overall system latency and more balanced load. Alternatively, a simple feedback mechanism can be set up, such as automatically and slightly increasing the value of β if the system frequently triggers alarms due to disk I / O bottlenecks, making the system more "averse" to high I / O load shards during routing.

[0059] In one embodiment of this disclosure, the configuration rules further include a hot and cold data scoring strategy, which includes the following strategy dimensions: ① Hot / Cold Data Criteria: This is used to define the quantitative conditions for classifying a data document as "hot data," such as an access frequency of >100 times / day and a data size of <10KB.

[0060] ② Target storage medium: This specifies the type of hardware to store hot and cold data in order to balance performance and cost, such as storing hot data in SSDs and cold data in HDDs.

[0061] ③ Migration trigger timing: This specifies when to initiate cold and hot data migration (or downgrade), such as performing cold and hot data reassessment and migration at 2:00 AM every day.

[0062] ④ Impact of routing weights: This specifies how hot and cold attributes affect the weight calculation of shard groups (i.e., the role of the γ parameter). For example, shard groups with hot data aggregation have higher cold and hot scores (ColdHotScore).

[0063] For example, a complete hot / cold policy could be: "If a document has been accessed more than 50 times in the past 24 hours, it is marked as 'hot data' and is preferentially stored in the SSD storage shard group; it is re-evaluated every 4 hours, and if it fails to meet the standard for 3 consecutive times, it is automatically migrated to the HDD storage shard group."

[0064] For example, the configuration rules are constructed in the following way: ① Static Configuration and Initialization: Administrators set basic rules through the configuration interface or API, such as: number of shard groups = 3, routing key = user ID. Based on the above configuration, the system will automatically create or allocate physical resources. For example, it will allocate several physical shards to each shard group (Group 0, 1, 2) and record their network access addresses. At this time, a static initial mapping relationship without weight information is generated, that is, one shard group corresponds to several shards.

[0065] ② Dynamic Calculation and Real-time Updates: After the expansion begins, the monitoring service continuously collects real-time performance metrics for each shard group, such as CPU utilization, disk I / O throughput / latency, etc. The hot / cold strategy engine analyzes the access patterns of data documents based on predefined hot / cold data scoring strategies (such as "accesses >100 in the past hour are hot data"), and calculates the hot / cold score of the data documents carried by each shard group. The weight calculation engine collects real-time performance metrics for each shard group and calculates the weight of each shard group (the weight of the shard group is updated in real time).

[0066] ③ Integrating the static skeleton from ① and the dynamic weights from ② results in a dynamically updated configuration principle. Its core fields include: shard group ID, shard group weight, the list of shards corresponding to each shard group, and the access address.

[0067] For example, the administrator creates a metadata table (index_extend_config) via API, defines the initial configuration of shard groups (shard group ID, list of shards corresponding to each shard group, hot and cold data scoring strategy); then loads the dynamic weighted sharding algorithm parameters (α, β, γ), generates the initial weight values ​​of shard groups (the weight of new shard groups is set to the highest by default); finally, initializes the routing rule table, and establishes the mapping relationship between shard groups and physical shards.

[0068] For example, the administrator defines the number of shard groups (e.g., 3 groups), the routing key (e.g., user_id), sets the weight parameters (α=0.6, β=0.3, γ=0.1), and finally outputs the shard group mapping rule table and the initial weight values.

[0069] Step 103: Write all newly generated data documents to both the old and new indexes of the ES cluster simultaneously.

[0070] In one embodiment of this disclosure, during the index expansion process, all newly generated data documents in the ES cluster are simultaneously written to both the old and new indexes of the ES cluster, and the document ID, version number, timestamp, etc. of the newly generated data documents are recorded in the transaction log. This dual-write mechanism ensures that all newly generated data documents are backed up in both the old and new indexes during the migration, preventing data loss and ensuring uninterrupted service. Step 104: Obtain the hot / cold rating of all historical data documents in the old index.

[0071] In one embodiment of this disclosure, obtaining the hot / cold rating of all historical data documents in the old index includes: Obtain the access frequency of each historical data document in the old index; Based on the access frequency and data size of each historical data document in the old index, obtain the hot / cold rating of all historical data documents.

[0072] Specifically, by statistically analyzing the access frequency and data size of all historical data documents in the old index, and based on the hot / cold scoring formula described in step 102, a hot / cold score is calculated for all historical data documents. The access frequency of all historical data documents can be statistically analyzed in conjunction with historical logs.

[0073] For example, by statistically analyzing the real-time write and read requests of old index data, the access pressure and patterns of the business on the old index can be perceived in real time, and "which data is hot" can be determined. This provides a real-time data source for obtaining hot and cold scores through historical logs, and helps to calibrate the hot and cold scores.

[0074] In one embodiment of this disclosure, the newly generated data documents written at the same time are also monitored during the process of counting real-time write and read requests of old index data, and are analyzed by obtaining hot and cold ratings through historical logs, and are included in the overall hot and cold rating system evaluation.

[0075] Step 105: Determine the migration priority of each historical data document based on its hot / cold rating.

[0076] In one embodiment of this disclosure, the migration priority of each historical data document is determined by statistically analyzing the hot and cold scores of all historical data documents. The higher the hot and cold score, the higher the migration priority, and the lower the hot and cold score, the lower the migration priority.

[0077] For example, the above-mentioned "hot and cold data scoring strategy" can also be used to determine whether historical data documents are hot or cold data. The above-mentioned "hot and cold data scoring strategy" defines the quantitative conditions for hot data.

[0078] For example, the above migration priority can also be calculated in the following way:

[0079] Step 106: Based on the migration priority of each historical data document, write each historical data document into the new index.

[0080] In one embodiment of this disclosure, writing each historical data document to a new index based on the migration priority of each historical data document includes: Based on the migration priority of each historical data document, the historical data documents are written into the new index in descending order of migration priority.

[0081] Specifically, based on the migration priority of each historical data document, the historical data documents are sorted from high to low to form a migration task queue. The historical data documents are then written to the new index according to the order of the historical data documents in the migration task queue. "Hot data" is written to the new documents first, while low-priority cold data is migrated asynchronously in a low-speed background thread.

[0082] Furthermore, index expansion methods based on elastic retrieval also include: During the process of writing historical data documents into the new index, the target shards in the new index are written to each historical data document based on the routing key and the weight of each shard group.

[0083] Specifically, combining the aforementioned "hot and cold data scoring strategy," all historical data documents are divided into hot data and cold data. Based on the weights of the shard groups, frequently accessed "hot data" is allocated to high-weight shard groups (high-performance storage), while infrequently accessed "cold data" is allocated to low-weight shard groups (low-cost storage). Then, a deterministic hash algorithm (such as hashing the routing key user_id and taking the modulo operation) is used to route historical data documents already written to a shard group to a specific shard within that shard group (i.e., target allocation). This ensures that data for the same user is always in the same shard group, which is crucial for accurate query location and avoiding cross-group queries. Simultaneously, it achieves load balancing, avoids the skew problem of traditional hash algorithms, and reduces the possibility of hotspot allocation.

[0084] In summary, according to the technical solution provided in the embodiments of this disclosure, this disclosure determines whether index expansion is needed by monitoring the resource indicators of the ES cluster. When index expansion is triggered, a new index is automatically created according to preset configuration rules. By writing new data to both the old and new indexes simultaneously and determining the migration method of historical data documents based on hot and cold data scoring, it can ensure the complete retention and normal writing of new data during the expansion process, and can also quickly migrate historical data documents in the old index to the new index based on the hot and cold data scoring. This ensures the stability of the read and write performance of the ES cluster while expanding, avoids the impact of expansion on business services, and optimizes the data storage and retrieval efficiency of the new index through the hierarchical migration of hot and cold data, reducing manual intervention.

[0085] By writing data documents to the new index using the above method, even with a large number of newly generated data documents, no data will be lost due to the dual-write mechanism. Furthermore, because hot data is migrated first, most query traffic can be quickly switched to the new cluster shard group with lower load, rapidly improving the overall system performance. Meanwhile, cold data is migrated "silently" in the background, with minimal preemption of critical business resources (CPU, I / O).

[0086] Furthermore, the above-mentioned index expansion method based on elastic retrieval also includes: Before performing a write operation, an optimistic locking mechanism is used to determine whether each historical data document and the newly generated data document has an updated version in the new index.

[0087] Specifically, optimistic locking is a non-blocking concurrency control mechanism based on version number verification. Its core is to assign each data document an incrementing version number as it is updated. When historical data documents are migrated to a new index or new data documents (i.e., newly generated data documents) are double-written to a new index, Elasticsearch's Painless script first compares the version number of the data to be written with the current version number of the document in the new index. Only when the version number to be written is updated (without conflict) will the data document writing and version number synchronization be performed. If an updated version already exists in the new index, the writing will be abandoned.

[0088] In summary, based on the technical solutions provided in the embodiments of this disclosure, this disclosure uses an optimistic locking mechanism to ensure the atomicity and consistency of data during the writing of new and old data to a new index, avoiding the problem of old version overwriting or repeated writing, and adapting to high-concurrency scenarios of index expansion.

[0089] Furthermore, the above-mentioned index expansion method based on elastic retrieval also includes: When the migration progress of historical data documents exceeds the first preset threshold and the health check of the new index passes, 5% of the read traffic data in the ES cluster will be switched to the new index.

[0090] Specifically, migration progress refers to the proportion of historical data documents written to the new index relative to all historical data documents in the old index. The first preset threshold is the critical point value for the migration progress that triggers the first read traffic data switch. This first preset threshold can be limited according to actual business needs. Before data documents are written to the new index, that is, before the data document migration begins, 100% of the read traffic is directed to the old index. After the data document migration begins, the migration progress of historical data documents is monitored in real time. When the migration progress reaches the first preset threshold, and the service capability of the new index is verified to be healthy, 5% of the read traffic data in the ES cluster is switched to the new index.

[0091] Health checks may include, but are not limited to, shard status, performance metrics, and data consistency in the new index.

[0092] In summary, according to the technical solution provided in the embodiments of this disclosure, this disclosure sets a first preset threshold. When the migration progress of historical data documents exceeds the first preset threshold, it switches to a new index by reading data with a small volume of traffic to verify the availability of the new index and lays the foundation for subsequent full traffic migration.

[0093] Furthermore, the above-mentioned index expansion method based on elastic retrieval also includes: After switching 5% of the read traffic data in the ES cluster to the new index, monitor the query latency and error rate of the new index. If everything is normal, gradually increase the read traffic data switched to the new index according to the preset ratio. After each increase, monitor the query latency and error rate of the new index at the preset time. After everything is normal, then execute the increase operation.

[0094] Specifically, the preset ratio can be set according to actual business needs, such as 10%, 20%, etc., and this disclosure does not limit it. After switching 5% of the read traffic data in the ES cluster to the new index, monitor the query latency and error rate of the new index. If everything is normal, gradually increase the read traffic data switched to the new microcosm based on the preset ratio of 5% of the read traffic data. After each increase, monitor the query latency and error rate of the new index for a preset time. If everything is normal, then execute the increase operation. The "preset time" is set to observe the new index stably for a period of time and avoid misjudgment based on a single point of data.

[0095] In summary, according to the technical solution provided in the embodiments of this disclosure, this disclosure achieves seamless traffic migration for users and without service interruption by gradually increasing the traffic to switch to the new index in a proportional manner, and by detecting the new index each time a switch is made.

[0096] Furthermore, the above-mentioned index expansion method based on elastic retrieval also includes: When the migration progress of historical data documents exceeds the second preset threshold, all read traffic data in the ES cluster will be switched to the new index.

[0097] Specifically, the second preset threshold is the critical value for the migration progress that triggers a full read traffic switch. The second preset threshold is set to 95%. When the migration progress of historical data documents exceeds the second preset threshold, all read traffic data in the Elasticsearch cluster will be switched to the new index. The old index will only be used for disaster recovery and will no longer serve regular queries.

[0098] In summary, according to the technical solution provided in the embodiments of this disclosure, most of the historical data documents in the old index have been migrated to the new index. When the new index is verified to be usable through a health check, 5% of the read traffic is first tentatively switched to the new index. The stability and availability of the new index are verified through small traffic volumes, laying the groundwork for the subsequent full traffic switch. Then, the read traffic is migrated by gradually increasing the proportion of read traffic. When the migration progress of historical data documents exceeds the second preset threshold, the full read traffic switch is achieved, ultimately achieving the expansion goal of "unnoticeable and low risk".

[0099] Furthermore, the above-mentioned index expansion method based on elastic retrieval also includes: During the process of writing historical data documents to a new index, a proxy server in the Elasticsearch cluster is determined based on a distributed lock to execute the writing of historical data documents to the new index.

[0100] Specifically, the core function of distributed locks is to prevent conflicts caused by multiple proxy servers concurrently executing the same data write task. The RedLock algorithm allows multiple proxy servers in the Elasticsearch cluster to compete for lock resources. Only the proxy server that acquires the lock can execute the write operation for that batch of historical data documents. Other proxy servers must wait for the lock to be released or re-compete using an exponential backoff strategy (retry intervals of 1s, 2s, 4s, etc.). The lock timeout is set to 30 seconds to reduce lock contention. This mechanism ensures that the same batch of data is processed by only one proxy server, avoiding problems such as duplicate data writes, version conflicts, or fragmented write chaos.

[0101] For example, based on the concept of shared reads and unique writes, each thread (i.e., the proxy server) in the Elasticsearch cluster writes its timestamp, the key to be acquired, and its own unique identifier to its corresponding document (task table). Then, after a delay plus a random amount of time, it re-queries all threads competing for the lock based on the key, and checks if its own timestamp is the smallest, while also ensuring that its timestamp is within the timeout range provided by the user. If its timestamp is the smallest and within the timeout range, it means it has acquired the lock, and it continuously updates the heartbeat of its document during this period.

[0102] In summary, according to the technical solution provided in the embodiments of this disclosure, this disclosure avoids duplicate data writing caused by multiple proxy servers concurrently executing the same write task through distributed locks, ensuring that only one proxy server processes the migration task of a specific index at a unified time.

[0103] Figure 2 This is a flowchart illustrating another index expansion method based on elastic retrieval provided in this embodiment of the disclosure.

[0104] like Figure 2 As shown, the method further includes the following steps: Step 201: During the process of writing each historical data document and newly generated data document into the new index, the writing status of each historical data document and newly generated data document is recorded through the task table.

[0105] In one embodiment of this disclosure, during the process of writing each historical data document and newly generated data document into a new index, a task table is used to record the writing status of each historical data document and newly generated data document in real time.

[0106] Step 202: When the writing of a historical data document or a newly generated data document is abnormally terminated, the writing status of each historical data document or newly generated data document is recorded in the task table to determine the breakpoint of the writing.

[0107] In one embodiment of this disclosure, the breakpoint refers to the current migration progress position when a data document experiences an abnormal termination. During the process of writing each historical data document and newly generated data document into the new index, if the writing of a certain historical data document or newly generated data document experiences an abnormal termination, the breakpoint of the writing of the data document that experienced the abnormal termination is determined based on the writing status of each historical data document or newly generated data document recorded in the task table.

[0108] Step 203: Based on the breakpoint, restart and write the historical data document or the newly generated data document.

[0109] In one embodiment of this disclosure, at the breakpoint, the task of writing to the historical data document or the newly generated data document is restarted.

[0110] In summary, according to the technical solution provided in the embodiments of this disclosure, this disclosure, by using a task table to record the writing status of each document in real time during the process of writing historical data documents and newly generated data documents into a new index, and accurately locating the writing breakpoint based on the writing status when an abnormal termination occurs, and then restarting the writing task of the corresponding document from the breakpoint, can effectively avoid the waste of resources from full re-migration after an anomaly, ensure the continuity and integrity of data writing, eliminate the problem of data omission or duplicate writing, provide reliable fault-tolerant support for the data migration stage of index expansion, and achieve the goal of "zero downtime and low risk" dynamic index expansion.

[0111] Furthermore, the above-mentioned index expansion method based on elastic retrieval also includes: By encrypting the data transmitted during the expansion process (such as historical data documents and newly generated data documents) using the AES-256 algorithm, the security of data transmission is ensured.

[0112] Specifically, AES-256 is a mainstream high-strength symmetric encryption algorithm that can prevent data from being stolen or tampered with during transmission.

[0113] The keys required in the above encryption process are not stored in a fixed manner, but are dynamically generated and distributed through a key management system, which reduces the risk of key leakage and further improves the security of data transmission.

[0114] Reference Figure 3 Overview of application scenarios according to embodiments of this disclosure.

[0115] Figure 3 This is a schematic diagram of the architecture of an index expansion system provided in an embodiment of this disclosure. Figure 3 As shown, the index expansion system 300 includes at least: a dynamic sharding mapping configuration module 301, a resource awareness triggering module 302, a data cutover and migration module 303, an intelligent routing and consistency guarantee module 304, a distributed lock and fault tolerance module 305, and a security and auditing module 306.

[0116] The dynamic sharding mapping configuration module 301 is used to define index expansion rules through the metadata table (index_extend_config), including the number of shards, routing key, weight parameters (α, β, γ), and hot / cold scoring strategy. Its core function is to divide the index into multiple logical shard groups, each containing several physical shards, then dynamically generate shard groups, and calculate the data distribution target based on consistent hashing and real-time load feedback.

[0117] Shard group weighting is the core design of the dynamic sharding mapping configuration module, used to intelligently allocate data to the optimal shard group, solving the problem of static sharding based on traditional hash modulo. The dynamic weighted sharding algorithm primarily allocates frequently accessed "hot data" to high-weight shard groups (high-performance storage) and infrequently accessed "cold data" to low-weight shard groups (low-cost storage) based on the real-time load (CPU, disk I / O) and hot / cold data score (based on LRU-K predicted access frequency). The formula for calculating shard group weights is:

[0118] Where CPUg represents the current CPU utilization of the shard group (normalized to 0~1), DiskIOg represents the disk I / O load of the shard group (normalized to 0~1), and ColdHotScoreg represents the cold / hot score of the shard group (calculated based on the access frequency of data within the group). For each data document, a cold / hot score is calculated based on its access frequency (number of accesses in the last 24 hours).

[0119] The hot / cold rating of a segment group is a weighted average of the ratings of all the documents it contains:

[0120] The resource-aware triggering module 302 supports both automatic and manual triggering mechanisms to ensure the flexibility and controllability of expansion tasks. Automatic triggering monitors storage resources (such as disk utilization and query latency) using a sliding window algorithm, calculates the average load within the window period, and initiates the expansion task when the load exceeds a preset threshold, avoiding accidental triggering due to instantaneous fluctuations. It also incorporates a Bayesian optimization algorithm to predict resource demand and dynamically adjust the triggering strategy. Manual triggering is provided to administrators through a management interface or API, allowing them to proactively initiate expansion based on business needs. The triggering formula is expressed as:

[0121] The data cutover and migration module 303 is responsible for performing data migration and traffic switching. Its core mechanisms include dual-write transaction logs and priority-driven incremental migration. During the cutover, new data is simultaneously written to both the old and new indexes via a dual-write mechanism, and changes are recorded in the transaction log (transaction index) to ensure data integrity. Atomicity verification is achieved through optimistic locking (CAS) using Elasticsearch's Painless scripts. Priority-driven incremental migration primarily calculates migration priority based on hot / cold scoring (access frequency, data size), prioritizing the migration of high-frequency, smaller documents. The priority migration formula is:

[0122] The intelligent routing and consistency guarantee module 304 dynamically adjusts the routing strategy for read and write requests to ensure data consistency and low latency. Write routing calculates the target shard index based on a hash algorithm, while read routing prioritizes querying the old index during the initial cutover phase, switching to the new index after completion. Theoretically, combining Multi-Version Concurrency Control (MVCC) and version number verification mechanisms, this can be expressed as:

[0123] Security and Audit Module 306: This module encrypts data transmission using AES-256 and combines it with RBAC (Role-Based Access Control) to restrict operation permissions. Keys are dynamically distributed via KMS (Key Management System), and operation logs record the entire lifecycle of the expansion task, supporting auditing and anomaly backtracking. Encryption strength is verified through entropy calculation.

[0124] This disclosure also provides an index expansion device based on elastic retrieval. Figure 4 A structural block diagram of an index expansion device based on flexible retrieval provided in this disclosure embodiment is shown below. Figure 4 As shown, the index expansion device 400 based on elastic retrieval includes: a first determining unit 401, a creating unit 402, a first writing unit 403, an obtaining unit 404, a second determining unit 405, and a second writing unit 406.

[0125] In one exemplary embodiment, the first determining unit 401 is configured to determine whether to trigger index expansion based on resource metrics of the Elastic Retrieval (ES) cluster. In one exemplary embodiment, the creation unit 402 is configured to automatically create a new index based on preset configuration rules when an index expansion is triggered; In one exemplary embodiment, the first write unit 403 is configured to simultaneously write all newly generated data documents to both the old and new indexes of the ES cluster; In one exemplary embodiment, the acquisition unit 404 is configured to acquire the hot and cold ratings of all historical data documents in the old index; In one exemplary embodiment, the second determining unit 405 is configured to determine the migration priority of each historical data document based on the hot and cold ratings of all historical data documents. In one exemplary embodiment, the second writing unit 406 writes each historical data document to a new index based on the migration priority of each historical data document.

[0126] In one exemplary embodiment, resource metrics include disk utilization and CPU load.

[0127] In one exemplary embodiment, the first determining unit 401 is further configured to: Based on the resource metrics of the ES cluster, determine the average load of the resource metrics. When the average load exceeds the trigger threshold, index expansion is triggered.

[0128] In one exemplary embodiment, the trigger threshold is calculated using the following formula:

[0129] in, As the trigger threshold, Predict the incremental load.

[0130] In one exemplary embodiment, the configuration rules include: the number of shard groups, the routing key, the weight of each shard group, and the hot and cold data migration strategy, wherein each shard group contains several shards.

[0131] In one exemplary embodiment, the weight of a shard group is determined based on the shard group's CPU utilization, disk I / O load, and hot / cold rating.

[0132] In one exemplary embodiment, the hot / cold rating of a shard group is determined based on the access frequency and size of each data document in the shard group.

[0133] Figure 5This is a hardware block diagram of an electronic device provided according to an embodiment of the present disclosure. The electronic device 500 according to an embodiment of the present disclosure includes at least a processor and a memory for storing computer-readable instructions. When the computer-readable instructions are loaded and executed by the processor, the processor performs the index expansion method based on flexible retrieval described in any of the preceding embodiments of the present disclosure.

[0134] Figure 5 The illustrated electronic device 500 specifically includes a central processing unit (CPU) 501, a graphics processing unit (GPU) 502, and a memory 503. These units are interconnected via a bus 504. The CPU 501 and / or GPU 502 can function as the aforementioned processor, and the memory 503 can function as the aforementioned memory storing computer-readable instructions. Furthermore, the electronic device 500 may also include a communication unit 505, a storage unit 506, an output unit 507, an input unit 508, and an external device 509, all of which are also connected to the bus 504.

[0135] Figure 6 This is a schematic diagram of a computer-readable storage medium provided in an embodiment of this disclosure. (As shown...) Figure 6 As shown, a computer-readable storage medium 600 according to an embodiment of this disclosure stores computer-readable instructions 601 thereon. When the computer-readable instructions 601 are executed by a processor, the index expansion method based on flexible retrieval described in any of the foregoing embodiments of this disclosure with reference to the above figures is performed. The computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, optical disk, magnetic disk, etc.

[0136] This disclosure further provides a computer program product, including a computer program that, when executed by a processor, implements the index expansion method based on elastic retrieval as described in any of the preceding embodiments of this disclosure.

[0137] For the above, please refer to the appendix. Figure 1 - Appendix Figure 6 This disclosure provides a detailed description of an index expansion device, electronic device, storage medium, and program product according to embodiments of the present disclosure. The technical solutions of this disclosure have the following technical effects: 1. Significantly reduced operational complexity: Traditional solutions rely on manual index creation, dual-write logic configuration, and traffic switching, which are cumbersome and have poor fault tolerance (such as data loss due to accidental alias switching). This proposal reduces manual intervention by 80% through a fully automated process (resource monitoring → expansion triggering → migration execution → traffic switching) and a self-healing fault tolerance mechanism (task breakpoint resumption, distributed locks).

[0138] 2. Significantly Improved Resource Utilization: The existing static sharding design leads to resource waste (30% of shards are idle for extended periods), and full migration (such as reindexing) consumes even higher I / O resources (peaking at 90%). This proposal optimizes resource utilization through dynamic sharding allocation (adjusting weights based on load and hot / cold data scores) and incremental migration. Businesses can allocate shards according to actual data volume, avoiding resource waste caused by pre-allocation, achieving a 40% improvement in resource utilization and a 70% reduction in I / O resource consumption.

[0139] 3. Significantly enhanced data distribution uniformity: Traditional hash algorithms (such as modulo) lead to severe data skew, with high-frequency data forming "hot spot shards". This proposal adopts a dynamic weighted sharding algorithm (combining CPU, disk I / O and hot / cold data scoring) and a two-level routing mechanism (shard group selection + intra-group load balancing). By dynamically selecting low-load shards to allocate data, the skew problem of traditional hash algorithms is solved. Through two-level routing (shard group i.e., index selection and intra-group shard allocation), the target shard is accurately located, reducing the possibility of hot spot shards.

[0140] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0141] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0142] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0143] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0144] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0145] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0146] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0147] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for expanding an index based on elastic retrieval, characterized in that, include: Based on resource metrics of the Elastic Search Elasticsearch cluster, determine whether to trigger index expansion; When index expansion is triggered, a new index is automatically created based on preset configuration rules; All newly generated data documents are simultaneously written to both the old and new indexes of the ES cluster; Retrieve the hot / cold ratings of all historical data documents in the old index; Based on the hot and cold ratings of all historical data documents, the migration priority of each historical data document is determined. Based on the migration priority of each historical data document, each historical data document is written into the new index.

2. The index expansion method based on elastic retrieval according to claim 1, characterized in that, The resource metrics include: disk utilization and CPU load.

3. The index expansion method based on elastic retrieval according to claim 2, characterized in that, The resource metrics based on the Elasticsearch cluster, used to determine whether to trigger index expansion, include: Based on the resource metrics of the ES cluster, determine the average load of the resource metrics; When the average load exceeds the trigger threshold, index expansion is triggered.

4. The index expansion method based on elastic retrieval according to claim 3, characterized in that, The trigger threshold is calculated using the following formula: in, The threshold for triggering the event is S, where S is the current load of the resource metric. Predict the incremental load.

5. The index expansion method based on elastic retrieval according to claim 1, characterized in that, The configuration rules include: number of shard groups, routing key, and weight of each shard group, wherein each shard group contains several shards.

6. The index expansion method based on elastic retrieval according to claim 5, characterized in that, The weights of the shard groups are dynamically updated based on the CPU utilization, disk I / O load, and hot / cold rating of the shard groups.

7. The index expansion method based on elastic retrieval according to claim 6, characterized in that, The hot / cold rating of the shard group is determined based on the access frequency and size of each data document in the shard group.

8. The index expansion method based on elastic retrieval according to claim 1, characterized in that, The process of obtaining the hot / cold rating of all historical data documents in the old index includes: Obtain the access frequency of each historical data document in the old index; Based on the access frequency and data size of each historical data document in the old index, obtain the hot / cold rating of all historical data documents.

9. The index expansion method based on elastic retrieval according to claim 5, characterized in that, The process of writing each historical data document into the new index based on the migration priority of each historical data document includes: Based on the migration priority of each historical data document, the historical data documents are written into the new index in descending order of migration priority.

10. The index expansion method based on elastic retrieval according to claim 9, characterized in that, The method further includes: During the process of writing the historical data documents into the new index, based on the routing key and the weight of each shard group, each historical data document is written into the target shard in the new index.

11. The index expansion method based on elastic retrieval according to claim 10, characterized in that, The method further includes: Before performing a write operation, an optimistic locking mechanism is used to determine whether each historical data document and the newly generated data document has an updated version in the new index.

12. The index expansion method based on elastic retrieval according to claim 1, characterized in that, The method further includes: When the migration progress of the historical data document exceeds the first preset threshold and the health check of the new index passes, 5% of the read traffic data in the ES cluster will be switched to the new index.

13. The index expansion method based on elastic retrieval according to claim 12, characterized in that, The method further includes: After switching 5% of the read traffic data in the ES cluster to the new index, monitor the query latency and error rate of the new index. If everything is normal, gradually increase the read traffic data switched to the new index according to the preset ratio. After each increase, monitor the query latency and error rate of the new index at the preset time. After everything is normal, then execute the increase operation.

14. The index expansion method based on elastic retrieval according to claim 13, characterized in that, When the migration progress of the historical data document exceeds the second preset threshold, all read traffic data in the ES cluster will be switched to the new index.

15. The index expansion method based on elastic retrieval according to claim 1, characterized in that, The method further includes: During the process of writing the historical data documents to the new index, a proxy server in the ES cluster is determined based on a distributed lock to execute the writing of the historical data documents to the new index.

16. The index expansion method based on elastic retrieval according to claim 1, characterized in that, The method further includes: During the process of writing historical data documents and newly generated data documents into the new index, the writing status of each historical data document and newly generated data document is recorded through a task table; When the writing of a historical data document or a newly created data document is abnormally terminated, Based on the task table records of the writing status of each historical data document or newly generated data document, determine the breakpoint of the write operation. Based on the breakpoint, restart writing historical data documents or newly generated data documents.

17. An index expansion device based on elastic retrieval, characterized in that, include: The first determining unit is configured to determine whether to trigger index expansion based on resource metrics of the Elastic Retrieval (ES) cluster. The creation unit is configured to automatically create a new index based on preset configuration rules when an index expansion is triggered; The first write unit is configured to simultaneously write all newly generated data documents to both the old and new indexes of the ES cluster; The acquisition unit is configured to acquire the hot and cold ratings of all historical data documents in the old index; The second determining unit is configured to determine the migration priority of each historical data document based on the hot and cold scores of all historical data documents. The second writing unit writes each historical data document into the new index based on the migration priority of each historical data document.

18. The index expansion device based on elastic retrieval according to claim 17, characterized in that, The resource metrics include: disk utilization and CPU load.

19. The index expansion device based on elastic retrieval according to claim 18, characterized in that, The first determining unit is further configured to: Based on the resource metrics of the ES cluster, determine the average load of the resource metrics; When the average load exceeds the trigger threshold, index expansion is triggered.

20. The index expansion device based on elastic retrieval according to claim 19, characterized in that, The trigger threshold is calculated using the following formula: in, The threshold for triggering the event is S, where S is the current load of the resource metric. Predict the incremental load.

21. The index expansion device based on elastic retrieval according to claim 17, characterized in that, The configuration rules include: the number of shard groups, the routing key, and the weight of each shard group, wherein each shard group contains several shards.

22. The index expansion device based on elastic retrieval according to claim 21, characterized in that, The weights of the shard groups are determined based on the CPU utilization, disk I / O load, and hot / cold rating of the shard group.

23. The index expansion device based on elastic retrieval according to claim 22, characterized in that, The hot / cold rating of the shard group is determined based on the access frequency and size of each data document in the shard group.

24. An electronic device, characterized in that, include: Memory, used to store computer-readable instructions; as well as A processor for executing the computer-readable instructions, causing the electronic device to perform the index expansion method based on elastic retrieval as described in any one of claims 1 to 18.

25. A non-transitory computer-readable storage medium for storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by a processor, the processor performs the index expansion method based on elastic retrieval as described in any one of claims 1 to 18.

26. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the index expansion method based on elastic retrieval as described in any one of claims 1 to 18.

Images