Index recovery method, device and equipment and computer readable storage medium

Abstract

The invention discloses an index recovery method, device and equipment and a computer readable storage medium, which are applied to the technical field of computers, and the method comprises the following steps: when a fault occurs, analyzing a data topological structure corresponding to an index data fragment stored by a node; based on the fragment access log of the node, calculating a heat value of the index data fragment by adopting a time decay model; determining a loading thread count according to the node resource state; determining the priorities of the index data fragments based on the business query priorities, the popularity values and the fragment memory efficiency, and determining a loading sequence according to the priorities; and according to the loading thread count and the loading sequence, loading the index data fragments in the nodes from the persistent storage to the cache, and constructing a memory finite state converter structure capable of being directly retrieved based on a data topological structure. According to the method, on the premise that the system architecture is not changed, the distributed retrieval system can be quickly recovered when the fault occurs, the unavailable time of the system is shortened, and the availability, stability and resource utilization rate of the system are improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an index recovery method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] In some distributed retrieval system failure scenarios, an index recovery process is required, namely disk read → generation of in-memory FST (Finite State Transformer). This is the core operation of index recovery and the priority and resource-constrained step from system startup to availability. Currently, the index recovery process not only consumes a large amount of system resources such as CPU (Central Processing Unit), I / O (Input / Output, i.e., read / write), and memory, but also leads to a decrease in system response speed, affecting the overall performance of the index cluster. Common optimization strategies, such as increasing cache capacity, using cache preloading technology, or optimizing the structure of the FST index, can improve performance to some extent, but they do not fundamentally solve the problem and usually require significant modifications to the system, further increasing the complexity of the original system.

[0003] Therefore, improving the efficiency of index recovery and the stability of the system are urgent technical problems that need to be solved. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide an index recovery method, apparatus, device and computer-readable storage medium, which solves the problems of high complexity and poor stability of the index recovery system in the prior art.

[0005] To solve the above-mentioned technical problems, the present invention provides an index recovery method, comprising:

[0006] When a failure occurs, the data topology structure corresponding to the index data shards stored in the node is parsed, and the hotness value of the index data shards is calculated using a time decay model based on the shard access logs of the node; the index data shards are finite state converter index data shards that contain finite state converter index structures and corresponding index data.

[0007] The number of loading threads is determined based on the node resource status, the priority of the index data shards is determined based on the business query priority, the popularity value, and the shard memory efficiency, and the loading order is determined based on the priority.

[0008] Based on the number of loading threads and the loading order, the index data shards in the node are loaded from persistent storage to the cache, and a directly searchable memory finite state converter structure is constructed based on the data topology.

[0009] On the one hand, it also includes:

[0010] Receive retrieval requests in real time and determine target index data fragments based on the terms or ranges in the retrieval requests;

[0011] If the target index data fragment has been loaded into the cache, then the data is obtained based on the memory finite state converter structure corresponding to the target index data fragment in the cache, and the retrieval request is responded to.

[0012] If the target index data shard is not loaded into the cache, the retrieval request is routed to the persistent storage, the storage file corresponding to the target index data shard is read, and parsed into a searchable temporary data structure to obtain the data corresponding to the target index data shard in response to the retrieval request.

[0013] On the one hand, when routing the retrieval request to the persistent storage, the method also includes:

[0014] Record the index data shard identifier corresponding to the retrieval request, and automatically trigger the preloading of the shard corresponding to the index data shard identifier when the node resource status is low load. The thread priority of the preloading is lower than the loading thread of the fault recovery. The low load status is the state where the comprehensive load factor calculated based on the preset resource monitoring index is less than or equal to the first threshold.

[0015] On the one hand, based on the sharded access logs of the aforementioned nodes, a time decay model is used to calculate the popularity value of the index data shards, including:

[0016] The access frequency and temporal locality of the index data fragments are recorded by a hot data tracker, and the heat value of the index data fragments is calculated based on the time decay model of the sliding window.

[0017] On the one hand, the heat value of the index data fragment is calculated based on the time decay model of the sliding window, including:

[0018] The window duration of the sliding window is set according to the temporal locality of node business queries, and the sliding window is divided into multiple statistical intervals according to the time interval.

[0019] The number of accesses to the index data fragments within each statistical interval is accumulated using a sliding window counter, and the number of accesses over a long period of time is retained when tracing back the window from the current point in time.

[0020] Based on the access count records, the time decay model is applied to the access counts of each statistical interval within the sliding window to calculate the popularity value;

[0021] As the sliding window moves over time, historical access records that exceed a long time range beyond the window are automatically removed, and the popularity value of each index data shard is recalculated based on the updated access count records.

[0022] On the one hand, the number of loading threads is determined based on the node resource status; the priority of the index data shards is determined based on the business query priority, the popularity value, and the shard memory efficiency; and the loading order is determined based on the priority, including:

[0023] The comprehensive load factor is calculated based on preset resource monitoring indicators; the preset resource monitoring indicators include any one or more of the following: CPU utilization, disk read / write utilization, free memory size, and current number of concurrent loading threads.

[0024] If the comprehensive load factor is less than or equal to the first threshold, then the node resource status is a low load state;

[0025] If the overall load factor is greater than the first threshold and less than the second threshold, then the node resource status is medium load.

[0026] If the overall load factor is greater than or equal to the second threshold, then the node resource status is a high load state; the second threshold is greater than the first threshold.

[0027] When the node resource status is in the low load state, the maximum number of loading threads is started according to the number of node central processing unit cores and hardware carrying capacity.

[0028] When the node resource status is in the medium load state, a loading thread is started that does not exceed half the number of node central processing unit cores;

[0029] If the node resource status is in the high load state, then start a single loading thread or pause the loading thread;

[0030] The fragment memory efficiency is calculated based on the total size of all the index data fragments and the size of each index data fragment.

[0031] Based on the popularity value, memory efficiency, and business query priority, calculate the priority score for each index data shard;

[0032] The index data shards are sorted in descending order according to their priority scores to determine the loading order of each index data shard.

[0033] On the one hand, it also includes:

[0034] Real-time monitoring of the loading progress of each index data shard and changes in node resource status;

[0035] When the loading progress of the first index data fragment exceeds a preset threshold, and the node resource status switches from low load to high load, the loading task of the first index data fragment is paused, and the first index data fragment is placed at the tail of the loading queue.

[0036] The present invention also provides an index recovery device, comprising:

[0037] The heat calculation module is used to parse the data topology structure corresponding to the index data shards stored by the node when a fault occurs, and calculate the heat value of the index data shards based on the shard access logs of the node using a time decay model; the index data shards are finite state converter index data shards that contain finite state converter index structures and corresponding index data.

[0038] The recovery strategy determination module is used to determine the number of loading threads based on the node resource status, determine the priority of the index data shards based on the business query priority and the popularity value, as well as the shard memory efficiency, and determine the loading order based on the priority.

[0039] The execution recovery module is used to load the index data shards in the node from persistent storage to cache according to the number of loading threads and the loading order, and to construct a directly searchable memory finite state converter structure based on the data topology.

[0040] The present invention also provides an index recovery device, comprising:

[0041] Memory, used to store computer programs;

[0042] A processor for implementing the index recovery method as described above when executing the computer program.

[0043] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the index recovery method described above.

[0044] The present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the index recovery method described above.

[0045] As can be seen from the above technical solution, this invention, when a fault occurs, parses the data topology structure corresponding to the index data shards stored in the node, and calculates the heat value of the index data shards based on the shard access logs of the node using a time decay model. The index data shards are finite state converter index data shards containing finite state converter index structures and corresponding index data. The number of loading threads is determined according to the node resource status, the priority of the index data shards is determined based on the business query priority, the heat value, and the shard memory efficiency, and the loading order is determined according to the priority. According to the number of loading threads and the loading order, the index data shards in the node are loaded from persistent storage to cache, and a directly searchable memory finite state converter structure is constructed based on the data topology structure. The beneficial effects of this invention are: without changing the system architecture, i.e., without adding extra system performance hardware, it improves the recovery efficiency of index data shards (including index structures and index data). Especially in high-frequency retrieval scenarios, it greatly shortens the time for the system to recover from a fault; it reduces the peak usage of disk read / write, CPU, and memory during the recovery process, avoids resource conflicts with other business tasks (such as real-time writing and query processing), and ensures the overall stability of the system. Ultimately, this allows the distributed retrieval system to recover faster when a failure occurs, reducing system downtime and improving system availability, stability, and resource utilization.

[0046] In addition, the present invention also provides an index recovery device, apparatus, and computer-readable storage medium, which also have the above-mentioned beneficial effects. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 A flowchart of an index recovery method provided in an embodiment of the present invention;

[0049] Figure 2 A flowchart illustrating an index recovery method provided in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the structure of an index recovery device provided in an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the structure of an index recovery device provided in an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] First, let's analyze some of the terms used in this application:

[0054] Distributed retrieval system: also known as full-text retrieval system, is a distributed platform system based on inverted index retrieval technology and used for fast data page retrieval.

[0055] Index data fragments: Data blocks in a full-text search system, typically defined during index initialization and cannot be changed afterward. At its core, they refer to the data units that carry the index data within the full-text search system. One of their key attributes is their strong binding to the index structure; they are usually defined and fixed during index initialization and cannot be arbitrarily changed.

[0056] FST (Finite State Transducer) is a trie-like data structure used in retrieval systems to build and store index structures.

[0057] In existing distributed retrieval systems, the conventional indexing process is data writing → generating an in-memory FST → persisting to disk. Key FST data resides in off-heap memory, enabling rapid data retrieval of the index dictionary and querying of response data upon receiving retrieval requests. However, in some failure scenarios, such as machine failures or power outages, this process needs to be reversed: disk reading → generating an in-memory FST to restore the in-memory index information. This operation is the main procedure for index recovery, the highest priority operation from initial startup to the recovery and availability phase, and also the operation with limited usage during this phase. For example, in normal scenarios, the distributed retrieval system generates the index based on data writing and caches it in memory, ensuring smooth indexing. However, in abnormal situations, such as node power failures and restarts or fault recovery, the FST index data from the data blocks already serialized to disk needs to be reloaded into the cache. This process not only consumes significant system resources but also slows down system response, impacting the overall performance of the index cluster. Current optimization strategies include increasing cache capacity, using cache preloading techniques, and optimizing the FST index structure. While these methods can improve system performance to some extent, they do not fundamentally solve the problem and usually require extensive modifications to the system, increasing its complexity.

[0058] To address the aforementioned problems, this invention provides an index recovery method that can improve the loading efficiency of index data shards and reduce resource consumption while ensuring system stability without altering the system architecture. Please refer to [link / reference needed] for details. Figure 1 , Figure 1 A flowchart illustrating an index recovery method provided in an embodiment of the present invention. The method may include:

[0059] S101: When a fault occurs, parse the data topology structure corresponding to the index data shards stored on the node, and calculate the hotness value of the index data shards based on the shard access logs of the node using a time decay model.

[0060] The execution subject in this embodiment is a terminal. This embodiment does not limit the type of terminal, as long as it can perform the index recovery method operation. It should be noted that the index data shard is a finite state converter index data shard containing a finite state converter index structure and corresponding index data. The fault in this step can be a node fault or an index data shard fault. That is, if it is a node fault, the parsing object is the faulty node; if it is an index data shard fault, the parsing object is the node where the faulty shard is located. The popularity value of the index data shard in this embodiment can be the popularity status within the index data shard; or it can be the overall popularity status of the entire index data shard based on the popularity status within the index data shard.

[0061] In this embodiment, the heat value can be displayed in the form of a heat map. The sharded FST data topology of the distributed retrieval system (i.e., the data topology structure corresponding to the index data shards) can be used to construct a heat map based on the topology structure information. This jointly controls the smooth decay of index heat (shard access logs of nodes) in the metadata (i.e., the time decay model), avoiding the problem of sudden changes in heat at the window boundary within a fixed time window. It also considers access frequency (number of times) and temporal locality (recentness), and can be approximated to avoid storing the timestamp of each access, resulting in extremely high efficiency.

[0062] The time decay model is an algorithmic model used to quantify the change in data access popularity over time. Its core logic is that recent access behavior contributes more to popularity, while the contribution of earlier access behavior gradually decreases over time, thus more accurately reflecting the current access level of data. For example, this model can introduce a time decay coefficient to assign different weights to access records at different points in time: accesses closer to the current time (e.g., within 1 hour) have higher weights; accesses further back in time (e.g., 24 hours ago) have weights that decrease exponentially.

[0063] Furthermore, when a failure occurs, the data topology structure corresponding to the index data shards stored in the parsing node is resolved. Specifically, this may include: when a failure occurs, resolving the data topology structure corresponding to the index data shards of the finite state converter on the disk through the metadata parser, and traversing the data topology structure based on the finite state machine to extract the triple sequence; the triple sequence includes state transition edges, output values, and termination flags.

[0064] Specifically, the FST metadata parser is used to parse the data topology structure of the disk (persistent storage) FST. Based on finite state machine traversal, the sequence of triples <state transition edge, output value, termination flag> is extracted to output the data content of the basic FST. The state transition edge describes the relationship between data, the output value corresponds to the specific index result, and the termination flag marks the end point of the query.

[0065] It's important to note that by parsing the FST's data topology and extracting the triple sequence, the originally unsearchable disk storage format (such as binary structured data) is transformed into basic data that can be used to construct the in-memory FST structure. This provides the foundation for subsequently building a directly searchable in-memory FST structure. In other words, only after extracting this information can a complete, directly queryable FST structure be reconstructed in memory (rather than the original, unsearchable storage format on disk), allowing the index to regain its ability to receive queries and return results after a failure. Furthermore, the triple sequence implicitly contains the structural characteristics of the index data shards (such as the number of state nodes and transition complexity). These characteristics can be combined with popularity values ​​and business priorities to help determine the loading efficiency of index data shards (e.g., simpler shards load faster), making subsequent priority-based loading more accurate and ultimately accelerating the overall index recovery process.

[0066] Furthermore, based on the node's sharded access logs, a time decay model is used to calculate the hotness value of the index data shards. Specifically, this may include: recording the access frequency and temporal locality of the index data shards through a hot data tracker, and calculating the hotness value of the index data shards using a sliding window-based time decay model.

[0067] Specifically, the access frequency and temporal locality of index data shards are recorded using a hot data tracker, and the heat value of the index data shards is calculated based on a sliding window time decay model. It can be understood that this method can be used to calculate the heat of the entire shard, as well as the heat of each data portion within the shard. The formula for calculating the heat value is: Where Σ represents the total heat value obtained by summing the results, e is the natural constant, that is, the base of the natural logarithm, which is approximately equal to 2.71828; λ is the decay rate / decay coefficient; and Δt is the time difference between the current time and the time when the event occurred.

[0068] Furthermore, metadata snapshot storage can be implemented. Specifically, the structured data of the FST (such as triple sequences) and the heat distribution information (heat values) of each index data shard are persistently stored using Parquet (a columnar storage file format). Simultaneously, a Bloom filter is configured for rapid hot shard location, and the identifying characteristics of hot shards are recorded in advance. In this way, Parquet columnar storage can efficiently compress the FST structure description and heat data, reducing storage consumption; it also supports on-demand reading of specific columns (such as reading only the heat values ​​of hot shards), reducing IO overhead and accelerating data loading. The Bloom filter can determine whether a shard is a hot shard in milliseconds, avoiding traversing all shard information, significantly improving the speed of hot shard identification, and providing support for loading priority sorting and rapid recovery of high-frequency query capabilities.

[0069] Furthermore, the above-mentioned calculation of the popularity value of index data shards based on the sliding window time decay model can specifically include the following steps: setting the window duration of the sliding window according to the temporal locality of node business queries, and dividing the sliding window into multiple statistical intervals according to time intervals; accumulating the number of accesses to index data shards in each statistical interval through a sliding window counter, and retaining the access count records over a long period of time when tracing back the window from the current time point; applying the time decay model to the access counts of each statistical interval within the sliding window based on the access count records, and calculating the popularity value; as the sliding window slides over time, automatically removing historical access records that exceed the long period of time of the window duration, and recalculating the popularity value of each index data shard based on the updated access count records.

[0070] For example, based on the recent business query patterns of nodes (e.g., most frequent queries in the last hour), a sliding window duration (e.g., 1 hour) is set, and this window is then divided into multiple smaller time intervals (e.g., 10-minute intervals). A counter is used to count the number of times each index data shard is accessed within each smaller time interval, retaining only access records from the hour preceding the current time, excluding earlier ones. Weights are assigned to the access counts of each time interval within the sliding window: for example, more recent access counts have higher weights (e.g., accesses from 10 minutes ago are counted as 100%, accesses from 20 minutes ago as 80%). The final popularity value is calculated using this time decay model. The sliding window slides with time (e.g., moving forward 10 minutes every 10 minutes), automatically deleting old records exceeding the 1-hour range, and recalculating the popularity value of each shard. In this way, by focusing only on recent valid accesses and avoiding interference from old data (such as accesses from a few days ago), the current popularity of shards can be accurately reflected. For example, shards that have been frequently queried recently will be judged as high-popularity. When loading index data shards later, high-popularity shards can be loaded first based on dynamic popularity, reducing business query waiting time (for example, shards frequently queried by users are loaded into the cache first). By splitting the statistical area and automatically removing old records, the storage and computation of massive amounts of historical data can be avoided, reducing the memory and computing power consumption of nodes. The window updates dynamically over time, which can quickly respond to changes in business query patterns (for example, the popular shards are different during morning and evening peak hours, and the popularity value can be adjusted in real time).

[0071] S102: Determine the number of loading threads based on the node resource status, determine the priority of index data shards based on business query priority and popularity value, and shard memory efficiency, and determine the loading order based on the priority.

[0072] This step monitors the node resource status (such as whether CPU and memory are under pressure). If resources are idle (low load), more threads are used to speed up loading. If resources are under pressure (high load), fewer threads or even a single thread are used to avoid consuming too many resources and affecting business operations. Furthermore, the loading order is determined by considering multiple dimensions such as business importance (prioritizing core business-related shards), recent access popularity (prioritizing high-population shards that are frequently accessed), and memory efficiency (prioritizing shards that consume less memory after loading and can be used quickly).

[0073] Furthermore, the number of loading threads is determined based on the node resource status, the priority of index data shards is determined based on business query priority and popularity, and shard memory efficiency, and the loading order is determined based on the priority. Specifically, this may include:

[0074] The comprehensive load factor is calculated based on preset resource monitoring indicators. These indicators include any one or more of the following: CPU utilization, disk read / write utilization, free memory size, and the number of currently concurrent loading threads. If the comprehensive load factor is less than or equal to a first threshold, the node's resource status is low load. If the comprehensive load factor is greater than the first threshold but less than a second threshold, the node's resource status is medium load. If the comprehensive load factor is greater than or equal to the second threshold, the node's resource status is high load. The second threshold is greater than the first threshold. When the node's resource status is low load, the load factor is calculated based on the number of CPU cores (N_cp). The maximum number of loading threads is determined by the node's resource status (u) and hardware capacity. When the node's resource status is medium load, no more than half the number of CPU cores are started. If the node's resource status is high load, a single loading thread is started or loading threads are paused. The shard memory efficiency is calculated based on the total size of all index data shards and the size of each index data shard. The priority score of each index data shard is calculated based on the popularity value, memory efficiency, and business query priority. The index data shards are sorted in descending order according to their priority scores to determine the loading order of each index data shard.

[0075] This step involves dynamic monitoring and analysis of node resources. With real-time performance as the guide, it dynamically refreshes the optimal loading strategy for index data shards. Based on the actual resources of the current node, it dynamically monitors and analyzes them in real time to build a real-time and accurate dynamic environment resource awareness. Based on the dynamic awareness results, it provides the index data shard loading strategy engine with accurate basic information for selecting loading strategies, ensuring the matching degree of the strategy selected for index recovery, improving loading speed while reducing the impact on current system business.

[0076] For example, (1) Input and status monitoring: Multiple system resource monitoring indicators are continuously monitored and represented as system status vector S(t): CPU utilization C_usage∈[0,1], disk IO utilization I_usage∈[0,1], free memory size M_free (in MB), current concurrent loading thread count T_current, business query request QPS (optional), business query average response time Avg_Latency (optional). (2) Calculate system load status: Provide a scalar value of the overall system pressure level. The comprehensive load factor is configured and adjusted according to the actual load status and resource status of the cluster so that the final analysis results are more in line with the actual system environment. For example: The formula for calculating the comprehensive load factor L is: L=α*C_usage+β*I_usage. Where, α+β=1. For example, α=0.6, β=0.4, α and β can be adjusted according to the actual hardware characteristics. According to the value of the comprehensive load factor L, the system load status is divided into three intervals:

[0077] Green zone (Low Load): L < θ_low (e.g., θ_low = 0.5);

[0078] Yellow zone (Medium Load): θ_low <= L < θ_high (e.g., θ_high = 0.8);

[0079] Red zone (High Load): L>=θ_high.

[0080] (3) Calculate the priority score of the index data shard: For each index data shard i in the queue, calculate its priority score P_i. This score is a weighted average of multiple factors. For example:

[0081] The formula for calculating the priority score P_i is:

[0082] P_i=w1*H_i+w2*(S_total / S_i)+w3*B_i.

[0083] Wherein, the heat value H_i is the heat value from the metadata, that is, the heat value calculated in step S101, which is normalized to [0, 1].

[0084] Shard memory efficiency S_total / S_i: S_total is an estimate of the total size of all index data shards, and S_i is the size of FST i, i.e., the size of the current index data shard. This item encourages loading smaller index data shards first, which can quickly complete more tasks, reduce queue depth, and improve overall performance.

[0085] Business Priority B_i: Business importance label from metadata, normalized to [0,1]. For example, an important business index is 1, and a regular log index is 0.2.

[0086] w1, w2, and w3 are weighting coefficients, and w1 + w2 + w3 = 1. For example, (0.7, 0.2, 0.1) indicates that popularity is given more weight.

[0087] (4) Based on the system load status and priority score P_i, determine the loading strategy. The strategy is provided as a real-time dynamic action and can be quickly obtained during fragment recovery. Refer to Table 1, which is an example table of loading strategies.

[0088] Table 1 Example of Loading Strategies

[0089]

[0090] (5) Queue Sorting: Sort the waiting queue in descending order according to P_i. Thread Pool Management: Based on the current maximum allowed concurrency (i.e., the number of loading threads) T_max, take T_max-T_current tasks from the head of the queue and submit them to the loading thread pool for execution. Where T_current is the current number of concurrent loading threads.

[0091] (6) Monitoring and Adjustment: Continuously monitor the system state S(t). If the state changes (e.g., from a low-load state to a high-load state), immediately reset the loading strategy, which may interrupt the currently executing low-priority loading task and reallocate resources.

[0092] S103: Based on the number of loading threads and the loading order, the index data fragments in the node are loaded from persistent storage to the cache, and a memory-based finite state converter structure that can be directly retrieved is constructed based on the data topology.

[0093] This step is the execution step. Based on the loading order and number of loading threads obtained in step S102, loading is performed to achieve index recovery. Furthermore, based on the topology structure parsed in step S101 (such as data relationships, state transition relationships, etc.), a finite state converter (FST) is built in memory. Once this structure is built, it can be used directly for fast data retrieval without having to slowly search on disk. For example, if the topology relationship where data A can be transferred to data B, and data B corresponds to output result C, an FST is built in memory according to this relationship. Subsequent queries from A to B can directly retrieve the answer instantly through the FST in memory, without needing to re-parse disk data.

[0094] In this embodiment, the number of loading threads refers to the number of concurrent loading threads. The loading order in this embodiment includes the loading order of each index data shard in the case of multiple index data shards, and the loading order within the index data shard in the case of a single index data shard.

[0095] For example, loading strategy parameters (including the number of loading threads and loading order) are defined as scenario standards, baseline strategy rules and loading scenario priority configurations are defined, and cache loading is performed according to the actual loading strategy requirements and strategy loading configuration during actual strategy execution. For example:

[0096] (1) Determine the current scenario: The system will first determine which loading scenario it belongs to and find the corresponding loading path.

[0097] (2) Single shard loading path: When restoring a single shard, the hot data distribution of the shard will be checked first (judged by real-time updated metadata), and the hot data will be loaded in the order of hot data first and cold data later, so as to quickly restore the query function of the shard.

[0098] (3) Multi-shard loading path: When multiple shards are restored at the same time, the system load status (such as resource usage) and task queue will be referenced, and the shards will be loaded from high to low in terms of popularity (hot shards first, cold shards later). This loading order ensures that 80% of high-frequency queries can be responded to normally.

[0099] (4) Optimized loading under low load: If the system resources are sufficient (low load), when initializing the cache loading, the state transition continuity of the index data shards will be sorted to reduce the disk read and write time (about 30%), making the cache loading faster.

[0100] Furthermore, it may also include: receiving retrieval requests in real time, determining the target index data shard based on the terms or terms in the retrieval request; if the target index data shard has been loaded into the cache, obtaining the data based on the memory finite state converter structure corresponding to the target index data shard in the cache, and responding to the retrieval request; if the target index data shard has not been loaded into the cache, routing the retrieval request to persistent storage, reading the storage file corresponding to the target index data shard, parsing it into a searchable temporary data structure, obtaining the data corresponding to the target index data shard, and responding to the retrieval request.

[0101] Specifically, during index recovery, even with only partial data recovery, stable data retrieval services can be provided. By receiving retrieval requests and designing retrieval information, the system locates whether the retrieval content has been loaded into the cache. If so, it quickly retrieves the index information from the cache for efficient subsequent extraction of actual data. Otherwise, the retrieval request is sent to disk for index and data retrieval, ensuring the validity and efficiency of the query. For example: a. Data nodes receive retrieval requests from the coordinating node (intermediate node) and distribute the corresponding retrieval requests to all index data shards. b. Each index data shard processes the retrieval request. Shards that are running normally and have not reverse-loaded FST (i.e., shards without failures) process retrieval requests normally. Shards that are reverse-flush FST (shards that have failed but recovered normally) check whether the Term or Range involved in the retrieval request has been loaded into the cache. If so, the cached index is directly queried; otherwise, the index and results are obtained using disk reads. The following example illustrates how disk reads can be used to obtain the index and results: the retrieval request is routed to persistent storage, the storage file corresponding to the target index data shard is read, and parsed into a searchable temporary data structure. c. Finally, the query results are summarized and returned to the coordinating node.

[0102] Furthermore, when routing retrieval requests to persistent storage, the process may also include: recording the index data shard identifier corresponding to the retrieval request, and automatically triggering the preloading of the shard corresponding to the index data shard identifier when the node resource status is low load, with the preloading thread having a lower priority than the loading thread during fault recovery; the low load status is defined as a state where the comprehensive load factor calculated based on preset resource monitoring indicators is less than or equal to a first threshold.

[0103] Specifically, when querying from disk, the identifiers of the index data shards involved in each retrieval request can be recorded first. When node resources are idle (overall load factor ≤ first threshold, i.e., low load state), these shards are automatically preloaded. The thread priority used for preloading is lower than the thread loading shards during fault recovery, avoiding impact on the core task of fault recovery. For example, if a user has queried the index data shard of product A, the system will record the identifier of this index data shard; when the node is not busy, this index data shard is preloaded into the cache, so that the next time the user queries, the response will be faster; and preloading will not compete with fault recovery for resources, prioritizing the core recovery task during faults. In this way, by triggering the preloading mechanism through missed requests, cold shards are actively converted into hot shards, gradually improving cache coverage; at the same time, by distinguishing by thread priority, preloading is avoided from interfering with the core loading task of fault recovery, optimizing retrieval response speed in the long term without affecting fault recovery efficiency.

[0104] Furthermore, after recording the index data shard identifier corresponding to the retrieval request, it may also include: counting the frequency of the index data shard identifier being recorded within a preset time period, and when the recording frequency is greater than or equal to a second threshold, triggering the preloading of the index data shard when the node resource status is in a low-load state; wherein, the preset time period is positively correlated with the window duration of the sliding window.

[0105] This step further optimizes the preloading trigger logic: instead of preloading any shard with a search request record, it first counts the number of times the shard is searched within a preset time (e.g., 1 hour, which is about the same as the sliding window duration for calculating popularity). Only when the number of times exceeds the second threshold (e.g., more than 5 times) does it indicate that the shard is indeed frequently searched by users, and only then is preloading triggered when the node is idle. This approach avoids invalid preloading, excludes shards that are occasionally queried, and only preloads frequently queried index data shards, reducing the waste of idle node resources (e.g., avoiding loading infrequently queried shards into the cache, thus consuming memory); by filtering through frequency thresholds, it ensures that the preloaded index data shards are those that users actually need frequently, increasing the probability of cache hits during subsequent user queries and further improving query speed; the preset time period is positively correlated with the sliding window duration (e.g., if the sliding window is 1 hour, the preset period is also 1 hour), ensuring that the preloading criteria are consistent with the time dimension of the popularity value calculation, avoiding preloading deviations caused by time dimension misalignment (e.g., not using a 1-day frequency to match an 1-hour popularity).

[0106] Furthermore, it may also include: real-time monitoring of the loading progress of each index data shard and changes in node resource status; when the loading progress of the first index data shard exceeds a preset threshold and the node resource status switches from a low load state to a high load state, the loading task of the first index data shard is paused and the first index data shard is placed at the tail of the loading queue.

[0107] Specifically, the loading progress of each index data shard is monitored in real time (e.g., how many percent has been loaded) and the node resource status (e.g., whether CPU / memory has changed from idle to strained). If a certain index data shard (i.e., the first index data shard) has been loaded to a considerable extent (progress exceeding a preset threshold, e.g., 70%), but the node resource status changes from idle (low load) to strained (high load), the loading of this index data shard is paused and placed at the end of the loading queue, to be processed again when resources become available. For example, if index data shard A is being loaded and is already 80% loaded, but the node suddenly becomes busy, the loading of index data shard A is stopped to avoid competing for resources with other important tasks, and it is retrieved from the end of the queue to continue loading when the node becomes idle. In this way, through the linkage mechanism between loading progress threshold and resource status, resource blocking caused by forcibly loading large shards under high load is avoided, while some loading results are reserved for retrieval use. In resource-scarce scenarios, the integrity of loading and business availability are balanced, reducing system response latency.

[0108] The index recovery method provided in this invention, when a fault occurs, parses the data topology structure corresponding to the index data shards stored on the node, and calculates the heat value of the index data shards using a time decay model based on the shard access logs of the node. The index data shards are finite state converter index data shards containing finite state converter index structures and corresponding index data. The number of loading threads is determined according to the node resource status, the priority of the index data shards is determined based on the business query priority, the heat value, and the shard memory efficiency, and the loading order is determined according to the priority. According to the number of loading threads and the loading order, the index data shards in the node are loaded from persistent storage to cache, and a directly searchable memory finite state converter structure is constructed based on the data topology structure. This method can improve the recovery efficiency of index data shards (including index structures and index data) without changing the system architecture, i.e., without adding extra system performance hardware. Especially in high-frequency retrieval scenarios, it greatly shortens the time for the system to recover from a fault; reduces the peak usage of disk read / write, CPU, and memory during the recovery process, avoids resource conflicts with other business tasks (such as real-time writing and query processing), and ensures the overall stability of the system. Ultimately, this allows the distributed retrieval system to recover faster when a failure occurs, reducing system downtime and improving system availability, stability, and resource utilization.

[0109] For a clearer understanding of this invention, please refer to the following details. Figure 2 , Figure 2 A flowchart illustrating an index recovery method provided in this embodiment of the invention may specifically include:

[0110] This invention mainly comprises four parts: a fast metadata maintenance module, a dynamic balancer, an index data sharding loading strategy engine, and a data retrieval adapter. These four parts optimize the loading of the FST index cache during failure scenarios in distributed retrieval systems. They employ a more efficient single-shard loading strategy and a shard-level data loading scheme to improve sharding recovery speed in retrieval system failure recovery scenarios, thereby enhancing system availability and stability.

[0111] 1) The fast-fetch metadata maintenance module is used to structure and maintain the data structure and thermal information of a single-shard FST, providing the data foundation for shard cache loading. It includes: an FST metadata parser to parse the FST data topology and extract the <state transition edge, output value, termination flag> triple sequence based on a priority state machine traversal; a thermal data tracker to record the access frequency and temporal locality of index data shards, tracking and identifying thermal data based on a sliding window time decay model; and finally, persistently storing the FST structure description and thermal distribution map as a metadata snapshot: columnar storage (Parquet format) and a Bloom filter (for fast location of thermal shards).

[0112] 2) The dynamic balancer can continuously monitor the resource usage status of each node in the distributed cluster and calculate the optimal sharding recovery strategy and supported sharding recovery capabilities for the current system. It monitors the resource usage of its own node and performs key indicator tracking and comprehensive analysis on the monitored node resource and business information. It outputs the node's performance resource indicators and provides the optimal FST loading strategy. Under resource-constrained node conditions (CPU, IO, memory), it maximizes the cache loading efficiency of the FST index while minimizing the impact on concurrently recovering business query requests.

[0113] 3) The data sharding loading strategy engine performs actual single-index data sharding FST and multi-index data sharding concurrent FST loading actions based on the optimal loading strategy provided by the dynamic balancer and the configured loading scheme. This ensures that single-node sharded cached data can be deployed and used as quickly as possible. The data sharding loading strategy engine, based on the corresponding loading strategies of the dynamic balancer, including sharding loading, cluster-level concurrent loading, and topology optimization, selects sharding loading strategies according to the current business scenario characteristics and data scenario, and performs loading actions based on the corresponding strategies. This allows for the rapid recovery of high-frequency query capabilities according to scenario characteristics and business needs, ensuring the timely deployment of critical business functions.

[0114] 4) The data retrieval adapter can quickly restore the data retrieval capability of index data shards during fault recovery scenarios, especially ensuring rapid business recovery in cluster scenarios with large data volumes and large index data shards, improving system robustness and stability. The data retrieval adapter forwards and processes retrieval requests. Its core function is to retrieve the structure and information of cached data from the FST, while forwarding uncacheed data to disk. The data retrieval adapter runs automatically when the index data shards start, acquiring all retrieval requests from the local node. By checking whether the Term or Range involved in the retrieval request already exists in the loaded FST cache, it directly retrieves the information from the FST structure in memory when the cache is hit, quickly returning the result; when the cache is not hit, the retrieval request is forwarded to disk, where the result is obtained by directly reading the Segment (index data shard) and FST file, ensuring the final query result.

[0115] As can be seen, the fast-flush metadata maintenance module is used to structure and maintain the data structure and data heat distribution information of a single-shard FST, providing the optimal path for the single-shard recovery process; the balancer, based on the resource monitoring status and analysis results of the node, performs shard data weight positioning and data recovery strategy evaluation, ensuring that the optimal strategy is used in the shard-level FST data recovery process; the data shard loading strategy engine performs single-shard and shard-level FST loading operations based on the fast-flush metadata record information and the shard loading strategy method output by the dynamic balancer, ensuring the rapid online deployment of index shard FSTs; the data retrieval adapter is suitable for use in fault recovery scenarios, assisting in the rapid transfer and execution of query requests through data checks, ensuring that the retrieval service can run stably and efficiently; through the above control process, the rapid recovery capability and business recovery speed of the distributed retrieval system in fault recovery scenarios can be improved, thereby improving the performance and stability of the cluster.

[0116] The index recovery device provided in the embodiments of the present invention will be described below. The index recovery device described below and the index recovery method described above can be referred to each other.

[0117] Please refer to the details. Figure 3 , Figure 3 A schematic diagram of an index recovery device provided in an embodiment of the present invention may include:

[0118] The heat calculation module 100 is used to parse the data topology structure corresponding to the index data shards stored by the node when a fault occurs, and calculate the heat value of the index data shards based on the shard access logs of the node using a time decay model; the index data shards are finite state converter index data shards that contain finite state converter index structures and corresponding index data.

[0119] The recovery strategy determination module 200 is used to determine the number of loading threads based on the node resource status, determine the priority of the index data shards based on the business query priority and the popularity value, as well as the shard memory efficiency, and determine the loading order based on the priority.

[0120] The recovery execution module 300 is used to load the index data shards in the node from persistent storage to cache according to the number of loading threads and the loading order, and to construct a directly searchable memory finite state converter structure based on the data topology.

[0121] Based on the above embodiments, the index recovery device may further include:

[0122] The request receiving module is used to receive retrieval requests in real time and determine the target index data fragments based on the terms or ranges in the retrieval requests.

[0123] The first response module is used to retrieve data based on the memory finite state converter structure corresponding to the target index data fragment in the cache if the target index data fragment has been loaded into the cache, and respond to the retrieval request.

[0124] The second response module is used to route the retrieval request to the persistent storage if the target index data shard is not loaded into the cache, read the storage file corresponding to the target index data shard, parse it into a searchable temporary data structure, and obtain the data corresponding to the target index data shard to respond to the retrieval request.

[0125] Based on the above embodiments, it may also include:

[0126] The preloading module is used to record the index data shard identifier corresponding to the retrieval request, and automatically trigger the preloading of the shard corresponding to the index data shard identifier when the node resource status is low load. The thread priority of the preloading is lower than the loading thread of the fault recovery. The low load status is the state where the comprehensive load factor calculated based on the preset resource monitoring index is less than or equal to the first threshold.

[0127] Based on the above embodiments, the heat calculation module 100 may include:

[0128] The heat calculation unit is used to record the access frequency and temporal locality of the index data fragment through a heat data tracker, and calculate the heat value of the index data fragment based on the time decay model of the sliding window.

[0129] Based on the above embodiments, the heat calculation unit may include:

[0130] The sub-unit is used to set the window duration of the sliding window based on the temporal locality of the node business query, and to divide the sliding window into multiple statistical intervals according to the time interval.

[0131] The statistics subunit is used to accumulate the number of accesses to the index data fragments within each of the statistical intervals through a sliding window counter, and to retain the access count record over a long period of time when tracing back the window from the current time point.

[0132] The calculation subunit is used to calculate the popularity value by applying the time decay model to the number of visits in each statistical interval within the sliding window based on the number of visits recorded.

[0133] The update subunit is used to automatically remove historical access records that exceed a long time range outside the window as the sliding window slides over time, and to recalculate the heat value of each index data fragment based on the updated access count records.

[0134] Based on the above embodiments, the recovery strategy determination module 200 may include:

[0135] The load calculation unit is used to calculate the comprehensive load factor based on preset resource monitoring indicators; the preset resource monitoring indicators include any one or several of the following: CPU utilization, disk read / write utilization, free memory size, and current number of concurrent loading threads.

[0136] The first judgment unit is used to determine the node resource status as low load if the comprehensive load factor is less than or equal to the first threshold.

[0137] The second judgment unit is used to determine the node resource status as medium load if the comprehensive load factor is greater than the first threshold and less than the second threshold.

[0138] The third judgment unit is used to determine the node resource status as high load state if the comprehensive load factor is greater than or equal to the second threshold; the second threshold is greater than the first threshold.

[0139] The first thread number determination unit is used to start the maximum number of loading threads based on the number of CPU cores and hardware capacity of the node when the node resource status is the low load status.

[0140] The second thread number determination unit is used to start loading threads that do not exceed half the number of CPU cores of the node when the node resource status is medium load status.

[0141] The third thread number determination unit is used to start a single loading thread or pause the loading thread if the node resource status is the high load status.

[0142] A memory efficiency calculation unit is used to calculate the memory efficiency of the segment based on the total size of all the index data segments and the size of each index data segment.

[0143] The priority score calculation unit is used to calculate the priority score of each index data shard based on the popularity value, the memory efficiency, and the business query priority.

[0144] The loading order determination unit is used to sort the index data fragments in descending order according to their priority scores, and to determine the loading order of the index data fragments.

[0145] Based on the above embodiments, the index recovery device may further include:

[0146] The monitoring module is used to monitor the loading progress of each index data shard and the changes in node resource status in real time.

[0147] The adjustment module is used to pause the loading task of the first index data fragment and place the first index data fragment at the tail of the loading queue when the loading progress of the first index data fragment exceeds a preset threshold and the node resource status switches from the low load state to the high load state.

[0148] It should be noted that the order of the modules and units in the above-mentioned index recovery device can be changed without affecting the logic.

[0149] The index recovery device provided in this embodiment of the invention uses a heat calculation module 100 to parse the data topology structure corresponding to the index data shards stored on the node when a fault occurs, and calculates the heat value of the index data shards using a time decay model based on the shard access logs of the node. The index data shards are finite state converter index data shards containing finite state converter index structures and corresponding index data. A recovery strategy determination module 200 determines the number of loading threads based on the node resource status, determines the priority of the index data shards based on the business query priority, the heat value, and the shard memory efficiency, and determines the loading order based on the priority. A recovery execution module 300 loads the index data shards in the node from persistent storage to cache according to the number of loading threads and the loading order, and constructs a directly searchable memory finite state converter structure based on the data topology structure. This device improves the recovery efficiency of index data shards (including index structures and index data) without changing the system architecture, i.e., without adding extra system performance hardware. Especially in high-frequency retrieval scenarios, it significantly shortens the time for the system to recover from failures; reduces peak disk read / write, CPU, and memory usage during recovery, avoids resource conflicts with other business tasks (such as real-time writing and query processing), and ensures overall system stability. Ultimately, it enables the distributed retrieval system to recover faster when failures occur, reduces system downtime, and improves system availability, stability, and resource utilization.

[0150] Figure 4 This is a schematic diagram of the structure of an index recovery device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the index recovery device includes:

[0151] Memory 60 is used to store computer programs;

[0152] The processor 61 is used to implement the steps of the index recovery method as described in the above embodiments when executing a computer program.

[0153] The index recovery device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0154] The processor 61 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 61 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 61 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 61 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0155] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 60 is used to store at least the following computer program 601, which, after being loaded and executed by the processor 61, is capable of implementing the relevant steps of the index recovery method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, and the storage method may be temporary or permanent storage. The operating system 602 may include Windows, Unix, Linux, etc. The data 603 may include, but is not limited to, data related to the index recovery method.

[0156] In some embodiments, the index recovery device may further include a display screen 62, an input / output interface 63, a communication interface 64, a power supply 65, and a communication bus 66.

[0157] Those skilled in the art will understand that Figure 4 The structure shown does not constitute a limitation on the index recovery device and may include more or fewer components than illustrated.

[0158] It is understood that if the index recovery method in the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk, or optical disk, and other media capable of storing program code.

[0159] Based on this, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the index recovery method described above.

[0160] The following describes a computer program product provided by an embodiment of this application. The computer program product described below can be referred to in conjunction with other embodiments described herein.

[0161] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned disclosed index recovery method.

[0162] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0163] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0164] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0165] The foregoing has provided a detailed description of an index recovery method, apparatus, device, and computer-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An index recovery method, characterized in that, include: When a failure occurs, the data topology structure corresponding to the index data shards stored in the parsed node is analyzed, and the hotness value of the index data shards is calculated using a time decay model based on the shard access logs of the node; the index data shards are finite state converter index data shards that contain finite state converter index structures and corresponding index data. The number of loading threads is determined based on the node resource status, the priority of the index data shards is determined based on the business query priority, the popularity value, and the shard memory efficiency, and the loading order is determined based on the priority. Based on the number of loading threads and the loading order, the index data shards in the node are loaded from persistent storage to the cache, and a directly searchable memory finite state converter structure is constructed based on the data topology.

2. The index recovery method according to claim 1, characterized in that, Also includes: Receive retrieval requests in real time and determine target index data fragments based on the terms or ranges in the retrieval requests; If the target index data fragment has been loaded into the cache, then data is obtained based on the memory finite state converter structure corresponding to the target index data fragment in the cache, and the retrieval request is responded to. If the target index data shard is not loaded into the cache, the retrieval request is routed to the persistent storage, the storage file corresponding to the target index data shard is read, and parsed into a searchable temporary data structure to obtain the data corresponding to the target index data shard in response to the retrieval request.

3. The index recovery method according to claim 2, characterized in that, When routing the retrieval request to the persistent storage, the method further includes: Record the index data shard identifier corresponding to the retrieval request, and automatically trigger the preloading of the shard corresponding to the index data shard identifier when the node resource status is low load. The thread priority of the preloading is lower than the loading thread of the fault recovery. The low load status is the state where the comprehensive load factor calculated based on the preset resource monitoring index is less than or equal to the first threshold.

4. The index recovery method according to claim 1, characterized in that, Based on the sharded access logs of the aforementioned nodes, the popularity value of the index data shards is calculated using a time decay model, including: The access frequency and temporal locality of the index data fragments are recorded by a hot data tracker, and the heat value of the index data fragments is calculated based on the time decay model of the sliding window.

5. The index recovery method according to claim 4, characterized in that, The heat value of the index data fragment is calculated based on the time decay model using a sliding window, including: The window duration of the sliding window is set according to the temporal locality of node business queries, and the sliding window is divided into multiple statistical intervals according to the time interval. The number of accesses to the index data fragments within each statistical interval is accumulated using a sliding window counter, and the number of accesses over a long period of time is retained when tracing back the window from the current point in time. Based on the access count records, the time decay model is applied to the access counts of each statistical interval within the sliding window to calculate the popularity value; As the sliding window moves over time, historical access records that exceed a long time range beyond the window are automatically removed, and the popularity value of each index data shard is recalculated based on the updated access count records.

6. The index recovery method according to claim 1, characterized in that, The number of loading threads is determined based on the node resource status. The priority of the index data shards is determined based on the business query priority, the popularity value, and the shard memory efficiency. The loading order is then determined according to the priority, including: The comprehensive load factor is calculated based on preset resource monitoring indicators; the preset resource monitoring indicators include any one or more of the following: CPU utilization, disk read / write utilization, free memory size, and current number of concurrent loading threads. If the comprehensive load factor is less than or equal to the first threshold, then the node resource status is a low load state; If the overall load factor is greater than the first threshold and less than the second threshold, then the node resource status is medium load. If the overall load factor is greater than or equal to the second threshold, then the node resource status is a high load state; the second threshold is greater than the first threshold. When the node resource status is in the low load state, the maximum number of loading threads is started according to the number of node central processing unit cores and hardware carrying capacity. When the node resource status is in the medium load state, a loading thread is started that does not exceed half the number of node central processing unit cores; If the node resource status is in the high load state, then start a single loading thread or pause the loading thread; The fragment memory efficiency is calculated based on the total size of all the index data fragments and the size of each index data fragment. Based on the popularity value, memory efficiency, and business query priority, calculate the priority score for each index data shard; The index data fragments are sorted in descending order according to their priority scores to determine the loading order of each index data fragment.

7. The index recovery method according to claim 6, characterized in that, Also includes: Real-time monitoring of the loading progress of each index data shard and changes in node resource status; When the loading progress of the first index data shard exceeds a preset threshold, and the node resource status switches from the low load state to the high load state, the loading task of the first index data shard is paused, and the first index data shard is placed at the tail of the loading queue.

8. An index recovery device, characterized in that, include: The heat calculation module is used to parse the data topology structure corresponding to the index data shards stored by the node when a fault occurs, and calculate the heat value of the index data shards based on the shard access logs of the node using a time decay model; the index data shards are finite state converter index data shards that contain finite state converter index structures and corresponding index data. The recovery strategy determination module is used to determine the number of loading threads based on the node resource status, determine the priority of the index data shards based on the business query priority and the popularity value, as well as the shard memory efficiency, and determine the loading order based on the priority. The execution recovery module is used to load the index data shards in the node from persistent storage to cache according to the number of loading threads and the loading order, and to construct a directly searchable memory finite state converter structure based on the data topology.

9. An index recovery device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the index recovery method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the index recovery method as described in any one of claims 1 to 7.

Images