Retrieval route optimization method and device based on distributed retrieval system

By introducing document status identifiers and dynamic routing algorithms into the distributed retrieval system, combined with node resource monitoring, the routing strategy for retrieval requests is optimized, solving the problems of resource waste and performance degradation, and achieving efficient resource utilization and stable retrieval performance.

CN120892453APending Publication Date: 2025-11-04JINAN INSPUR DATA TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202510989654.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-07-17
Publication Date
2025-11-04

AI Technical Summary

Technical Problem

Existing distributed retrieval systems suffer from resource waste, performance degradation, and increased request response latency when processing retrieval requests. This is especially true in large-scale and high-concurrency scenarios, where static routing and resource allocation mechanisms are difficult to adapt, leading to data inconsistency and resource skew.

Method used

By introducing document status identification, dynamic routing algorithms, and node resource monitoring mechanisms, and by acquiring configuration information, inverted metadata, and optimal execution sharding formulas, we can achieve accurate routing and efficient processing of retrieval requests, thereby optimizing retrieval performance and resource utilization.

Benefits of technology

It improves the retrieval performance and resource utilization of the distributed retrieval system, ensuring the system's stability and efficiency, especially in large-scale and high-concurrency scenarios, reducing resource consumption and response latency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120892453A_ABST
    Figure CN120892453A_ABST
Patent Text Reader

Abstract

The invention relates to a retrieval routing optimization method and device based on a distributed retrieval system, and relates to the field of computer technology distributed retrieval, the method comprises the following steps: obtaining inverted metadata of a document, including routing information, fragmentation information and a document storage identifier, judging whether the document is a new document, and combining the resource state and availability of each node to optimize the retrieval routing optimization of the document. And determining an optimal routing path by adopting an optimal execution fragmentation formula. A retrieval request is split into a plurality of sub-requests and is directionally issued to a target fragment, and the coordination node integrates return results of the fragments and then returns the return results to the client. The device comprises an information integration module, an inverted metadata recorder and a retrieval routing control module, and supports a dynamic configuration routing strategy, a document state refreshing mechanism and performance monitoring parameters. By introducing the document state identifier, the dynamic routing algorithm and the node resource monitoring mechanism, accurate routing and efficient processing of the retrieval request are realized, so that the retrieval performance and the resource utilization rate of the system are improved.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to a distributed retrieval system, and more particularly to a retrieval routing optimization method and device. BACKGROUND

[0002] As a core technology in big data processing and storage systems, distributed retrieval systems are widely used in scenarios such as log analysis, search engines, real-time data queries, etc. In related technologies, through the collaborative work of the inverted index mechanism, data sharding strategy and replica synchronization mechanism, an efficient full-text retrieval architecture is constructed. Specifically, this system covers the entire process from document index import to the full process of retrieval request distribution, including document ID hash routing, master-slave shard read-write separation, translog and segment refresh mechanism, etc. With the continuous growth of data size, the traditional retrieval routing strategy gradually exposes performance bottlenecks, and it is urgent to introduce more refined routing control and resource scheduling mechanisms at the system level to improve overall retrieval efficiency and cluster stability.

[0003] However, in the existing retrieval routing method, a fixed routing strategy based on document ID hash is directly used, and dynamic routing judgment is not combined with document state identification and replica synchronization state, which may cause retrieval requests to be broadcast to all master-slave shards globally, causing unnecessary network and computing resource consumption. Specifically, before the document completes the translog or segment refresh, its replica may not have been synchronized, and if the retrieval request is randomly issued to the replica shard, it will cause data inconsistency problems and affect retrieval accuracy. In addition, due to the lack of dynamic perception and balanced scheduling of node resource states (such as CPU, memory, disk IO, network bandwidth, etc.), the system is prone to resource skew in high-concurrency scenarios, thereby reducing overall throughput and response efficiency. This static routing and resource allocation mechanism cannot adapt to large-scale and high-concurrency retrieval requirements, limiting the performance optimization space of the system. SUMMARY

[0004] The present disclosure aims to at least partially solve one of the technical problems in the related art.

[0005] The present disclosure proposes a retrieval routing optimization method based on a distributed retrieval system, which aims to solve the problems of resource waste, performance degradation, and request response delay increase in existing distributed retrieval systems when processing retrieval requests. By introducing document state identification, dynamic routing algorithm, and node resource monitoring mechanism, accurate routing and efficient processing of retrieval requests are achieved, thereby improving the retrieval performance and resource utilization of the system.

[0006] Another purpose of the present disclosure is to propose a retrieval routing optimization device based on a distributed retrieval system.

[0007] To achieve the above object, the application provides a retrieval route optimization method based on a distributed retrieval system, comprising:

[0008] obtaining configuration information of the distributed retrieval system; wherein the configuration information comprises an enabling state of a retrieval route function, resource monitoring indexes, a document state refreshing strategy and a node state reporting frequency;

[0009] in response to a retrieval request sent by a client, obtaining inverted metadata of a document based on the configuration information, wherein the inverted metadata comprises route information, shard information and a document storage identifier of the document; determining whether the document is a new document according to the document storage identifier, if the document is a new document, directing the retrieval request to a primary shard where the document is located; if the document is a historical document, selecting a primary shard or a secondary shard as a target shard according to resource states and availability of each node and shard in a current cluster;

[0010] determining an optimal route path of the retrieval request by an optimal execution shard formula based on the resource states and the availability; in response to the determination of the optimal route path, splitting the retrieval request into a plurality of sub-requests and directing and issuing the sub-requests to the target shard, the target shard performing a retrieval operation after receiving the sub-requests and returning a retrieval result to a coordination node; the coordination node integrating the retrieval result and returning the integrated retrieval result to the client.

[0011] The retrieval route optimization method based on the distributed retrieval system can further have the following additional technical features:

[0012] In an embodiment of the application, the inverted metadata recorder records route information, shard information and a document storage identifier of a document in a document indexing stage, and the document storage identifier is used to identify whether the document has completed data refreshing and is synchronized to all copies.

[0013] In an embodiment of the application, the resource states comprise at least one of CPU usage, memory usage, disk IO usage, network bandwidth usage, request quantity, query delay, retrieval delay, index delay and refreshing delay of each node.

[0014] In an embodiment of the application, the optimal execution shard formula is:

[0015] TargetRoute = argmin(S * (1 - CheckpointWeight * IsNewDocument) + ReplicaWeight * (1 - IsReplicaAvailable) + ShardStateWeight * (1 - IsShardOpen)),

[0016] wherein S is a node resource state score, CheckpointWeight is a document state weight, ReplicaWeight is a replica availability weight, ShardStateWeight is a shard state weight, and argmin represents a minimization function.

[0017] In an embodiment of the present application, the formula for calculating the node resource state score S is:

[0018] S = ∑(P_i * W_i) / ∑W_i

[0019] wherein P_i is the value of the i-th node performance indicator, and W_i is the weight of the i-th node performance indicator.

[0020] In an embodiment of the present application, the search request is split into multiple sub-requests and sent to the target shard in the form of batch requests to improve the request processing efficiency.

[0021] In an embodiment of the present application, the method further comprises providing default and customized search routing parameter indicators and performance analysis indicator settings through an information integration mode for dynamically adjusting the routing strategy and resource monitoring rules.

[0022] In an embodiment of the present application, the information integration mode supports configuring a routing function switch, a document state refresh strategy, a node state reporting frequency, and a performance indicator weight to adapt to search requirements in different business scenarios.

[0023] To achieve the above purpose, another aspect of the present application provides a search routing optimization device based on a distributed search system, comprising:

[0024] an information integration module configured to configure the enabling state of the search routing function, resource monitoring indicators, a document state refresh strategy, and a node state reporting frequency;

[0025] The inverted metadata recorder is used to obtain inverted metadata of a document according to configuration information of the information integration module in response to a retrieval request sent by a client, wherein the inverted metadata comprises routing information, shard information and document storage identification of the document; the document storage identification is used to determine whether the document is a new document, if the document is a new document, the retrieval request is directed to a primary shard where the document is located; if the document is a historical document, a primary shard or a secondary shard is selected as a target shard according to resource states and availability of each node and shard in a current cluster.

[0026] The retrieval routing control module determines an optimal routing path of the retrieval request by optimally executing a shard formula based on the resource states and availability; in response to determination of the optimal routing path, the retrieval request is split into a plurality of sub-requests, and the sub-requests are directed and issued to the target shard; the target shard executes a retrieval operation after receiving the sub-requests, and returns a retrieval result to the coordination node; the coordination node integrates the retrieval result, and returns the integrated retrieval result to the client.

[0027] The retrieval routing optimization method and device based on the distributed retrieval system according to the embodiments of the present application record routing information, shard information and document storage identification of a document in a document indexing stage, and dynamically select an optimal retrieval path according to a document state, availability of a copy and a node resource state in a retrieval stage, so as to realize accurate routing and load balancing of a retrieval request, and improve overall performance and stability of the system.

[0028] Additional aspects and advantages of the present application will be made apparent by the following description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0029] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0030] Figure 1 is a flowchart of a retrieval routing optimization method based on a distributed retrieval system according to an embodiment of the present application;

[0031] Figure 2 is a flowchart of another retrieval routing optimization method based on a distributed retrieval system according to an embodiment of the present application;

[0032] Figure 3 is a structure diagram of a retrieval routing optimization device based on a distributed retrieval system according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0034] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings and in combination with the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0035] A retrieval routing optimization method and device based on a distributed retrieval system according to an embodiment of the present application are described below with reference to the drawings.

[0036] The technical terms mentioned in the present application are introduced below.

[0037] Distributed retrieval system: full-text retrieval system, a distributed platform system based on inverted index technology and used for fast retrieval of data pages;

[0038] Data shard: data block of the full-text retrieval system, generally determined when the index initialization definition is determined and cannot be changed after determination;

[0039] Index master and secondary shard: data block and copy, both of which can provide index read task, and the master shard provides data write task.

[0040] Translog and segment refresh: both of which represent the result of data warehousing and copy synchronization, and after the refresh action is completed, the copies of the data block can be retrieved to the warehoused information by default.

[0041] Figure 1 The flowchart of the retrieval routing optimization method based on the distributed retrieval system according to the embodiment of the present application is shown in Figure 1 and Figure 2 , including:

[0042] S1, obtaining configuration information of the distributed retrieval system; wherein the configuration information includes the enabled state of the retrieval routing function, the resource monitoring index, the document state refresh strategy and the node state reporting frequency.

[0043] In an embodiment of the present application, step S1 is used for information configuration of the distributed retrieval system, and the configured information includes but is not limited to the enabled state of the retrieval routing function, the resource monitoring index, the document state refresh strategy and the node state reporting frequency.

[0044] In an embodiment of the present application, the information configuration is based on the information integration to achieve the related configuration switch processing, performance index configuration, index parameter basic integration analysis, etc. of the entire search routing flow.

[0045] It can be understood that the search routing control enable switch is used to configure whether the system uses the search routing function, the new document data identification binding module is used to configure the refresh dependent module of the new document data identification, the cluster performance resource monitoring parameter configuration is used to set the performance parameters of the disk IO, memory, CPU, network, single node running read task and write task, etc. to determine the performance evaluation parameters used, the state and information monitoring, summarization and integration of the performance monitoring elements are used to summarize the resource state of the evaluation node to assist the dynamic adjustment of the query copy of the index routing algorithm, the performance integration rule parameter integration is used to evaluate the resource pressure of the node and cluster to balance the search task, etc.

[0046] Specifically, the information integration of the embodiment of the present application mainly functions to configure the parameter configuration and control information monitoring mode in the search routing flow, the non-self-defined scene runs with the default default configuration, and the user can also independently configure according to the cluster or business situation of the user, so that the search routing can run in a more matched configuration and mode of the actual scene of the cluster, such as the configuration content mainly including the search routing switch constraint, so that the user can enable the routing search according to the demand, the self-defined resource monitoring index such as the average value standard and peak value standard of the CPU, memory, disk IO and network bandwidth occupation of the cluster level and node level, the search routing shard adjustment can be performed according to the actual node resource and business situation during the routing search, and the checkpoint refresh configuration of the document warehouse identification can be configured according to the cluster pressure state to configure the frequency and time node of the identification refresh to ensure the stable operation of the cluster. It can be understood that the information integration steps of the embodiment of the present application include but are not limited to the following sub-steps:

[0047] S101, search routing enable switch rout_index = on / off, search routing function switch parameter, default off state, open search routing function effective;

[0048] S102, checkpoint refresh configuration of the document warehouse identification, default following translog refresh, if the cluster performance is good or the available resource is more, the self-adjustment setting can be set to follow the segment refresh;

[0049] S103, define the main resource and behavior analysis parameter indicators, such as defining the maximum IOPS of the node disk and defining the disk resource usage standard as the IOPS percentage value of read-write IO, defining the usage percentage value of memory / CPU / network bandwidth, node running pressure state such as total request quantity and Query latency query delay / Fetch latency extraction delay / Indexing latency indexing delay, i.e. write delay / Flush latency data flushing delay, etc. Performance reference indicators;

[0050] S104, define the rule constraints of information analysis, such as the indicators of memory, network, CPU, disk IO contain instantaneous indicators and interval indicators, and the overall indicator group is composed according to the weight algorithm, such as the average value of node memory / CPU / network bandwidth / disk IO occupancy is less than 70%, the peak value of memory / CPU / network bandwidth / disk IO within 1 minute does not exceed 90%, and the latency related parameters do not exceed the congestion threshold, that is, it can be determined as a usable business node, and all node resource consumption comparison analysis can be provided.

[0051] The information integration step of the embodiment of the application provides default and self-defined search routing parameter indicator and performance analysis indicator settings, provides data basis and analysis basis for search routing optimization. At the same time, the dynamic resource information and analysis strategy configuration of the cluster are collected and integrated, so as to ensure that the configuration of the search routing process is more adaptive and effective. That is, the default and self-configuration entrance of various parameters and strategies required in the search routing optimization process is provided, which is used for configuring all runtime resource configurations, resource rolling analysis strategies, checkpoint flushing strategies, information interaction reporting frequency, etc. in the process, and each parameter strategy can be freely adjusted according to the cluster and environment state, so as to ensure that the search routing process runs in the optimal way.

[0052] S2, in response to the search request sent by the client, obtaining the inverted metadata of the document based on the configuration information, wherein the inverted metadata contains the routing information, the sharding information and the document storage identifier of the document; determining whether the document is a new document according to the document storage identifier, if it is a new document, directing the search request to the master shard where the document is located; if it is a historical document, selecting a master shard or a secondary shard as a target shard according to the resource state and availability of each node and shard in the current cluster.

[0053] In an embodiment of the present application, the inverted metadata recorder can be used to respond to the retrieval request sent by the client. The inverted metadata recorder of the embodiment of the present application is mainly introduced based on the original inverted information record for new routing, sharding and document warehouse identification checkpoint. Since the explicit data location state information is added in the index inverted metadata, the basic data location analysis and retrieval routing adaptation can be quickly performed during retrieval, and the data processing performance is improved.

[0054] It can be understood that the inverted metadata recorder records the routing sharding information and the document warehouse identification while recording the inverted information of the index document based on the index process, and refreshes the document warehouse identification according to the configured location. For example, when the document index_a is indexed, the index_a is split according to the word segmentation rule and inverted data is generated, the routing information and the sharding information of the document are added in the inverted record, and the warehouse identification checkpoint of the document is recorded. The routing information and the sharding information of the document are determined when the document is stored in the shard, and the warehouse identification is updated when the data is written according to the configured segment or translog refresh warehouse. Based on the inverted information and the warehouse identification, the basic routing and sharding information of the warehoused document data can be recorded, and the warehouse identification of the refreshed data is identified to identify the retrieval path of the new and old data.

[0055] Specifically, the inverted metadata recorder mainly processes and synchronizes the information such as retrieval routing, sharding, document warehouse identification based on the inverted mechanism of the original distributed retrieval system. During the indexing of all documents, after the stored index is split based on the word segmenter and the like, the original routing information and the sharding information of the document, and the state of the warehouse checkpoint are initialized before the inverted index table is warehoused. These warehouse information parameters will occupy a small amount of node performance during warehousing, but can effectively balance the retrieval task distribution of the retrieval cluster through relevant recording information during the subsequent retrieval process, and optimize the efficiency and stability of the retrieval task operation. The checkpoint needs to refresh the record state, and the initial warehouse in is converted into flush following the refresh mechanism of the translog or segment, so as to change the identification state and be used for dynamic allocation of retrieval routing. The operation of the inverted metadata recorder based on the embodiment of the present application in response to the retrieval request sent by the client based on the configuration information includes but is not limited to the following sub-steps:

[0056] S201, inverted routing metadata, when the document is indexed, the document is allocated for sharding storage routing, the relevant routing information calculated according to the original hash is recorded, and is stored in the inverted table of the document to establish a subordinate relationship with the document id, so as to facilitate the main routing guidance during retrieval;

[0057] S202, inverted slice metadata, after the document index routing is generated, the document is located in a specific slice, the slice position of the document index is recorded, and the inverted table of the document is also stored to establish a subordinate relationship with the document id, which is used for identifying the node position information of the slice and analyzing the retrieval task volume when the retrieval routing is rebroadcast;

[0058] S203, storage identification checkpoint, when the inverted metadata of the index is recorded, the initialization value in is stored, which is used to identify that the index data is in the initial storage state, and the data is temporarily in the unsafe state of segment or translog not flushed, and it is ensured that the master slice exists index data, and only when the checkpoint is flushed to the flush state, it represents that the data has been flushed to all replicas, and can be marked as historical data;

[0059] S204, checkpoint flushing, as the index data is stored and landed, the checkpoint stage is obtained from the information integration module segment or translog, so that when the index data is flushed from segment or translog, the index storage identification of the same batch is updated in batches to flush, which represents that the data has been completed flushing to the disk, and is synchronized to all secondary shards, and can be queried in the secondary shards.

[0060] The inverted metadata recorder of the embodiment of the application introduces new routing, slicing and document storage identification checkpoint based on the original inverted information recording. Since the explicit data position state information is added in the index inverted metadata, the basic data position analysis and retrieval routing adaptation can be quickly performed during retrieval, and the data processing performance is improved.

[0061] The initial metadata recording and update maintenance can also be performed according to the actual application scene, so as to provide accurate analysis basic data information for subsequent retrieval routing algorithm. Therefore, the metadata is recorded and flushed according to the actual data application scene and data state, the routing algorithm is assisted to perform accurate routing analysis calculation, so as to improve the calculation speed and accuracy of the routing algorithm.

[0062] S3, based on the resource state and availability, determine the optimal routing path of the retrieval request by the optimal execution slicing formula; in response to the determination of the optimal routing path, split the retrieval request into multiple sub-requests, and direct the sub-requests to the target slice; the target slice performs a retrieval operation after receiving the sub-requests, and returns the retrieval result to the coordination node; the coordination node integrates the retrieval result, and returns the integrated retrieval result to the client.

[0063] It can be understood that the retrieval routing control is directly located to the routing information, the shard information and the document storage identifier of the relevant document data by the inverted record information when the client initiates the retrieval process. The retrieval routing control device can perform shard routing control according to the routing and shard information, in combination with the resource status of the current cluster and the node. The selection of the shard master and the shard replica of the retrieval information position is mainly controlled according to the resource health status of the node and the availability or enablement status of the shard. For part of the data with the storage identifier, the shard is located according to the in / flush state of the storage identifier checkpoint. The in state represents that the data storage binding stage is segment or translog for flush. At this time, the data is not flushed to the disk. There may be differences between the master shard and the replica shard. The data needs to be sent to the master shard for retrieval. The flush state data represents that the data is synchronized for storage. The master shard and the replica shard are consistent. The data can be directed to any corresponding master shard or replica shard for retrieval according to the demand. In this way, the shard retrieval demand and the node performance use of the cluster can be balanced, and the normal resources and performance space required for the operation of the node business can be protected in time.

[0064] In an embodiment of the present application, the retrieval routing control is used for the metadata routing of the comprehensive cluster, the retrieval request state and the comprehensive performance distribution of all nodes of the cluster. The request sent by the client is directed for retrieval routing. The request path is balanced according to the shard node distribution state and the cluster node resource state during retrieval, so as to reduce the business pressure of the resource inclined node. The whole process starts from the client initiating the retrieval process. The request is located to the inverted index metadata through the retrieval condition. After the responsive document information and the routing shard information are obtained, the best retrieval path of the retrieval request is calculated through the sub-routing algorithm according to the shard distribution, the document data amount of the node where the shard is located, the node resource status and the like. The index position of the document data is directed. The overall rule follows the optional shard of the historical data. The data with the checkpoint identifier in needs to be inquired to the master shard where the index data is located, so as to ensure the reliability of the document query. The retrieval routing control device is assumed by any one coordination node. The device is usually operated on the client receiving the request. The resource collection, analysis and processing of all nodes depend on the operation of each node. The analysis data of each node needs to be synchronized to the routing control device of all coordination nodes according to the polling_detection interval, so as to ensure the reliability of the performance data. Specifically, the retrieval routing control steps of the embodiment of the present application include but are not limited to the following sub-steps:

[0065] S301, each node collects and analyzes node basic information, including all monitoring indicators defined by the information integration module, including all data information such as memory / CPU / network bandwidth / disk IO / request quantity and Query latency query delay / Fetch latency extraction delay / Indexing latency indexing delay / Flush latency data flushing delay;

[0066] S302, each node analyzes the basic performance indicators according to the performance data collected by the node, including extracting the mean value and peak value in the period of all memory / CPU / network bandwidth / disk IO / request quantity and Query latency query delay / Fetch latency extraction delay / Indexing latency indexing delay / Flush latency data flushing delay indicators, and archiving and packaging the analysis results information according to the definition standard of the information integration module, and synchronizing to all coordination nodes, the synchronization frequency is configured by the polling_detection parameter refresh interval, to ensure the timeliness and reliability of the data;

[0067] S303, the retrieval routing control device of the coordination node synchronously receives the performance indicator information of each node in real time, when there is a retrieval request of the client in the coordination node, the retrieval inverted metadata table is called according to the retrieval condition, the retrieval shard information is preliminarily integrated, including all data shards, available primary shards, replica shards and the like (filtering unassigned, closed and the like unavailable shard information);

[0068] S304, the request is split into sub-requests, that is, the retrieval request usually contains one or more retrieval conditions, such as multiple word retrieval of the same retrieval content, which is split into multiple single word requests, and converted into bulk request for batch delivery;

[0069] S305, for each retrieval request, the dynamic information data collected in advance is comprehensively used to process the retrieval routing algorithm, and the optimal execution shard formula of the request is located:

[0070] TargetRoute=argmin(S\*(1-

[0071] CheckpointWeight\*IsNewDocument)+ReplicaWeight\*(1-

[0072] IsReplicaAvailable)+ShardStateWeight\*(1-IsShardOpen))

[0073] For example, the routing formula parameters: S: node resource status, the calculation formula is: S = ∑(P_i\*W_i) / ∑W_i; CheckpointWeight: checkpoint weight, determined according to the checkpoint value (in or flush); IsNewDocument: whether it is a new document (checkpoint value is in); ReplicaWeight: replica shard weight, determined according to the availability of the replica shard; IsReplicaAvailable: whether the replica shard is available; ShardStateWeight: shard state weight, determined according to the shard state (open, close, UNassign); IsShardOpen: whether the shard is in the open state; argmin: minimization function.

[0074] For example, the node resource performance state formula:

[0075] S = ∑(P_i\*W_i) / ∑W_i

[0076] P_i: the value of node performance indicator i; W_i: the weight of node performance indicator i;

[0077] Wherein, the node basic information: N = {M, C, N, D, R, QL, FL, IL, SL};

[0078] M: memory usage, C: CPU usage, N: network bandwidth usage, D: disk IO usage, R: request quantity, QL: Query latency (query delay), FL: Fetch latency (fetch delay), IL: Indexing latency (indexing delay);

[0079] Wherein, the node performance indicator: P = {μ_M, μ_C, μ_N, μ_D, μ_R, μ_QL, μ_FL, μ_IL, μ_SL}; μ_X: the mean value of indicator X;

[0080] Wherein, the node performance indicator weight: W = {w_M, w_C, w_N, w_D, w_R, w_QL, w_FL, w_IL, w_SL}; w_X: the weight of indicator X (according to information integration configuration).

[0081] For example, other parameter explanations:

[0082] CheckpointWeight = 1, if the checkpoint value is in (new document) CheckpointWeight = 0, if the checkpoint value is flush (data has been synchronized to all replica shards);

[0083] ReplicaWeight = 1 if replica is available, ReplicaWeight = 0 if replica is not available;

[0084] ShardStateWeight = 1 if shard state is open, ShardStateWeight = 0 if shard state is close or UNassign.

[0085] S306, according to the routing algorithm result, the document index extraction of the corresponding master and replica shards is guided to all split retrieval requests, and is returned to the coordination node for retrieval result integration, and finally returned to the client.

[0086] The retrieval routing control of the embodiment of the application controls according to the resource strategy control and the inverted metadata record related to the retrieval routing of information integration, optimizes the split client retrieval request on the basis of dynamically monitoring the node resource state, and guides the retrieval request based on the optimal routing algorithm, so that the retrieval process is fast and efficient, and the business performance distribution of the cluster can be better balanced, the performance efficiency of the cluster is optimized, and the resource utilization is improved.

[0087] Meanwhile, the retrieval routing control performs comprehensive resource information dynamic analysis and retrieval routing generation according to the resource configuration, analysis strategy and inverted metadata record update information pre-configured, accurately guides and efficiently searches the retrieval demand of the client, saves resource consumption while ensuring the fast and efficient operation of the retrieval demand, improves the retrieval capacity of the cluster, and optimizes the operation performance and resource use. Therefore, the retrieval request is accurately balanced and optimized control processing through retrieval request splitting, accurate routing algorithm calculation and retrieval path shunting, the retrieval efficiency is improved, the resource use is better optimized, and the performance of the cluster is improved.

[0088] In summary, the information integration of the embodiment of the present application is mainly used for searching the basic dependent index and data analysis strategy of the route, used for comprehensively configuring the basic application scene parameter configuration of the cluster, and used for collecting and analyzing the index data of the performance data of the basic cluster by applying the configuration, and finally, the integrated data is used for searching the route strategy reference of the control module. The inverted metadata recorder is used for recording and refreshing the inverted metadata, the route, the sharding information, and the checkpoint information, and is used for recording and identifying the basic outer route information and the data state information of the original document index, so that the basic information can be acquired conveniently and quickly during the searching route processing. The searching route control is used for performing the operations such as the cluster basic performance information summarization, the searching request splitting, the quick route calculation, and the request data processing based on the information integration module and the inverted metadata recording information, so that the corresponding data is positioned and searched in the quick route mode, the searching efficiency is improved, and the performance resource consumed by the searching is optimized.

[0089] According to the searching route optimization method based on the distributed searching system of the embodiment of the present application, the searching route strategy of the distributed searching system is dynamically and directionally optimized, the request searching activity is more efficient according to the real-time running state of the cluster by optimizing the index inverted information and the searching route algorithm, the searching performance of the searching cluster is improved, and the resource utilization of the cluster is optimized. And the searching speed of the searching system is improved by the control process, the performance running stability of the cluster is ensured, and especially in the large-scale scene, the resource utilization rate and the running efficiency of the cluster are ensured.

[0090] In order to realize the above-mentioned embodiment, as shown in Figure 3 The embodiment also provides a searching route optimization device 10 based on a distributed searching system, which comprises:

[0091] The information integration module 100 is used for configuring the enabling state of the searching route function, the resource monitoring index, the document state refreshing strategy, and the node state reporting frequency.

[0092] The inverted metadata recorder 200 is used for acquiring the inverted metadata of the document according to the configuration information of the information integration module in response to the searching request sent by the client, wherein the inverted metadata comprises the route information, the sharding information, and the document storage identification of the document; the document is judged to be a new document according to the document storage identification, if the document is a new document, the searching request is directed to the master shard where the document is located; if the document is a historical document, the master shard or the slave shard is selected as the target shard according to the resource state and the availability of each node and shard in the current cluster.

[0093] The search routing control module 300 determines an optimal routing path of the search request by executing a sharding formula based on the resource state and availability, splits the search request into a plurality of sub-requests in response to the determination of the optimal routing path, and directs the sub-requests to the target shards, the target shards perform a search operation after receiving the sub-requests and return a search result to the coordination node, and the coordination node integrates the search result and returns the integrated search result to the client.

[0094] Further, the inverted metadata recorder 200 is connected with the information integration module 100 to determine a refresh time of a document storage identification according to a configured document state refresh strategy, the refresh strategy including following a translog refresh or following a segment refresh.

[0095] The search routing optimization device based on the distributed search system according to the embodiment of the application dynamically optimizes the search routing strategy of the distributed search system, performs more efficient request search activities according to the real-time running state of the cluster through index inverted information optimization and search routing algorithm optimization, and improves the search performance of the search cluster and optimizes the resource utilization of the cluster. The control process can improve the search speed of the search system, ensure the stable performance of the cluster, and especially in a large-scale scenario, ensure the resource utilization and operation efficiency of the cluster.

[0096] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0097] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

Claims

1. A retrieval routing optimization method based on a distributed retrieval system, characterized in that, Includes the following steps: Obtain the configuration information of the distributed retrieval system; wherein, the configuration information includes the activation status of the retrieval routing function, resource monitoring indicators, document status refresh strategy, and node status reporting frequency; In response to a retrieval request sent by the client, the inverted metadata of the document is obtained based on the configuration information. The inverted metadata includes the document's routing information, shard information, and document entry identifier. Based on the document entry identifier, it is determined whether the document is a new document. If it is a new document, the retrieval request is directed to the primary shard where the document is located. If it is a historical document, the primary shard or secondary shard is selected as the target shard based on the resource status and availability of each node and shard in the current cluster. Based on the resource status and availability, the optimal routing path for the retrieval request is determined using the optimal execution sharding formula. In response to the determination of the optimal routing path, the retrieval request is split into multiple sub-requests, and the sub-requests are directed to the target shard. Upon receiving the sub-requests, the target shard performs a retrieval operation and returns the retrieval results to the coordinating node. The coordinating node integrates the retrieval results and returns the integrated retrieval results to the client.

2. The method according to claim 1, characterized in that, The inverted metadata recorder records the document's routing information, shard information, and document entry identifier during the document indexing stage. The document entry identifier is used to identify whether the document has completed data refresh and synchronization to all copies.

3. The method according to claim 1, characterized in that, The resource status includes at least one of the following: CPU utilization, memory utilization, disk I / O utilization, network bandwidth utilization, number of requests, query latency, retrieval latency, index latency, and refresh latency for each node.

4. The method according to claim 1, characterized in that, The optimal execution partitioning formula is: TargetRoute=argmin(S\*(1-CheckpointWeight\*IsNewDocument)+ReplicaWeight\*(1-IsReplicaAvailable)+ShardStateWeight\*(1-IsShardOpen)), Where S is the node resource status score, CheckpointWeight is the document status weight, ReplicaWeight is the replica availability weight, ShardStateWeight is the shard status weight, and argmin represents the minimization function.

5. The method according to claim 4, characterized in that, The formula for calculating the node resource status score S is: S=∑(P_i×W_i) / ∑W_i Where P_i is the value of node performance index i, and W_i is the weight of node performance index i.

6. The method according to claim 1, characterized in that, The retrieval request is split into multiple sub-requests and sent to the target shard in batches to improve request processing efficiency.

7. The method according to claim 1, characterized in that, The method further includes: providing default and custom retrieval routing parameter indicators and performance analysis indicator settings through an information integration mode, for dynamically adjusting routing strategies and resource monitoring rules.

8. The method according to claim 7, characterized in that, The information integration mode supports configuring routing function switches, document status refresh strategies, node status reporting frequency, and performance indicator weights to adapt to retrieval needs in different business scenarios.

9. A retrieval route optimization device based on a distributed retrieval system, characterized in that, include: The information integration module is used to configure the activation status of the retrieval routing function, resource monitoring indicators, document status refresh strategy, and node status reporting frequency. The inverted metadata logger is used to respond to retrieval requests sent by clients. It obtains the inverted metadata of a document based on the configuration information of the information integration module. The inverted metadata includes the document's routing information, shard information, and document entry identifier. Based on the document entry identifier, it determines whether the document is a new document. If it is a new document, the retrieval request is directed to the primary shard where the document is located. If it is a historical document, the primary or secondary shard is selected as the target shard based on the resource status and availability of each node and shard in the current cluster. The retrieval routing control module determines the optimal routing path for the retrieval request based on the resource status and availability using an optimal execution sharding formula. In response to the determination of the optimal routing path, the retrieval request is split into multiple sub-requests, and the sub-requests are directed to the target shard. Upon receiving the sub-requests, the target shard performs a retrieval operation and returns the retrieval results to the coordinating node. The coordinating node integrates the retrieval results and returns the integrated retrieval results to the client.

10. The apparatus according to claim 9, characterized in that, The inverted metadata logger is connected to the information integration module to determine the refresh timing of the document entry identifier according to the configured document status refresh strategy. The refresh strategy includes refreshing with the translog or refreshing with the segment.

Citation Information

Cited By

  • Data routing method and device based on metadata index, equipment and medium

    CN121681614A

  • Data routing methods, devices, equipment, and media based on metadata indexes

    CN121681614B