Speed precision balance hybrid retrieval method and system based on relevancy
By adopting a phased search strategy and weighted fusion results method in the search system, combined with the adjustment of the equilibrium factor α, the problem of difficulty in finding a balance between speed and accuracy in the prior art is solved, and flexible search results output is achieved.
Patent Information
- Application Number
- CN202411386383.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-02-14
AI Technical Summary
Existing search systems are difficult to find a flexible balance between speed and accuracy, and cannot effectively meet the personalized needs of different users for speed and accuracy.
The speed-precision balanced hybrid search method based on correlation is adopted. The preliminary screening is performed with a fast but low-precision algorithm first, and then the secondary search is performed with a slow but high-precision algorithm, and the balance factor is introduced through weighted superimposition and fusion results. α to adjust the search strategy.
It is realized that roughly relevant result sets are given at the first time and then deep processing is carried out to obtain both fast and accurate search results, which enhances the flexibility and adaptability of the search system.
Smart Images
Figure CN119441381A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of retrieval technology, and in particular to a speed-accuracy balanced hybrid retrieval method and system based on relevance. Background Art
[0002] In the digital age, the amount of information is growing exponentially, and retrieval systems have become an essential tool for us to obtain knowledge and data. An efficient retrieval system can ensure that we can quickly locate the required content in the vast ocean of information, which is crucial for various fields such as academic research, business decision-making, and technological development. It not only improves the efficiency of information acquisition, but also helps us extract valuable insights from data and promote the innovation and application of knowledge. Many retrieval systems use a single retrieval strategy, such as relying only on keyword matching or statistical-based methods. Although these methods can return results quickly, they often lack deep semantic understanding, resulting in low relevance of the returned results or missing potentially important information. In addition, a single retrieval strategy is difficult to meet the dual requirements of speed and accuracy at the same time. For example, although vector-based semantic retrieval has high accuracy, it has high computational cost and slow speed; while traditional keyword matching methods are fast but lack accuracy. These deficiencies limit the effectiveness of retrieval systems in practical applications.
[0003] Nowadays, with the continuous improvement of users' personalized demand for retrieval, different users have different requirements for speed and accuracy, which has become a problem that needs to be measured and adjusted in retrieval. Although some hybrid retrieval technologies have emerged, which also consider the balance between speed and accuracy, they are more inclined to a single optimization speed or a single optimization rate, or to seek a fixed balance between the two. This does not solve the problem of different user needs and is not conducive to user retrieval personalization.
[0004] In view of this, a speed-accuracy balanced hybrid retrieval method and system based on relevance is needed. Summary of the invention
[0005] In view of the fact that the existing technology has not considered the problem of hybrid retrieval and the flexible adjustment of the speed and accuracy of the retrieval results, the present invention provides a hybrid retrieval method and system based on the balance of speed and accuracy of relevance, which can effectively combine the advantages of speed and accuracy by first using a fast but low-precision algorithm for preliminary screening, and then using a slow but high-precision algorithm for secondary retrieval. This phased retrieval strategy can give a rough set of relevant results at the first time, and then further process these results through a more accurate algorithm, and finally obtain a retrieval result that is both fast and accurate. More importantly, by introducing the balance factor α, users can adjust the retrieval strategy according to specific needs. By simply changing one parameter, they can make a trade-off between speed and accuracy, making the retrieval system more flexible and adaptable. The specific technical solution is as follows:
[0006] A speed-accuracy balance hybrid retrieval method based on relevance includes the following steps:
[0007] A preliminary search is performed through a first-class search algorithm to obtain a first-class search result, and based on the first-class search result, a second-class search is performed through a second-class search algorithm to obtain a second-class search result;
[0008] The first-category search results and the second-category search results are merged by weighted superposition to obtain the final search results, wherein the weights of the first-category search results and the second-category search results are both provided with a balance factor for adjusting the proportion of speed and accuracy;
[0009] The relevance of each final search result is calculated, and graded based on the set relevance threshold, and finally the search results are output in order of level.
[0010] Preferably, the first type of retrieval algorithm uses BM25, inverted index, TF-IDF, hash search, binary search or approximate nearest neighbor search, and the second type of retrieval algorithm uses HNSW, Faiss, Exact k-NN, Proxima, NSG, EFANNA or NGT.
[0011] Preferably, the fusion is performed by weighted superposition as follows:
[0012] R fusion =w fast ·R fast +w slow ·R slow ;
[0013] In the formula, R fusion represents the fused result set, w fast is the weight coefficient of a type of retrieval algorithm, R fast is the output result set of a type of retrieval algorithm, w slow is the weight coefficient of the two-category retrieval algorithm, R slowIt is the output result set of the second-category retrieval algorithm.
[0014] Preferably, the weights of the first-class search results and the second-class search results are both provided with a balancing factor for adjusting the proportion of speed and accuracy, and the balancing factor is added to the weight as follows:
[0015]
[0016] In the formula, w fast is the weight coefficient of a type of retrieval algorithm, w slow is the weight coefficient of the two-category retrieval algorithm, S fast is the relevance score of the output results of a type of retrieval algorithm, S slow is the relevance score of the output results of the two-category retrieval algorithm, and α is the balance factor.
[0017] Preferably, the relevance score is as follows:
[0018]
[0019] In the formula, S fast is the relevance score of the output results of a type of retrieval algorithm, S slow is the relevance score of the output results of the two-category retrieval algorithm, Precision fast is the precision of a class of retrieval algorithms, slow is the precision of the two-category retrieval algorithm, Recall fast is the recall rate of a retrieval algorithm. slow is the recall rate of the two-category retrieval algorithm.
[0020] Preferably, the relevance of the final search result is obtained based on a relevance score or cosine similarity, Euclidean distance, Manhattan distance, Jaccard similarity coefficient or point mutual information.
[0021] A speed-accuracy balance hybrid retrieval system based on relevance, applied to the above method, comprises:
[0022] A type of search module, which performs a preliminary search through a type of search algorithm and obtains a type of search results;
[0023] The second-class search module further searches the first-class search results through the second-class search algorithm to obtain the second-class search results;
[0024] The weighted fusion module fuses the first-category search results with the second-category search results through weighted superposition to obtain the final search results. The weights of the first-category search results and the second-category search results are both equipped with a balance factor for adjusting the proportion of speed and accuracy.
[0025] The hierarchical sorting module calculates the relevance of each final search result, and performs hierarchical processing based on the set relevance threshold, and finally outputs the search results in order of level.
[0026] Preferably, the first type of retrieval algorithm uses BM25, inverted index, TF-IDF, hash search, binary search or approximate nearest neighbor search, and the second type of retrieval algorithm uses HNSW, Faiss, Exact k-NN, Proxima, NSG, EFANNA or NGT.
[0027] A computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the speed-accuracy balanced hybrid retrieval method based on relevance as described above.
[0028] A processor, characterized in that the processor is used to run a program, wherein when the program is running, the speed-accuracy balanced hybrid retrieval method based on relevance as described above is executed.
[0029] Compared with the prior art, the present invention has the following beneficial effects:
[0030] The present invention can effectively combine the advantages of speed and accuracy by first using a fast but inaccurate algorithm for preliminary screening, and then using a slow but accurate algorithm for secondary retrieval. This phased retrieval strategy can give a rough set of relevant results at the first time, and then further process these results through a more accurate algorithm, ultimately obtaining fast and accurate retrieval results. More importantly, by introducing the balance factor α, users can adjust the retrieval strategy according to specific needs. By simply changing one parameter, they can make a trade-off between speed and accuracy, making the retrieval system more flexible and adaptable. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.
[0032] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION
[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0034] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0035] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0036] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0037] In one embodiment of the present invention, a speed-accuracy balance hybrid retrieval method based on relevance is provided. Figure 1 As shown, the following steps are included:
[0038] Step 1: Perform a preliminary search using a first-class search algorithm and obtain a first-class search result. Based on the preliminary search result, perform a further search using a second-class search algorithm to obtain a second-class search result.
[0039] Among them, the first type of search algorithms include BM25, inverted index, TF-IDF, hash search, binary search, and approximate nearest neighbor search (ANN). According to the actual application needs, one of these algorithms can be selected for use. The characteristics of this type of algorithm are fast but not particularly high accuracy. Its main function is to implement preliminary queries to obtain a relatively broad and low-precision search range, which is convenient for subsequent second-type searches (slow but high-precision).
[0040] The following is a brief introduction to several first-class retrieval algorithms.
[0041] First is BM25, the retrieval steps are as follows:
[0042] 1. Get the document collection and user query, build an inverted index, and record all documents where each word appears.
[0043] 2. Calculate the IDF (Inverse Document Frequency) value of the query term and the document.
[0044] 3. For each query word, calculate its TF (term frequency) in each document.
[0045] 4. Use the BM25 formula to calculate the score of the query term and each document.
[0046] 5. Sort the documents according to their scores.
[0047] Inverted index operation steps:
[0048] 1. Segment the document collection.
[0049] 2. Build an inverted index, including terms and corresponding document lists.
[0050] 3. Segment the user's query.
[0051] 4. Find the list of documents for each query term in the inverted index.
[0052] 5. Merge document lists and remove duplicates.
[0053] 6. Sort the results by relevance.
[0054] TF-IDF operation steps:
[0055] 1. Calculate the TF (term frequency) of each word in the document collection.
[0056] 2. Calculate the IDF (Inverse Document Frequency) of each word.
[0057] 3. Multiply TF and IDF to get the weight of each word in each document.
[0058] 4. Use the same TF-IDF calculation method for the words in the user query.
[0059] 5. Compare the query vector and document vector and calculate the similarity.
[0060] 6. Sort documents by similarity.
[0061] Hash search operation steps:
[0062] 1. Calculate hash values for keywords in the document.
[0063] 2. Map the hash values to the slots in the hash table.
[0064] 3. Calculate the hash value of the keywords queried by the user.
[0065] 4. Find the corresponding slot in the hash table.
[0066] 5. Return the list of documents in the slot.
[0067] Binary search operation steps:
[0068] 1. Make sure the document collection is sorted.
[0069] 2. Set the low and high pointers of the search range.
[0070] 3. Calculate the middle position and compare the query term with the middle term.
[0071] 4. If there is a match, the search is successful; if there is no match, the search range is adjusted based on the comparison result.
[0072] 5. Repeat steps 3 and 4 until a match is found or the search range is invalid.
[0073] Approximate nearest neighbor search (ANN) operation steps:
[0074] 1. Use hashing techniques to map data points into a low-dimensional space.
[0075] 2. Build a hash table, with each bucket storing data points with similar distances.
[0076] 3. Hash the query point and find the corresponding bucket.
[0077] 4. Find the most similar data point within the bucket.
[0078] Among the above algorithms, BM25 and inverted index are particularly suitable for fast retrieval of text data, while hash search and binary search are suitable for situations with specific data structures (such as hash tables or ordered arrays). TF-IDF is suitable for scenarios where document weights need to be calculated quickly. ANN is suitable for approximate search in high-dimensional space, which can sacrifice a little accuracy while ensuring relatively high speed. Choose one to implement preliminary retrieval and preliminary data screening according to actual application needs.
[0079] The second-class search algorithms include: HNSW, Faiss, Exact k-NN, Proxima, NSG, EFANNA, and NGT. The main function of these second-class search algorithms is to achieve more refined search, so that the search results meet customer needs in terms of semantics and other aspects.
[0080] The following are several retrieval algorithms with high accuracy but low speed, as well as their specific operation steps:
[0081] HNSW (Hierarchical Navigable Small World) operation steps:
[0082] 1. Build a multi-layer graph structure, each layer is a small-world network.
[0083] 2. Each node is connected to its neighbors in each layer of the graph.
[0084] 3. Search in a hierarchical manner, first perform a rough search at a high level, then perform a precise search at a low level.
[0085] 4. While searching, quickly jump to other areas by jumping sideways.
[0086] Faiss (Facebook AI Similarity Search) operation steps:
[0087] 1. Use inverted index and quantization techniques to build indexes.
[0088] 2. For each query, first search for candidate sets in the quantized index.
[0089] 3. Perform secondary retrieval in the candidate set using exact distance calculation.
[0090] 4. Return the most similar documents.
[0091] Exact k-NN operation steps:
[0092] 1. For each query, calculate the distance to all documents.
[0093] 2. Sort by distance and select the k closest documents.
[0094] Proxima operation steps:
[0095] 1. Use a composite index structure that combines multiple retrieval algorithms.
[0096] 2. Support streaming indexing and online updates.
[0097] 3. Improve retrieval efficiency through heterogeneous computing.
[0098] 4.Support joint retrieval of tags and vectors.
[0099] NSG (Navigating Small World) operation steps:
[0100] 1. Build a small-world network where each node is connected to its neighbors.
[0101] 2. Search by random walk.
[0102] 3. While searching, quickly jump to other areas by jumping sideways.
[0103] EFANNA operation steps:
[0104] 1. Construct an approximate k-nearest neighbor graph.
[0105] 2. Use a greedy algorithm to search.
[0106] 3. Quickly locate possible neighbors through pre-computed candidate sets.
[0107] NGT (Neighborhood Graph Technology)* Operation steps:
[0108] 1. Build an index based on the nearest neighbor graph.
[0109] 2. Use graph traversal algorithm to search.
[0110] 3. Quickly locate neighbors through graph connectivity.
[0111] Among these algorithms, HNSW and Faiss are particularly suitable for approximate search of large-scale data sets, Proxima achieves a good balance between accuracy and performance, and Exact k-NN provides high-precision but slower search. NSG, EFANNA, and NGT perform well in specific scenarios.
[0112] Since the first-class search and the second-class search each have their own advantages, the first-class search is fast but not accurate, while the second-class search is accurate but not fast. In actual applications, some users need search speed and can sacrifice the corresponding search accuracy, some users need search accuracy and can ignore search speed, and some users need a balance between the two. In other words, different users have different standards and requirements for search speed and accuracy. How to balance and adjust the search speed and accuracy is also the problem to be solved at the end of this embodiment. Based on this, step two is provided.
[0113] Step 2: The first-category search results and the second-category search results are weighted and superimposed to obtain the final search results, wherein the weights of the first-category search results and the second-category search results are both provided with a balance factor for adjusting the proportion of speed and accuracy;
[0114] The two types of search results are merged as follows:
[0115] R fusion =w fast ·R fast +w slow ·R slow ;
[0116] In the formula, R fusion represents the fused result set, w fast is the weight coefficient of a type of retrieval algorithm, R fastis the output result set of a type of retrieval algorithm, w slow is the weight coefficient of the two-category retrieval algorithm, R slow It is the output result set of the second-category retrieval algorithm.
[0117] Among them, the weights of the first-class search results and the second-class search results are both equipped with a balance factor for adjusting the proportion of speed and accuracy. The balance factor is added to the weight as follows:
[0118]
[0119] In the formula, w fast is the weight coefficient of a type of retrieval algorithm, w slow is the weight coefficient of the two-category retrieval algorithm, S fast is the relevance score of the output results of a type of retrieval algorithm, S slow is the relevance score of the output results of the two-class retrieval algorithm, and α is the balance factor. The relevance score is generated based on precision and recall. Precision is the ratio of the number of correctly predicted positive classes to the total number of predicted positive classes, and recall is the ratio of the number of correctly predicted positive classes to the total number of actual positive classes. The specific relevance score is as follows:
[0120]
[0121] In the formula, S fast is the relevance score of the output results of a type of retrieval algorithm, S slow is the relevance score of the output results of the two-category retrieval algorithm, Precision fast is the precision of a class of retrieval algorithms, slow is the precision of the two-category retrieval algorithm, Recall fast is the recall rate of a retrieval algorithm. slow is the recall rate of the two-category retrieval algorithm.
[0122] In information retrieval, precision and recall are two very important performance indicators. Their calculation is an existing technology and will not be described in detail here.
[0123] In step 2, the added balance factor α is used to adjust the weights of the fast search algorithm and the slow search algorithm in the final result. It is not simply added to the denominator, but it achieves a balance between speed and accuracy by affecting the calculation of weights. Specifically, the balance factor α affects the influence of the relevance scores of the two algorithms in the final weight calculation. fast and S slowThe balancing factor α is added to the denominator of the sum of the scores of the two algorithms. The larger the value of α, the greater its proportion in the denominator, which makes the scores of the two algorithms less influential than α, that is, the weight w fast and w slow On the contrary, if the α value is smaller, its influence on the final weight is smaller, and the influence of the scores of the two algorithms on the final weight is greater.
[0124] Therefore, by adjusting α, we can control the weight of fast and slow searches in the final results:
[0125] If we want faster retrieval speed, we can increase the value of α, so that the weight w of the fast retrieval algorithm fast will be relatively high, and the weight w of the slow retrieval algorithm slow Will be relatively low.
[0126] If we want to improve the accuracy of the retrieval, we can reduce the value of α, so that the weight w of the slow retrieval algorithm slow will be relatively high, the weight of the fast retrieval algorithm w fast Will be relatively low.
[0127] In this way, the balance factor α is actually a regulator that allows us to make a trade-off between speed and accuracy, rather than simply increasing or decreasing the weight of both retrieval algorithms at the same time. This trade-off is based on the relevance scores of the algorithms, rather than simply increasing or decreasing their weight.
[0128] Let's use a specific example to illustrate how the balance factor α affects the weight of speed and accuracy. Suppose we have the following situation:
[0129] The relevance score S of a fast search algorithm (such as BM25) fast =0.8;
[0130] Relevance score S for slow retrieval algorithms (such as BERT-based vector search) slow =0.9;
[0131] We set two different balance factor values: α1 = 0.1 and α2 = 1.0;
[0132] When α1=0.1:
[0133]
[0134] In this case, the slow retrieval algorithm is weighted slightly higher than the fast retrieval algorithm, which means that we value precision more.
[0135] When α2=1.0:
[0136]
[0137] Now, the weights of both algorithms have been reduced, but the weight of the fast retrieval algorithm has been reduced more, indicating that speed is now more valued than accuracy.
[0138] Specific impact:
[0139] Speed: weight w fast The higher case of α1=0.1 will make the system tend to use a fast retrieval algorithm, thereby increasing the retrieval speed but may sacrifice some accuracy.
[0140] Precision: Weight w slow The higher case of α2 = 1.0 will cause the system to favor a slow retrieval algorithm, thereby improving retrieval accuracy, but this usually comes at the expense of retrieval speed.
[0141] By adjusting the balance factor α, we can make a trade-off between speed and accuracy. A smaller α value gives a relatively higher weight to the fast retrieval algorithm, which helps to improve the retrieval speed; a larger α value gives a relatively higher weight to the slow retrieval algorithm, which helps to improve the retrieval accuracy. This adjustment is based on the relevance score of the algorithm, rather than simply increasing or decreasing their weight. By changing a balance factor α, the weight between speed and accuracy can be changed, thereby achieving the effect of weighing the two according to different needs.
[0142] Step 3: Calculate the relevance of each final search result, and perform hierarchical processing based on the set relevance threshold, and finally output the search results in order of level.
[0143] In this step, a relevance threshold θ can be set according to business needs. This relevance threshold θ can be made based on historical data and user feedback or relevant experience. The effect of this relevance threshold θ is mainly to classify the obtained search results, such as dividing them into three levels of "high", "medium" and "low" or into several levels of "correct", "wrong" and "uncertain". In short, by setting the relevance threshold θ, the upper limit, lower limit, value range, etc. in the threshold comparison can be obtained, and the effect of grading the search results can be achieved according to needs and experience.
[0144] The relevance of the search results can be obtained based on the relevance score or cosine similarity, Euclidean distance, Manhattan distance, Jaccard Similarity or pointwise mutual information (PMI). Since this is a prior art, it will not be elaborated here.
[0145] In one embodiment of the present invention, a speed-accuracy balance hybrid retrieval system based on relevance is provided, which is applied to the above method and includes:
[0146] A type of search module, which performs a preliminary search through a type of search algorithm and obtains a type of search results;
[0147] The second-class search module further searches the first-class search results through the second-class search algorithm to obtain the second-class search results;
[0148] The weighted fusion module fuses the first-category search results with the second-category search results through weighted superposition to obtain the final search results. The weights of the first-category search results and the second-category search results are both equipped with a balance factor for adjusting the proportion of speed and accuracy.
[0149] The hierarchical sorting module calculates the relevance of each final search result, and performs hierarchical processing based on the set relevance threshold, and finally outputs the search results in order of level.
[0150] In summary, the present invention can effectively combine the advantages of speed and accuracy by first using a fast but inaccurate algorithm for preliminary screening, and then using a slow but highly accurate algorithm for secondary retrieval. This phased retrieval strategy can give a rough set of relevant results at the first time, and then further process these results through a more accurate algorithm, ultimately obtaining fast and accurate retrieval results. More importantly, by introducing the balancing factor α, users can adjust the retrieval strategy according to specific needs. By simply changing one parameter, they can make a trade-off between speed and accuracy, making the retrieval system more flexible and adaptable.
[0151] Those of ordinary skill in the art will appreciate that the units of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0152] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored, etc.
[0153] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nlyMemory), random access memory (RAM, RandomAccessMemory), mobile hard disk, magnetic disk or optical disk, etc., which can store program code.
[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.
Claims
1. A speed-accuracy balance hybrid retrieval method based on relevance, characterized in that: The following steps are involved: A preliminary search is performed through a first-class search algorithm to obtain a first-class search result, and based on the first-class search result, a second-class search algorithm is used to further search to obtain a second-class search result; The first-category search results and the second-category search results are merged by weighted superposition to obtain the final search results, wherein the weights of the first-category search results and the second-category search results are both provided with a balance factor for adjusting the proportion of speed and accuracy; The relevance of each final search result is calculated, and graded based on the set relevance threshold, and finally the search results are output in order of level.
2. The speed-accuracy balance hybrid retrieval method based on relevance according to claim 1 is characterized in that: The first type of retrieval algorithm uses BM25, inverted index, TF-IDF, hash search, binary search or approximate nearest neighbor search, and the second type of retrieval algorithm uses HNSW, Faiss, Exact k-NN, Proxima, NSG, EFANNA or NGT.
3. The speed-accuracy balance hybrid retrieval method based on relevance according to claim 1 is characterized in that: The fusion through weighted superposition is as follows: R fusion =w fast ·R fast +w slow ·R slow ; In the formula, R fusion represents the fused result set, w fast is the weight coefficient of a type of retrieval algorithm, R fast is the output result set of a type of retrieval algorithm, w slow is the weight coefficient of the two-category retrieval algorithm, R slow It is the output result set of the second-category retrieval algorithm.
4. The speed-accuracy balance hybrid retrieval method based on relevance according to claim 3 is characterized in that: The weights of the first and second category search results are both equipped with a balance factor to adjust the weight of speed and accuracy. The balance factor is added to the weight as follows: In the formula, w fast is the weight coefficient of a type of retrieval algorithm, w slow is the weight coefficient of the two-category retrieval algorithm, S fast is the relevance score of the output results of a type of retrieval algorithm, S slow is the relevance score of the output results of the two-category retrieval algorithm, and α is the balance factor.
5. The speed-accuracy balance hybrid retrieval method based on relevance according to claim 4 is characterized in that: The relevance scores are as follows: In the formula, S fast is the relevance score of the output results of a type of retrieval algorithm, S slow is the relevance score of the output results of the two-category retrieval algorithm, Precision fast is the precision of a class of retrieval algorithms, slow is the precision of the two-category retrieval algorithm, Recall fast is the recall rate of a retrieval algorithm. slow is the recall rate of the two-category retrieval algorithm.
6. The speed-accuracy balance hybrid retrieval method based on relevance according to claim 1 is characterized in that: The relevance of the final search results is obtained based on the relevance score or cosine similarity, Euclidean distance, Manhattan distance, Jaccard similarity coefficient or point mutual information.
7. A speed-accuracy balanced hybrid retrieval system based on relevance, characterized in that: The method applied to any one of claims 1 to 6 comprises: A type of search module, which performs a preliminary search through a type of search algorithm and obtains a type of search results; The second-class search module further searches the first-class search results through the second-class search algorithm to obtain the second-class search results; The weighted fusion module fuses the first-category search results with the second-category search results through weighted superposition to obtain the final search results. The weights of the first-category search results and the second-category search results are both equipped with a balance factor for adjusting the proportion of speed and accuracy. The hierarchical sorting module calculates the relevance of each final search result, and performs hierarchical processing based on the set relevance threshold, and finally outputs the search results in order of level.
8. The speed-accuracy balance hybrid retrieval system based on relevance according to claim 7, characterized in that: The first type of retrieval algorithm uses BM25, inverted index, TF-IDF, hash search, binary search or approximate nearest neighbor search, and the second type of retrieval algorithm uses HNSW, Faiss, Exact k-NN, Proxima, NSG, EFANNA or NGT.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the speed-accuracy balanced hybrid retrieval method based on relevance as described in any one of claims 1 to 6.
10. A processor, characterized in that: The processor is used to run a program, wherein the program executes the speed-accuracy balance hybrid retrieval method based on relevance as described in any one of claims 1 to 6 when running.