Multi-stage mixed retrieval method and system

Abstract

The invention provides a multi-stage mixed retrieval method and system. The method comprises the following steps: acquiring an original database; obtaining retrieval content; rewriting the retrieval content by using the large model to obtain N query questions; performing full-text retrieval on the original database by utilizing the N query questions to obtain a first retrieval result; performing vector retrieval on the original database by using the N query questions to obtain a second retrieval result; performing weighted sorting on the first retrieval result and the second retrieval result, screening the sorted retrieval results according to a length threshold value, and defining M retrieval results after screening as final retrieval results; and taking the final retrieval result and the N query questions as cue words, transmitting the cue words to the large model, and outputting an answer result by the large model. According to the method, the advantages of full-text retrieval and natural language retrieval are combined, and the retrieval efficiency and the retrieval accuracy are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of information retrieval, and in particular relates to a multi-stage hybrid retrieval method and system. Background Art

[0002] With the advent of the information explosion era, the accumulation of massive data has brought huge challenges to information retrieval. Traditional retrieval methods mainly include the following:

[0003] 1. Full-text search. Full-text search mainly indexes documents, matches them based on keywords, and quickly locates documents containing keywords. Its advantages are fast speed and high accuracy. Its disadvantages are insufficient understanding of user query intent and difficulty in processing queries with complex semantics and ambiguous expressions. For example, if a user queries "the latest developments in artificial intelligence", full-text search may return all documents containing the two keywords "artificial intelligence" and "latest developments", but it cannot truly understand that the user wants to know the latest research results in the field of artificial intelligence.

[0004] 2. Natural language retrieval. Natural language retrieval is mainly about understanding the semantics of user queries and searching based on the semantics. It uses natural language processing technology to analyze user queries, identify keywords, semantic relationships, intents, etc., so as to find results that are more in line with user intent. Its advantage is that it can understand the semantics of user queries and return more accurate results. Its disadvantage is that it is slow and has limited processing capabilities for fuzzy queries. For example, if a user queries "how to improve writing skills", natural language retrieval may need to identify that the user's intention is "improve writing skills" and return relevant tutorials, but its retrieval efficiency is often not as good as full-text retrieval.

[0005] 3. Hybrid search. Hybrid search combines the advantages of full-text search and natural language search, which can effectively make up for the shortcomings of the two technologies and provide more comprehensive and efficient information retrieval services. Its advantages are: combining the accuracy of full-text search and the semantic understanding ability of natural language search, hybrid search can return search results that are more in line with user intentions; using the fast indexing mechanism of full-text search, hybrid search can effectively improve the search speed and meet the user's demand for timely information acquisition; hybrid search supports multiple search methods, such as keyword search, natural language search, semantic search, etc., which can meet the search needs of different users. At present, hybrid search mainly focuses on the following directions:

[0006] 1) Simple hybrid search based on full-text search and semantic matching: First, perform full-text search and natural language search separately, then simply mix the results of the two and send them to the big model for semantic similarity matching. Although the search results of the two are also combined, the combination is simple and crude, and the effect is not ideal.

[0007] 2) Hybrid retrieval based on deep learning: Use deep learning technology to extract features of user queries and documents, and build models for prediction, so as to achieve more accurate semantic matching. However, this retrieval technology requires high time and computing costs, and cannot take advantage of the advantages of full-text retrieval. Summary of the invention

[0008] In view of the defects in the prior art, the present invention provides a multi-stage hybrid retrieval method, which improves the retrieval efficiency and retrieval accuracy.

[0009] In a first aspect, a multi-stage hybrid retrieval method comprises:

[0010] Obtaining an original database; the original database includes multiple text paragraphs and corresponding feature vectors;

[0011] Get the search content;

[0012] Rewrite the search content using the big model to get N query questions;

[0013] Performing full-text search on the original database using N query questions respectively to obtain the first search result;

[0014] Using N query questions respectively to perform vector search on the original database to obtain a second search result;

[0015] Perform weighted sorting on the first search result and the second search result, filter the sorted search results according to the length threshold, and define the filtered M search results as the final search results;

[0016] The final retrieval results and N query questions are passed to the big model as prompt words, and the big model outputs the answer results.

[0017] Furthermore, the method for obtaining the first search result specifically includes:

[0018] Match each query question with a text paragraph in the original database;

[0019] A plurality of text paragraphs with the highest matching degree are defined as the first search results.

[0020] Furthermore, the method for obtaining the second search result specifically includes:

[0021] Match each query question with the feature vector in the original database;

[0022] The text paragraphs corresponding to the multiple feature vectors with the highest matching degree are defined as the second search result.

[0023] Furthermore, after obtaining the first search result, the method further includes:

[0024] Matching each query question with the feature vector of each text paragraph in the first search result;

[0025] The text paragraphs corresponding to the multiple feature vectors with the highest matching degree are defined as the second search result.

[0026] Furthermore, the method for obtaining the final search results specifically includes:

[0027] Set the weight of the first search result and the weight of the second search result;

[0028] Multiplying the matching degree of each text paragraph in the first search result and the second search result by the corresponding weight respectively to obtain the matching score of each text paragraph;

[0029] Sort the text paragraphs in order of matching scores from high to low;

[0030] Identify the length of all text paragraphs;

[0031] The first M text paragraphs are defined as the final retrieval results, where the sum of the lengths of all text paragraphs in the final retrieval results is not greater than a length threshold.

[0032] In a second aspect, a multi-stage hybrid retrieval system includes:

[0033] Original database: includes multiple text paragraphs and corresponding feature vectors;

[0034] Query acquisition unit: used to obtain search content;

[0035] Query rewriting unit: used to rewrite the search content using the large model to obtain N query questions;

[0036] Full-text search unit: used to perform full-text search on the original database using N query questions respectively to obtain a first search result;

[0037] A natural language retrieval unit: used to perform vector retrieval on the original database using N query questions respectively to obtain a second retrieval result;

[0038] Sorting unit: used to perform weighted sorting on the first search result and the second search result, filter the sorted search results according to the length threshold, and define the M filtered search results as the final search results;

[0039] Question-answering unit: used to pass the final retrieval results and N query questions as prompt words to the big model, which then outputs the answer results.

[0040] Furthermore, the full-text retrieval unit is specifically used for:

[0041] Match each query question with a text paragraph in the original database;

[0042] A plurality of text paragraphs with the highest matching degree are defined as the first search results.

[0043] Furthermore, the natural language retrieval unit is specifically used for:

[0044] Match each query question with the feature vector in the original database;

[0045] The text paragraphs corresponding to the multiple feature vectors with the highest matching degree are defined as the second search result.

[0046] Furthermore, the natural language retrieval unit is also used for:

[0047] Matching each query question with the feature vector of each text paragraph in the first search result;

[0048] The text paragraphs corresponding to the multiple feature vectors with the highest matching degree are defined as the second search result.

[0049] Furthermore, the sorting unit is specifically used for:

[0050] Set the weight of the first search result and the weight of the second search result;

[0051] Multiplying the matching degree of each text paragraph in the first search result and the second search result by the corresponding weight respectively to obtain the matching score of each text paragraph;

[0052] Sort the text paragraphs in order of matching scores from high to low;

[0053] Identify the length of all text paragraphs;

[0054] The first M text paragraphs are defined as the final retrieval results, where the sum of the lengths of all text paragraphs in the final retrieval results is not greater than a length threshold.

[0055] It can be seen from the above technical solution that the multi-stage hybrid search method and system provided by the present invention conducts full-text search and natural language search in stages, first uses full-text search and natural language search to quickly locate relevant text paragraphs, and then sorts and screens the search results. This method combines the advantages of full-text search and natural language search, and improves search efficiency and search accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0057] Figure 1 A flowchart of a multi-stage hybrid retrieval method provided in an embodiment.

[0058] Figure 2 A flowchart of a method for obtaining final search results provided in an embodiment.

[0059] Figure 3 A module block diagram of a multi-stage hybrid retrieval system provided in an embodiment. DETAILED DESCRIPTION

[0060] The embodiments of the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and the protection scope of the present invention cannot be limited thereto. It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should be the common meanings understood by those skilled in the art to which the present invention belongs.

[0061] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0062] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0063] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0064] Example:

[0065] A multi-stage hybrid retrieval method, see Figure 1 ,include:

[0066] Obtaining an original database; the original database includes multiple text paragraphs and corresponding feature vectors;

[0067] Get the search content;

[0068] Rewrite the search content using the big model to get N query questions;

[0069] Performing full-text search on the original database using N query questions respectively to obtain the first search result;

[0070] Using N query questions respectively to perform vector search on the original database to obtain a second search result;

[0071] Perform weighted sorting on the first search result and the second search result, filter the sorted search results according to the length threshold, and define the filtered M search results as the final search results;

[0072] The final retrieval results and N query questions are passed to the big model as prompt words, and the big model outputs the answer results.

[0073] In this embodiment, the original database is created based on the text paragraphs obtained after the original document is split. The original database also includes feature vectors extracted from each text paragraph.

[0074] In this embodiment, since the search content input by the user may not be accurate when searching, it is impossible to match some more accurate search results. Therefore, before searching, the method needs to use the big model to rewrite the user's search content to achieve the purpose of clear expression and better matching of the search. When the big model rewrites the search content, it can distinguish the instruction information and non-instruction information in the search content and derive keywords. For example, the search content "When was the bicycle invented" is converted into {"keywords_en":["bicycles","invented","when"],"keywords_zh":["bicycle","invent","time"]}. Then the big model uses appropriate prompt words to generate N similar or synonymous query questions based on the obtained keywords. For example, the obtained query questions include "When did rickshaws appear". In order to achieve the purpose of balancing cost and accuracy, N can be determined according to the configuration of server resources, and is generally set to 3.

[0075] In this embodiment, after the search content is rewritten, full-text search and vector search are performed on the N query questions obtained, respectively, to obtain two search results. Then, according to the Token length limit of the large model, the two search results are weighted, sorted and trimmed to obtain the final search result. Finally, the final search result and the N query questions are passed to the large model as prompt words, and the large model outputs the answer result.

[0076] This method conducts full-text search and natural language search in stages, first using full-text search and natural language search to quickly locate relevant text paragraphs, and then sorting and screening the search results. This method combines the advantages of full-text search and natural language search, improving search efficiency and accuracy.

[0077] Furthermore, in some embodiments, the method for obtaining the first search result specifically includes:

[0078] Match each query question with a text paragraph in the original database;

[0079] A plurality of text paragraphs with the highest matching degree are defined as the first search results.

[0080] In this embodiment, the full-text search may adopt an inverted index method to match each query question with a text paragraph in the original database, and obtain multiple text paragraphs with the highest matching degree as the first search result.

[0081] Furthermore, in some embodiments, the method for obtaining the second search result specifically includes:

[0082] Match each query question with the feature vector in the original database;

[0083] The text paragraphs corresponding to the multiple feature vectors with the highest matching degree are defined as the second search result.

[0084] In this embodiment, vector retrieval is a retrieval based on semantic similarity, which matches each query question with a feature vector in the original database, and defines the text paragraphs corresponding to the multiple feature vectors with the highest matching degree as the second retrieval result.

[0085] Furthermore, in some embodiments, after obtaining the first search result, the method further includes:

[0086] Matching each query question with the feature vector of each text paragraph in the first search result;

[0087] The text paragraphs corresponding to the multiple feature vectors with the highest matching degree are defined as the second search result.

[0088] In this embodiment, if resources are tight, or the first search result obtained by full-text search matches a large number of text paragraphs, the first search result can be directly used as the original data for vector search, that is, each query question is matched with the feature vector of each text paragraph in the first search result, and the text paragraphs corresponding to the multiple feature vectors with the highest matching degree are defined as the second search result.

[0089] Further, in some embodiments, see Figure 2 , the method for obtaining the final search results specifically includes:

[0090] Set the weight of the first search result and the weight of the second search result;

[0091] Multiplying the matching degree of each text paragraph in the first search result and the second search result by the corresponding weight respectively to obtain the matching score of each text paragraph;

[0092] Sort the text paragraphs in order of matching scores from high to low;

[0093] Identify the length of all text paragraphs;

[0094] The first M text paragraphs are defined as the final retrieval results, where the sum of the lengths of all text paragraphs in the final retrieval results is not greater than a length threshold.

[0095] In this embodiment, since the length of tokens that the large model can process is limited, the method needs to weight and trim the above two search results according to the length threshold required by the large model, and intercept as many search results as possible while meeting the length threshold of the large model. The method first sets the weight of the first search result and the weight of the second search result. For example, the weight of the first search result is set to 6, and the weight of the second search result is set to 4. Then, the matching degree of each text paragraph in the first search result and the second search result is multiplied by the corresponding weight to obtain the matching score of each text paragraph. For example, the first search result a has a matching degree of 0.92, and the matching score = 0.92*6 = 5.52; the second search result b has a matching degree of 0.9, and the matching score = 0.9*4 = 3.6; the first search result c has a matching degree of 0.5, and the matching score = 0.5*6 = 3; the second search result d has a matching degree of 0.6, and the matching score = 0.6*4 = 2.4; then the above four search results are ranked as follows: first search result a, second search result b, first search result c, second search result d. Then identify the length of all text paragraphs. For example, the lengths of the above four search results are: the first search result a has a length of 2000, the second search result b has a length of 3000, the first search result c has a length of 500, and the second search result d has a length of 3000.

[0096] In this embodiment, assuming that the token length limit is 8192 and the query length is 2000, the length of the search result obtained shall not be greater than 8192-2000=6192. If the length threshold of 6192 is not exceeded, the first M text paragraphs are taken as the final search results. For example, for the above four search results, since the total length of the first search result a, the second search result b, the first search result c, and the second search result d = 2000+3000+500+3000=8500, which exceeds the length threshold of 6192, this method can only define the first search result a, the second search result b, and the first search result c as the final search results.

[0097] A multi-stage hybrid retrieval system, see Figure 3 ,include:

[0098] Original database: includes multiple text paragraphs and corresponding feature vectors;

[0099] Query acquisition unit: used to obtain search content;

[0100] Query rewriting unit: used to rewrite the search content using the large model to obtain N query questions;

[0101] Full-text search unit: used to perform full-text search on the original database using N query questions respectively to obtain a first search result;

[0102] A natural language retrieval unit: used to perform vector retrieval on the original database using N query questions respectively to obtain a second retrieval result;

[0103] Sorting unit: used to perform weighted sorting on the first search result and the second search result, filter the sorted search results according to the length threshold, and define the M filtered search results as the final search results;

[0104] Question-answering unit: used to pass the final retrieval results and N query questions as prompt words to the big model, which then outputs the answer results.

[0105] Furthermore, in some embodiments, the full-text retrieval unit is specifically used to:

[0106] Match each query question with a text paragraph in the original database;

[0107] A plurality of text paragraphs with the highest matching degree are defined as the first search results.

[0108] Furthermore, in some embodiments, the natural language retrieval unit is specifically used to:

[0109] Match each query question with the feature vector in the original database;

[0110] The text paragraphs corresponding to the multiple feature vectors with the highest matching degree are defined as the second search result.

[0111] Furthermore, in some embodiments, the natural language retrieval unit is further configured to:

[0112] Matching each query question with the feature vector of each text paragraph in the first search result;

[0113] The text paragraphs corresponding to the multiple feature vectors with the highest matching degree are defined as the second search result.

[0114] Furthermore, in some embodiments, the sorting unit is specifically used to:

[0115] Set the weight of the first search result and the weight of the second search result;

[0116] Multiplying the matching degree of each text paragraph in the first search result and the second search result by the corresponding weight respectively to obtain the matching score of each text paragraph;

[0117] Sort the text paragraphs in order of matching scores from high to low;

[0118] Identify the length of all text paragraphs;

[0119] The first M text paragraphs are defined as the final retrieval results, where the sum of the lengths of all text paragraphs in the final retrieval results is not greater than a length threshold.

[0120] The system provided in the embodiment of the present invention is briefly described. For matters not mentioned in the embodiment part, reference may be made to the corresponding contents in the aforementioned embodiment.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A multi-stage hybrid retrieval method, characterized in that: include: Acquire an original database; the original database includes a plurality of text paragraphs and corresponding feature vectors; Get the search content; Rewriting the search content using the large model to obtain N query questions; Performing a full-text search on the original database using the N query questions respectively to obtain a first search result; Using the N query questions respectively to perform vector search on the original database to obtain a second search result; Performing weighted sorting on the first search result and the second search result, screening the sorted search results according to a length threshold, and defining the screened M search results as final search results; The final search result and the N query questions are passed to the large model as prompt words, and the large model outputs the answer result.

2. The multi-stage hybrid retrieval method according to claim 1, characterized in that: The method for obtaining the first search result specifically includes: Matching each of the query questions with the text paragraphs in the original database; A plurality of text paragraphs with the highest matching degree are defined as the first search results.

3. The multi-stage hybrid retrieval method according to claim 2, characterized in that: The method for obtaining the second search result specifically includes: Matching each of the query questions with the feature vectors in the original database; The text paragraphs corresponding to the multiple feature vectors with the highest matching degree are defined as the second search results.

4. The multi-stage hybrid retrieval method according to claim 2, characterized in that: After obtaining the first search result, the method further includes: Matching each of the query questions with the feature vector of each text paragraph in the first search result; The text paragraphs corresponding to the multiple feature vectors with the highest matching degree are defined as the second search results.

5. The multi-stage hybrid retrieval method according to claim 3, characterized in that: The method for obtaining the final search result specifically includes: Setting a weight of the first search result and a weight of the second search result; Multiplying the matching degree of each text paragraph in the first search result and the second search result by the corresponding weight respectively to obtain the matching score of each text paragraph; Sort the text paragraphs in order of matching scores from high to low; Identify the length of all text paragraphs; The first M text paragraphs are defined as the final search results, wherein the sum of the lengths of all text paragraphs in the final search results is not greater than the length threshold.

6. A multi-stage hybrid retrieval system, characterized in that: include: Original database: includes multiple text paragraphs and corresponding feature vectors; Query acquisition unit: used to obtain search content; Query rewriting unit: used to rewrite the search content using the large model to obtain N query questions; A full-text search unit: used to perform a full-text search on the original database using the N query questions respectively to obtain a first search result; A natural language search unit: configured to perform vector search on the original database using the N query questions respectively to obtain a second search result; A sorting unit: used for weighted sorting of the first search result and the second search result, screening the sorted search results according to a length threshold, and defining the screened M search results as the final search results; Question and answer unit: used to pass the final search result and the N query questions as prompt words to the big model, and the big model outputs the answer result.

7. The multi-stage hybrid retrieval system according to claim 6, characterized in that: The full-text retrieval unit is specifically used for: Matching each of the query questions with the text paragraphs in the original database; A plurality of text paragraphs with the highest matching degree are defined as the first search results.

8. The multi-stage hybrid retrieval system according to claim 7, characterized in that: The natural language retrieval unit is specifically used for: Matching each of the query questions with the feature vectors in the original database; The text paragraphs corresponding to the multiple feature vectors with the highest matching degree are defined as the second search results.

9. The multi-stage hybrid retrieval system according to claim 7, characterized in that: The natural language retrieval unit is also used for: Matching each of the query questions with the feature vector of each text paragraph in the first search result; The text paragraphs corresponding to the multiple feature vectors with the highest matching degree are defined as the second search results.

10. The multi-stage hybrid retrieval system according to claim 8, characterized in that: The sorting unit is specifically used for: Setting a weight of the first search result and a weight of the second search result; Multiplying the matching degree of each text paragraph in the first search result and the second search result by the corresponding weight respectively to obtain the matching score of each text paragraph; Sort the text paragraphs in order of matching scores from high to low; Identify the length of all text paragraphs; The first M text paragraphs are defined as the final search results, wherein the sum of the lengths of all text paragraphs in the final search results is not greater than the length threshold.

Images