Information retrieval enhancement method and system based on multi-path recall mode

Abstract

The invention provides an information retrieval enhancement method and system based on a multi-path recall mode, and the method comprises the following steps: analyzing a user search term to obtain retrieval result data; performing multi-path recall on the retrieval result data to obtain recall information; sorting the recall information to obtain a sorting result; and reordering the ordering result to obtain a final retrieval result. According to the technical scheme, the search result data are obtained by analyzing the user search term; performing multi-path recall on the retrieval result data to obtain recall information; sorting the recall information to obtain a sorting result; reordering the ordering result to obtain a final retrieval result; according to the method, multiple information retrieval strategies are integrated in a flexible and expansible manner, so that the information retrieval precision and efficiency are improved, and the expandability of the information retrieval strategies is not lost.

Description

Background Art

[0002] Traditional search engines encounter the problem of low recall rate when facing massive data, especially when dealing with complex or long-tail queries, the single text matching effect is often not ideal. This is mainly due to the following reasons:

[0003] 1. Data quality issues

[0004] Incomplete data: Traditional search engines may not be able to index all relevant data, resulting in some relevant information being missed;

[0005] Data accuracy: Errors or inconsistencies in the data may also affect the relevance of search results;

[0006] Data timeliness: For queries with high timeliness requirements, if the search engine fails to update its index data in a timely manner, the returned results will be outdated and no longer meet user needs.

[0007] 2. Limitations of text matching algorithms

[0008] Single strategy: Traditional search engines often use a single text matching algorithm, such as keyword matching, vector space model, etc. These algorithms cannot accurately capture the user's true intention when processing complex or long-tail queries;

[0009] Insufficient semantic understanding: Due to the lack of in-depth understanding of query statements and document content, traditional retrieval engines are unable to accurately identify synonyms, near-synonyms, or context-related words, which affects the recall rate.

[0010] 3. Diversified user interests

[0011] Wide range of interests: Users’ interests are very wide and diverse, and a single recall strategy cannot cover all relevant interests;

[0012] Personalized needs: Different users have different needs and expectations for the same query. Traditional search engines are often unable to customize recall based on users' personalized needs. Summary of the invention

[0013] The present application provides an information retrieval enhancement method and system based on a multi-way recall mode, which is used to flexibly and scalably integrate multiple information retrieval strategies to improve the accuracy and efficiency of information retrieval.

[0014] In a first aspect, a method for enhancing information retrieval based on a multi-way recall mode is provided, comprising the following steps:

[0015] Analyze user search terms to obtain search result data;

[0016] Performing multi-channel recall on the search result data to obtain recall information;

[0017] Sorting the recall information to obtain a sorting result;

[0018] The sorting results are re-sorted to obtain final retrieval results.

[0019] In the above technical scheme, the search results data are obtained by analyzing the user's search terms; the search result data is recalled in multiple ways to obtain recall information; the recall information is sorted to obtain the sorted results; the sorted results are re-sorted to obtain the final search results; multi-way recall significantly improves the comprehensiveness and accuracy of recall by executing multiple search strategies in parallel; multi-way recall combines text and vector recall to better capture the semantic information of the text and provide more accurate search results; multi-way recall technology can combine the user's personal preferences and historical behaviors, provide personalized search results through vector recall, and enhance user experience; multi-way recall has better scalability and efficiency, can handle large-scale data and high-concurrency requests, and ensure search performance; multi-way recall allows different strategies to be adopted according to different situations and needs, and different strategies form complementary effects, enhance the recall effect, and improve the accuracy of information retrieval; it flexibly and scalably integrates multiple information retrieval strategies, improves the accuracy and efficiency of information retrieval, and does not lose the scalability of information retrieval strategies.

[0020] In a specific embodiment, it also includes:

[0021] The data is cleaned and structured and stored in the database.

[0022] In a specific implementation scheme, the steps of cleaning and structuring the data and storing it in a database include:

[0023] Clean, segment, and replace synonyms on documents, store the processed data in Elasticsearch, and build the document segmentation and inverted index;

[0024] The document content is extracted and unstructured data is converted. The data features are extracted using a vectorization model, the text is converted into vector form, and stored in the Milvus database.

[0025] In a specific implementation scheme, the vectorized model includes Langchain, M3E, and BGE.

[0026] In a specific implementation scheme, the step of analyzing the user search terms to obtain the search result data specifically includes:

[0027] Generate user search terms through word segmentation, synonym replacement, and query rewriting for search in elasticsearch;

[0028] By using the vectorization conversion model embedding, the user's search terms are converted into vectors to generate semantic vectors for searching in the vector library.

[0029] In a specific implementation method, the step of performing multi-channel recall on the search result data to obtain recall information includes:

[0030] Recall data from the milvus vector library through a search algorithm;

[0031] Or you can search from elasticsearch with keywords to return the recalled information.

[0032] In a specific implementation scheme, the step of sorting the recall information to obtain the sorting result is specifically as follows:

[0033] After recalling the topM results, the results are preliminarily sorted through the coarse ranking model and the topN results are returned.

[0034] In a specific implementation scheme, the step of re-sorting the sorting results to obtain the final search results is specifically:

[0035] Based on the topN results returned in the rough ranking, the re-ranking technology is used to evaluate the relevance of the context, and finally the topK results are returned as the final retrieval results.

[0036] In a specific implementation scheme, the reordering technology includes a reordering model BC Embedding and a large language model LLM.

[0037] In a second aspect, an information retrieval enhancement system based on a multi-way recall mode is provided, comprising:

[0038] Data preprocessing and structuring module, used to clean and structure the data and store it in the database;

[0039] Search word analysis module, used to analyze user search words and obtain search result data;

[0040] A multi-channel recall module, used for performing multi-channel recall on the search result data to obtain recall information;

[0041] A sorting module, used to sort the recall information to obtain a sorting result;

[0042] The rearrangement module is used to re-arrange the sorting results to obtain the final search results.

[0043] In the above technical scheme, the search results data are obtained by analyzing the user's search terms; the search result data is recalled in multiple ways to obtain recall information; the recall information is sorted to obtain the sorted results; the sorted results are re-sorted to obtain the final search results; multi-way recall significantly improves the comprehensiveness and accuracy of recall by executing multiple search strategies in parallel; multi-way recall combines text and vector recall to better capture the semantic information of the text and provide more accurate search results; multi-way recall technology can combine the user's personal preferences and historical behaviors, provide personalized search results through vector recall, and enhance user experience; multi-way recall has better scalability and efficiency, can handle large-scale data and high-concurrency requests, and ensure search performance; multi-way recall allows different strategies to be adopted according to different situations and needs, and different strategies form complementary effects, enhance the recall effect, and improve the accuracy of information retrieval; it flexibly and scalably integrates multiple information retrieval strategies, improves the accuracy and efficiency of information retrieval, and does not lose the scalability of information retrieval strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of an information retrieval enhancement method based on a multi-channel recall mode provided in an embodiment of the present application;

[0045] Figure 2 A structural block diagram of an information retrieval enhancement system based on a multi-channel recall mode provided in an embodiment of the present application;

[0046] Figure 3 A specific flow chart of the information retrieval enhancement method based on the multi-way recall mode provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The present application is further described in detail below through the accompanying drawings and embodiments. Through these descriptions, the characteristics and advantages of the present application will become clearer and more specific.

[0048] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0049] In addition, the technical features involved in different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0050] To facilitate understanding of the information retrieval enhancement method and system based on the multi-way recall mode provided in the embodiment of the present application, its application scenario is first explained. The information retrieval enhancement method and system based on the multi-way recall mode provided in the embodiment of the present application are used to flexibly and scalably integrate multiple information retrieval strategies to improve the accuracy and efficiency of information retrieval. When faced with massive amounts of data, traditional retrieval engines will encounter the problem of low recall rate, especially when dealing with complex or long-tail queries, the effect of single text matching is often unsatisfactory. This is mainly due to the following reasons: 1. Data quality issues: Incomplete data: Traditional retrieval engines may not be able to index all relevant data, resulting in some relevant information being omitted; Data accuracy: Errors or inconsistencies in the data may also affect the relevance of the retrieval results; Data timeliness: For queries with high timeliness requirements, if the retrieval engine fails to update its index data in a timely manner, the returned results are outdated and no longer meet user needs. 2. Limitations of text matching algorithms: Single strategy: Traditional search engines often use a single text matching algorithm, such as keyword matching, vector space model, etc. These algorithms cannot accurately capture the user's true intention when processing complex or long-tail queries; Insufficient semantic understanding: Due to the lack of in-depth understanding of query statements and document content, traditional search engines cannot accurately identify synonyms, near-synonyms or context-related words, which affects the recall rate. 3. Diversified user interests: Wide range of interests: Users' interests are very wide and diverse, and a single recall strategy cannot cover all relevant interests; Personalized needs: Different users have different needs and expectations for the same query, and traditional search engines often cannot customize recall according to the user's personalized needs. To this end, the embodiment of the present application provides an information retrieval enhancement method and system based on a multi-way recall mode, which flexibly and scalably integrates multiple information retrieval strategies to improve the accuracy and efficiency of information retrieval. The following is a detailed description of the embodiment in conjunction with specific drawings.

[0051] refer to Figures 1 to 3 , Figure 1 A flowchart of an information retrieval enhancement method based on a multi-channel recall mode provided in an embodiment of the present application; Figure 2 A structural block diagram of an information retrieval enhancement system based on a multi-channel recall mode provided in an embodiment of the present application; Figure 3 A specific flow chart of the information retrieval enhancement method based on the multi-way recall mode provided in an embodiment of the present application.

[0052] exist Figure 1 In the embodiment of the present application, a method for enhancing information retrieval based on a multi-path recall mode is provided, comprising the following steps:

[0053] Analyze user search terms to obtain search result data;

[0054] Performing multi-channel recall on the search result data to obtain recall information;

[0055] Sorting the recall information to obtain a sorting result;

[0056] The sorting results are re-sorted to obtain final retrieval results.

[0057] In the above technical scheme, the search results data are obtained by analyzing the user's search terms; the search result data is recalled in multiple ways to obtain recall information; the recall information is sorted to obtain the sorted results; the sorted results are re-sorted to obtain the final search results; multi-way recall significantly improves the comprehensiveness and accuracy of recall by executing multiple search strategies in parallel; multi-way recall combines text and vector recall to better capture the semantic information of the text and provide more accurate search results; multi-way recall technology can combine the user's personal preferences and historical behaviors, provide personalized search results through vector recall, and enhance user experience; multi-way recall has better scalability and efficiency, can handle large-scale data and high-concurrency requests, and ensure search performance; multi-way recall allows different strategies to be adopted according to different situations and needs, and different strategies form complementary effects, enhance the recall effect, and improve the accuracy of information retrieval; it flexibly and scalably integrates multiple information retrieval strategies, improves the accuracy and efficiency of information retrieval, and does not lose the scalability of information retrieval strategies.

[0058] In a specific embodiment, it also includes:

[0059] The data is cleaned and structured and stored in the database.

[0060] In a specific implementation scheme, the steps of cleaning and structuring the data and storing it in a database include:

[0061] Clean, segment, and replace synonyms on documents, store the processed data in Elasticsearch, and build the document segmentation and inverted index;

[0062] The document content is extracted and unstructured data is converted. The data features are extracted using a vectorization model, the text is converted into vector form, and stored in the Milvus database.

[0063] In a specific implementation scheme, the vectorized model includes Langchain, M3E, and BGE.

[0064] In a specific implementation scheme, the step of analyzing the user search terms to obtain the search result data specifically includes:

[0065] Generate user search terms through word segmentation, synonym replacement, and query rewriting for search in elasticsearch;

[0066] By using the vectorization conversion model embedding, the user's search terms are converted into vectors to generate semantic vectors for searching in the vector library.

[0067] In a specific implementation method, the step of performing multi-channel recall on the search result data to obtain recall information includes:

[0068] Recall data from the milvus vector library through a search algorithm;

[0069] Or you can search from elasticsearch with keywords to return the recalled information.

[0070] In a specific implementation scheme, the step of sorting the recall information to obtain the sorting result is specifically as follows:

[0071] After recalling the topM results, the results are preliminarily sorted through the coarse ranking model and the topN results are returned.

[0072] In a specific implementation scheme, the step of re-sorting the sorting results to obtain the final search results is specifically:

[0073] Based on the topN results returned in the rough ranking, the re-ranking technology is used to evaluate the relevance of the context, and finally the topK results are returned as the final retrieval results.

[0074] In a specific implementation scheme, the reordering technology includes a reordering model BC Embedding and a large language model LLM.

[0075] Specifically, refer to Figure 3 The specific process of the information retrieval enhancement method based on the multi-way recall mode includes:

[0076] 1. Data preprocessing and structuring into database: First, the data needs to be cleaned and structured to make it into a format suitable for model processing and stored in the relevant database. This application uses two paths to process data:

[0077] Path 1: Clean, segment, and replace synonyms on documents, store the processed data in Elasticsearch, and create a segmentation and inverted index for the documents;

[0078] Path 2: Extract content from documents and convert unstructured data, use vectorized models (such as Langchain, M3E, BGE, etc.) to extract data features, convert text into vector form, and store it in the Milvus database for subsequent retrieval.

[0079] 2. User search term processing (i.e. query analysis): Generate queries for searching in elasticsearch through word segmentation, synonym replacement, query rewriting, etc.; convert search terms into vectors using the vectorization conversion model embedding to generate semantic vectors for searching in the vector library.

[0080] 3. Multi-channel recall (recall module): Information searched through multi-channel recall. For example, data is recalled from the milvus vector library through a search algorithm, or data is retrieved from elasticsearch in combination with keywords, and the recalled information is returned.

[0081] The search algorithms include: Depth-First Search (DFS), Breadth-First Search (BFS), backtracking algorithm, binary search algorithm and hash table search algorithm.

[0082] 4. Sorting module: After recalling the topM possibly relevant results, it is necessary to preliminarily sort the results through a coarse sorting model, quickly filter out relatively few results from a large number of candidate results, and return the topN results.

[0083] 5. Result re-ranking: Based on the top N results returned in rough ranking, re-ranking technology (implemented by re-ranking model (BC Embedding) or large language model (LLM)) is used to evaluate the relevance of these contexts and finally return the top K results.

[0084] In this embodiment, 1. Traditional search engines often encounter the problem of low recall rate when facing massive data. Multi-way recall significantly improves the comprehensiveness and accuracy of recall by executing multiple search strategies in parallel.

[0085] 2. For complex or long-tail queries, single text matching is not ideal. Multi-way recall combines text and vector recall to better capture the semantic information of the text and provide more accurate search results.

[0086] 3. Multi-way recall technology can combine users’ personal preferences and historical behaviors, provide personalized search results through vector recall, and enhance user experience.

[0087] 4. Multi-way recall has better scalability and efficiency, can handle large-scale data and high-concurrency requests, and ensures search performance.

[0088] 5. Multi-way recall allows different strategies to be adopted according to different situations and needs. Different strategies form a complementary effect, enhance the recall effect, and improve the accuracy of information retrieval.

[0089] 6. Traditional single-strategy recall may not cover the diverse interests of users. Multi-way recall breaks this limitation by executing multiple strategies in parallel.

[0090] 7. With the development of technology and the accumulation of data, new recall strategies and algorithms can be continuously added to the multi-channel recall system to continuously optimize the recall effect.

[0091] In the above technical scheme, the search results data are obtained by analyzing the user search terms; the search result data is recalled in multiple ways to obtain the recall information; the recalled information is sorted to obtain the sorted results; the sorted results are re-sorted to obtain the final search results; and a variety of information retrieval strategies are flexibly and scalably integrated to improve the accuracy and efficiency of information retrieval without losing the scalability of the information retrieval strategy.

[0092] exist Figure 2 In the embodiment of the present application, an information retrieval enhancement system based on a multi-path recall mode is provided, comprising:

[0093] Data preprocessing and structuring module, used to clean and structure the data and store it in the database;

[0094] Search word analysis module, used to analyze user search words and obtain search result data;

[0095] A multi-channel recall module, used for performing multi-channel recall on the search result data to obtain recall information;

[0096] A sorting module, used to sort the recall information to obtain a sorting result;

[0097] The rearrangement module is used to re-arrange the sorting results to obtain the final search results.

[0098] In the above technical scheme, the search results data are obtained by analyzing the user's search terms; the search result data is recalled in multiple ways to obtain recall information; the recall information is sorted to obtain the sorted results; the sorted results are re-sorted to obtain the final search results; multi-way recall significantly improves the comprehensiveness and accuracy of recall by executing multiple search strategies in parallel; multi-way recall combines text and vector recall to better capture the semantic information of the text and provide more accurate search results; multi-way recall technology can combine the user's personal preferences and historical behaviors, provide personalized search results through vector recall, and enhance user experience; multi-way recall has better scalability and efficiency, can handle large-scale data and high-concurrency requests, and ensure search performance; multi-way recall allows different strategies to be adopted according to different situations and needs, and different strategies form complementary effects, enhance the recall effect, and improve the accuracy of information retrieval; it flexibly and scalably integrates multiple information retrieval strategies, improves the accuracy and efficiency of information retrieval, and does not lose the scalability of information retrieval strategies.

[0099] Those skilled in the art will appreciate that the present application may be implemented as a system, method or computer program product.

[0100] Therefore, the present disclosure may be specifically implemented in the following forms, namely: it may be completely hardware, it may be completely software (including firmware, resident software, microcode, etc.), or it may be a combination of hardware and software, generally referred to herein as a "circuit", "module" or "system". In addition, in some embodiments, the present application may also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium may contain computer-readable program code.

[0101] Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0102] Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art can change, modify, replace and modify the above embodiments within the scope of the present application. On this basis, a variety of replacements and improvements can be made to the present application, all of which fall within the scope of protection of the present application.

Claims

1. An information retrieval enhancement method based on a multi-way recall mode, characterized in that: The following steps are involved: Analyze user search terms to obtain search result data; Performing multi-channel recall on the search result data to obtain recall information; Sorting the recall information to obtain a sorting result; The sorting results are re-sorted to obtain final retrieval results.

2. The information retrieval enhancement method based on the multi-way recall mode according to claim 1 is characterized in that: Also includes: The data is cleaned and structured and stored in the database.

3. The information retrieval enhancement method based on the multi-way recall mode according to claim 2 is characterized in that: The steps of cleaning and structuring the data and storing it in the database include: Clean, segment, and replace synonyms on documents, store the processed data in Elasticsearch, and build the document segmentation and inverted index; The document content is extracted and unstructured data is converted. The data features are extracted using a vectorization model, the text is converted into vector form, and stored in the Milvus database.

4. The information retrieval enhancement method based on the multi-way recall mode according to claim 3 is characterized in that: The vectorized models include Langchain, M3E, and BGE.

5. The information retrieval enhancement method based on the multi-way recall mode according to claim 4 is characterized in that: The steps of analyzing the user's search terms to obtain the search result data specifically include: Generate user search terms through word segmentation, synonym replacement, and query rewriting for search in elasticsearch; By using the vectorization conversion model embedding, the user's search terms are converted into vectors to generate semantic vectors for searching in the vector library.

6. The information retrieval enhancement method based on the multi-way recall mode according to claim 5 is characterized in that: The step of performing multi-channel recall on the search result data to obtain recall information includes: Recall data from the milvus vector library through a search algorithm; Or you can search from elasticsearch with keywords to return the recalled information.

7. The information retrieval enhancement method based on the multi-way recall mode according to claim 6 is characterized in that: The step of sorting the recall information to obtain a sorting result is specifically as follows: After recalling the topM results, the results are preliminarily sorted through the coarse ranking model and the topN results are returned.

8. The information retrieval enhancement method based on the multi-way recall mode according to claim 7 is characterized in that: The steps of re-sorting the sorting results to obtain the final search results are specifically: Based on the topN results returned in the rough ranking, the re-ranking technology is used to evaluate the relevance of the context, and finally the topK results are returned as the final retrieval results.

9. The information retrieval enhancement method based on the multi-way recall mode according to claim 8 is characterized in that: The reordering technology includes a reordering model BC Embedding and a large language model LLM.

10. An information retrieval enhancement system based on a multi-way recall mode, characterized in that: include: Data preprocessing and structuring module, used to clean and structure the data and store it in the database; Search word analysis module, used to analyze user search words and obtain search result data; A multi-channel recall module, used for performing multi-channel recall on the search result data to obtain recall information; A sorting module, used to sort the recall information to obtain a sorting result; The rearrangement module is used to re-arrange the sorting results to obtain the final search results.

Images