An information retrieval method, device, program product, and storage medium

Abstract

The application provides an information retrieval method, device, program product and storage medium; the method comprises the following steps: receiving to-be-retrieved content, and analyzing the to-be-retrieved content to obtain first information related to the to-be-retrieved content; determining a first retrieval tool set according to the first information, and using each retrieval tool in the first retrieval tool set to retrieve the to-be-retrieved content to obtain an initial retrieval result set corresponding to the to-be-retrieved content; judging whether it is necessary to determine a first retrieval sub-content set corresponding to the to-be-retrieved content and determine a second retrieval tool set according to the initial retrieval result set and the to-be-retrieved content; when it is judged that it is necessary to determine the first retrieval sub-content set and determine the second retrieval tool set, using each retrieval tool in the second retrieval tool set to retrieve corresponding first retrieval sub-contents in the first retrieval sub-content set to obtain a first intermediate retrieval result set; and determining a final retrieval result corresponding to the to-be-retrieved content according to the initial retrieval result set and the first intermediate retrieval result set.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to an information retrieval method, device, program product, and storage medium. Background Technology

[0002] Currently, there are many information retrieval methods, such as using search engines to retrieve information by matching keywords in user queries with document content, direct question-answering systems based on large language models (LLMs) that directly utilize the powerful generative capabilities of large language models to answer user questions, and retrieval augmented generation (RAG) systems that combine information retrieval and LLM generative capabilities to compensate for the shortcomings of pure LLMs in terms of knowledge timeliness and factual accuracy. IBM Deep Search uses artificial intelligence technology to focus on collecting, transforming, managing, and ultimately searching large collections of documents. However, the retrieval results from these information retrieval methods still cannot meet user needs. Therefore, providing more accurate retrieval results has become an urgent issue to be addressed. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this application provide an information retrieval method, device, program product, and storage medium.

[0004] The information retrieval method provided in this application includes: Receive the content to be searched and analyze the content to be searched to obtain first information related to the content to be searched, wherein the first information is information for obtaining search results corresponding to the content to be searched; Based on the first information, a first set of search tools is determined, and each search tool in the first set of search tools is used to search the content to be searched to obtain an initial set of search results corresponding to the content to be searched. Based on the initial search result set and the content to be searched, determine whether it is necessary to determine the first set of search sub-contents corresponding to the content to be searched and the second set of search tools; When it is determined that it is necessary to determine the first set of search sub-contents and the second set of search tools, each search tool in the second set of search tools is used to retrieve the corresponding first sub-content in the first set of search sub-contents to obtain the first intermediate set of search results. The final search result corresponding to the content to be searched is determined based on the initial search result set and the first intermediate search result set.

[0005] The information retrieval device provided in this application includes: a processor and a memory, the memory being used to store computer programs, and the processor being used to call and run the computer programs stored in the memory to execute the above-described information retrieval method.

[0006] This application provides a computer program product, comprising: a computer program that, when executed by a processor, implements the aforementioned information retrieval method.

[0007] The computer-readable storage medium provided in this application is used to store a computer program that causes a computer to execute the above-described information retrieval method.

[0008] In the technical solution of this application, the following steps are taken: receiving content to be searched and analyzing the content to be searched to obtain first information related to the content to be searched, wherein the first information is information for obtaining search results corresponding to the content to be searched; determining a first set of search tools based on the first information, and using each search tool in the first set of search tools to search the content to be searched to obtain an initial set of search results corresponding to the content to be searched; determining whether it is necessary to determine a first set of search sub-contents and a second set of search tools based on the initial set of search results and the content to be searched; when it is determined that it is necessary to determine the first set of search sub-contents and the second set of search tools, using each search tool in the second set of search tools to search the corresponding first sub-contents in the first set of search sub-contents to obtain a first intermediate set of search results; and determining the final search result corresponding to the content to be searched based on the initial set of search results and the first intermediate set of search results. In this way, by judging the initial search results of the content to be searched, it is determined whether to continue searching. Once it is determined that to continue searching, the updated set of search tools and search sub-contents are used to obtain the intermediate set of search results for the content to be searched, thereby dynamically realizing information extraction and improving the accuracy of the final search results. At the same time, one or more search tools are called to obtain search results, which greatly improves the comprehensiveness and depth of information acquisition. It can flexibly cope with different types of information needs and data sources, and also improves the accuracy of the final search results. Attached Figure Description

[0009] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0010] Figure 1 This is a flowchart illustrating the information retrieval method provided in the embodiments of this application. Figure 1 ; Figure 2 This is a schematic diagram of the structure of the information retrieval system provided in the embodiments of this application; Figure 3 This is a flowchart illustrating the information retrieval method provided in the embodiments of this application. Figure 2 ; Figure 4This is a schematic diagram of the structural composition of the information retrieval device provided in the embodiments of this application; Figure 5 This is a schematic structural diagram of an information retrieval device provided in an embodiment of this application; Figure 6 This is a schematic structural diagram of the chip according to an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0012] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0013] It should also be noted that the terms "first," "second," and "third" used in the embodiments of this application are only used to distinguish similar objects and do not represent a specific order of objects. It is understood that "first," "second," and "third" can be interchanged in a specific order or sequence where permissible, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein. The term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. It should also be understood that the "instruction" mentioned in the embodiments of this application can be a direct instruction, an indirect instruction, or an indication of an association relationship. For example, A instructing B can mean that A directly instructs B, for example, B can be obtained through A; it can also mean that A indirectly instructs B, for example, A instructs C, and B can be obtained through C; or it can mean that there is an association relationship between A and B. It should also be understood that the term "correspondence" mentioned in the embodiments of this application may indicate a direct or indirect correspondence between the two, or an association between the two, or a relationship such as instruction and being instructed, configuration and being configured.

[0014] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and they all fall within the protection scope of the embodiments of this application.

[0015] Existing methods for information retrieval mainly include traditional keyword search, direct question-answering systems based on large language models (LLM), retrieval-enhanced generation (RAG) systems, and deep retrieval. These search methods are described below: Traditional keyword search: Traditional search engines primarily retrieve information by matching keywords in a user's query with document content. These search engines rely on pre-built indexes and ranking algorithms to assess document relevance based on surface characteristics such as keyword frequency and location, and then provide search results. Specifically, the user enters one or more keywords as a query, the system searches a vast document index for documents containing these keywords, and then ranks the retrieved documents using various relevance algorithms (e.g., TF-IDF, PageRank, etc.), presenting the most relevant results to the user.

[0016] Direct question-answering systems based on large language models (LLM): These systems directly leverage the powerful generative capabilities of large language models to answer user questions. During the pre-training phase, LLMs have learned from massive amounts of text data and accumulated extensive knowledge. When a question is received, they attempt to generate an answer directly based on their internal knowledge base and reasoning ability. Specifically, the user submits a natural language question to the LLM, which uses its vast number of parameters and the knowledge encoded in the training data to directly generate a text as an answer.

[0017] Search Enhancement Generation (RAG) System: The RAG system combines information retrieval and LLM (Language and Literature Management) generation capabilities, aiming to compensate for the shortcomings of pure LLM in terms of knowledge timeliness and factual accuracy. Its basic idea is that when a user submits a query, the system first retrieves relevant documents or text fragments from an external knowledge base. This retrieved information is then provided to the LLM as additional context, and the LLM generates an answer based on this external information. Specifically, the implementation involves: Query embedding and search: After a user submits a query, the system generates an embedding vector for that query. Subsequently, the system searches a vector database for text fragments semantically most similar to the query embedding. This method utilizes semantic similarity rather than simple keyword matching to find relevant content; Context retrieval: Relevant text fragments are used as contextual information and input into the LLM along with the user's query; Answer generation: The LLM uses this externally provided contextual information, combined with its own language generation capabilities, to generate a coherent and fact-based answer.

[0018] Deep search: Some deep search technologies leverage artificial intelligence to focus on collecting, transforming, managing, and ultimately searching large collections of documents, such as patent documents and research papers. These systems aim to make specific information that is too technical or difficult to process for traditional search tools accessible. They collect data from diverse sources, including public, private, structured, and unstructured data, and use advanced AI methods to convert PDF documents into easily parsed JSON format, forming a unified schema. Subsequently, the system applies specialized natural language processing and computer vision machine learning algorithms to these documents, ultimately constructing a searchable knowledge graph. Specifically, deep search implementations emphasize data ingestion, standardization, knowledge graph construction, and knowledge graph-based deep search. It excels in handling unstructured data and building structured knowledge representations.

[0019] However, these search methods also have drawbacks, specifically: Traditional RAG systems struggle with complex and multi-hop problems. While they can enhance LLM responses with external data, they underperform when dealing with complex, multi-hop problems requiring the connection of multiple information fragments or reasoning across multiple documents. This is primarily because they typically rely on traditional vector similarity searches, which, while finding semantically similar content, lack an understanding of how facts are connected, making them ill-suited for problems requiring multi-step logical reasoning or involving multiple entities. For example, if a problem requires retrieving an entity from document A, then using that entity to query document B, and finally synthesizing information from both to arrive at an answer, traditional RAG systems often fall short. Furthermore, existing RAG systems lack iterative query refinement and contextual filtering mechanisms. They typically perform a single retrieval and pass the result to the LLM for response. When the initial retrieval results are irrelevant, incomplete, or inaccurate, the system lacks effective mechanisms to dynamically modify or enhance the initial query to iteratively optimize search results and filter contextual evidence. This means the quality of the LLM's response is largely limited by the quality of the initial retrieval; if the initial information is biased or insufficient, subsequent generation may also be inaccurate or incomplete. This one-shot retrieval model limits the system's adaptability and accuracy in complex scenarios. Difficulty in dynamically selecting and orchestrating multiple search tools: Existing systems typically rely on single or limited retrieval strategies, such as pure vector search or keyword search. They cannot intelligently select and coordinate the use of multiple heterogeneous search tools (such as vector search, keyword search, web crawlers, structured database queries, etc.) based on the complexity of the question, the type of information required, or intermediate inference results. This limitation restricts the comprehensiveness and depth of information retrieval, making it difficult to handle complex queries that require information from different types of data sources (such as unstructured text, structured tables, and real-time web page information). Susceptibility to training data bias or initial retrieval bias: LLMs themselves may have inherent biases towards specific keywords or contexts. Without dynamically updated internal knowledge and effective self-correction mechanisms, existing systems struggle to effectively mitigate these biases. If the initial retrieval results are biased or inaccurate, this bias can be amplified, leading to biased or inaccurate answers from the LLM. Insufficient contextual understanding and semantic relationship recognition: Traditional keyword search and some RAG systems have limitations in understanding the deep contextual meaning of queries and recognizing complex semantic relationships between data. These methods often remain at the level of superficial keyword matching, failing to extract deeper insights from complex datasets, resulting in search results lacking detail and relevance. Therefore, maximizing the accuracy of information retrieval becomes a crucial consideration. To address this, the following technical solutions, as proposed in this application, are presented.

[0020] It should be noted that the retrieval tool in the embodiments of this application can also be called a search tool.

[0021] Figure 1 This is a flowchart illustrating the information retrieval method provided in the embodiments of this application. Figure 1 ,like Figure 1 As shown, the information retrieval method includes the following steps: Step 101: Receive the content to be searched and analyze it to obtain the first information related to the content to be searched.

[0022] The first piece of information is the information that yields the search results corresponding to the content to be searched.

[0023] In some implementations, the first information includes one or more of the following: intent information of the content to be retrieved, key entity information of the content to be retrieved, topic information of the content to be retrieved, complexity information of the content to be retrieved, tool information required by the content to be retrieved, and information on the sources of information related to the content to be retrieved.

[0024] In some implementations, the information retrieval method is applied to an information retrieval system, which receives content to be retrieved.

[0025] In some implementations, the content to be retrieved can be in the form of text, images, or voice. When the information retrieval system receives an image, it will identify the content to be retrieved in the image and generate the corresponding text. When the large language model receives voice, it will convert the voice into text. This application does not specifically limit the specific recognition and conversion methods.

[0026] For example, a user submits a natural language query, such as "What materials does a certain compound react with at different temperatures, and what are the mechanisms of these reactions?" Here, the natural language query is the content to be retrieved.

[0027] In some implementations, the step of analyzing the content to be retrieved to obtain first information related to the content to be retrieved includes: The content to be retrieved is input into a large language model, which analyzes the content to be retrieved to obtain first information related to the content to be retrieved.

[0028] In some implementations, the information retrieval system includes a large language model (LLM). After receiving the information to be retrieved, the information retrieval system inputs the content to be retrieved into the large language model, and the large language model analyzes the content to be retrieved to obtain first information related to the content to be retrieved.

[0029] Specifically, LLM performs preliminary analysis of user queries to understand their intent and identify key entities, topics, and potential complexities (e.g., determining whether it is a multi-hop problem requiring multiple steps of reasoning).

[0030] In some implementations, the large language model, based on preliminary analysis, determines what types of tools and information sources might be needed. For example, when the search query is "What materials does a certain compound react with at different temperatures, and what are the mechanisms of these reactions?", the large language model determines that chemical databases, materials science literature, and explanations of reaction mechanisms might be needed.

[0031] Step 102: Determine the first set of search tools based on the first information, and use each search tool in the first set of search tools to search for the content to be searched to obtain the initial set of search results corresponding to the content to be searched.

[0032] In some embodiments, the information retrieval system includes a multi-source search tool module, which includes various retrieval tools, such as vector retrieval tools, keyword retrieval tools, web crawler retrieval tools, and structured data query tools. The specific retrieval tools included can be determined according to the actual situation, and this application does not make specific limitations in this regard.

[0033] In some implementations, vector retrieval tools are used to retrieve text fragments or documents based on semantic similarity. They can understand the deeper meaning of queries and find relevant content even if keywords do not match exactly. Keyword retrieval tools are used to precisely match literal keywords, particularly suitable for finding specific entities, phrases, or known information, compensating for the lack of literal precision in pure vector search. Web crawler retrieval tools are used to obtain the latest or specific domain-specific publicly available information from the internet, enabling the system to access real-time or dynamic information beyond pre-trained data. Structured data query tools are used to retrieve structured information from databases or knowledge graphs, which is crucial for complex queries that require connecting multiple facts and performing multi-hop reasoning, as knowledge graphs can explicitly represent the relationships between entities. Furthermore, the multi-source search tool module also includes hybrid search capabilities, meaning the information retrieval system supports hybrid search methods combining vector search and keyword search to compensate for the shortcomings of pure vector search in certain literal queries, thereby improving the comprehensiveness and accuracy of the retrieval.

[0034] In some implementations, the weight of each search tool is determined based on the historical final search result set and the historical search content set, with each historical final search result in the historical final search result set corresponding one-to-one with each historical search content in the historical search content set.

[0035] Here, the historical final search result set is the final search result set obtained by the information retrieval system after processing each historical search content in the historical search content set. The information retrieval result system determines the weight of each search tool based on each historical final search result.

[0036] In some implementations, type information for each historical content to be retrieved is obtained, a set of retrieval tools used to obtain each historical final retrieval result is obtained, and weight information for each retrieval tool is determined based on the type information and the set of retrieval tools.

[0037] The type information here refers to the task type corresponding to the content to be retrieved. For example, the task type is a multi-hop task.

[0038] Here, for each historical content to be searched, after obtaining its corresponding final search results, the weight of the search tool in the search process for that type of information will be increased if the information was retrieved using the search tool, otherwise the weight of the search tool in the search process for that type of information will be decreased.

[0039] Specifically, in the initial stage of system operation, each retrieval tool has an initial weight. As the system processes each query, the LLM will "reward" (increase weight) or "penalize" (decrease weight) a tool's weight based on which tool's information is ultimately used in answer synthesis. This mechanism enables the system to continuously learn and optimize its tool selection strategy, prioritizing the use of tools that perform well on specific task types.

[0040] In some implementations, the sum of the weights of the various retrieval tools equals 1.

[0041] In some implementations, determining a first set of search tools based on the first information and using the first set of search tools to retrieve the content to be searched to obtain an initial set of search results corresponding to the content to be searched includes: The large language model determines a first set of search tools based on the first information, and calls each search tool in the first set of search tools to search for the content to be searched, thereby obtaining the initial search results corresponding to each search tool; The initial set of search results is determined based on the initial search results corresponding to each search tool.

[0042] In some implementations, the large language model selects a corresponding retrieval tool from the multi-source search tool module based on the first information to form a first retrieval tool set. Each retrieval tool in the first retrieval tool set is used to retrieve the content to be retrieved to obtain an initial retrieval result set corresponding to the content to be retrieved.

[0043] It is understandable that the initial search result set here includes the initial search results obtained by each search tool by searching the content to be searched.

[0044] In some implementations, a set of search tools to be determined is determined based on the type information of the content to be searched and the first information. The top N search tools are selected according to the weight ranking of each search tool in the set of search tools to be determined. The first set of search tools includes the top N search tools, where N is a positive integer.

[0045] It is understandable that different search tools have different weights for different types of search content. After obtaining the content to be searched, the type information of the content to be searched is determined, and then the weights of the search tools corresponding to that type information are sorted to obtain a set of search tools.

[0046] In some implementations, the initial search result set includes one or more of the following: document information, web page information, image information, video information, and knowledge graph entries. The specific information retrieved can be determined based on the actual situation, and this application does not impose any specific limitations on it.

[0047] In some implementations, the initial search result set is sent to a large language model, which then parses and understands the initial search result set.

[0048] In some implementations, the large language model calls one or more tools from the multi-source search tool module, that is, determines the first set of search tools, combines the initial search results retrieved by the first set of search tools, and sends them to the large language model, which processes them to extract key information, facts, entities and their relationships.

[0049] In some implementations, after obtaining the initial set of search results, the large language model converts the data into a unified representation format to facilitate subsequent reasoning and integration.

[0050] In some implementations, the uniform representation format is JSON.

[0051] In some implementations, the LLM invokes one or more tools from the multi-source search tool module. For example, the information retrieval system might first invoke a vector search tool to perform a vector search to retrieve document fragments related to the semantics of compounds and reactions from an internal document library; simultaneously, it might invoke a keyword search tool to perform a keyword search to ensure accurate matching of compound names or specific reaction types. The retrieved raw data, such as document content, web page information, and knowledge graph entries, is passed to the LLM for parsing and understanding. The LLM extracts key information, facts, entities, and their relationships from this data and may convert it into a unified internal representation format (such as JSON) to facilitate subsequent reasoning and integration.

[0052] Step 103: Based on the initial search result set and the content to be searched, determine whether it is necessary to determine the first set of search sub-contents corresponding to the content to be searched and the second set of search tools.

[0053] Here, the first search sub-content is a portion of the search content to be searched, which is more specific sub-query information of the search content, that is, information that was not found in the initial search result set or information that needs to be searched again.

[0054] In some implementations, determining whether it is necessary to determine the first set of search sub-contents corresponding to the search content and the second set of search tools based on the initial search result set and the search content includes: The large language model determines the quality information of the initial search result set based on the initial search result set and the content to be searched; The large language model determines, based on the quality information, whether it is necessary to determine the first set of search sub-contents corresponding to the content to be searched and the second set of search tools corresponding to the first set of search sub-contents.

[0055] Here, the second set of search tools and the first set of search tools can be completely the same, partially the same, or completely different. The second set of search tools is determined based on the first set of search sub-contents.

[0056] In some implementations, after obtaining the initial search result set, the LLM constructs the search results for the content to be searched based on the initial search result set.

[0057] It is understandable that the search results here are answers to the user's search query.

[0058] In some implementations, during the process of constructing the search results for the content to be searched based on the initial search result set, it is determined whether it is necessary to determine the first set of search sub-contents corresponding to the content to be searched and the second set of search tools corresponding to the first set of search sub-contents based on the initial search result set and the content to be searched.

[0059] In some implementations, when the initial search result set meets a second condition, the quality information of the initial search result set is classified as first-level quality; otherwise, it is classified as second-level quality. The second condition includes one or more of the following: the initial search result set contains information gaps; the initial search result set contains contradictory information; the initial search result set does not include all the key points of the content to be searched. The first-level quality refers to the quality of search results where the content to be searched cannot be obtained, while the second-level quality refers to the quality of search results where the content to be searched can be obtained.

[0060] It is understandable that all the key points of the content to be searched are the key to fully representing the user's search intent, which means there is still the possibility of further in-depth exploration.

[0061] In some implementations, when the quality information of the initial search result set is at the first level, the large language model determines that it is necessary to determine the first search sub-content set corresponding to the content to be searched; and determines the second search tool set based on the first search sub-content set; when the quality information of the initial search result set is at the second level, the large language model determines that it is not necessary to determine the first search sub-content set and the second search tool set corresponding to the first search sub-content set.

[0062] In some implementations, when it is determined that it is not necessary to determine the first set of search sub-contents corresponding to the content to be searched and the second set of search tools corresponding to the first set of search sub-contents, the final set of search results is determined based on the initial set of search results.

[0063] In some implementations, the quality information includes completeness, accuracy, and consistency information of the initial search results.

[0064] In some implementations, a completeness score, an accuracy score, and a coherence score of the initial search result set are obtained based on the initial search result set and the content to be searched. A comprehensive score of the initial search result set is then determined based on the completeness score, accuracy score, and coherence score, wherein the comprehensive score is used to represent the quality information of the initial search result set.

[0065] In some implementations, when the overall score is greater than or equal to a third threshold, it is determined that it is not necessary to determine the first set of search sub-contents corresponding to the content to be searched and the second set of search tools corresponding to the first set of search sub-contents; when the overall score is less than the third threshold, it is determined that it is necessary to determine the first set of search sub-contents corresponding to the content to be searched and the second set of search tools corresponding to the first set of search sub-contents.

[0066] In some implementations, the completeness score, accuracy score, and coherence score correspond to a first weight, a second weight, and a third weight, respectively. The comprehensive score of the initial search result set is determined based on the completeness score, the accuracy score, the coherence score, the first weight, the second weight, and the third weight.

[0067] Among them, completeness is used to determine whether the initial search result set is missing key information of the content to be searched, accuracy is used to determine whether the initial search result set is correct, and coherence is used to determine the logical degree of the initial search result set.

[0068] In some implementations, the LLM performs preliminary reasoning based on the extracted information, attempting to construct an answer to the user's query. During this process, the LLM simultaneously evaluates the sufficiency and accuracy of the current information, i.e., assessing the quality of the initial search result set. Specific quality assessment can be based on whether the initial search result set contains information gaps, contradictions, or aspects requiring further investigation. Specifically, the system identifies whether multi-hop reasoning is needed to fully answer the question. For example, if the first round of searching only finds the reactants of a compound but not the reaction mechanism, an information gap related to "mechanism" is identified. If the information is insufficient or the question is complex, the LLM identifies "information gaps" that need to be filled or "sub-problems" that need to be addressed, i.e., redefining the search sub-content corresponding to the content to be searched.

[0069] In some implementations, when it is determined that it is not necessary to determine the first set of search sub-contents and the second set of search tools, the final search result is determined based on the initial set of search results.

[0070] Step 104: When it is determined that the first set of search sub-contents and the second set of search tools need to be determined, use each search tool in the second set of search tools to retrieve the corresponding first search sub-content in the first set of search sub-contents to obtain the first intermediate search result set.

[0071] In some implementations, when the quality of the initial search result set is determined to be low, a second set of search tools needs to be determined, and each search tool in the second set of search tools is used to perform a new round of information retrieval and analysis to obtain a first intermediate set of search results.

[0072] Here, the first intermediate search result set is a supplement to the aforementioned initial search result set.

[0073] Specifically, when an information gap is detected or further exploration is needed, the LLM dynamically refines the original query or generates new, more specific subqueries based on the currently accumulated knowledge and preliminary reasoning results. The LLM then intelligently selects or switches to more suitable search tools based on the refined query and current information needs; that is, it determines whether the first set of search tools needs to be updated to obtain a second set. For example, if knowledge in a specific domain is needed, a knowledge graph tool might be invoked; if the latest information is needed, a web crawler might be invoked; if facts need to be verified, multiple search tools might be invoked for cross-validation.

[0074] Here, each first search sub-content corresponds to an intermediate search result, and each intermediate search result constitutes the first intermediate search result set.

[0075] Step 105: Determine the final search results corresponding to the content to be searched based on the initial search result set and the first intermediate search result set.

[0076] In some implementations, determining the final search result corresponding to the content to be searched based on the initial search result set and the first intermediate search result set includes: Obtain the first evaluation value of the initial search result set; Obtain a second evaluation value for the first search result set, wherein the first search result set includes the initial search result set and the first intermediate search result set; When the difference between the first evaluation value and the second evaluation value is less than the first threshold, the final search result is determined based on the first search result set.

[0077] It is understandable that the first set of search results here is the current set of search results for the content to be searched.

[0078] In some implementations, the first evaluation value and the second evaluation value are confidence scores.

[0079] In some implementations, the first threshold is 0.05.

[0080] Specifically, if the second evaluation value of the first search result set is less than the first evaluation value of the initial search result set, and the difference is less than 0.05, the quality of the first search result set is considered to be good, and the final search result is determined based on the first search result set.

[0081] In some embodiments, the method further includes: when the difference between the first evaluation value and the second evaluation value is greater than or equal to a first threshold, iteratively executing the first process until the first condition is met; The first process includes: Based on the first set of search results and the content to be searched, determine whether it is necessary to determine the second set of search sub-contents corresponding to the content to be searched and the third set of search tools; When it is determined that it is necessary to determine the second search sub-content set and the third search tool set, the first search result set is updated according to the third search tool set, the second search sub-content set and the first search result set; The first condition includes one or more of the following conditions: The number of times the first process is executed iteratively is greater than or equal to the second threshold; The difference between the evaluation value of the first set of search results obtained in this round and the evaluation value of the first set of search results obtained in the previous round is less than the first threshold.

[0082] In some implementations, updating the first search result set based on the third search tool set, the second search sub-content set, and the first search result set includes: Use each search tool in the third search tool set to search for the corresponding second search sub-content in the second search sub-content set to obtain the second intermediate search result set; Update the first search result set based on the second intermediate search result set.

[0083] It can be understood that here, the updated first search result set includes the initial search result set, the first intermediate search result set, and the second intermediate search result set. That is, after each round of executing the first process, the obtained intermediate search result set is merged into the current search result set to update the current search result set.

[0084] In some implementations, when the number of iterations of the first process is greater than or equal to a second threshold, the iteration is stopped, and the final search result is determined based on the first search result set obtained in the last round.

[0085] In some implementations, when the difference between the evaluation value of the first search result set obtained in the current round and the evaluation value of the first search result set obtained in the previous round is less than a first threshold, the final search result is determined based on the first search result set obtained in the current round. Here, the first search result set obtained in the current round is the last obtained first search result set.

[0086] It is understandable that the first set of search results in each round includes the set of intermediate search results obtained in the current round and the set of search results obtained in each previous round.

[0087] It should also be understood that the process of determining whether to determine the second set of search sub-contents and the third set of search tools corresponding to the searched content based on the first set of search results and the searched content is consistent with the aforementioned process of determining whether to determine the first set of search sub-contents and the second set of search tools corresponding to the searched content based on the initial set of search results and the searched content. That is, it involves judging the quality information of the first set of search results, then judging whether to determine the second set of search sub-contents, and finally judging whether to determine the third set of search tools. In other words, each iteration requires judging the quality information of the current set of search results after the current iteration, and based on the quality information, judging whether to determine new search sub-contents and a new set of search tools.

[0088] In some implementations, when the difference between the first evaluation value and the second evaluation value is less than a first threshold, the final search result set can be determined directly based on the current search result set, that is, the final search result is determined based on the first search result set. When the difference between the first evaluation value and the second evaluation value is greater than or equal to the first threshold, the second search sub-content corresponding to the content to be searched and the third search tool set are determined based on the first search result set. The search tools in the third search tool set are then used to re-search to obtain the second intermediate search result set, and the first search result set is updated. This process continues until the difference between the evaluation value of the first search result set after this round of execution and the evaluation value of the first search result set obtained in the previous round is less than the first threshold, or the number of times the first process is iterated is greater than or equal to the second threshold. In this case, the first search result set obtained in the last round of execution of the first process is used to determine the final search result.

[0089] It can be understood that the difference between the first evaluation value and the second evaluation value being less than the first threshold means that the difference between the second evaluation value and the first evaluation value is less than the first threshold.

[0090] It should also be understood that the first set of search results obtained in this round is the first set of search results updated after the first process is executed in this round, and the first set of search results obtained in the previous round is the first set of search results updated after the first process is executed in the previous round. Each update of the first set of search results is to add the intermediate set of search results obtained in this round by using the search tool to search the corresponding search sub-content to the current set of search results obtained in the previous round.

[0091] It is understandable that this involves a new round of information retrieval and reading through new search sub-content and search tools, thus forming a closed-loop iterative process. This cycle will continue until LLM determines that it has obtained enough information or reached a preset confidence threshold, or the information gain (marginal gain) brought by further iterations has become negligible.

[0092] Specifically, the confidence score is evaluated as follows: After each iteration, LLM generates a confidence score for the accumulated knowledge base and preliminary answer. This score can be a value between 0 and 1, used to evaluate the completeness, accuracy, and consistency of the answer; Marginal gain calculation: The confidence score of the current iteration is compared with the score of the previous iteration. If the difference between the two scores (i.e., the marginal gain) is less than a preset threshold (i.e., the aforementioned first threshold), it indicates that the performance improvement brought by continuing iteration is negligible; Termination condition: The confidence score of the current iteration is compared with the score of the previous iteration. That is, when the confidence marginal gain of two consecutive iterations is lower than the preset threshold, or when the maximum number of iterations is reached, the iteration is terminated to prevent infinite loops in extreme cases.

[0093] It is understandable that the final search results here are the answers to the searched content.

[0094] In some implementations, the information retrieval system includes a feedback and self-correction module that evaluates each retrieval result and determines whether the information retrieval system needs to be updated.

[0095] Specifically, information retrieval systems continuously evaluate task completion and the correctness of answers through reflection mechanisms. For example, LLM might check the logical consistency of the generated answer or perform cross-validation with multiple information sources. If errors or inconsistencies are found, the system adjusts its reasoning path or re-executes steps, thus achieving self-correction. This helps mitigate potential biases in LLM and improves the reliability of the final answer.

[0096] The feedback and self-correction module uses a multi-dimensional weighted scoring formula to evaluate the quality of the current answer. This formula comprehensively considers multiple quantifiable indicators, such as the completeness of the answer (whether it covers all sub-questions), accuracy (whether there are factual conflicts), and logical coherence. When the comprehensive score reaches a preset threshold, the system determines that the answer is sufficient and terminates the iteration. Specifically, the evaluation of whether the current retrieval and reasoning results are sufficient to complete the task, or whether there are information gaps, contradictions, or aspects requiring further exploration, is typically performed by LLM (Limited Language Management) and may be combined with preset evaluation criteria. Through targeted detection of information gaps and re-query mechanisms, the system can mitigate potential biases of LLM towards specific keywords or contexts. By evaluating task completion and correctness, it provides feedback for iterative improvement and self-correction. Based on the feedback, the system can adjust its internal strategies, such as optimizing text segmentation strategies, improving retrieval strategies, or adjusting the answer generation method, to continuously improve performance.

[0097] It is understandable that large language models can also be used to determine the quality of the initial search result set and the subsequent process of updating the current search result set.

[0098] In some embodiments, the method further includes: Store the initial search result set, the first intermediate search result set, and the final search result; Present the final search results and the source information of the final search results.

[0099] In some implementations, the information retrieval system includes a knowledge base and a memory module for storing the initial set of retrieval results, the first set of intermediate retrieval results, and the final retrieval results.

[0100] Here, the knowledge base and memory module are responsible for storing and managing the knowledge accumulated by the system during the iteration process, as well as maintaining the context of the dialogue or task. Understandably, multiple sets of intermediate retrieval results will be stored in the knowledge base and memory module.

[0101] In some implementations, when the difference between the first evaluation value and the second evaluation value is greater than or equal to the first threshold, after iteratively executing the first process, the set of intermediate retrieval results obtained in each round will also be stored.

[0102] Specifically, in each iteration, LLM integrates newly acquired and verified information into its dynamically evolving internal knowledge base. This enables the system to continuously learn and optimize its understanding of the world.

[0103] In some implementations, the information retrieval system includes a presentation module for presenting the final retrieval results and the source information of the final retrieval results.

[0104] Specifically, once the LLM determines that it has acquired sufficient information and completed all necessary reasoning steps, it integrates the knowledge accumulated throughout the iterations to generate a comprehensive, accurate, and coherent final answer. The system can provide citations or links to the sources of the answer to enhance transparency and credibility. For example, it can list the cited documents, web pages, or knowledge graph entries.

[0105] The technical solution provided in this application embodiment receives content to be searched and analyzes the content to be searched to obtain first information related to the content to be searched, the first information being used to obtain search results corresponding to the content to be searched; determines a first set of search tools based on the first information, and uses each search tool in the first set of search tools to search the content to be searched to obtain an initial set of search results corresponding to the content to be searched; determines whether it is necessary to determine the search sub-content corresponding to the content to be searched and update the first set of search tools based on the initial set of search results and the content to be searched; when it is necessary to determine the search sub-content and update the first set of search tools, uses each search tool in the updated first set of search tools to search the search sub-content to obtain an intermediate set of search results; and determines the final search result corresponding to the content to be searched based on the initial set of search results and the intermediate set of search results. Thus, by judging the initial search results for the content to be searched, it is determined whether further searching is needed. Once it is determined that further searching is needed, an updated set of search tools and search sub-contents are used to obtain an intermediate set of search results for the content to be searched. This dynamically achieves information extraction, improves the accuracy of the final search results, and simultaneously calls one or more search tools to obtain search results, greatly improving the comprehensiveness and depth of information acquisition. This allows for flexible responses to different types of information needs and data sources, and also improves the accuracy of the final search results.

[0106] Based on this, the information retrieval method provided in the embodiments of this application will be further described.

[0107] This application proposes a deep information retrieval system, the core of which is to mimic the process of complex research and reasoning by humans, and to gradually approach the best answer through a continuous feedback loop, thereby achieving the following functions: Iterative Search and Reasoning Loop: The core of deep information retrieval lies in its unique iterative process, namely, a "loop of search, reading, and reasoning." This is fundamentally different from the traditional one-off retrieval model. In each iteration, the system performs preliminary reasoning based on the acquired information, identifies information gaps or areas requiring further exploration, then dynamically refines the query, and performs the search and reading again until the answer reaches a preset confidence level or the problem is fully resolved. This cyclical mechanism allows the system to optimize search results through continuous feedback and adjustment, thereby handling complex problems that are difficult to solve using traditional methods.

[0108] The use of multi-step tools: Deep information retrieval emphasizes using an intelligent agent framework, with a large language model (LLM) as the core controller, to dynamically select and orchestrate multiple tools to solve complex problems. This means that the system is not limited to a single retrieval method, but can intelligently invoke different external toolchains, such as vector search, keyword search, web crawlers, or even specific structured database query tools, depending on the complexity of the problem, the type of information required, or intermediate inference results. This use of multi-step tools enables the system to handle complex problems that require progressively refined information to obtain an accurate answer, such as multi-hop question answering.

[0109] Agent-Driven Control Flow: This application employs a novel LLM-driven agent framework. In this framework, the LLM is not merely a tool for generating text; it acts more like a decision-maker, dynamically determining the application's control flow. This means the LLM can autonomously route (choose the next action), select tools (determine which tool to invoke), and determine whether the currently generated answer is sufficient or requires further work. This agent capability enhances the system's reasoning ability and adaptability by leveraging dynamically evolving knowledge to iteratively refine queries and filter contextual evidence.

[0110] Figure 2 This is a schematic diagram of the structure of the information retrieval system provided in the embodiments of this application, such as... Figure 2 As shown, it includes a large language model (LLM) core module, a multi-source search tool module, a knowledge base and memory module, and a feedback and self-correction module, which work together to achieve deep information retrieval capabilities. A detailed description follows: Large Language Model (LLM) Core Modules: As the "brain" of the entire deep information retrieval system, the LLM core module is responsible for advanced decision-making, reasoning, and coordination.

[0111] Reasoning and Planning: LLM can perform deep reasoning on retrieved information, identify the relationships between information, and formulate multi-step plans to achieve the goal of solving complex problems. This includes breaking down complex problems into manageable subproblems and planning the steps to solve these subproblems.

[0112] Tool selection and orchestration: LLM intelligently selects and invokes appropriate external tools based on the progress and intermediate results of the current task. For example, it invokes vector search when semantic similarity information is needed, web crawlers when the latest publicly available information is needed, and knowledge graphs when precise facts are needed.

[0113] Query Refinement and Sub-problem Generation: Based on the reasoning results and the information obtained, LLM can dynamically refine the original query or decompose a complex problem into multiple more specific and easier-to-handle subqueries or sub-problems.

[0114] Answer synthesis: After all the necessary information gathering and reasoning steps are completed, LLM will synthesize information from all sources to generate a coherent, accurate and comprehensive final answer.

[0115] Multi-source search tool module: This system introduces a dynamic tool weighting mechanism. In the early stages of system operation, each tool has an initial weight. As the system processes each query, the LLM will "reward" (increase weight) or "penalize" (decrease weight) the weight of the tool based on which tool's information is ultimately used to synthesize the answer. This mechanism enables the system to continuously learn and optimize its tool selection strategy, prioritizing the use of tools that perform well on specific task types.

[0116] Vector search tools: used to retrieve text fragments or documents based on semantic similarity. They can understand the deeper meaning of the query and find relevant content even if the keywords do not match exactly.

[0117] Keyword search tool: Used for precise matching of literal keywords, especially suitable for finding specific entities, phrases or known information. It makes up for the lack of literal precision in pure vector search.

[0118] Web crawler (Web Search) tools: used to retrieve the latest or publicly available information in a specific field from the Internet, enabling the system to access real-time or dynamic information beyond pre-trained data.

[0119] Structured data query tools: These are used to retrieve structured information from databases or knowledge graphs. This is crucial for complex queries that require connecting multiple facts and performing multi-hop reasoning, because knowledge graphs can explicitly represent the relationships between entities.

[0120] Hybrid search capability: This system supports a hybrid search method that combines vector search and keyword search to compensate for the shortcomings of pure vector search in certain literal queries, thereby improving the comprehensiveness and accuracy of the retrieval.

[0121] Knowledge base and memory module: This module is responsible for storing and managing the knowledge accumulated by the system during the iteration process, as well as maintaining the context of dialogues or tasks.

[0122] Dynamically evolving internal knowledge: The system continuously acquires, verifies, and integrates new information during the iteration process. This information is stored and used to guide subsequent searches and reasoning. This knowledge is dynamically updated rather than static.

[0123] Short-term memory: allows an agent to access information acquired in earlier steps of the current task sequence, which is crucial for maintaining the coherence of multi-step reasoning.

[0124] Long-term memory: enables agents to recall information from past interactions, such as historical dialogue records or similar problems solved before, which helps the system learn and adapt over a longer time scale.

[0125] Knowledge graphs: As a special form of knowledge representation, knowledge graphs are used to store entities and their relationships. They are particularly suitable for multi-hop reasoning and complex data connections because their structure can clearly show the correlation between information.

[0126] Knowledge Management Mechanism: To ensure the quality and timeliness of the knowledge base, the system has the following built-in management mechanisms: Expired information elimination: The system adds a timestamp or expiration date to each knowledge entry. When processing queries with high real-time requirements (such as news or stock information), the system will prioritize the use of the latest information and regularly clean up outdated and irrelevant data.

[0127] Conflict Fact Handling: When new information retrieved conflicts with existing facts in the knowledge base, the system does not simply replace them. Instead, LLM evaluates the credibility of the source of the new information, the cross-validation of multiple sources, and may coexist conflicting facts, while marking their source and conflict nature. This allows the system to provide a more comprehensive perspective or explain uncertainty when generating answers, thereby improving reliability.

[0128] Feedback and self-correction module: This module uses a multi-dimensional weighted scoring formula to evaluate the quality of the current answer. The formula takes into account multiple quantifiable indicators, such as the completeness of the answer (whether it covers all sub-questions), accuracy (whether there are any factual conflicts), and logical coherence. When the comprehensive score reaches a preset threshold, the system determines that the answer is sufficient and terminates the iteration.

[0129] Results evaluation: Evaluating whether the current search and reasoning results are sufficient to complete the task, or whether there are information gaps, contradictions, or aspects that need further investigation. This is usually done by LLM and may be combined with pre-set evaluation criteria.

[0130] Bias mitigation and self-correction: Through targeted detection of information gaps and re-query mechanisms, the system can mitigate potential biases in LLM regarding specific keywords or contexts. Feedback is provided to iteratively improve and self-correct by evaluating task completion and accuracy.

[0131] Adaptive adjustments: Based on feedback, the system can adjust its internal strategies, such as optimizing text segmentation strategies, improving retrieval strategies, or adjusting answer generation methods, to continuously improve performance.

[0132] Figure 3 This is a flowchart illustrating the information retrieval method provided in the embodiments of this application. Figure 2 ,like Figure 3 As shown, it includes the following steps: Step 301: Initial query reception and analysis.

[0133] A user submits a natural language query, such as: "What materials does a certain compound react with at different temperatures, and what are the mechanisms of these reactions?". The LLM core module performs a preliminary analysis of the user's query to understand their intent, identify key entities, topics, and potential complexities (e.g., determining if it is a multi-hop problem requiring multiple steps of reasoning). Based on this preliminary analysis, LLM makes an initial determination of what types of tools and information sources might be needed, such as chemical databases, materials science literature, and explanations of reaction mechanisms.

[0134] Step 302: Initial information retrieval and reading.

[0135] LLM invokes one or more tools from the multi-source search tool module. For example, the system might first perform a vector search to retrieve document fragments related to compound and reaction semantics from an internal document repository; simultaneously, it might perform a keyword search to ensure precise matching of compound names or specific reaction types. The retrieved raw data (such as document content, web page information, and knowledge graph entries) is passed to the LLM core module for "reading" (i.e., parsing and understanding). From this data, LLM extracts key information, facts, entities, and their relationships, and may convert them into a unified internal representation format (such as JSON) to facilitate subsequent reasoning and integration.

[0136] Step 303: Preliminary reasoning and information gap detection.

[0137] The core LLM module performs initial reasoning based on the extracted information, attempting to construct an answer to the user's query. During this process, LLM simultaneously evaluates the sufficiency and accuracy of the current information, identifying any information gaps, contradictions, or aspects requiring further investigation. Specifically, the system identifies whether multi-hop reasoning is needed to fully answer the question. For example, if the first round of search only finds the reactants of a compound but not the reaction mechanism, the "mechanism" information gap is identified. If the information is insufficient or the question is complex, LLM identifies "information gaps" that need to be filled or "sub-problems" that need to be addressed.

[0138] Step 304: Query refinement and tool selection.

[0139] This step is the core iterative loop process. If an information gap is detected in step 303 or further exploration is needed, the LLM will dynamically refine the original query or generate new, more specific subqueries based on the currently accumulated knowledge and preliminary reasoning results. The LLM intelligently selects or switches to more suitable search tools based on the refined query and current information needs. For example, if knowledge in a specific domain is needed, a knowledge graph tool may be invoked; if the latest information is needed, a web crawler may be invoked; if facts need to be verified, multiple tools may be invoked for cross-validation. The system returns to step 302, using the new query and tools for a new round of information retrieval and reading, thus forming a closed-loop iterative process. This loop continues until the LLM determines that sufficient information has been obtained or a preset confidence threshold has been reached, or the information gain (marginal gain) from further iterations is negligible. Confidence score evaluation: After each iteration, the LLM can generate a confidence score for the currently accumulated knowledge base and preliminary answers. This score can be a value between 0 and 1, used to evaluate the completeness, accuracy, and consistency of the answer; Marginal gain calculation: compare the confidence score of the current iteration with the score of the previous iteration. If the difference between the two scores (i.e., the marginal gain) is less than a preset threshold (e.g., 0.05), it means that the performance improvement from continuing iteration is negligible; Termination condition: compare the confidence score of the current iteration with the score of the previous iteration. The confidence marginal gain of two consecutive iterations is lower than the preset threshold. The maximum number of iterations is reached to prevent infinite loops in extreme cases.

[0140] Step 305: Knowledge Accumulation and Self-Correction.

[0141] In each iteration, LLM integrates newly acquired and verified information into its dynamically evolving internal knowledge base. This allows the system to continuously learn and optimize its understanding of the world. The system continuously evaluates task completion and the correctness of answers through reflection mechanisms. For example, LLM might check the logical consistency of the generated answer or cross-validate it with multiple information sources. If errors or inconsistencies are found, the system adjusts its reasoning path or re-executes steps, thus achieving self-correction. This helps mitigate potential biases in LLM and improves the reliability of the final answer.

[0142] Step 306: Answer synthesis and presentation.

[0143] Once the LLM determines that it has acquired sufficient information and completed all necessary reasoning steps, it will synthesize the knowledge accumulated throughout the iterations to generate a comprehensive, accurate, and coherent final answer. The system can provide citations or links to the sources of the answer to enhance transparency and credibility. For example, it can list the cited documents, web pages, or knowledge graph entries.

[0144] This application provides a method and system for effectively handling complex, multi-hop question-and-answer deep information retrieval, overcoming the limitations of existing RAG systems in complex reasoning. It implements an iterative, dynamic query refinement and context optimization mechanism to continuously improve the accuracy and relevance of information retrieval, ensuring that LLM (Learning Query Model) is always generated based on high-quality, optimized information. An agent-driven multi-tool collaborative framework is developed, capable of flexibly selecting and orchestrating various heterogeneous search tools based on question requirements and intermediate reasoning results, thereby obtaining more comprehensive and in-depth information. A dynamic knowledge accumulation and self-correction mechanism is established to reduce system bias towards specific keywords or contexts, improving the accuracy and reliability of answers. The system's ability to understand contextual meaning and semantic relationships is enhanced, enabling the extraction of deeper insights from complex data and providing more valuable answers.

[0145] This application provides an LLM-driven iterative "search, reasoning" loop mechanism. Specifically, the core of this application lies in its unique iterative process driven by a large language model (LLM) as the core controller. This process dynamically refines information retrieval through continuous "search, read, and reasoning" loops until the optimal answer is found. This represents a fundamental breakthrough from the traditional one-time retrieval mode, enabling the system to gradually optimize information through multiple rounds of feedback and adjustment, thereby effectively handling complex problems.

[0146] Dynamic multi-tool collaboration and orchestration within an agent framework: In this embodiment, the LLM acts as an agent, possessing the ability to intelligently select, configure, and invoke various heterogeneous search tools based on the current problem and intermediate inference results. These tools include, but are not limited to, vector search, keyword search, web crawling, structured database queries, and knowledge graph queries. This dynamic, seamless switching and efficient collaboration between tools significantly improves the comprehensiveness and depth of information acquisition, enabling it to flexibly respond to different types of information needs and data sources.

[0147] The system employs a feedback-based query refinement and contextual evidence filtering mechanism: It possesses strong self-evaluation capabilities, identifying information gaps or deficiencies in the current information acquisition process and dynamically modifying or enhancing the initial query or generating new subqueries accordingly. Simultaneously, the LLM intelligently filters and selects the most relevant contextual evidence, effectively reducing the interference of redundant and inaccurate information on subsequent reasoning and ensuring the quality of the context input to the LLM.

[0148] Dynamic knowledge accumulation and self-correction capabilities: During the iterative information acquisition and reasoning process, the system can continuously accumulate and update its internal knowledge. Furthermore, by introducing a reflection mechanism, the system can self-evaluate and correct its own reasoning process and results. This capability helps mitigate the inherent biases that may exist in LLM and continuously improves the accuracy and reliability of the final answer, enabling the system to learn and adapt.

[0149] Effective handling of complex multi-hop question answering: Through the combination of the aforementioned iterative mechanism and multi-tool collaboration, the embodiments of this application can systematically decompose and solve complex multi-hop problems requiring multi-step reasoning and cross-source information integration. This is a pain point that existing RAG systems struggle to address effectively. The embodiments of this application provide a more robust and accurate solution, capable of deeply uncovering the implicit relationships between information.

[0150] The advantages of this application's embodiments are: significantly improving the accuracy and precision of complex and multi-hop question answering. Specifically, the core of this application's embodiments is an LLM-driven iterative "search, reasoning" loop, and dynamic multi-tool collaboration based on an intelligent agent framework. Through dynamic query refinement and multi-step tool collaboration, the system can deeply understand the multi-level semantics of complex questions and connect multiple information fragments across documents and modalities. This mechanism overcomes the information fragmentation and insufficient precision caused by single retrieval when traditional RAG systems handle multi-hop problems. Each iteration allows the system to adjust its strategy based on the results of the previous round, gradually focusing on and refining the required information. Compared to traditional RAG systems, the iterative nature of deep information retrieval systems can significantly improve retrieval accuracy and provide more comprehensive and accurate answers. It is particularly suitable for complex problems that require deep understanding and gradual refinement of information, such as exploring complex topics in scientific research or gaining multi-dimensional insights in business analysis.

[0151] Enhanced contextual understanding and semantic relationship recognition capabilities: Deep information retrieval systems utilize advanced machine learning algorithms and natural language processing technologies to go beyond simple keyword matching. They can deeply understand the contextual meaning of queries, identify deep semantic relationships between data, and extract deeper insights from complex datasets. This is thanks to the powerful semantic understanding capabilities of LLM and the assistance of structured knowledge such as knowledge graphs. The system can provide more detailed and relevant search results, avoiding the biases or omissions caused by the lack of contextual understanding in traditional searches. For example, when dealing with technical terms or ambiguous words, deep information retrieval systems can accurately determine their meaning based on the context, thereby providing more accurate information.

[0152] Effectively reducing LLM illusion and improving response reliability: Deep information retrieval ensures that LLM responses are always based on verified, high-quality external data through iterative information retrieval and strict context filtering. At the same time, dynamic knowledge accumulation and reflection mechanisms enable the system to identify and correct any biases or errors that may exist in LLM. The system actively evaluates the correctness of the answers and corrects them when necessary. This significantly reduces the risk of LLM generating inaccurate or false information and greatly improves the authority and credibility of the system output.

[0153] Enhancing the system's adaptability, flexibility, and scalability: As an intelligent agent, LLM can dynamically select and orchestrate various search tools and adjust its information acquisition strategy based on real-time feedback. This modular and intelligent tool management enables the system to easily integrate new data sources and retrieval technologies. For example, when new data formats or retrieval APIs emerge, the system can adapt quickly. The system can flexibly cope with different types of problems and data sources, and has stronger generalization capabilities and future scalability. This means that there is no need for large-scale system modifications for every new scenario, thereby reducing maintenance costs and extending the technology lifecycle.

[0154] The technical solution of this application has broad market prospects and significant commercial value, mainly due to its disruptive efficiency improvement and cost control capabilities. Through an LLM-driven intelligent agent framework, it transforms traditional, expensive, and fragile enterprise knowledge management processes into a highly automated, self-optimizing "cognitive engine." Specifically, this system can dynamically coordinate multiple tools such as vector databases, web crawlers, and structured queries, accurately solving complex, multi-hop problems across multiple domains through an iterative "search-reasoning" loop. This not only improves information processing efficiency by several orders of magnitude, but more importantly, it significantly reduces reliance on continuous optimization by AI experts, transforming a "cost center" into a "value engine."

[0155] Figure 4 This is a schematic diagram of the structural composition of the information retrieval device provided in the embodiments of this application, as shown below. Figure 4 As shown, the information retrieval device includes: The retrieval unit 401 is configured to receive content to be retrieved, analyze the content to be retrieved to obtain first information related to the content to be retrieved, wherein the first information is information for obtaining retrieval results corresponding to the content to be retrieved; determine a first set of retrieval tools based on the first information, and use each retrieval tool in the first set of retrieval tools to retrieve the content to be retrieved to obtain an initial set of retrieval results corresponding to the content to be retrieved; determine whether it is necessary to determine a first set of retrieval sub-contents and a second set of retrieval tools based on the initial set of retrieval results and the content to be retrieved; when it is determined that it is necessary to determine the first set of retrieval sub-contents and the second set of retrieval tools, use each retrieval tool in the second set of retrieval tools to retrieve the corresponding first sub-contents in the first set of retrieval sub-contents to obtain a first intermediate set of retrieval results; and determine the final retrieval result corresponding to the content to be retrieved based on the initial set of retrieval results and the first intermediate set of retrieval results.

[0156] In some embodiments, the retrieval unit 401 is used to input the content to be retrieved into a large language model and analyze the content to be retrieved to obtain first information related to the content to be retrieved.

[0157] In some embodiments, the retrieval unit 401 is configured to determine a first set of retrieval tools based on the first information, and call each retrieval tool in the first set of retrieval tools to retrieve the content to be retrieved, thereby obtaining an initial retrieval result corresponding to each retrieval tool; and determine the initial set of retrieval results based on the initial retrieval result corresponding to each retrieval tool.

[0158] In some embodiments, the retrieval unit 401 is configured to determine the quality information of the initial retrieval result set based on the initial retrieval result set and the content to be retrieved; and to determine, based on the quality information, whether it is necessary to determine the first retrieval sub-content set corresponding to the content to be retrieved and the second retrieval tool set corresponding to the first retrieval sub-content set.

[0159] In some embodiments, the retrieval unit 401 is configured to obtain a first evaluation value of the initial retrieval result set; obtain a second evaluation value of the first retrieval result set, wherein the first retrieval result set includes the initial retrieval result set and the first intermediate retrieval result set; and determine the final retrieval result based on the first retrieval result set when the difference between the first evaluation value and the second evaluation value is less than a first threshold.

[0160] In some embodiments, the retrieval unit 401 is configured to iteratively execute a first process until a first condition is met when the difference between the first evaluation value and the second evaluation value is greater than or equal to a first threshold. The first process includes: determining whether it is necessary to determine a second set of search sub-contents and a third set of search tools corresponding to the search content based on the first retrieval result set and the content to be retrieved; when it is determined that it is necessary to determine the second set of search sub-contents and the third set of search tools, updating the first retrieval result set based on the third set of search tools, the second set of search sub-contents, and the first retrieval result set; the first condition includes one or more of the following conditions: the number of times the first process is iteratively executed is greater than or equal to a second threshold; the difference between the evaluation value of the first retrieval result set obtained in the current round and the evaluation value of the first retrieval result set obtained in the previous round is less than a first threshold.

[0161] In some embodiments, the apparatus further includes: a storage unit 402; the storage unit 402 is used to store the initial search result set, the first intermediate search result set, and the final search result; the apparatus further includes: a presentation unit 403; the presentation unit 403 is used to present the final search result and the source information of the final search result.

[0162] Those skilled in the art should understand that Figure 4 The functions of each unit in the information retrieval device shown can be understood by referring to the relevant descriptions of the aforementioned method. Figure 4 The functions of each unit in the information retrieval device shown can be implemented by a program running on a processor or by specific logic circuits.

[0163] Figure 5 This is a schematic structural diagram of an information retrieval device 500 provided in an embodiment of this application. Figure 5 The information retrieval device 500 shown includes a processor 510, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0164] Optionally, such as Figure 5 As shown, the information retrieval device 500 may further include a memory 520. The processor 510 can retrieve and run computer programs from the memory 520 to implement the methods described in this embodiment.

[0165] The memory 520 can be a separate device independent of the processor 510, or it can be integrated into the processor 510.

[0166] Optionally, such as Figure 5As shown, the information retrieval device 500 may also include a transceiver 530, and the processor 510 may control the transceiver 530 to communicate with other devices. Specifically, it may send information or data to other devices or receive information or data sent by other devices.

[0167] The transceiver 530 may include a transmitter and a receiver. The transceiver 530 may further include antennas, and the number of antennas may be one or more.

[0168] The information retrieval device 500 can implement the corresponding processes implemented by the information retrieval device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0169] Figure 6 This is a schematic structural diagram of the chip according to an embodiment of this application. Figure 6 The chip 600 shown includes a processor 610, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0170] Optionally, such as Figure 6 As shown, chip 600 may further include memory 620. Processor 610 can retrieve and run computer programs from memory 620 to implement the methods described in this embodiment.

[0171] The memory 620 can be a separate device independent of the processor 610, or it can be integrated into the processor 610.

[0172] Optionally, the chip 600 may also include an input interface 630. The processor 610 can control the input interface 630 to communicate with other devices or chips; specifically, it can acquire information or data sent by other devices or chips.

[0173] Optionally, the chip 600 may also include an output interface 640. The processor 610 can control the output interface 640 to communicate with other devices or chips, specifically, to output information or data to other devices or chips.

[0174] This chip can implement the corresponding processes implemented by the information retrieval device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0175] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0176] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0177] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0178] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0179] This application also provides a computer program product, including a computer program.

[0180] When executed by a processor, the computer program implements the corresponding processes of the information retrieval device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0181] This application also provides a computer-readable storage medium for storing computer programs.

[0182] The computer program causes the computer to execute the corresponding processes implemented by the information retrieval device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0183] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0184] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0185] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0186] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0187] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0188] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0189] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application. Moreover, without conflict, the various embodiments and / or the technical features in the various embodiments described in this application can be arbitrarily combined with the prior art, and the resulting technical solutions should also fall within the protection scope of this application.

[0190] It should be understood that in the various method embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0191] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An information retrieval method, characterized in that, The method includes: Receive the content to be searched and analyze the content to be searched to obtain first information related to the content to be searched, wherein the first information is information for obtaining search results corresponding to the content to be searched; Based on the first information, a first set of search tools is determined, and each search tool in the first set of search tools is used to search the content to be searched to obtain an initial set of search results corresponding to the content to be searched. Based on the initial search result set and the content to be searched, determine whether it is necessary to determine the first set of search sub-contents corresponding to the content to be searched and the second set of search tools; When it is determined that it is necessary to determine the first set of search sub-contents and the second set of search tools, each search tool in the second set of search tools is used to retrieve the corresponding first sub-content in the first set of search sub-contents to obtain the first intermediate set of search results. The final search result corresponding to the content to be searched is determined based on the initial search result set and the first intermediate search result set.

2. The method according to claim 1, characterized in that, The analysis of the content to be searched to obtain first information related to the content to be searched includes: The content to be retrieved is input into a large language model, which analyzes the content to be retrieved to obtain first information related to the content to be retrieved.

3. The method according to claim 2, characterized in that, The step of determining a first set of search tools based on the first information, and using the first set of search tools to search for the content to be searched to obtain an initial set of search results corresponding to the content to be searched, includes: The large language model determines a first set of search tools based on the first information, and calls each search tool in the first set of search tools to search for the content to be searched, thereby obtaining the initial search results corresponding to each search tool; The initial set of search results is determined based on the initial search results corresponding to each search tool.

4. The method according to claim 2, characterized in that, The step of determining whether it is necessary to determine the first set of search sub-contents corresponding to the search content and the second set of search tools based on the initial search result set and the search content includes: The large language model determines the quality information of the initial search result set based on the initial search result set and the content to be searched; The large language model determines, based on the quality information, whether it is necessary to determine the first set of search sub-contents corresponding to the content to be searched and the second set of search tools corresponding to the first set of search sub-contents.

5. The method according to claim 1, characterized in that, The step of determining the final search result corresponding to the content to be searched based on the initial search result set and the first intermediate search result set includes: Obtain the first evaluation value of the initial search result set; Obtain a second evaluation value for the first search result set, wherein the first search result set includes the initial search result set and the first intermediate search result set; When the difference between the first evaluation value and the second evaluation value is less than the first threshold, the final search result is determined based on the first search result set.

6. The method according to claim 5, characterized in that, The method further includes: When the difference between the first evaluation value and the second evaluation value is greater than or equal to the first threshold, the first process is executed iteratively until the first condition is met; The first process includes: Based on the first set of search results and the content to be searched, determine whether it is necessary to determine the second set of search sub-contents corresponding to the content to be searched and the third set of search tools; When it is determined that it is necessary to determine the second search sub-content set and the third search tool set, the first search result set is updated according to the third search tool set, the second search sub-content set and the first search result set; The first condition includes one or more of the following conditions: The number of times the first process is executed iteratively is greater than or equal to the second threshold; The difference between the evaluation value of the first set of search results obtained in this round and the evaluation value of the first set of search results obtained in the previous round is less than the first threshold.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Store the initial search result set, the first intermediate search result set, and the final search result; Present the final search results and the source information of the final search results.

8. An information retrieval device, characterized in that, include: A processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the method as described in any one of claims 1 to 7.

9. A computer program product, characterized in that, include: A computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 7.

Images