Multi-source knowledge enhanced large language model question and answer method, device and equipment and medium

Abstract

The invention discloses a multi-source knowledge-enhanced big language model question and answer method, device and equipment and a medium, and the method comprises the steps: inputting a user question into a big language model, and obtaining an initial answer and a cooperation strategy deduced by the big language model for the user question; after taking the initial answer as a current candidate answer, extracting a target knowledge source type in a collaborative strategy and generating a corresponding calling expression; according to the calling expressions, target knowledge sources matched with the target knowledge source types are called respectively, and an enhanced retrieval document set is obtained; re-inputting the user question and the enhanced retrieval document set into the large language model to obtain a new answer and a collaboration strategy; and taking the new answer as the current candidate answer, repeating the operation of extracting the target knowledge source type, and determining a target answer from all the candidate answers as a feedback result when an iteration ending condition is met. According to the technical scheme, the question and answer accuracy can be optimized through iterative retrieval and knowledge enhancement.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large language model question answering method, device, equipment and medium with multi-source knowledge enhancement. Background Art

[0002] In recent years, large language models (LLMs) have made significant progress in tasks such as text generation and question-answering systems, but it is still difficult to ensure the optimality and interpretability of the output results.

[0003] In the existing technology, Retrieval-Augmented Generation (RAG) is an effective means to improve the knowledge boundaries and factual accuracy of language models, and is widely used in various knowledge-intensive tasks. It enhances the accuracy of output content through the "retrieval-splicing-generation" approach.

[0004] In the process of realizing the present invention, the inventors discovered that although traditional RAG systems can optimize output content to a certain extent, they mostly adopt a one-time information splicing method, which has obvious limitations in the accuracy, completeness and interpretability of the output results, and it is difficult to meet users' growing demand for high-quality question and answer results. Summary of the Invention

[0005] Embodiments of the present invention provide a large language model question answering method, apparatus, device, and medium enhanced with multi-source knowledge, which can significantly improve the accuracy and reliability of the question answering system in complex knowledge-intensive tasks.

[0006] According to one aspect of an embodiment of the present invention, a large language model question answering method enhanced with multi-source knowledge is provided, the method comprising:

[0007] Input the user question into the large language model, obtain the initial answer inferred by the large language model for the user question, and the collaborative strategy that matches the current candidate answer;

[0008] After taking the initial answer as the current candidate answer, extract the target knowledge source types in the collaborative strategy of the current candidate answer, and generate a call expression corresponding to each target knowledge source type based on the current candidate answer;

[0009] According to each calling expression, target knowledge sources matching each target knowledge source type are called respectively to obtain an enhanced retrieval document set;

[0010] Re-input the user question and the enhanced retrieval document collection into the large language model to obtain the new answer inferred by the large language model and the collaborative strategy that matches the new answer;

[0011] After taking the new answer as the current candidate answer, return to execute the operation of extracting at least one target knowledge source type in the collaborative strategy of the current candidate answer, and when the end iteration condition is met, determine the target answer among all candidate answers as the feedback result to the user question.

[0012] According to another aspect of an embodiment of the present invention, a large language model question answering device enhanced with multi-source knowledge is provided, the device comprising:

[0013] The initial reasoning module is used to input the user question into the large language model, obtain the initial answer inferred by the large language model for the user question, and the collaborative strategy that matches the current candidate answer;

[0014] The collaborative strategy parsing module is used to extract the target knowledge source types in the collaborative strategy of the current candidate answer after taking the initial answer as the current candidate answer, and generate a call expression corresponding to each target knowledge source type based on the current candidate answer;

[0015] A multi-source knowledge retrieval module is used to call target knowledge sources that match the target knowledge source type according to each call expression to obtain an enhanced retrieval document set;

[0016] An iterative optimization module, which is used to re-input user questions and enhanced retrieval document sets into the large language model, obtain new answers inferred by the large language model, and a collaborative strategy that matches the new answers;

[0017] The result evaluation output module is used to take the new answer as the current candidate answer, return to execute the operation of extracting at least one target knowledge source type in the collaborative strategy of the current candidate answer, and when the end iteration condition is met, determine the target answer among all candidate answers as the feedback result to the user question.

[0018] According to another aspect of an embodiment of the present invention, an electronic device is provided, the electronic device including:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a large language model question answering method with multi-source knowledge enhancement as described in any embodiment of the present invention.

[0022] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a large language model question answering method with multi-source knowledge enhancement as described in any embodiment of the present invention when executed.

[0023] According to another aspect of an embodiment of the present invention, a computer program product is provided, comprising computer instructions, which implement the steps of the method according to any embodiment of the present invention when executed by a processor.

[0024] The technical solution of the embodiment of the present invention first inputs the user question into the large language model, obtains the initial answer generated by the model and the corresponding collaborative strategy, then uses the initial answer as the current candidate answer, parses its collaborative strategy to determine the required target knowledge source type, and generates a corresponding call expression for each type, and then retrieves relevant information from the matching knowledge source based on these call expressions to form an enhanced retrieval document set, then re-inputs the user question and the retrieval results into the large language model to obtain the updated answer and the new collaborative strategy. After using the new answer as the current candidate answer, the above-mentioned knowledge source extraction and retrieval process will be executed cyclically, and when the preset iteration termination condition is met, the target answer will be determined among all candidate answers as the feedback result for the user question. This new enhanced large language model question-answering method can effectively break through the limitations of traditional methods, and significantly improve the accuracy and completeness of the answer through multi-source knowledge collaborative retrieval. Its iterative optimization mechanism ensures that the system can continuously improve the output quality, and the dynamic knowledge source selection strategy also makes the answer more comprehensive.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 This is a flowchart of a large language model question answering method with multi-source knowledge enhancement according to the first embodiment of the present invention;

[0028] Figure 2 is a flowchart of another large language model question answering method with multi-source knowledge enhancement provided by Example 2 of the present invention;

[0029] Figure 3 is a flowchart of another large language model question answering method with multi-source knowledge enhancement provided by Example 3 of the present invention;

[0030] Figure 4 2. This is a schematic diagram of the structure of a large language model question answering device with multi-source knowledge enhancement according to the fourth embodiment of the present invention;

[0031] Figure 5 It is a structural diagram of an electronic device for implementing a large language model question answering method with multi-source knowledge enhancement according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Example 1

[0035] Figure 1 This is a flowchart of a large language model question answering method enhanced by multi-source knowledge provided in Example 1 of the present invention. This embodiment is applicable to situations where complex and knowledge-intensive tasks are to be processed. The method can be performed by a large language model question answering device enhanced by multi-source knowledge. The device can be implemented in the form of hardware and / or software and can generally be configured in an electronic device.

[0036] Correspondingly, such as Figure 1 As shown, the method includes:

[0037] S110: Input the user question into the large language model, obtain the initial answer inferred by the large language model for the user question, and the collaborative strategy that matches the current candidate answer.

[0038] Among them, a large language model (LLM) can be understood as a deep learning model based on a deep neural network architecture (e.g., Transformer) with a large number of model parameters (e.g., more than 1 billion). The large language model can be a generative system, that is, after training with massive data, it can realize natural language understanding, generation, and reasoning to obtain new content for multi-source input data such as text, images, or audio and video. Large language models often have strong context understanding capabilities and can handle diverse tasks such as question answering, translation, and summarization. Typically, a large language model can be an autoregressive language model that supports setting the context size.

[0039] In an embodiment of the present invention, when a user asks a question, the question is first input into the LLM for preliminary processing. At this stage, the LLM relies solely on the parameterized knowledge and built-in knowledge base learned during its training process to generate an initial answer. The initial answer reflects the LLM's initial understanding of the question. At the same time, a collaborative strategy related to the initial answer is generated to determine which retrieval methods the LLM will subsequently call to further supplement and optimize the initial answer.

[0040] The collaborative strategy can be understood as the LLM selecting the subsequent search method to further enrich and improve the answer based on the current input question and the initial answer generated. This collaborative strategy specifically defines the acquisition channels, or types of one or more knowledge sources, used by the LLM to retrieve documents before generating the next round of answers.

[0041] Specifically, commonly used knowledge bases (or knowledge graphs) and search engine configuration information are pre-integrated into the LLM as various knowledge sources, including the access methods and applicable scenarios of various knowledge sources, so that the LLM can independently select the search method that matches the answer.

[0042] S120. After taking the initial answer as the current candidate answer, extract each target knowledge source type in the collaborative strategy of the current candidate answer, and generate a call expression corresponding to each target knowledge source type according to the current candidate answer.

[0043] In an embodiment of the present invention, after obtaining the initial answer generated by LLM, the answer will be marked as the current candidate answer, and then the collaborative strategy associated with the candidate answer will be parsed to extract the target knowledge source types preliminarily selected by LLM from all types of knowledge sources based on the problem characteristics and candidate answers. For each selected target knowledge source type, a corresponding standardized calling expression will be generated. These expressions specify in detail the access method, query parameters and result format requirements of the knowledge source.

[0044] Among them, the knowledge source type can be understood as a classification system of various pre-configured external knowledge bases, including but not limited to: structured knowledge sources (such as knowledge graph databases such as Wikidata), unstructured knowledge sources (such as open network resources such as academic paper libraries and news websites), and functional knowledge sources (such as professional calculation tools, unit converters and other auxiliary modules). Each knowledge source type has a specific data description, including its applicable field (such as professional fields such as science and technology, medicine, and finance), data organization form (structured tables or unstructured text), timeliness characteristics (real-time updates or periodic snapshots), and access interface specifications (API (Application Programming Interface) calling method, authentication mechanism, etc.).

[0045] Call expressions can be understood as standardized instruction templates for interacting with external knowledge sources. They define access protocol specifications, parameterized query structures (translating natural language into query statements executable by a specific knowledge source), and result processing requirements (including data formats, required fields, and cleansing rules). These expressions contain core components such as knowledge source identifiers, authentication information, query templates, parameter binding rules, and result processing logic. These instructions are controlled by the Prompt module, generating structured call instructions. For example, "Use Wikidata to query the origin of entity X" or "Call Bing to retrieve articles related to X."

[0046] S130 , calling target knowledge sources that match the target knowledge source types respectively according to each calling expression to obtain an enhanced retrieval document set.

[0047] In an embodiment of the present invention, a connection is established with various target knowledge sources through a standardized interface protocol and a search is performed. For structured knowledge sources (such as knowledge graphs), the parameterized query template is converted into a specific query language (such as SPARQL) and the request is submitted through the API endpoint; for unstructured knowledge sources (such as academic databases), a keyword combination that conforms to its search syntax is constructed, and filtering conditions such as timeliness and field are added; for functional knowledge sources (such as computing tools), parameters are directly passed in to perform calculations. All search results are organized as enhanced search documents.

[0048] It is understandable that enhanced retrieval documents integrate multi-source data from different knowledge sources, including structured entity relationships provided by knowledge graphs, in-depth discussions in academic literature, and professional analysis results generated by real-time computing tools. This multi-dimensional knowledge fusion method fundamentally ensures that the generated answers have factual accuracy (from structured knowledge sources), theoretical depth (from academic literature) and computational precision (from professional tools).

[0049] Furthermore, enhanced retrieval documents can encapsulate multi-source data through a unified JSON-LD format, achieving standardized processing while maintaining the structural characteristics of each knowledge source data: all documents contain basic metadata fields (such as knowledge source type and timestamp) and provide a consistent access interface through a standardized document parsing API; structured knowledge source data retains its attribute graph structure, unstructured documents maintain the original text paragraphs and annotation information, and the calculation tool results fully preserve the numerical precision and unit annotations.

[0050] S140: Re-input the user question and the enhanced retrieval document set into the large language model to obtain a new answer inferred by the large language model and a collaborative strategy matching the new answer.

[0051] In an embodiment of the present invention, an iterative reasoning process of knowledge enhancement is achieved by re-inputting the user question and the enhanced retrieval document set into the large language model. Specifically, the user question and the enhanced retrieval document set are first input into the LLM, where the enhanced retrieval documents have been sorted and labeled according to relevance and credibility. The LLM will perform multi-level deep reasoning: first, the consistency between the retrieval document and the initial answer is analyzed to identify potential knowledge conflicts or supplementary points; then, evidence is synthesized based on the credibility weights of different knowledge sources (such as academic papers have higher weights than ordinary web pages); finally, a new answer that has been verified by knowledge is generated, and the source basis of the key conclusions is annotated.

[0052] Furthermore, a new generation of collaborative strategies will be dynamically generated based on this round of reasoning process, which will achieve optimization in multiple dimensions: based on the actual contribution of each knowledge source in this round of retrieval, the weight distribution of subsequent retrievals will be automatically adjusted, for example, the query priority of high-value knowledge sources that provide key evidence will be significantly improved; for sub-fields of problems that have not yet been fully answered, the original query will be intelligently refined into more targeted sub-queries, for example, the broad "quantum computing applications" will be automatically converted into precise retrieval instructions such as "the latest progress in quantum computing in drug molecule simulation."

[0053] S150. After taking the new answer as the current candidate answer, return to execute the operation of extracting at least one target knowledge source type in the collaborative strategy of the current candidate answer, and when the end iteration condition is met, determine the target answer among all candidate answers as the feedback result to the user question.

[0054] In an embodiment of the present invention, after the newly generated answer is set as the current candidate answer, the next round of knowledge enhancement process will be automatically triggered. First, the collaborative strategy corresponding to the candidate answer is parsed, the target knowledge source type with the highest weight in the strategy is prioritized, and an optimized call expression is generated for the identified knowledge gap. These expressions will add targeted constraints on the basis of retaining the effective search parameters of the previous round. For example, when it is detected that the answer needs to be supplemented with experimental data, the limiting conditions of the relevant research field will be automatically added to the call expression. Through this iterative optimization mechanism, the knowledge completeness and accuracy of the answer can be continuously improved.

[0055] Furthermore, the next round of knowledge enhancement process is initiated. First, the latest collaborative strategy associated with the candidate answer is parsed, and the target knowledge source type with the highest priority in the strategy is extracted (for example, when it is detected that the answer lacks relevant expressions of "drug molecular simulation", academic paper databases and patent libraries will be prioritized as the next round of search targets). For each selected knowledge source type, the knowledge gap marker in the current answer (such as "[To be supplemented] clinical trial data") will be combined to dynamically generate optimized call expressions. These expressions will inherit the valid parameters of the previous round of search (such as the time range limited to 2020-2023), while adding targeted constraints.

[0056] Furthermore, when the preset maximum number of retrieval rounds is reached (usually 4 rounds), the answer integration stage begins: first, the corresponding candidate answers generated in each round of iteration are summarized (4 in total); then, these candidate answers are deeply analyzed, focusing on comparing the key differences between answers in different rounds, including data differences and differences in opinions; finally, the reliability of the evidence cited in each answer is comprehensively evaluated, and the optimal answer is determined as the final target answer to be output to the user.

[0057] The technical solution of the embodiment of the present invention inputs the user question into the model to obtain the initial answer and collaborative strategy, and after taking the answer as the current candidate answer, parses its collaborative strategy to determine the target knowledge source type and generates the corresponding call expression, and then calls the matching knowledge source to obtain an enhanced retrieval document set, and then re-inputs the user question and retrieval document into the model to obtain the updated answer and new collaborative strategy, and after taking the new answer as the current candidate answer, the above-mentioned knowledge source extraction and retrieval process is executed cyclically. When the iterative termination condition is met, the system comprehensively analyzes all candidate answers and finally determines the optimal target answer as the feedback result. This solution fundamentally improves the performance of the question-answering system through innovative multi-source knowledge collaborative retrieval and iterative optimization mechanism. This solution can also intelligently integrate the precise relationship network of structured knowledge graphs, the in-depth theoretical explanation of unstructured literature, and the quantitative analysis results of real-time computing tools, breaking through the limitations of traditional single knowledge sources and ensuring that the answers are both comprehensive and timely.

[0058] Example 2

[0059] Figure 2 This is a flowchart of another large language model question answering process enhanced with multi-source knowledge, provided in Example 2 of the present invention. This example is optimized based on the above examples. In this example, the operation of "generating a call expression corresponding to each target knowledge source type based on the current candidate answer" is specifically refined.

[0060] S210: Input the user question into the large language model, obtain the initial answer inferred by the large language model for the user question, and the collaborative strategy that matches the current candidate answer.

[0061] S220: Use the initial answer as the current candidate answer.

[0062] S230: extract the target knowledge source types in the collaborative strategy of the current candidate answer, and input the current candidate answer into a pre-built named entity recognition model to obtain the key entities contained in the current candidate answer.

[0063] Among them, entities can be understood as noun components in the text that represent specific things or concepts. These components contain key information. Internships can be divided into five categories: the first category is specific entities, including people, institutions, places and other objects with clear references; the second category is time entities, covering time information such as years, dates, and periods; the third category is domain concepts, which include professional terms and technical terms in various disciplines; the fourth category is quantitative entities, involving numerical information such as measurement values ​​and percentages; the fifth category is relational entities, which represent specific connections between objects.

[0064] In this embodiment, once the current candidate answer is obtained, the Named Entity Recognition (NER) process is immediately initiated. This process is implemented using a pre-trained professional NER model, such as the W2NER model, an innovative entity recognition model based on word-pair relationship classification. W2NER simultaneously solves the dual tasks of entity recognition and inter-entity relationship determination by constructing a word-pair relationship matrix, providing high-precision entity detection capabilities.

[0065] Specifically, the model will perform word segmentation, part-of-speech tagging, and dependency analysis on candidate answers, then identify entity fragments with specific meanings in the text and label them with predefined category labels. For example, for the answer fragment "According to a report in the journal Nature in 2023, CRISPR gene editing technology has achieved a breakthrough in clinical trials," the model will identify: "2023" (time entity), "Nature journal" (institutional entity), "CRISPR gene editing technology" (science and technology entity), and "clinical trials" (medical field entity). These recognition results will be output in a structured JSON format for use by subsequent modules.

[0066] S240: Perform entity disambiguation on each identified key entity based on the task context that matches the user question.

[0067] In this example, key entities extracted through NER (such as names, places, and terms) may have semantic ambiguity or ambiguous references. For example, "apple" may refer to a fruit, a technology company, or the name of a movie, and "Java" may represent a programming language or an Indonesian island. The core goal of entity disambiguation is to accurately associate the identified entities with unique and clear semantic concepts based on the specific context of the current task, avoiding errors in subsequent retrieval or reasoning due to ambiguity.

[0068] Specifically, the overall semantics of the user's question and the conversation history are combined to determine the potential candidate meanings of the entity. At the same time, the entity is compared with the entries in the structured knowledge base (such as Wikipedia), and the most likely candidate is screened by the common probability of the contextual keywords.

[0069] Optionally, if the confidence level of automatic ambiguity resolution is insufficient, an interactive confirmation process can be triggered to obtain explicit instructions from the user by asking precise questions.

[0070] S250. Obtain a call template that matches each target knowledge source type, wherein the call template includes at least one blank item for filling entity content.

[0071] Among them, the call template can be understood as a standardized interface framework for interacting with external knowledge sources. Its essence is to convert unstructured natural language queries into executable instructions. This type of template adopts a "fill-in-the-blank" design, consisting of two parts: a fixed structure and variable parameters. The fixed part clearly defines the access protocol and basic call format of the knowledge source (such as API path and fixed query parameters), while the variable part uses placeholders to mark the fields that need to be dynamically filled. Each blank item has strict semantic type constraints (such as location, time, etc.). For example, when processing "Query Shanghai tomorrow's weather", the weather API template "https: / / api.weather.com / v1?location=&date=" will be matched, the disambiguated "Shanghai" will be filled in the location parameter, and the calculated date will be filled in the date parameter, finally generating a standardized call instruction.

[0072] In this embodiment, the knowledge source call preparation phase retrieves a corresponding call template for each identified target knowledge source type. These structured call templates not only include the standard access format and required parameters for a specific knowledge source, but also feature clear semantics within their blank entries. Based on the results of entity recognition and disambiguation, the verified entity content is accurately filled into the corresponding blank entries.

[0073] S260. Obtain the entity content required for each call template from each key entity that has completed the disambiguation process, fill in each call template, and obtain the call expression corresponding to each target knowledge source type.

[0074] In this embodiment, based on the key entities after disambiguation processing, the call template corresponding to the target knowledge source is automatically matched, and the verified entity content is accurately filled into the designated blank items of the template. This process strictly follows the principle of semantic type matching, and at the same time performs necessary format conversion on the entity content, such as converting the natural language expression "tomorrow" into the standard date format "2023-11-20", or converting Chinese place names into pinyin form. The final generated call expression not only retains the standard structure of the template, but also fully embeds the processed entity parameters, forming a standardized instruction that can be directly executed, thereby ensuring the accuracy and execution efficiency of the subsequent knowledge acquisition process.

[0075] S270. Call the target knowledge sources that match the target knowledge source types according to the call expressions to obtain an enhanced retrieval document set.

[0076] In this embodiment, for each parameter-filled call expression, the corresponding access protocol and execution engine can be selected according to the corresponding knowledge source type: for RESTful API (Representational State Transfer API) type knowledge sources, an HTTP request can be initiated and the returned JSON or XML format data can be parsed; for database type knowledge sources, parameterized query statements are executed to obtain structured records; and for academic resources such as professional document libraries, a constructed search formula can be submitted and a collection of related documents can be received.

[0077] Furthermore, all acquired raw data undergoes unified formatting and conversion into an internal standard document representation, ultimately forming an enhanced retrieval document collection. This collection not only contains the latest information obtained from external knowledge sources but also retains the source metadata of each document, providing a traceable basis for subsequent answer generation and quality verification.

[0078] S280: Re-input the user question and the enhanced retrieval document set into the large language model to obtain a new answer inferred by the large language model and a collaborative strategy that matches the new answer.

[0079] S290. Check whether the condition for ending the iteration is met: if not, execute S2100; if so, execute S2110.

[0080] S2100: After setting the new answer as the current candidate answer, return to S230.

[0081] S2110. Determine a target answer from all candidate answers as a feedback result to the user's question.

[0082] The technical solution of the embodiment of the present invention is to input the user question into the large language model to obtain the initial answer and collaborative strategy, take the initial answer as the current candidate answer and then extract the target knowledge source type in its collaborative strategy, then input the current candidate answer into the named entity recognition model to obtain the key entity, perform disambiguation on the entity in combination with the task context, then obtain the call template corresponding to each target knowledge source type and extract the required content from the disambiguated entity to fill in the template and generate a call expression, call the target knowledge source according to the expression to obtain an enhanced retrieval document set, then re-input the user question and the document set into the large language model to obtain a new answer and collaborative strategy, take the new answer as the current candidate answer and return the extracted target The operation of the knowledge source type is iteratively executed until the end condition is met, and finally the target answer is determined from all candidate answers as the feedback result. Through the collaborative processing of named entity recognition and context disambiguation, this solution can accurately identify and verify the key information elements in the candidate answers, effectively solve the semantic ambiguity problems of polysemous words and vague references, and the templated call expression generation mechanism based on the disambiguation results ensures the standardization and security of knowledge source calls, and realizes the flexible adaptation of different types of knowledge sources. Through the dynamically constructed enhanced retrieval document collection, it breaks through the knowledge limitations of the large language model itself, obtains the latest and most relevant external knowledge support, and significantly improves the accuracy and timeliness of answer generation.

[0083] Optionally, based on the above embodiments, inputting the user question into the large language model, obtaining the initial answer inferred by the large language model for the user question, and the collaborative strategy matching the initial answer with the current candidate answer may include:

[0084] Input the user's question into the large language model and obtain the initial answer inferred by the large language model for the user's question;

[0085] According to multiple types of knowledge sources, a prompt word template is constructed, and the prompt word template is provided to a large language model, and the large language model is obtained to generate and output a collaborative strategy based on at least one type of knowledge source randomly selected from the multiple types of knowledge sources.

[0086] Typically, a user's question is input into an LLM, which then infers based on its internal knowledge to generate an initial answer. This step relies on the LLM's natural language understanding and generation capabilities to ensure the semantic plausibility and fluency of the initial answer. However, due to potential knowledge limitations and timeliness issues with LLMs, further enhancement requires the introduction of external knowledge sources.

[0087] Generally speaking, specific prompt word templates are constructed based on various types of knowledge sources. These templates clearly define the applicable scenarios and call conditions for each type of knowledge source. When a collaborative strategy needs to be generated, the user question and these prompt word templates are input into the LLM. The LLM analyzes the semantic features of the question, randomly selects at least one knowledge source type from a variety of preset knowledge source types, and outputs a corresponding collaborative strategy. This strategy specifies the specific knowledge source type to be called, providing clear direction for subsequent retrieval operations.

[0088] Optionally, based on the above embodiments, the user question and the enhanced search document set are re-inputted into the large language model to obtain a new answer inferred by the large language model, and a collaborative strategy matching the new answer may include:

[0089] Re-input the user question and the enhanced retrieval document collection into the large language model to obtain the new answer inferred by the large language model;

[0090] Invoke the reflection function of the large language model to detect whether the new answer has information gaps;

[0091] The reflection function is achieved by setting a preset prompt word template with review-guiding semantics;

[0092] If so, determining a collaborative strategy that matches the new answer based on all types of knowledge sources and at least one type of knowledge source in the most recently generated collaborative strategy;

[0093] If not, a collaborative strategy matching the new answer is determined based on at least one type of knowledge source in the most recently generated collaborative strategy.

[0094] Generally speaking, during the answer quality verification phase, the LLM's built-in reflection function is invoked to comprehensively test the completeness of newly generated answers. This function is implemented through pre-designed prompt word templates with review-guided semantics. These templates use natural language guidance, such as "Please systematically evaluate whether the following answer fully covers all key points of the question, whether there are any factual omissions or logical loopholes, and whether the latest data support is needed." The answer to be tested is combined with the verification instructions and re-entered into the model for analysis. Based on its semantic understanding capabilities, the LLM conducts a multi-dimensional review of the answer content, including assessments of key element integrity, factual accuracy, logical coherence, and timeliness. It ultimately outputs a detailed quality defect report, providing precise improvement directions for subsequent answer optimization iterations, thereby ensuring the reliability and completeness of the final output answer.

[0095] Furthermore, the knowledge source selection strategy is dynamically adjusted based on the reflection results: if information gaps are detected in the new answer, all available knowledge source types and the knowledge source types in the most recent collaborative strategy are comprehensively evaluated, and the optimal knowledge source combination is re-determined through multi-dimensional matching; if the answer integrity verification is passed, the knowledge source types that have been verified to be effective in the most recent collaborative strategy are preferentially used, and only appropriate optimization is required based on the current answer characteristics.

[0096] Optionally, based on the above embodiments, a target answer is determined from all candidate answers as a feedback result to the user's question, including:

[0097] The semantic re-ranking mechanism is used to re-rank the candidate answers, and the first candidate answer after re-ranking is used as the target answer as the feedback result to the user's question.

[0098] Generally speaking, when determining the final feedback answer, a semantic reranking mechanism based on DPR (Dense Passage Retrieval) can be used to rerank all candidate answers. This mechanism calculates the semantic relevance score of each candidate answer to the user's question through a pre-trained deep semantic model, while comprehensively considering multi-dimensional features such as the coherence of the answer itself, the completeness of the information, and the credibility of external knowledge support to generate a comprehensive ranking weight. The DPR model densely matches the semantic representation vector of the candidate answer with the question vector, and captures fine-grained semantic associations through the attention mechanism to ensure that the ranking result is consistent with the question intent and has high-quality content characteristics. Finally, the candidate answer ranked first after reranking is determined as the target answer as the final feedback to the user's question.

[0099] Example 3

[0100] Figure 3 This is a flowchart of another large language model question answering process enhanced with multi-source knowledge, provided in Example 3 of the present invention. This example is optimized based on the above examples. In this example, the operation of "calling target knowledge sources that match each target knowledge source type based on each call expression to obtain an enhanced retrieval document set" is specifically refined.

[0101] S310: Input the user question into the large language model, obtain the initial answer inferred by the large language model for the user question, and the collaborative strategy that matches the current candidate answer.

[0102] S320: After taking the initial answer as the current candidate answer, extract each target knowledge source type in the collaborative strategy of the current candidate answer, and generate a call expression corresponding to each target knowledge source type according to the current candidate answer.

[0103] S330: Input the call expressions corresponding to each target knowledge source type into the matching target knowledge source respectively, and obtain at least one candidate retrieval document fed back by each target knowledge source for the call expression.

[0104] In this embodiment, after the generation of the call expression is completed, each expression is sent to its corresponding target knowledge source for execution. These knowledge sources include but are not limited to specific data platforms such as structured databases, professional API interfaces and academic literature libraries. A connection is established according to the communication protocols of different knowledge sources, and the formatted call expression is submitted as an input parameter. After receiving the request, each knowledge source will perform a retrieval operation based on its internal data processing logic and return one or more alternative documents that match the query conditions. These documents may be presented as structured data in JSON or XML format, original text content or professional document records in a specific encoding. They are uniformly parsed and standardized to ensure that documents from different sources can be converted into a unified representation that can be processed internally, providing multi-source heterogeneous but standardized data support for subsequent answer generation and quality verification.

[0105] S340: After filtering out documents with low relevance to the user question and the current candidate answer from the candidate retrieval documents, the remaining candidate retrieval documents are organized into an enhanced retrieval document set.

[0106] In this embodiment, the candidate documents retrieved directly from multiple knowledge sources often have uneven quality. Some documents may only be superficially related to the user's question and lack substantive relevance (such as advertisements), or contain information that contradicts the current candidate answer. If these low-quality documents are used directly for answer generation without screening, it will not only reduce processing efficiency, but may also lead to factual errors or logical confusion in the final answer. Therefore, a strict document screening module is established to ensure that the knowledge input into the answer generation process is high-relevance, high-credibility, and high-quality content.

[0107] Furthermore, by introducing a critical language model (such as the Self-RAG model), alternative retrieval documents are intelligently screened. First, the semantic relevance score of each alternative document with the user question and the current candidate answer is calculated using the semantic understanding ability of the model. Based on the preset threshold, low-relevance documents with scores below the standard are automatically filtered out. At the same time, the retained documents are evaluated for credibility to ensure that their content is both consistent with the question intent and factually accurate. Subsequently, the screened high-quality documents are prioritized according to semantic relevance to form a structured enhanced retrieval document collection, which not only contains the most relevant multi-source knowledge, but also retains the confidence score and source information of each document, providing rigorously screened knowledge support for subsequent answer optimization.

[0108] S350: Re-input the user question and the enhanced retrieval document set into the large language model to obtain a new answer inferred by the large language model and a collaborative strategy matching the new answer.

[0109] S360. After taking the new answer as the current candidate answer, return to execute the operation of extracting at least one target knowledge source type in the collaborative strategy of the current candidate answer, and when the end iteration condition is met, determine the target answer among all candidate answers as the feedback result to the user question.

[0110] The technical solution of the embodiment of the present invention is to input the user question into the large language model to obtain the initial answer and collaborative strategy, take the initial answer as the current candidate answer, and then extract the target knowledge source type in its collaborative strategy, generate call expressions corresponding to each knowledge source type according to the candidate answer, and input these expressions into the matching knowledge source to obtain alternative retrieval documents, filter out low-relevance documents and form an enhanced retrieval document set, then re-input the user question and document set into the large language model to obtain a new answer and collaborative strategy, take the new answer as the current candidate answer and return to the operation of extracting the target knowledge source type, and iterate until the end condition is met, and finally determine the target answer from all candidate answers as the feedback result. This solution ensures the efficiency and pertinence of the retrieval process by accurately matching the knowledge source type and executing the call expression, and adopts a strict document filtering mechanism to effectively eliminate low-relevance information, significantly improving the accuracy of knowledge retrieval. The enhanced retrieval document set finally formed not only optimizes the quality of knowledge acquisition, but also provides multi-source knowledge support for subsequent answer generation, thereby comprehensively improving the question-answering performance and reliability of the model.

[0111] Optionally, based on the above embodiments, filtering out documents with low relevance to the user question and the current candidate answer from the candidate search documents may include:

[0112] Organize each candidate retrieval document with the user question and the current candidate answer to obtain relevance detection data;

[0113] Input each relevance detection data into the pre-trained evaluation language model, and obtain the relevance evaluation result of the evaluation language model for each relevance detection data;

[0114] According to the relevance evaluation results, the documents with low relevance to the user question and the current candidate answer are filtered out from the candidate retrieval documents.

[0115] Generally speaking, each candidate document is structured and matched with key elements of the user's question and the current candidate answer, generating triplet detection data consisting of the document content, the question focus, and the answer's proposition. This process extracts core information fragments from the document and aligns them multi-dimensionally with the question intent and the candidate answer's arguments. It also incorporates conversation context and domain characteristics to form a complete relevance detection dataset, providing comprehensive input for subsequent in-depth analysis.

[0116] Furthermore, these detection data are fed into a pre-trained critical language model for deep semantic analysis. The vector similarity between the document content and the question and answer is calculated through the encoder architecture. A comprehensive evaluation is performed from three dimensions: question relevance, answer support, and context adaptability. The overall relevance score of each document is then output.

[0117] Furthermore, a dynamic filtering mechanism is implemented based on the model's evaluation results. By setting relevance thresholds and conflict detection rules, low-quality documents are automatically eliminated and the knowledge organization structure is optimized. Highly relevant documents are prioritized and formed into an enhanced search set, while fundamentally conflicting or redundant content is strictly filtered. This process ensures that the final knowledge set used for answer optimization is both highly accurate and information-dense, significantly improving factual accuracy and reasoning reliability.

[0118] Example 4

[0119] Figure 4 This is a structural diagram of a large language model question answering device with multi-source knowledge enhancement provided by the fourth embodiment of the present invention. Figure 4 As shown, the device includes:

[0120] Initial reasoning module 410, used to input the user question into the large language model, obtain the initial answer inferred by the large language model for the user question, and the collaborative strategy that matches the current candidate answer;

[0121] The collaborative strategy parsing module 420 is used to extract the target knowledge source types in the collaborative strategy of the current candidate answer after taking the initial answer as the current candidate answer, and generate a call expression corresponding to each target knowledge source type according to the current candidate answer;

[0122] The multi-source knowledge retrieval module 430 is used to call target knowledge sources that match the target knowledge source type according to each call expression to obtain an enhanced retrieval document set;

[0123] Iterative optimization module 440 is used to re-input the user question and the enhanced search document set into the large language model to obtain a new answer inferred by the large language model and a collaborative strategy that matches the new answer;

[0124] The result evaluation output module 450 is used to take the new answer as the current candidate answer, return to execute the operation of extracting at least one target knowledge source type in the collaborative strategy of the current candidate answer, and when the end iteration condition is met, determine the target answer among all candidate answers as the feedback result for the user question.

[0125] The technical solution of the embodiment of the present invention first inputs the user question into the large language model, obtains the initial answer generated by the model and the corresponding collaborative strategy, then uses the initial answer as the current candidate answer, parses its collaborative strategy to determine the required target knowledge source type, and generates a corresponding call expression for each type, and then retrieves relevant information from the matching knowledge source based on these call expressions to form an enhanced retrieval document set, then re-inputs the user question and retrieval results into the large language model to obtain the updated answer and the new collaborative strategy. After using the new answer as the current candidate answer, the system will cyclically execute the above-mentioned knowledge source extraction and retrieval process, and when the preset iteration termination condition is met, determine the target answer among all candidate answers as the feedback result for the user question. This new enhanced large language model question-answering method can effectively break through the limitations of traditional methods, and significantly improve the accuracy and completeness of the answer through multi-source knowledge collaborative retrieval. Its iterative optimization mechanism ensures that the system can continuously improve the output quality, and the dynamic knowledge source selection strategy also makes the answer more comprehensive.

[0126] Based on the above embodiments, the initial reasoning module 410 is specifically configured to:

[0127] Input the user's question into the large language model and obtain the initial answer inferred by the large language model for the user's question;

[0128] According to multiple types of knowledge sources, a prompt word template is constructed, and the prompt word template is provided to a large language model, and the large language model is obtained to generate and output a collaborative strategy based on at least one type of knowledge source randomly selected from the multiple types of knowledge sources.

[0129] Based on the above embodiments, the collaborative strategy analysis module 420 is specifically configured to:

[0130] Input the current candidate answer into the pre-built named entity recognition model to obtain the key entities contained in the current candidate answer;

[0131] Perform entity disambiguation on each identified key entity based on the task context that matches the user's question;

[0132] Obtaining a call template that matches each target knowledge source type, wherein the call template includes at least one blank item for filling entity content;

[0133] The entity content required for each call template is obtained from each key entity that has completed the disambiguation process, and each call template is filled to obtain the call expression corresponding to each target knowledge source type.

[0134] Based on the above embodiments, the multi-source knowledge retrieval module 430 may further include:

[0135] A retrieval document generating unit is used to input the call expressions corresponding to each target knowledge source type into the matching target knowledge source, and obtain at least one candidate retrieval document fed back by each target knowledge source for the call expression;

[0136] The noise filtering unit is used to filter out documents with low relevance to the user question and the current candidate answer from each candidate retrieval document, and organize the remaining candidate retrieval documents into an enhanced retrieval document set.

[0137] Based on the above embodiments, the noise filtering unit is specifically configured to:

[0138] Organize each candidate retrieval document with the user question and the current candidate answer to obtain relevance detection data;

[0139] Input each relevance detection data into the pre-trained evaluation language model, and obtain the relevance evaluation result of the evaluation language model for each relevance detection data;

[0140] According to the relevance evaluation results, the documents with low relevance to the user question and the current candidate answer are filtered out from the candidate retrieval documents.

[0141] Based on the above embodiments, the iterative optimization module 440 is specifically configured to:

[0142] Re-input the user question and the enhanced retrieval document collection into the large language model to obtain the new answer inferred by the large language model;

[0143] Invoke the reflection function of the large language model to detect whether the new answer has information gaps;

[0144] The reflection function is achieved by setting a preset prompt word template with review-guiding semantics;

[0145] If so, determining a collaborative strategy that matches the new answer based on all types of knowledge sources and at least one type of knowledge source in the most recently generated collaborative strategy;

[0146] If not, a collaborative strategy matching the new answer is determined based on at least one type of knowledge source in the most recently generated collaborative strategy.

[0147] Based on the above embodiments, the result evaluation output module 450 is specifically configured to:

[0148] The semantic reordering mechanism is used to reorder the candidate answers, and the first candidate answer after reordering is used as the target answer as the feedback result for the user's question.

[0149] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0150] Example 5

[0151] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0152] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0153] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0154] The processor 11 may be a variety of general and / or specialized processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a large language model question answering method enhanced by multi-source knowledge, that is:

[0155] Input the user question into the large language model, obtain the initial answer inferred by the large language model for the user question, and the collaborative strategy that matches the current candidate answer;

[0156] After taking the initial answer as the current candidate answer, extract the target knowledge source types in the collaborative strategy of the current candidate answer, and generate a call expression corresponding to each target knowledge source type based on the current candidate answer;

[0157] According to each calling expression, target knowledge sources matching each target knowledge source type are called respectively to obtain an enhanced retrieval document set;

[0158] Re-input the user question and the enhanced retrieval document collection into the large language model to obtain the new answer inferred by the large language model and the collaborative strategy that matches the new answer;

[0159] After taking the new answer as the current candidate answer, return to execute the operation of extracting at least one target knowledge source type in the collaborative strategy of the current candidate answer, and when the end iteration condition is met, determine the target answer among all candidate answers as the feedback result to the user question.

[0160] In some embodiments, a large language model question answering method enhanced by multi-source knowledge may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the large language model question answering method enhanced by multi-source knowledge described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute a large language model question answering method enhanced by multi-source knowledge in any other appropriate manner (for example, by means of firmware).

[0161] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0162] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0163] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0164] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0165] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0166] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0167] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0168] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A large language model question answering method enhanced by multi-source knowledge, characterized by: include: Input the user question into the large language model, obtain the initial answer inferred by the large language model for the user question, and the collaborative strategy that matches the current candidate answer; After taking the initial answer as the current candidate answer, extract the target knowledge source types in the collaborative strategy of the current candidate answer, and generate a call expression corresponding to each target knowledge source type based on the current candidate answer; According to each calling expression, target knowledge sources matching each target knowledge source type are called respectively to obtain an enhanced retrieval document set; Re-input the user question and the enhanced retrieval document collection into the large language model to obtain the new answer inferred by the large language model and the collaborative strategy that matches the new answer; After taking the new answer as the current candidate answer, return to execute the operation of extracting at least one target knowledge source type in the collaborative strategy of the current candidate answer, and when the end iteration condition is met, determine the target answer among all candidate answers as the feedback result to the user question.

2. The method according to claim 1, characterized in that Input the user question into the large language model, obtain the initial answer inferred by the large language model for the user question, and the collaborative strategy that matches the current candidate answer, including: Input the user's question into the large language model and obtain the initial answer inferred by the large language model for the user's question; According to multiple types of knowledge sources, a prompt word template is constructed, and the prompt word template is provided to a large language model, and the large language model is obtained to generate and output a collaborative strategy based on at least one type of knowledge source randomly selected from the multiple types of knowledge sources.

3. The method according to claim 1, characterized in that Generate a call expression corresponding to each target knowledge source type based on the current candidate answer, including: Input the current candidate answer into the pre-built named entity recognition model to obtain the key entities contained in the current candidate answer; Perform entity disambiguation on each identified key entity based on the task context that matches the user's question; Obtaining a call template that matches each target knowledge source type, wherein the call template includes at least one blank item for filling entity content; The entity content required for each call template is obtained from each key entity that has completed the disambiguation process, and each call template is filled to obtain the call expression corresponding to each target knowledge source type.

4. The method according to claim 1, wherein According to each call expression, the target knowledge source matching each target knowledge source type is called respectively to obtain an enhanced retrieval document set, including: Inputting the call expressions corresponding to each target knowledge source type into the matching target knowledge source, and obtaining at least one candidate retrieval document fed back by each target knowledge source for the call expression; After filtering out documents with low relevance to the user question and the current candidate answer from each candidate retrieval document, the remaining candidate retrieval documents are organized into an enhanced retrieval document set.

5. The method according to claim 4, characterized in that Filter out documents with low relevance to the user's question and the current candidate answer from each candidate retrieval document, including: Organize each candidate retrieval document with the user question and the current candidate answer to obtain relevance detection data; Input each relevance detection data into the pre-trained evaluation language model, and obtain the relevance evaluation result of the evaluation language model for each relevance detection data; According to the relevance evaluation results, the documents with low relevance to the user question and the current candidate answer are filtered out from the candidate retrieval documents.

6. The method according to any one of claims 2 to 5, characterized in that: Re-input the user question and the enhanced retrieval document collection into the large language model to obtain the new answer inferred by the large language model and the collaborative strategy matching the new answer, including: Re-input the user question and the enhanced retrieval document collection into the large language model to obtain the new answer inferred by the large language model; Invoke the reflection function of the large language model to detect whether the new answer has information gaps; The reflection function is achieved by setting a preset prompt word template with review-guiding semantics; If so, determining a collaborative strategy that matches the new answer based on all types of knowledge sources and at least one type of knowledge source in the most recently generated collaborative strategy; If not, a collaborative strategy matching the new answer is determined based on at least one type of knowledge source in the most recently generated collaborative strategy.

7. The method according to claim 1, characterized in that Determine the target answer from all candidate answers as a result of feedback to the user's question, including: The semantic re-ranking mechanism is used to re-rank the candidate answers, and the first candidate answer after re-ranking is used as the target answer as the feedback result to the user's question.

8. A large language model question answering device enhanced by multi-source knowledge, characterized in that: The device comprises: The initial reasoning module is used to input the user question into the large language model, obtain the initial answer inferred by the large language model for the user question, and the collaborative strategy that matches the current candidate answer; The collaborative strategy parsing module is used to extract the target knowledge source types in the collaborative strategy of the current candidate answer after taking the initial answer as the current candidate answer, and generate a call expression corresponding to each target knowledge source type based on the current candidate answer; A multi-source knowledge retrieval module is used to call target knowledge sources that match the target knowledge source type according to each call expression to obtain an enhanced retrieval document set; An iterative optimization module, which is used to re-input user questions and enhanced retrieval document sets into the large language model, obtain new answers inferred by the large language model, and a collaborative strategy that matches the new answers; The result evaluation output module is used to take the new answer as the current candidate answer, return to execute the operation of extracting at least one target knowledge source type in the collaborative strategy of the current candidate answer, and when the end iteration condition is met, determine the target answer among all candidate answers as the feedback result to the user question.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a large language model question answering method with multi-source knowledge enhancement according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a large language model question answering method with multi-source knowledge enhancement according to any one of claims 1 to 7 when executed.

Images