A universal adaptive question answering method, system, storage medium and electronic device

Abstract

The present invention proposes a universal adaptive question-answering method, system, storage medium and electronic device, which belong to the field of artificial intelligence technology, and include: acquiring data content, dividing the data content into manageable units, extracting text summaries from the data content through a large language model, and converting them into vector representations to obtain vector data; creating an inverted index containing metadata, and introducing a content keyword field to achieve rapid filtering and retrieval based on content keywords, and using a vector database to store vector data; constructing a retriever proxy module, document retrieval and rearrangement, filtering the most relevant documents through a post-processor, constructing a relevance identifier, constructing an answer generation module, and using LLMs to combine user queries with corresponding most relevant documents; enhancing the model's comprehension ability; optimizing the model's use of external knowledge, improving the knowledge retrieval performance of different tasks, and demonstrating its potential and adaptability.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a universal adaptive question-answering method, system, storage medium and electronic device. Background Art

[0002] Large language models (LLMs), such as GPT-4, have made progress in tasks such as question answering (QA), but their knowledge can be incomplete and rely on parametric memory. Retrieval-augmented generation (RAG) methods address this issue by integrating non-parametric knowledge retrieval modules. However, existing RAG techniques face challenges such as performance degradation due to unified retrieval strategies, inclusion of irrelevant text, and difficulty in integrating retrieved knowledge. Proposed improvements include hierarchical processing, relevance evaluation, and consistency preference alignment. Despite these efforts, existing techniques still fall short in addressing these difficulties. Summary of the invention

[0003] The present invention aims at the problems in the prior art and proposes a general adaptive question answering method, system, storage medium and electronic device to overcome the limitations of the current retrieval-generation model. This is an innovative method for overcoming the limitations of the existing retrieval-generation model. The present invention integrates different retrieval strategies for different questions, processes irrelevant documents through post-processing operations, and adopts the Chain-of-Thought (COT) strategy to capture external textual knowledge. In addition, a content generation model and a strategy for aligning the consistency of retrieval information are introduced. Experimental results show that the present method is superior to existing methods in processing complex tasks.

[0004] In a first aspect, the present invention provides a general adaptive question answering method, comprising:

[0005] Obtain data content, divide the data content into manageable units, extract text summaries from the data content through a large language model, and convert them into vector representations to obtain vector data;

[0006] Create an inverted index containing metadata, introduce a content keyword field to achieve fast filtering and retrieval based on content keywords, and use a vector database to store vector data;

[0007] Build a retriever proxy module to correct spelling, grammatical errors, and missing information in queries using fine-tuned LLM and LORA techniques, classify queries based on complexity, and optimize queries through query rewriting, expansion, and intent recognition techniques;

[0008] Document retrieval and re-ranking: Retrieve documents by combining sparse retrieval, dense retrieval and hybrid retrieval, and select the optimal retrieval channel according to the characteristics of the query question. Re-rank the retrieved documents through the BGE-Reranker algorithm, put the documents with high relevance at the top, and select the top K documents as the most relevant documents;

[0009] The most relevant documents are filtered through a post-processor, and key summary information is extracted from the filtered most relevant documents, and then compressed and integrated;

[0010] Build a relevance discriminator to analyze the relevance of the most relevant documents retrieved to the query question, and send them to the answer generation module for processing if the content is relevant;

[0011] Build an answer generation module that combines user queries with the corresponding most relevant documents using LLMs.

[0012] Furthermore, the retriever agent module is constructed to correct spelling, grammatical errors and missing information using fine-tuned LLM and LORA technologies; to classify queries based on complexity, combining manual and automated annotations to achieve accurate classification; to optimize queries through query rewriting, expansion and intent recognition techniques to match user intent and system capabilities; and to assign queries to corresponding processing modules.

[0013] Furthermore, the document retrieval and re-ranking uses algorithms such as TF-IDF and BM25 for keyword matching and similarity-based document retrieval; uses the BGE-M3 model and vectorized documents for semantic similarity retrieval; combines sparse and dense methods to improve accuracy and relevance; selects the optimal retrieval channel based on query features; re-ranks the retrieved documents through the BGE-Reranker algorithm to place the most relevant documents at the top; selects and filters the top k most relevant documents to improve accuracy and relevance to users.

[0014] Furthermore, the most relevant documents are filtered through the post-processor, and the documents are carefully checked using keywords and metadata related to the question, and irrelevant information is removed to ensure the accuracy and relevance of the final result; key summary information is extracted from the filtered documents, and compressed and integrated to provide users with a detailed and efficient overview of the content.

[0015] Furthermore, the relevance identifier is used to analyze the relevance of the retrieved documents to the query question; if the document is determined to be irrelevant, the retrieval module will be restarted to obtain more accurate results, and advanced query techniques and hybrid retrieval strategies will be used in the re-retrieval process to improve accuracy; a large language model is used to capture semantic information and understand the intent of the question; the iterative retrieval process will continue until the preset conditions or the maximum number of iterations are reached; even after multiple iterations, if relevant content is still not available, the system will turn to the answer generation module; if the content is considered relevant, it will be sent to the answer generation module for further processing and generation of user information.

[0016] Furthermore, in the answer generation module, LLMs are used to combine user queries with the corresponding most relevant documents; the model is fine-tuned through supervised training; GPT-4 is used to process the most relevant documents to create a training dataset consisting of question and answer pairs; interference information is used to form triplets containing questions, answers, and related document blocks, and the triplets are used as units in the dataset; and the thought chain method is used for answer generation.

[0017] In a second aspect, the present invention provides a general adaptive question answering system, comprising:

[0018] The acquisition module is configured to: acquire data content, divide the data content into manageable units, extract a text summary of the data content through a large language model, and convert it into a vector representation to obtain vector data;

[0019] The vector database is configured to: create an inverted index containing metadata, introduce a content keyword field to achieve fast filtering and retrieval based on content keywords, and use the vector database to store vector data;

[0020] The retriever agent module is configured to: correct query spelling, grammatical errors, and missing information using fine-tuned LLM and LORA techniques, classify queries based on complexity, and optimize queries using query rewriting, expansion, and intent recognition techniques;

[0021] The document retrieval and re-ranking module is configured to: retrieve documents by combining sparse retrieval, dense retrieval and hybrid retrieval, and select the optimal retrieval channel according to the characteristics of the query question, re-rank the retrieved documents by the BGE-Reranker algorithm, put the documents with high relevance at the top, and select the top K documents as the most relevant documents;

[0022] The post-processor is configured to: filter the most relevant documents, extract key summary information from the filtered most relevant documents, and compress and integrate them;

[0023] A relevance discriminator configured to: analyze the relevance of the retrieved most relevant document to the query question and send it to the answer generation module for processing if the content is relevant;

[0024] The answer generation module is configured to combine user queries with the corresponding most relevant documents using LLMs.

[0025] In a third aspect, the present invention provides a computer-readable storage medium, the computer-readable storage medium including a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the general adaptive question-answering method described in the first aspect.

[0026] In a fourth aspect, the present invention provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the general adaptive question-answering method described in the first aspect.

[0027] Compared with the prior art, the advantages and positive effects of the present invention are:

[0028] The present invention provides a general adaptive question answering method, system, storage medium and electronic device, which are dedicated to dealing with the large model self-knowledge. The framework of the present invention consists of three components: vector storage, retrieval agent and answer generation agent, which is designed to improve the accuracy and precision of the response of the question answering system; the process starts with the vector storage module, which uses OCR and NLP to preprocess and integrate data from different sources; the generated data blocks are converted into vector representations and stored in the Milvus vector database; when a user query is received, the retrieval agent refines it and evaluates its complexity; for complex queries, the query conversion module is used to perform sub-question decomposition and query rewriting; then, the refined query is cross-referenced with the vector storage using a hybrid retrieval strategy; the retrieved documents are evaluated by the answer generation agent using a relevance identifier; if the document is not relevant, the retrieval process will be restarted. Relevant documents are combined with questions to generate hints, which are input into a large language model to generate answers; the quality discriminator ensures the quality of the answer, and for low-quality content, it triggers repeated iterations of the retrieval agent or answer generation module. It is a comprehensive and innovative framework that improves problem-solving capabilities and the relevance of retrieved content through iterative evaluation of retrieval, and solves complex challenges; it includes question correction and transformation, which enhances the model's understanding ability by decomposing and rewriting complex queries; it combines COT and contrastive learning mechanisms to optimize the model's use of external knowledge; it provides innovative solutions for key modules, improves the knowledge retrieval performance of different tasks, and demonstrates its potential and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 Schematic diagram of the general adaptive question answering method of the present invention. DETAILED DESCRIPTION

[0031] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0032] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0033] Embodiment 1, as Figure 1 As shown, this embodiment provides a general adaptive question answering method, which is dedicated to dealing with large models of self-knowledge. The framework of the present invention consists of three components: vector storage, retrieval agent and answer generation agent, such as Figure 1 As shown. It aims to improve the accuracy and precision of the responses of question-answering systems. The process starts with the vector storage module, which preprocesses and synthesizes data from different sources using OCR and NLP. The generated data blocks are converted into vector representations and stored in the Milvus vector database. When a user query is received, the retrieval agent refines it and evaluates its complexity. For complex queries, the query transformation module is used for sub-question decomposition and query rewriting. The refined query is then cross-referenced with the vector storage using a hybrid retrieval strategy. The retrieved documents are evaluated by the answer generation agent using a relevance discriminator. If the document is not relevant, the retrieval process is restarted. Relevant documents are combined with questions to generate hints, which are fed into a large language model to generate answers. The quality discriminator ensures the quality of the answer, and for low-quality content, it triggers repeated iterations of the retrieval agent or the answer generation module. The specific details are as follows:

[0034] Preprocessing: Data integration and key information extraction: Extract text from images and PDFs through OCR technology, and then segment the content into manageable units for semantic capture.

[0035] Vectorization technology: Use vectorization technology to improve retrieval accuracy, including training the BGE-M3 model to improve retrieval accuracy and indexing efficiency.

[0036] Text Summarization: Leverage Large Language Models (LLMs) for text summarization to generate high-quality vector representations for efficient information retrieval and organization.

[0037] Index construction and data storage: Data vectorization and inverted index creation: Create an inverted index containing metadata and introduce content keyword fields to achieve fast filtering and retrieval based on content keywords.

[0038] HNSW algorithm: Uses the Hierarchical Navigable Small World algorithm for nearest neighbor search to ensure high recall.

[0039] Milvus Vector Database: Use the Milvus database for data storage of large-scale vector retrieval tasks to achieve efficient storage and access to vectorized data.

[0040] (iii) Retrieval Agent Module: Query Correction: Use fine-tuned LLM and LORA technologies to correct spelling, grammatical errors and missing information to improve the accuracy and comprehensibility of queries.

[0041] Query Classification: Classify queries based on complexity, combining human and automated annotations for accurate classification.

[0042] Query classification and routing: Ensures queries are handled and resources are allocated appropriately.

[0043] Query optimization technology: Optimize queries through query rewriting, expansion, and intent recognition to match user intent and system capabilities.

[0044] Optimize routing agents: Intelligently assign queries to appropriate processing modules to improve accuracy and efficiency. Specifically, based on COT prompt engineering, a large language model is used to reasonably allocate processing modules based on query complexity.

[0045] (IV) Document retrieval and re-ranking: Sparse retrieval: Use algorithms such as TF-IDF and BM25 for keyword matching and similarity-based document retrieval;

[0046] Dense retrieval: Use the BGE-M3 model and vectorized documents for semantic similarity retrieval.

[0047] Hybrid retrieval: Combining sparse and dense methods to improve accuracy and relevance.

[0048] Proxy routing method: select the optimal retrieval channel based on query characteristics.

[0049] BGE-Reranker algorithm: Reranks the retrieved documents, putting the most relevant documents at the top.

[0050] Selection and Filtering: Select and filter the top k most relevant documents to improve accuracy and relevance to users.

[0051] (V) Post-processor: Filtering: Use keywords and metadata related to the problem to carefully check the documents and remove irrelevant information to ensure the accuracy and relevance of the final results.

[0052] Information compression: Extract key summary information from filtered documents, compress and integrate it to provide users with a detailed and efficient overview of the content.

[0053] (vi) Relevance Identifier: Document Relevance Analysis: After retrieval, the system applies a content relevance identifier to analyze the relevance of the retrieved documents to the original question.

[0054] Re-search: If the document is determined to be irrelevant, the search module will be restarted to obtain more accurate results, and advanced query techniques and hybrid search strategies will be used during the re-search process to improve accuracy.

[0055] Semantic Understanding and Intent Capture: Use Large Language Models (LLMs) to capture semantic information and understand the intent of the question.

[0056] Iterative retrieval process: The iterative retrieval process continues until a preset condition or a maximum number of iterations (usually five) is reached.

[0057] Answer Generation Module Transfer: Even after multiple iterations, if no relevant content is available, the system will transfer to the Answer Generation Module. If the content is considered relevant, it is sent to the Answer Generation Module for further processing and generating information for the user.

[0058] (VII) Answer Generation: Use of LLMs: In the answer generation module, LLMs are used to combine user queries with corresponding answers to produce more precise and comprehensive responses.

[0059] Model fine-tuning: Fine-tune the model through supervised training to improve its understanding and generation capabilities of retrieved knowledge.

[0060] Training dataset creation: Use GPT-4 to process relevant documents and create a training dataset consisting of question and answer pairs.

[0061] Triple formation: Triples containing question, answer, and related document chunks are formed using noise information, and these triplets serve as units in the dataset.

[0062] COT method: Use the Chain-of-Thought (COT) method for answer generation to ensure consistency between search and generation models, thereby improving the consistency, accuracy, and reliability of content generation.

[0063] Through these steps, the present invention can ensure that the retrieved content is not only relevant but also accurate, while the generated answers are both precise and comprehensive, providing users with valuable and personalized information.

[0064] Take the example of intelligent document retrieval and enhanced generation in a specific scientific field:

[0065] Preprocessing stage: In this stage, the present embodiment integrates data sources, extracts key information, and uses vectorization technology to improve retrieval accuracy. First, the present embodiment loads and parses data from various channels, including images and text extraction from PDFs, using OCR technology. Then, the present embodiment divides the content into manageable units through document chunking to capture semantics. Finally, the present embodiment uses a contrastive learning strategy to train the BGE-M3 model to improve retrieval accuracy and indexing efficiency. In addition, the present embodiment uses large language models (LLMs) for text summary extraction to generate high-quality vector representations for effective information retrieval and organization. By integrating these technologies, the present embodiment is able to efficiently retrieve documents and provide related information services.

[0066] Index construction and data storage phase: In this phase, the present embodiment focuses on achieving efficient retrieval by building indexes and storing data. The present embodiment performs data vectorization and creates an inverted index with metadata. The present embodiment also introduces a content keyword field to enable fast filtering and retrieval based on specific content keywords. To ensure a high recall rate, the present embodiment uses the hierarchical navigable small world (HNSW) algorithm for nearest neighbor search. In terms of data storage, the present embodiment utilizes the Milvus vector database, which is designed for vector query and retrieval. With its support for large-scale vector retrieval tasks, the present embodiment is able to efficiently store and access vectorized data. By implementing these techniques, the present embodiment builds an efficient indexing and storage system to quickly and accurately retrieve documents based on user queries.

[0067] Question optimization stage: In this stage, the present embodiment implements four steps to improve the accuracy and efficiency of query processing. First, the present embodiment uses a large language model (LLM) fine-tuned by LORA technology to correct spelling, grammatical errors, and missing information in user queries. This enhances the accuracy and comprehensibility of queries. Secondly, the present embodiment uses the LLM model to classify queries, and classifies queries according to complexity by combining manual and automated annotations to achieve accurate classification. Third, the present embodiment classifies and redirects queries to ensure appropriate processing and resource allocation. Fourth, the present embodiment optimizes queries through techniques such as query rewriting, expansion, and intent recognition to match user intent with system capabilities. Finally, an optimized routing agent intelligently assigns queries to appropriate processing modules to improve accuracy and efficiency. These steps together improve the accuracy and efficiency of query processing.

[0068] Recall and reranking phase: In this phase, the present embodiment adopts multiple strategies to enhance the retrieval process. The present embodiment starts with sparse retrieval, using algorithms such as TF-IDF and BM25 for keyword matching and similarity-based document retrieval. Dense retrieval uses the BGE-M3 model and vectorized documents for semantic similarity retrieval. Hybrid retrieval combines sparse and dense methods to improve accuracy and relevance. To ensure accuracy, the present embodiment uses a proxy routing method to select the optimal retrieval channel based on query features. Then, the present embodiment uses the BGEReranker algorithm to rerank the retrieved documents, putting the most relevant documents at the front. Finally, the present embodiment selects and filters the most relevant k documents to improve accuracy and relevance to the user. By integrating these strategies, the present embodiment enhances the effectiveness and accuracy of the retrieval process in the question-and-answer system.

[0069] Post-processing stage: In the post-processing stage, the present embodiment optimizes the retrieved documents through filtering and information compression. The filtering stage involves carefully examining the documents using keywords and metadata related to the question to remove irrelevant information and ensure the accuracy and relevance of the final results. Next, in the information compression step, the present embodiment extracts key summary information from the filtered documents. This information is compressed and integrated to provide a comprehensive but concise summary of relevant content. These post-processing techniques enhance the quality and usability of retrieved information in the question-answering system by providing users with a detailed and efficient overview of the content.

[0070] Relevance determination stage: After retrieval, the system of this embodiment applies a content relevance identifier to analyze the relevance of the retrieved documents to the original question. If the document is determined to be irrelevant, the retrieval module will restart to obtain more accurate results. During the re-retrieval process, advanced query techniques and hybrid retrieval strategies are used to improve accuracy. In order to capture semantic information and understand the intent of the question, this embodiment uses a large language model (LLM). This iterative retrieval process will continue until the preset conditions or the maximum number of iterations (usually five) are reached. Even if no relevant content is obtained after multiple iterations, the system will turn to the answer generation module. If the content is considered relevant, it will be sent to the answer generation module for further processing and generate user-specific information. The system of this embodiment ensures that relevant and accurate content is retrieved to provide users with valuable and personalized information.

[0071] Answer generation stage: In the answer generation module, this embodiment uses large language models (LLMs) to combine user queries with corresponding answers to produce more accurate and comprehensive responses. This embodiment fine-tunes the model through supervised training to improve its understanding of retrieved knowledge. GPT-4 processes relevant documents to create a training dataset containing question and answer pairs. Interference information is used to form triplets consisting of questions, answers, and related document fragments. These triplets serve as units in the dataset of this embodiment. This embodiment uses the COT method for answer generation to ensure consistency between the search and generation models to improve the consistency, accuracy, and reliability of content generation.

[0072] Answer evaluation phase: In this phase, the present embodiment implements an advanced answer quality assessment model to evaluate the answers generated by the system. This model is designed to identify any inaccuracies in the answers. If the model determines that the answer is inaccurate, the system will initiate a secondary generation process to provide more precise information. Conversely, if the answer is verified as correct by the evaluation model, it will be presented directly to the user. This ensures that the user receives high-quality and accurate feedback without further refinement. By adopting this answer quality assessment model, the present embodiment is committed to improving the reliability and accuracy of the responses generated by the system, thereby improving the overall user experience.

[0073] Embodiment 2: This embodiment provides a general adaptive question answering system, including:

[0074] The acquisition module is configured to: acquire data content, divide the data content into manageable units, extract a text summary of the data content through a large language model, and convert it into a vector representation to obtain vector data;

[0075] The vector database is configured to: create an inverted index containing metadata, introduce a content keyword field to achieve fast filtering and retrieval based on content keywords, and use the vector database to store vector data;

[0076] The retriever agent module is configured to: correct query spelling, grammatical errors, and missing information using fine-tuned LLM and LORA techniques, classify queries based on complexity, and optimize queries using query rewriting, expansion, and intent recognition techniques;

[0077] The document retrieval and re-ranking module is configured to: retrieve documents by combining sparse retrieval, dense retrieval and hybrid retrieval, and select the optimal retrieval channel according to the characteristics of the query question, re-rank the retrieved documents by the BGE-Reranker algorithm, put the documents with high relevance at the top, and select the top K documents as the most relevant documents;

[0078] The post-processor is configured to: filter the most relevant documents, extract key summary information from the filtered most relevant documents, and compress and integrate them;

[0079] A relevance discriminator configured to: analyze the relevance of the retrieved most relevant document to the query question and send it to the answer generation module for processing if the content is relevant;

[0080] The answer generation module is configured to combine user queries with the corresponding most relevant documents using LLMs.

[0081] Embodiment 3: This embodiment provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the general adaptive question-answering method described in Embodiment 1.

[0082] Embodiment 4: This embodiment provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the general adaptive question-answering method described in Embodiment 1.

[0083] The electronic device may include: a processor, a memory and a communication unit. These components communicate via one or more buses. Those skilled in the art will appreciate that the structure of the electronic device does not limit the embodiments of the present invention, and it may be a bus structure or a star structure, or a combination of certain components, or different component arrangements.

[0084] The communication unit is used to establish a communication channel so that the electronic device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0085] The processor is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor can only include a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0086] The memory is used to store the execution instructions of the processor. The memory can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0087] When the execution instructions in the memory are executed by the processor, the electronic device is enabled to execute part or all of the steps of Embodiment 1.

[0088] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A general adaptive question answering method, characterized in that: include: Obtain data content, divide the data content into manageable units, extract text summaries from the data content through a large language model, and convert them into vector representations to obtain vector data; Create an inverted index containing metadata, introduce a content keyword field to achieve fast filtering and retrieval based on content keywords, and use a vector database to store vector data; Build a retriever proxy module to correct spelling, grammatical errors, and missing information in queries using fine-tuned LLM and LORA techniques, classify queries based on complexity, and optimize queries through query rewriting, expansion, and intent recognition techniques; Document retrieval and re-ranking, combining sparse retrieval, dense retrieval and hybrid retrieval to retrieve documents, and selecting the optimal retrieval channel according to the characteristics of the query question, re-ranking the retrieved documents through the BGE-Reranker algorithm, placing the documents with high relevance at the top, and selecting the top K documents as the most relevant documents; the document retrieval and re-ranking, using TF-IDF and BM25 algorithms for keyword matching and similarity-based document retrieval; using the BGE-M3 model and vectorized documents for semantic similarity retrieval; combining sparse and dense methods to improve accuracy and relevance; selecting the optimal retrieval channel according to the query characteristics; re-ranking the retrieved documents through the BGE-Reranker algorithm, placing the most relevant documents at the top; selecting and filtering the top k most relevant documents to improve accuracy and relevance to users; The most relevant documents are filtered through a post-processor, and key summary information is extracted from the filtered most relevant documents, and then compressed and integrated; Build a relevance discriminator to analyze the relevance of the most relevant documents retrieved to the query question, and send them to the answer generation module for processing if the content is relevant; Construct an answer generation module and use LLMs to combine user queries with the corresponding most relevant documents; In the answer generation module, use LLMs to combine user queries with the corresponding most relevant documents; Fine-tune the model through supervised training; Use GPT-4 to process the most relevant documents and create a training dataset consisting of question and answer pairs; Use interference information to form triples containing questions, answers, and related document blocks, and triples are used as units in the dataset; Use the thought chain method for answer generation; The retriever agent module is constructed to correct spelling, grammatical errors and missing information using fine-tuned LLM and LORA techniques; Classify queries based on complexity, combining manual and automated annotations for accurate classification; optimize queries through query rewriting, expansion, and intent recognition techniques to match user intent and system capabilities; assign queries to corresponding processing modules; The post-processor filters the most relevant documents, carefully checks the documents using keywords and metadata related to the question, removes irrelevant information, and ensures the accuracy and relevance of the final results; Extract key summary information from filtered documents, compress and integrate them to provide users with a detailed and efficient overview of the content; The relevance identifier is used to analyze the relevance of the retrieved documents to the query question; If a document is determined to be irrelevant, the retrieval module will be restarted to obtain more accurate results, and advanced query techniques and hybrid retrieval strategies will be used during the re-retrieval process to improve accuracy; large language models are used to capture semantic information and understand the intent of the question; The iterative retrieval process continues until the preset condition or the maximum number of iterations is reached; even after multiple iterations, if relevant content is still not available, the system will turn to the answer generation module; if the content is considered relevant, it will be sent to the answer generation module for further processing and generation of user information.

2. A general adaptive question answering system, characterized in that: include: The acquisition module is configured to: acquire data content, divide the data content into manageable units, extract a text summary of the data content through a large language model, and convert it into a vector representation to obtain vector data; The vector database is configured to: create an inverted index containing metadata, introduce a content keyword field to achieve fast filtering and retrieval based on content keywords, and use the vector database to store vector data; The retriever agent module is configured to: correct query spelling, grammatical errors, and missing information using fine-tuned LLM and LORA techniques, classify queries based on complexity, and optimize queries using query rewriting, expansion, and intent recognition techniques; The document retrieval and rearrangement module is configured to: retrieve documents by combining sparse retrieval, dense retrieval and hybrid retrieval, and select the optimal retrieval channel according to the characteristics of the query question, re-rank the retrieved documents by the BGE-Reranker algorithm, put the documents with high relevance at the top, and select the top K documents as the most relevant documents; the document retrieval and rearrangement uses TF-IDF and BM25 algorithms for keyword matching and similarity-based document retrieval; uses the BGE-M3 model and vectorized documents for semantic similarity retrieval; combines sparse and dense methods to improve accuracy and relevance; selects the optimal retrieval channel according to the query characteristics; re-ranks the retrieved documents by the BGE-Reranker algorithm, puts the most relevant documents at the top; selects and filters the top k most relevant documents to improve accuracy and relevance to users; The post-processor is configured to: filter the most relevant documents, extract key summary information from the filtered most relevant documents, and compress and integrate them; A relevance discriminator configured to: analyze the relevance of the retrieved most relevant document to the query question and send it to the answer generation module for processing if the content is relevant; The answer generation module is configured to: combine user queries with corresponding most relevant documents using LLMs; in the answer generation module, combine user queries with corresponding most relevant documents using LLMs; fine-tune the model through supervised training; process the most relevant documents using GPT-4 to create a training dataset consisting of question and answer pairs; use interference information to form triples containing questions, answers, and related document blocks, with the triples serving as units in the dataset; and generate answers using a thought chaining approach; Build a retriever agent module to correct spelling, grammatical errors and missing information using fine-tuned LLM and LORA techniques; Classify queries based on complexity, combining manual and automated annotations for accurate classification; optimize queries through query rewriting, expansion, and intent recognition techniques to match user intent and system capabilities; assign queries to corresponding processing modules; The post-processor filters the most relevant documents, carefully checks the documents using keywords and metadata related to the question, removes irrelevant information, and ensures the accuracy and relevance of the final results; Extract key summary information from filtered documents, compress and integrate them to provide users with a detailed and efficient overview of the content; The relevance identifier is used to analyze the relevance of the retrieved documents to the query question; If a document is determined to be irrelevant, the retrieval module will be restarted to obtain more accurate results, and advanced query techniques and hybrid retrieval strategies will be used during the re-retrieval process to improve accuracy; large language models are used to capture semantic information and understand the intent of the question; The iterative retrieval process continues until the preset condition or the maximum number of iterations is reached; even after multiple iterations, if relevant content is still not available, the system will turn to the answer generation module; if the content is considered relevant, it will be sent to the answer generation module for further processing and generation of user information.

3. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the general adaptive question-answering method described in claim 1.

4. An electronic device, characterized in that: The invention comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the universal adaptive question answering method described in claim 1.

Images