Interactive question-answer system with retrieval-augmented generation for personalized database

Abstract

An interactive question-answer system with retrieval-augmented generation for a personalized database according to the present invention comprises: an input unit which allows a user to input a question or map, through a tagging box system, a portion of text of a document of interest to a keyword to which a tag is assigned and store the mapped text in a personal archive; and an RAG pipeline which retrieves and injects information by using the personal archive and an external database for the question input through the input unit, and generates an answer on the basis of the information. Therefore, the accuracy and reliability of information can be significantly improved.

Description

Interactive question-and-answer system with search augmentation generation for personalized databases

[0001] The present invention relates to an interactive question-and-answer system with retrieval-augmented generation for a personalized database, and more specifically, to an interactive question-and-answer system with retrieval-augmented generation for a personalized database of a novel approach that combines retrieval-augmented generation (RAG) with a personalized database system to improve the precision and relevance of information retrieval.

[0002] Conventional Large Language Models (LLMs) demonstrate excellent text generation capabilities, but they suffer from limitations in knowledge accuracy and hallucination problems.

[0003] In addition, over the past few years, large-scale language models (LLM) and natural language processing technologies have brought innovation to the field of artificial intelligence by utilizing vast datasets and powerful computational resources.

[0004] OpenAI's Generative Pre-trained Transformer (GPT) series is a prime example; GPT-1, released in 2018, demonstrated high performance in natural language processing and generation tasks by utilizing the Transformer architecture and transfer learning techniques. Subsequently, GPT-2 further expanded the model's capabilities, and GPT-3, equipped with billions of parameters, significantly improved the ability to generate complex and diverse information.

[0005] These advancements have had a profound impact on various natural language processing applications, such as translation, question answering (QA), document summarization, and content creation. Currently, LLM is performing complex tasks like automatic translation, QA, document summarization, and content creation across multiple fields, including healthcare, education, and science.

[0006] While these pre-trained LLMs can generate increasingly realistic text, their ability to access and accurately manipulate knowledge remains limited, presenting a problem where it is difficult to verify the accuracy and authenticity of results and not easy to update the model to reflect the latest information.

[0007] As a result, while LLM provides convenience, it has the problem of generating inappropriate or error-containing responses in certain situations.

[0008] To examine the aforementioned problems in more detail, while Large-Scale Language Models (LLMs) have driven innovation in the field of Natural Language Processing (NLP), they suffer from issues such as false information, logical errors, and the generation of non-existent data that is consistent but not based on actual facts.

[0009] For example, it can provide information about people or events that do not exist.

[0010] Hallucinations include actual hallucinations containing information that is not factual, logical hallucinations containing answers that are logically inconsistent or meaningless, and structural hallucinations containing responses that are incorrect or inappropriate in format.

[0011] This problem has become more pronounced recently due to ChatGPT's tendency to generate plausible-sounding false information.

[0012] Hallucinations can cause serious problems because they contribute to the spread of misinformation and undermine the credibility of content generated in LLM.

[0013] To investigate this phenomenon, Hanna et al. entered 54 prompts from various domains into ChatGPT and confirmed that the overall success rates of ChatGPT-3.5 and ChatGPT-4 were approximately 61% and 72%, respectively, and the hallucination rates were 39% and 28%, respectively.

[0014] These results were obtained through prompt-dependent experiments but represent the hallucination rate of LLM.

[0015] The causes of hallucinations are diverse, and the following factors have been cited as major causes.

[0016] Training Data Bias: Most LLMs are trained based on vast amounts of text data collected from the Internet. This data inevitably contains incomplete or biased information, which poses a problem that increases the likelihood of the model learning incorrect patterns.

[0017] Model's inference method: LLM works by predicting the next word.

[0018] Therefore, it attempts to generate the most valid word within a given context, but this has the problem of potentially generating false information out of context.

[0019] Interaction Limitations: LLM generates information through interaction with human users, but this interaction can be limited. Consequently, there is a problem in that responses generated based on this information may be inaccurate, and it can be difficult to obtain accurate information.

[0020] (Patent Document 1) Korean Registered Patent Publication No. 10-2192796 (December 14, 2020)

[0021] To solve the aforementioned problems, the present invention aims to provide an interactive question-and-answer system equipped with search augmentation generation for a personalized database that solves the aforementioned problems and improves search quality by combining a large-scale language model (LLM) with structured personalized data, and enables users to efficiently search their own data repositories by tagging keywords within personal documents and organizing them into context-based categories.

[0022] Furthermore, to solve the aforementioned problems, the present invention aims to provide an interactive question-and-answer system equipped with Retrieval-Augmented Generation for a personalized database, which enables users to efficiently search their document repository through a keyword and context hierarchy constructed by applying the Retrieval-Augmented Generation (RAG) technique to a personalized database system, thereby tagging keywords set by the user through the QA process based on context, and allowing information to be managed as if it were structured into categories such as dates and topics.

[0023] In addition, to solve the aforementioned problems, the present invention aims to provide an interactive question-and-answer system equipped with search augmentation generation for a personalized database, which implements a RAG pipeline for context-based search enhancement and applies a NoSQL database to continuously update the user's search and conversation history, and is equipped with a personalized DB and a QA system.

[0024] Furthermore, to solve the aforementioned problems, the present invention aims to provide an interactive question-and-answer system equipped with search augmentation generation for a personalized database, which enables interactive search by understanding embedded personal information and prioritizing related documents when an individual or team structures and tags documents they wish to preserve and updates the database, thereby allowing a user to form a personal knowledge base by selecting specific sentences from personal documents and saving them along with keywords and links in a system called a TaggingBox.

[0025] Furthermore, to solve the aforementioned problems, the present invention aims to provide an interactive question-and-answer system equipped with search augmentation generation for a personalized database using RAG to implement search augmentation in a personalized database service and to resolve issues such as hallucinations, lack of updates, and unclear sources of the existing ChatGPT-3.5 model.

[0026] To achieve the above-mentioned purpose, the interactive question-answer system with search augmentation generation for a personalized database according to the present invention is characterized by comprising: an input unit that allows a user to input a question or map part of text of a document of interest to tagged keywords through a tagging box system and save it to a personal archive; and a RAG pipeline that retrieves and injects information using the personal archive and an external database for a question input through the input unit, and generates an answer based on the information.

[0027] To achieve the above-mentioned purpose, the RAG pipeline of an interactive question-answer system equipped with search augmentation generation for a personalized database according to the present invention is characterized by comprising: an SQL database that stores documents and information personalized according to personal identity; a vector database that reconstructs information by converting data chunks extracted from the SQL database into embedding vectors; and a QA generator that generates answers to questions by utilizing Large Language Models (LLMs) based on a context selected from the vector database.

[0028] To achieve the above-mentioned purpose, the SQL database of an interactive question-answer system equipped with search augmentation generation for a personalized database according to the present invention is characterized by extracting information matching a user ID from a user information table and dividing it into chunks for further processing when a question is entered into a user interface through the input unit.

[0029] To achieve the above-mentioned objective, the vector database of an interactive question-answer system equipped with search augmentation generation for a personalized database according to the present invention is characterized by using similarity search between vector data to process user queries.

[0030] To achieve the above-mentioned purpose, the RAG pipeline of an interactive question-answer system with search augmentation generation for a personalized database according to the present invention is characterized by further including MongoDB, which can store chat records of QA and generate answers that the user can use later.

[0031] To achieve the above-mentioned purpose, the RAG pipeline of an interactive question-answer system equipped with search augmentation generation for a personalized database according to the present invention further includes a LangChain, which is a framework for building an environment by integrating external tools, wherein the LangChain is characterized by extracting relevant data from a user information table through a query when extracting data from the SQL database.

[0032] To achieve the above-mentioned purpose, the LangChain of an interactive question-answer system with search augmentation generation for a personalized database according to the present invention is characterized by splitting the extracted data into claims and transmitting them to the vector database (searcher) where embeddings are generated.

[0033] To achieve the above-mentioned purpose, the LangChain of an interactive question-answer system with search augmentation generation for a personalized database according to the present invention is characterized by collecting user data from the SQL database and returning it in JSON format consisting of key-value pairs.

[0034] To achieve the above-mentioned purpose, the QA generator of an interactive question-answer system with search augmentation generation for a personalized database according to the present invention is characterized by generating an answer by taking into account the entire context provided in a prompt that serves as an interface between a user question and a large-scale language model.

[0035] To achieve the above-mentioned purpose, the QA generator of an interactive question-answer system equipped with search augmentation generation for a personalized database according to the present invention is characterized by excluding information not specified in context from answers and specifying the source of information in all answers.

[0036] To achieve the above-mentioned purpose, the QA generator of an interactive question-answer system equipped with search augmentation generation for a personalized database according to the present invention is characterized by notifying the user that, when an answer cannot be found, the user must search for tags or content in a designated tagging box to obtain an accurate result.

[0037] To achieve the above-mentioned purpose, the RAG pipeline of an interactive question-answer system equipped with search augmentation generation for a personalized database according to the present invention is characterized by further including an evaluation unit that evaluates the performance of the system based on reliability, answer relevance, contextual precision, and contextual recall.

[0038] Unlike the LLM-only generation method, the interactive question-and-answer system equipped with search augmentation generation for a personalized database according to the present invention has the effect of significantly increasing the accuracy and reliability of information by searching for relevant information from an external database and reflecting it in the answer.

[0039] In addition, the interactive question-and-answer system equipped with search augmentation generation for a personalized database according to the present invention has the effect of increasing transparency by specifying the source when providing an answer, thereby allowing the user to directly verify the basis of the response.

[0040] In addition, the interactive question-and-answer system equipped with search augmentation generation for a personalized database according to the present invention has the effect of providing customized answers to questions by individual or team by searching based on personalized documents and keywords directly saved by the user in the tagging box.

[0041] In addition, the interactive question-and-answer system equipped with search augmentation generation for a personalized database according to the present invention can frequently update a pre-built personalized DB and an external latest DB, thereby having the effect of reducing the problem of information omission and the risk of hallucination associated with LLM after the time of construction.

[0042] In addition, the interactive question-and-answer system with search augmentation generation for a personalized database according to the present invention has the effect of enabling continuous and natural interaction by effectively maintaining the conversation context through MongoDB-based chat history management.

[0043] Figure 1 is a diagram illustrating the difference between large-scale language model (LLM) processing using search augmentation generative (RAG) and large-scale language model (LLM) processing without using it.

[0044] FIG. 2 is a structural diagram of an interactive question-answer system with search augmentation generation for a personalized database according to the present invention.

[0045] Figure 3 is a diagram illustrating an example of user data extracted in JSON format when, for example, the UserID is 2190.

[0046] Figure 4 is a diagram illustrating a document embedding workflow.

[0047] Figure 5 is a diagram illustrating the process of processing user prompts and generating answers using LangChain with MongoDB.

[0048] Figure 6 is a diagram showing an example of a prompt template developed in the present invention.

[0049] Figure 7 is a diagram illustrating an example of a custom prompt created for QA work.

[0050] FIG. 8 is a diagram evaluating the performance of a system according to the present invention using reliability, answer relevance, contextual precision, and contextual recall.

[0051] Figure 9 is a graph showing the accuracy of the results generated for each combination of LLM and embedding module.

[0052] Figure 10 is a graph showing the result of the relevance of answers to a common sense data set.

[0053] Figure 11 is a graph showing the context-reproduction scores of each model combination.

[0054] Figure 12 is a graph showing the context-precision scores of all models.

[0055] Figure 13 is a graph comparing the processing time per data item for each model combination.

[0056] Figure 14 is a graph showing the results of the reliability evaluation for the news article dataset.

[0057] Figure 15 is a graph of the response regarding the relevance results for the news article dataset.

[0058] Figure 16 is a graph of the context recall results for the news article dataset.

[0059] Figure 17 is a graph of the Contex precision results for the news article dataset.

[0060] Figure 18 is a graph of the processing time per data item for various LLM and embedding combinations for a news article dataset.

[0061] Figure 19 is a graph showing the comparative evaluation of processing time, cost, and performance indicators for Ada-002 + GPT-3.5-Turbo and Ada-002 + GPT-4.

[0062] *Explanation of major symbols in the drawings*

[0063] 100 : Input section

[0064] 200: RAG Pipeline

[0065] 210 : SQL Database

[0066] 211 : User Information Table

[0067] 212 : LangChain

[0068] 220 : Vector Database

[0069] 230 : QA Generator

[0070] 240 : MongoDB

[0071] 250 : Evaluation Department

[0072] Terms and words used in this specification and claims should not be interpreted as being limited to their ordinary or dictionary meanings, and should be interpreted in a meaning and concept consistent with the technical spirit of the invention, based on the principle that the inventor can appropriately define the concept of the terms to best describe his invention.

[0073] Therefore, the embodiments described in this specification and the configurations illustrated in the drawings are merely one preferred embodiment of the present invention and do not represent all of the technical ideas of the present invention; thus, it should be understood that various equivalents and modifications that can replace them may exist at the time of filing this application.

[0074] Hereinafter, an interactive question-and-answer system with search augmentation generation for a personalized database according to the present invention will be described in detail with reference to the attached drawings.

[0075] Before describing the interactive question-and-answer system with search augmented generation for a personalized database according to the present invention, the Retrieval-Augmented Generation (RAG) technique is described.

[0076] Currently, LLM developers are seeking to resolve issues of hallucination, lack of up-to-date information, and opacity of answer basis through Retrieval-Augmented Generation (RAG) techniques.

[0077] The aforementioned RAG is a technology that combines LLM with external knowledge databases to enhance response quality and relevance, and is particularly useful in situations requiring up-to-date and accurate information, such as academic research, customer service, and content creation.

[0078] The modern data environment contains vast amounts of information, so rapid and accurate searching is essential for effectively utilizing data.

[0079] As the demand for customized and up-to-date information increases, LLMs that hold information within parameters can also utilize RAGs to provide rapid updates and personalized search.

[0080] The aforementioned RAG searches for and injects necessary information, eliminating the need to directly store sensitive data in model parameters. Consequently, it is attracting attention as a core technology that provides personalized search functions not only for companies and institutions but also for personal information search services.

[0081] The general process of the above RAG and the differences between an LLM implementation using the RAG and an LLM implementation not using the RAG are illustrated in Fig. 1.

[0082] Consider a scenario where a user asks ChatGPT about the number of international students at a university. Since ChatGPT relies on pre-trained data, it cannot provide the latest data initially.

[0083] As shown on the left side of Fig. 1, ChatGPT cannot provide exact numbers and outputs only abstract information. In contrast, as shown on the right side of Fig. 1, RAG can retrieve knowledge from external databases and can provide exact numbers by referencing university databases.

[0084] Therefore, RAG improves ChatGPT's search performance in a manner similar to providing personalized learning materials for information retrieval.

[0085] Recently, modular RAG technology has emerged that enhances the flexibility of RAG by integrating new modules and other technologies such as fine-tuning.

[0086] For example, the implementation and advantages and disadvantages of pure, advanced, and modular RAG models were analyzed for RAG applications. The basic processes of RAG were classified, and various search and generator augmentation methodologies were investigated to introduce practical RAG applications and benchmarks.

[0087] We investigated the impact of search augmentation on LLM and analyzed RAG performance in terms of noise, denial, information integration, and semantic robustness. To evaluate RAG in English and Chinese, we experimented with four separate testbeds based on the aforementioned basic functions. The results showed that while LLM exhibits a certain level of noise robustness, it still faces significant challenges in denial, information integration, and falsification processing.

[0088] These results suggest the need to design applications more deeply and utilize databases to enhance the benefits of RAG.

[0089] In this invention, an application system capable of maximizing the use of RAG by building and searching for personalized information was implemented.

[0090] We implemented a QA method to provide accurate information on typhoon disasters by integrating domain segmentation and RAG.

[0091] T5 LLM was trained using typhoon disaster information from open-source databases such as Baidu Encyclopedia and Wikipedia.

[0092] The RAG module was used to improve answers to user prompts by searching for semantically similar phrases in an external knowledge base.

[0093] We evaluated the Typhoon Agent (Typhoon-T5) using a similarity matching method and laid the foundation for integrating LLM and disaster information.

[0094] Therefore, using RAG together with LLM can effectively improve search performance for specific topics.

[0095] We reviewed search-augmented text generation and other notable approaches for various text generation tasks, including conversation-response generation and machine translation.

[0096] It summarized the various components of search-augmented text generation, including search metrics, search sources, and integration paradigms, and provided useful information for developing application-specific topics in the Retrieval-Augmented Generation (RAG) process.

[0097] Therefore, research is actively underway on the effects of compensating for the shortcomings of LLM, such as hallucinations, and improving search efficiency.

[0098] The present invention has demonstrated that by effectively implementing RAG, it is possible to build a personalized database that provides differentiated search and QA services and improves search accuracy.

[0099] An interactive question-and-answer system with search augmentation generation for a personalized database according to the present invention will be described in detail.

[0100] FIG. 2 is a structural diagram of an interactive question-answer system with search augmentation generation for a personalized database according to the present invention.

[0101] As illustrated in FIG. 2, an interactive question-and-answer system with search augmentation generation for a personalized database according to the present invention includes an input unit (100) and a RAG pipeline (200).

[0102] The user can access an interactive question-and-answer system equipped with search augmentation generation for a personalized database according to the present invention through the input unit (100) and select and save specific sentences, related tags, and links from a document.

[0103] This is implemented through a system called TaggingBox, which allows users to map parts of text from documents of interest to tagged keywords and save them to a personal archive.

[0104] This refers to a customized knowledge base classified according to the specific purposes of an individual or team. It is part of a large-scale knowledge management engine where numerous documents required by an individual or team are organized and stored, and keywords used to tag specific sentences serve as notes.

[0105] The goal of the present invention is to use a RAG pipeline (200) to design a personalized database construction and search technology to develop information retrieval technology without the disadvantages of existing LLM.

[0106] The above RAG pipeline (200) refers to a series of processing systems in which a large language model (LLM) does not generate an answer alone, but rather utilizes an external database (especially a vector DB) to retrieve and inject suitable information, and generates an answer based on this information.

[0107] The above RAG pipeline (200) uses three main components—an SQL database (210), a vector database (220), and a QA generator (230)—to effectively process user questions and generate relevant answers.

[0108] The above SQL database (210) stores documents and information personalized according to personal identity.

[0109] The user information table of the above SQL database (210) manages information of users registered through the "Tagging Box" system.

[0110] When a question is entered into the user interface through the input section (100) of the QA system, the SQL database (210) extracts information matching the user ID from the user information table (211) and divides it into chunks for further processing.

[0111] The above vector database (220) reconstructs information by converting data chunks extracted from the above SQL database (210) into embedding vectors.

[0112] The above vector database (220) can be used to process unstructured data by storing data as vectors.

[0113] The above vector database (220) uses similarity search between vector data to process user queries, which is more flexible than simple string matching and can recognize documents with similar meanings, thereby greatly improving search accuracy.

[0114] Searching for similarity between the above vector data has the advantage of being able to return results more flexibly than using exact matching for the query.

[0115] In other words, data can be transformed from a high-dimensional space to a low-dimensional vector through vector embedding of information.

[0116] Data dimensions are reduced, but important information and data patterns are preserved.

[0117] Through this, the computer can effectively analyze data and identify similarities or patterns between vectors.

[0118] Finally, the above QA generator (230) utilizes an LLM such as GPT-3.5-Turbo to generate accurate and useful answers to questions.

[0119] This process is performed based on the context selected from the above vector database (220), and the final answer is returned to the user.

[0120] The above RAG pipeline (200) further includes MongoDB (240), which stores the chat history of the QA and generates answers that the user can use later.

[0121] The LLM of the above QA generator (230) does not store the state, so it does not remember previous messages of the conversation.

[0122] Therefore, as it is necessary to maintain a record of previous messages in the conversation and provide context for the LLM, the above MongoDB (240) stores the chat history.

[0123] Previous context information can be stored in a permanent database and used to restore context in new conversations, enabling the implementation of scenarios where users' questions and answers can be summarized and tracked in the history.

[0124] An easy way to implement the above RAG is to use LangChain, a powerful framework that builds the environment by integrating external tools.

[0125] The above LangChain is a framework designed to simplify application creation using LLM (Large Language Model).

[0126] Extracting data from the SQL database (210): In this step, the LangChain extracts relevant data from the user information table (211) through a query.

[0127] The above data includes document-related information, such as documents tagged by the user.

[0128] Chunking and Embedding: The extracted data is divided into chunks using the integrated framework of the LangChain above and transmitted to the vector database (searcher: 220) where embeddings are generated.

[0129] These embeddings transform document content into high-dimensional vectors to improve information retrieval and matching.

[0130] Answer generation: When a user enters a prompt into the system through the input section (100), the system searches for relevant context in the stored vector database (200) and uses it as input to generate the optimal answer.

[0131] The above LangChain manages the flow used in this process.

[0132] The above LangChain is a software development kit that simplifies the integration of LLM and its applications, and its importance is growing as the use of LLM increases.

[0133] The above LangChain can split, combine, and filter documents. Data is collected from the existing SQL database (210) via an API and returned in JSON (JavaScript Object Notation) format, and consists of key-value pairs as shown in FIG. 3.

[0134] Figure 3 is a diagram illustrating an example of user data extracted in JSON format when, for example, the UserID is 2190.

[0135] The unique identifier number corresponds to a specific SQL database (210) table and generates user information in the form of a title and tag name.

[0136] The key information used to build a personalized database is the number of items containing tag information, such as keywords set by individuals to classify documents.

[0137] This is called "TB (Tagging Box) search" and is provided as an internet web service.

[0138] In practice, the tagging box is implemented as a hyperlink that references personalized information.

[0139] Figure 4, illustrating a document embedding workflow, shows that a Context, which stores context created or referenced by an individual and extracted from the SQL database (210), contains most of the document content.

[0140] Based on this information, set the maximum length of the document to 1000 characters and split the document for processing.

[0141] A JSON-based TextSplitter takes extracted data as direct input, selectively extracts and combines necessary data, and returns it in JSON format.

[0142] The above LangChain divides the partitioned document into chunks of a specific size and allows each chunk to be processed independently.

[0143] After splitting, the document is embedded through the text-embedding-ada-002 model of the OpenAI API and stored in the vector database (220).

[0144] This data helps place the most relevant information at the beginning of the QA prompt after a search.

[0145] In an interactive question-answer system with search augmentation generation for a personalized database according to the present invention, user prompts are very important as they directly affect the ability to effectively respond to user questions.

[0146] Figure 5 is a diagram illustrating the process of processing user prompts and generating answers using LangChain (212) with MongoDB (240).

[0147] Based on the searched questions, the context for each question was established and organized into a "prompt" template.

[0148] The above prompt serves as an interface between the user question and the LLM and is used as the basis for generating answers.

[0149] An interactive question-answer system with search augmentation generation for a personalized database according to the present invention retrieves a list of documents extracted through the LangChain (212), configures them into a prompt format, and transmits them to a Large Language Model (LLM) of the QA generator (230).

[0150] The ability of the aforementioned Large Language Models (LLMs) to generate contextually appropriate answers depends heavily on the quality and structure of the provided prompts.

[0151] The above prompt plays an important role in verifying not only that the answer is accurate but also that it is relevant to the question.

[0152] FIG. 6 shows an example of a prompt template developed in the present invention. When using prompts, the following principles must be strictly observed.

[0153] Context analysis: The above QA generator (230) must generate an answer by taking into account the entire context provided in the prompt.

[0154] This ensures the accuracy of the answer, minimizes the transmission of incorrect information, and allows the answer to be aligned with the intent of the question.

[0155] Information restriction: The above QA generator (230) must not include information in the answer that is not specified in the context.

[0156] This allows answers to be generated based solely on the input data, and since the system cannot access tables unrelated to the question, data leakage can be prevented.

[0157] Source attribution: The above QA generator (230) specifies the source of information in all answers so that the user can understand the source or refer back to the information in the tagging box.

[0158] Uncertainty awareness: If the above QA generator (230) cannot find an answer, it informs the user that they need to search for tags or content in a designated box to obtain a more accurate result.

[0159] A prompt is generated as shown in Fig. 6, and a prompt generated for use in Large Language Models (LLM) is shown in Fig. 7. Fig. 7 is an example of a custom prompt generated for QA tasks.

[0160] History management: The QA system developed in the present invention is implemented to manage chat records using MongoDB (240), a NoSQL-based database that stores key-value data in JSON format.

[0161] Because there is no fixed schema, various types of data can be processed quickly and flexibly.

[0162] This system receives a question and retrieves the last five conversation histories from the MongoDB (240) and inputs them into a Large Language Model (LLM) along with a QA prompt.

[0163] Afterwards, data is extracted using a predefined SQL query.

[0164] The above Large Language Models (LLM) analyze conversation history, generate context-appropriate responses, deliver them to the user, and store them in the above MongoDB (240).

[0165] Through the flexible data processing capabilities of the above MongoDB (240), the interactive question-answer system with search augmentation generation for the personalized database according to the present invention can store user interaction history and generate customized responses based thereon.

[0166] Personalized RAG-based response: An interactive question-answer system with search augmented generation for a personalized database according to the present invention is a Retrieval-Augmented Generation (RAG) based question-answer (QA) system that generates an accurate answer for a specific situation by utilizing tag keywords extracted from a stored document and the SQL database (210).

[0167] When the above RAG pipeline (200) is applied to the QA system, the user can receive answers based on previously established information.

[0168] On the other hand, typical ChatGPT services generally provide answers based on learned information.

[0169] This demonstrates that the developed interactive QA system can reflect personalized information through RAG.

[0170] Evaluation (RAGA Framework): The interactive question-answer system with search augmentation generation for a personalized database according to the present invention further includes an evaluation unit (250), said evaluation unit (250) evaluated the performance of the interactive question-answer system with search augmentation generation for a personalized database according to the present invention using a RAGA framework, which is a framework focused on evaluating the search and generation functions of the RAG system.

[0171] The evaluation of each component of the above RAG pipeline (200) can be divided into two parts: answer generation and document search.

[0172] The generation process shown in Fig. 8 consists of two indicators.

[0173] Reliability evaluates the relevance between the retrieved documents and the generated answers, and Answer Relevance evaluates the relevance between the generated answers and the questions.

[0174] During the search process, documents retrieved for a question are evaluated based on context precision and recall.

[0175] The mathematical formulas for the evaluation indicators are summarized in [Table 1].

[0176] [Table 1] below shows the formulas used to evaluate answer generation and document retrieval in the RAGAs framework.

[0177]

[0178] In [Table 1] above, Eg i is the embedding of the generated question i, and N is the total number of generated questions. Eo is the embedding value for the original question, and k is the total number of chunks.

[0179] The above reliability indicator evaluates the consistency of whether the generated answer provides relevant information based on the given context.

[0180] The generated answer is calculated by comparing it with the context, and a higher probability indicates higher reliability of the answer.

[0181] To calculate the above confidence, the set of assertions of the generated answers is first identified, and then each assertion is cross-validated with the given context to determine whether the assertion can be inferred from the context.

[0182] The above answer relevance indicates the relevance of the generated answer to the question originally presented.

[0183] It evaluates the extent to which the answer meets the requirements of the question, and a high score indicates a complete and clear answer to the given question without redundant or unnecessary information.

[0184] To calculate this, Large Language Models (LLMs) are used to generate multiple questions suitable for the generated answers, and relevance is evaluated by measuring the mean cosine similarity between the generated questions and the original questions.

[0185] The above contextual precision represents the proportion of retrieved documents containing content related to the question.

[0186] The aforementioned contextual precision is used to evaluate the accuracy with which a search system curates relevant documents and provides them to the user.

[0187] The aforementioned context recall comprehensively evaluates whether the retrieved documents contain the information necessary to construct the answer to a given question.

[0188] The aforementioned context recall evaluates the performance of the search system during document search by assessing whether a document contains sufficient background information and the correct answer to a question.

[0189] The data used in the experiment was evaluated using the common sense dataset, a personalized database of "Tag Box" provided by AIHub.

[0190] This dataset consists of 100 "Question", "Actual Truth", and "Context" items, and is composed of three columns with these titles.

[0191] The "Question" column contains questions generated for the experiment, and the "True Truth" column contains the actual fact-based correct answers for each question.

[0192] The "Context" column consists of text data saved by the user and serves as the basis for questions and answers.

[0193] Specific examples are provided in Table A1 of Appendix B to aid in understanding the role of each column.

[0194] Based on this, the search performance of the RAGAs framework was evaluated.

[0195] The experiment setup used an NVIDIA RTX 3070 GPU and an Anaconda environment, and the experiment used the Ollama (0.2.5) library.

[0196] The versions of the vector database (220) used in the present invention, Chroma DB, and RAGA and LangChain that support the RAG evaluation process are shown in detail in [Table 2].

[0197] ComponentsVersionCPUAMD Ryzen 9 5900X 12-Core Processor (Advanced Micro Devices, Inc., Santa Clara, CA, USA)RAM32 GBGPUNVIDIA GeForce RTX 3070Anaconda Python3.9.19Ollama0.2.5RAGA0.1.9Chroma DB0.4.23LangChain0.2.8

[0198] To implement the RAG of an interactive question-answer system with search augmented generation for a personalized database according to the present invention, GPT-3.5-Turbo was used as the LLM and text-embedding-ada-002 as the embedding model to verify the entire process from user query to answer generation.

[0199] The scalability of the RAG system across various LLMs was verified by reconstructing the RAG process using various combinations of 5 LLMs and 2 embedding models.

[0200] In particular, it was validated using GPT-3.5-Turbo, Gemma-2-9b, Llama-3-8B, Mistral-7B, Qwen2-7B LLM, OpenAI's "text-embedding-ada-002", and the domestic Korean "snunlp / KR-SBERT-V40K-klueNLI-augSTS" embedding models.

[0201] In this invention, the performance of the QA system was evaluated using the two public data sources mentioned above.

[0202] The results of the system's response contextual relevance, accuracy, and reliability evaluation were analyzed using the RAGAs framework.

[0203] First, the evaluation results are presented using a common sense dataset.

[0204] The above general knowledge dataset consists of question-answer pairs for WIKI text, where the question is related to the content of the WIKI text and the answer is the corresponding answer pair for the WIKI text.

[0205] The information from the above general knowledge dataset was processed in the user's "tag box" and stored in a personalized database. This allowed us to evaluate how well the QA system generates accurate and relevant answers to user input questions.

[0206] Figure 9 shows the accuracy of the results generated for each LLM and embedding module combination. Ada-002 + GPT-3.5-Turbo showed the highest accuracy of 0.51.

[0207] Figure 9 is a graph showing the results of a reliability evaluation for a common-sense data set.

[0208] This figure is significantly higher than other model combinations, suggesting reliability.

[0209] Combinations using Llama-3-8B, Ada-002 + Llama-3-8B, and KR-SBERT + Llama-3-8B also showed high accuracy of 0.46 and 0.45, respectively, suggesting that Llama-3-8B provides higher accuracy than other models.

[0210] Figure 10 is a graph showing the result of answer relevance for a common sense data set. Figure 10 shows the answer relevance.

[0211] Similar to the accuracy results, the Ada-002 + GPT-3.5-Turbo combination showed the highest score of 0.86, demonstrating superiority over other model combinations in terms of relevance.

[0212] On the other hand, the combinations of Ada-002 + Llama-3-8B and KR-SBERT + Qwen2-7B showed low relevance scores of 0.39 and 0.42, respectively, suggesting that the response relevance of these combinations needs improvement.

[0213] Most model combinations obtained a relevance score of about 0.5, but showed relatively low performance.

[0214] Figure 11 shows the context-recall scores of each model combination. The combinations of KR_SBERT + Gemma-2-9b, KR_SBERT + Llama-3-8B, and KR_SBERT + Mistral-7B show the highest recall score of 0.82, indicating that they are effective in reconstructing information in a given context.

[0215] On the other hand, the Ada-002 + GPT-3.5-Turbo combination showed a relatively low recall score of 0.67, which suggests that the information-recall ability of this combination is weak.

[0216] Figure 12 shows the context-precision scores of all models.

[0217] The combination of Ada-002 + GPT-3.5-Turbo and Ada-002 + Qwen2-7B shows the highest precision of 0.79, indicating that these combinations can provide very accurate information for a given context.

[0218] On the other hand, the KR-SBERT + Mistral-7B combination shows a relatively low precision of 0.63, which suggests that this combination needs improvement in terms of contextual precision.

[0219] In addition, the response times of all combinations were analyzed.

[0220] Figure 13 compares the processing time per data item for each model combination.

[0221] Among the KR-SBERT combinations, KR-SBERT + GPT-3.5-Turbo showed the fastest processing time at 0.92 seconds. KR-SBERT demonstrated faster processing speed because it performs inference locally.

[0222] On the other hand, KR-SBERT + Gemma-2-9B showed the slowest processing time at 5.57 seconds, which may be due to the large size of Gemma-2-9B. Among the Ada-002 combinations, Ada-002 + GPT-3.5-Turbo showed a relatively fast processing time of 1.39 seconds, suggesting that fast processing is possible even in API communication environments.

[0223] On the other hand, Ada-002 + Gemma-2-9B shows the slowest processing time at 6.16 seconds.

[0224] These differences are due to the fact that KR-SBERT is used locally, whereas Ada-002 requires API communication.

[0225]

[0226] Overall, Ada-002+GPT-3.5-Turbo demonstrated the best performance, outperforming other methods across various performance metrics including accuracy, answer relevance, and contextual precision, and showed the second-lowest processing time.

[0227] In addition, the KR-SBERT+GPT-3.5-Turbo combination showed the lowest processing time, suggesting that it is a useful alternative.

[0228] Next, additional experiments were conducted and performance was evaluated using a news article dataset along with the existing public dataset.

[0229] This news article dataset, consisting of 450,000 news articles, is used as a training set for the development of machine reading comprehension systems.

[0230] It includes articles in 9 categories and provides the correct answer to the question and the context in which the answer was derived, enabling evaluation through RAGA.

[0231] As can be seen in Figure 14, the Ada-002 + GPT-3.5-Turbo combination recorded the highest confidence score of 0.47.

[0232] This indicates that this combination generates a response more faithful to the original information compared to other models.

[0233] In contrast, the KR-SBERT + GPT-3.5-Turbo combination shows the lowest confidence score of 0.28.

[0234] In Figure 15, the Ada-002 + GPT-3.5-Turbo combination shows the highest answer relevance with a score of 0.82.

[0235] This indicates a very high degree of agreement with the original question, suggesting that this combination can provide a clear and accurate answer to user inquiries.

[0236] Conversely, the KR-SBERT + Mistral 7B combination recorded the lowest score on this indicator.

[0237] Figure 16 evaluated the situational reproducibility performance, and the Ada-002 + GPT-3.5-Turbo combination showed low performance with a score of 0.51.

[0238] On the other hand, the KR-SBERT + GPT-3.5-Turbo combination recorded the highest score of 0.68.

[0239] In Figure 17, situational accuracy was evaluated, and the Ada-002 + GPT-3.5-Turbo combination received a score of 0.73.

[0240] These combinations show a sufficiently reliable level of precision in document search, but the KR-SBERT + Mistral2-7B combination achieved the highest precision with a score of 0.87.

[0241] Figure 18 visualizes the data processing time for each embedding combination.

[0242] The KR-SBERT + GPT-3.5-Turbo combination recorded the fastest processing time of 0.95 seconds, which is likely the result of running it directly on a local machine.

[0243] In contrast, the Ada-002 + GPT-3.5-Turbo combination took slightly longer at 1.56 seconds, but maintained a very efficient processing speed while using API communication.

[0244] Therefore, Ada-002 + GPT-3.5-Turbo is a balanced choice in terms of performance and speed.

[0245] The Ada-002 + GPT-3.5-Turbo combination showed balanced results in terms of performance and processing speed, but additional experiments were conducted to examine the performance of the new GPT-4 model.

[0246] GPT-4 has been improved in various aspects, including enhanced context processing, increased accuracy, and reduced hallucination rates.

[0247] Experiments were conducted using GPT-4 to evaluate the impact of these improvements on actual performance.

[0248] This experiment aimed to compare the performance of GPT-4 and GPT-3.5-Turbo and evaluate the performance of the two models in a real-world operating environment.

[0249] According to the results shown in Figure 19, when processing 100 data points, GPT-4 incurs a cost approximately 8137.5 higher than GPT-3.5-Turbo.

[0250] In addition, GPT-4 is about 111.5% slower than GPT-3.5-Turbo in terms of processing time.

[0251] Here, processing time refers to the time it takes to process one piece of data.

[0252] These results show that cost and time efficiency can be a major issue in real-world operating environments where cost and time efficiency are critical.

[0253] In the RAGAs evaluation, GPT-3.5-Turbo showed better performance in terms of confidence, answer relevance, and contextual accuracy by 31.3%, 21.1%, and 1.9%, respectively.

[0254] These results suggest that GPT-3.5-Turbo is more suitable for meeting research requirements where response consistency and accuracy are important.

[0255] Conversely, GPT-4 showed 12.4% better performance in context recall, but this advantage did not lead to a significant improvement in overall system performance.

[0256] Therefore, while GPT-4 offers improved performance in terms of context processing and accuracy, cost efficiency and processing speed are critical factors in real-world applications.

[0257] Therefore, Ada-002 + GPT-3.5-Turbo may be more suitable for situations where cost and processing time efficiency are important.

[0258] Although the technical concept of the present invention has been described above together with the accompanying drawings, this is merely an illustrative explanation of preferred embodiments of the present invention and is not intended to limit the invention. Furthermore, it is evident that anyone with ordinary knowledge in the technical field to which the present invention pertains can make various modifications and imitations within the scope of the technical concept of the present invention without departing from its scope.

Claims

1. An input section that allows a user to input a question or save a portion of text from a document of interest to a personal archive by mapping it to tagged keywords through a tagging box system, and An interactive question-answer system with search augmentation generation for a personalized database, characterized by including a RAG pipeline that retrieves and injects information using the personal archive and external database for a question input through the input unit, and generates an answer based on the information.

2. In Paragraph 1, The above RAG pipeline is SQL database that stores documents and information personalized based on personal identity; A vector database that reconstructs information by converting data chunks extracted from the above SQL database into embedding vectors; and An interactive question-answer system with search augmentation generation for a personalized database, characterized by including a QA generator that generates answers to questions by utilizing Large Language Models (LLMs) based on a context selected from the vector database.

3. In Paragraph 2, The above SQL database is An interactive question-answer system with search augmentation generation for a personalized database, characterized by extracting information matching a user ID from a user information table and splitting it into chunks for further processing when a question is entered into a user interface through the above-mentioned input unit.

4. In Paragraph 2, The above vector database is An interactive question-answer system with search augmentation generation for a personalized database characterized by using similarity search between vector data to process user queries.

5. In Paragraph 1, The above RAG pipeline is An interactive question-answer system with search augmentation generation for a personalized database, characterized by further including NoSQL that stores QA chat history and can generate answers for the user to use later.

6. In Paragraph 2, In Article 1, The above RAG pipeline is An interactive question-and-answer system for a personalized database featuring search augmentation generation, further comprising LangChain, a framework for building an environment by integrating external tools, wherein LangChain is characterized by extracting relevant data from a user information table through a query when extracting data from the SQL database.

7. In Paragraph 6, The above LangChain is An interactive question-and-answer system with search augmentation generation for a personalized database, characterized by splitting extracted data into chunks and transmitting them to the vector database (searcher) where embeddings are generated.

8. In Paragraph 6, The above LangChain is An interactive question-and-answer system with search augmentation generation for a personalized database, characterized by collecting user data from the above-mentioned SQL database and returning it in JSON format consisting of key-value pairs.

9. In Paragraph 2, The above QA generator An interactive question-answer system with search augmentation generation for a personalized database, characterized by generating answers by considering the full context provided in a prompt that serves as an interface between a user question and a large-scale language model.

10. In Paragraph 2, The above QA generator An interactive question-answer system with search augmentation generation for a personalized database, characterized by excluding information not specified in context from answers and specifying the source of information in all answers.

11. In Paragraph 2, The above QA generator An interactive question-answer system with search augmentation for a personalized database, characterized by notifying the user that if an answer cannot be found, they should search for tags or content in a designated tagging box to obtain an accurate result.

12. In Paragraph 2, The above RAG pipeline is An interactive question-answer system with search augmentation generation for a personalized database, characterized by further including an evaluation unit that evaluates the performance of the system based on reliability, answer relevance, contextual precision, and contextual recall.

Images