Method and device for providing configuration document correction for improving chatbot performance

Abstract

Disclosed is a configuration document correction providing method for improving conversation performance of a retrieval-augmented generation (RAG) chatbot that performs document-based question answering, by combining a technology for generating a linguistic response suitable for a question prompt by using a large language model (LLM) and a technology for refining the question prompt by extracting a document-based question context through RAG.

Description

Method and device for providing configuration document corrections to improve chatbot performance

[0001] The present disclosure relates to a method and apparatus for providing correction of a configuration document for improving chatbot performance, and more particularly, to a method and apparatus for correcting a configuration document of a RAG module constituting a part of a Retrieval Augmented Generation (RAG) chatbot server and a Large Language Model (LLM) for answering questions, in order to improve the answer quality of the chatbot server. In addition, the present disclosure relates to a method and apparatus for correcting a configuration document loaded into a RAG module as a part of a chatbot server, in order to improve the answer quality of the RAG chatbot server, which is a combination of a Large Language Model (LLM) for generating an answer to a query prompt and a RAG module for refining a query prompt by extracting a question context based on a loaded document.

[0002] Large Language Models (LLMs) are typically based on deep learning models. These models, trained on large amounts of language data, can generate linguistic output or responses that most naturally correspond to arbitrary linguistic input or prompts. By incorporating these LLM capabilities into a chatbot server and configuring a chatbot interface that allows questions and answers to be exchanged between the chatbot server and the user, an LLM chatbot, such as ChatGPT, or a generative AI chatbot, can be implemented.

[0003] However, when asked questions about data it has not learned, the LLM chatbot may make incorrect inferences or exhibit hallucinations, such as responding to non-existent information as if it were true.

[0004] To reduce this illusion and improve LLM chatbots so they can provide specific answers that align with the intent of the user's query, rather than simply providing general, broad responses, it is generally known that extracting contextual information from each question and constructing a query prompt that incorporates this information is more effective than simply inputting the user's question as a query prompt. This technique, which extracts the context of a user's query and then combines it with the question to generate a more sophisticated query prompt, is called Retrieval Augmented Generation (RAG). This is because the act of querying can be considered a type of information retrieval.

[0005] Therefore, to implement an improved LLM chatbot through such query prompt generation, the chatbot system can be improved and configured to receive user questions between the chatbot interface and the LLM, additionally extract the context of the question, generate a query prompt, input it into the LLM, and return the resulting answer as a chatbot response to the user. In this case, the RAG module is installed as part of the chatbot server, and the chatbot server with this structure is called a RAG chatbot server.

[0006] A key difference between LLM and RAG chatbots is that while LLM chatbots require no separate preparation for users to initiate a question-and-answer conversation, RAG chatbots require a corpus prepared in advance in the RAG module of the chatbot server to extract question context before the conversation begins. In other words, to extract appropriate question context for each question, a corpus, or linguistic data sufficiently containing relevant knowledge about the user's question, is required. Documents are a form of corpus, and it is desirable to select documents that sufficiently contain knowledge about the user's question topic for extracting question context, which helps LLM generate high-quality answers.

[0007] These documents are input into the RAG module of the chatbot server, which receives user questions, constructs query prompts, and returns LLM responses to the chatbot interface. They are then loaded in vectorized form, thereby forming the RAG module's vector database. These documents are then used to extract context related to the user question. These documents are referred to as "configuration documents" throughout this specification.

[0008] A common method for extracting question context from questions is to mathematically calculate vector similarity. The constituent documents are vectorized into fine-grained chunks using the same encoding method used by LLM to convert linguistic data into mathematical vectors during training, and then loaded into the RAG module, forming the RAG module's vector database. Each user question is then vectorized using the same encoding method, and the most similar document fragments are extracted by matching them against the RAG module's vector database using the vector similarity calculation method. This is considered the question context for the question. This question context is then fused with the user question by the RAG module and generated as a query prompt.

[0009] Therefore, the question-answer conversation procedure of the RAG chatbot is as follows: the user loads a configuration document into the RAG module of the chatbot server as a preparation step before starting the conversation, and whenever a user question is entered, the chatbot server extracts the context related to the question from the loaded configuration document, merges it with the user question to generate a query prompt, inputs it into the LLM, and provides the generated LLM response as the chatbot's answer to the user.

[0010] RAG technology improves the response performance of LLM by structuring query prompts in this way, enabling it to provide more reliable and appropriate responses to user questions.

[0011] More specifically, RAG is a technology that refines the query prompt input into LLM to obtain an appropriate answer to the user's question. To this end, prior to the question-answer conversation of the RAG chatbot, a corpus or document containing sufficient knowledge about the question the user will ask is vectorized and loaded into the RAG module as a constituent document. During the conversation, the user's question is vectorized in the same way, and the context appropriate to the question among the contents of the constituent document is extracted through matching based on vector similarity, and this is input into LLM as a query prompt together with the user's question, thereby inducing a more appropriate LLM response.

[0012] The problem to be solved is to provide a method and device for providing configuration document correction for chatbot performance improvement that can improve the response quality of a chatbot server by identifying contents requiring correction through monitoring of user feedback on previous responses of the chatbot and chatbot-response pairs in the configuration document of a chatbot server's RAG module, which was difficult to improve with the existing RAG chatbot algorithm, extracting them in the form of correction documents, and using these correction documents to correct the configuration documents to create improved configuration documents, which are then loaded into the RAG module of the chatbot server.

[0013] A method for providing a configuration document correction for improving chatbot performance according to one embodiment is a method for providing a configuration document correction for improving the conversational performance of a RAG chatbot that performs document-based question answering by combining a technology for generating a natural language response to a query prompt using a Large Language Model (LLM) and a technology for refining a prompt by extracting a document-based question context through Retrieval Augmented Generation (RAG), the method comprising the steps of: preparing a RAG module of a RAG chatbot server to be operable in response to a user question; selecting a document containing knowledge that answers a question or can be referenced through a chatbot interface and inputting it as a configuration document into the RAG module so that a question context can be extracted through vector matching of each user question; and loading the input configuration document into a vector database of the RAG module; inputting a question into the RAG chatbot server through the chatbot, receiving an answer generated through the LLM, and repeating the question input and answer reception if there is no problem with the answer; A step for a user to input user feedback regarding the appropriateness of a received chatbot response through a chatbot interface when the user determines that correction or supplementation is required; a step for receiving a chatbot response corresponding to the user feedback from the chatbot; a step for determining whether the user feedback and the chatbot response include an intention to correct a previous chatbot response; a step for extracting new correction content from the user feedback and the chatbot response when it is determined that the user feedback and the chatbot response include an intention to correct; a step for creating a new correction document when the extracted correction content is the first correction content derived from the conversation, and a step for adding the extracted correction content to an existing correction document when it is not;And it may include a step of creating an improved version configuration document that corrects an existing configuration document by reflecting the contents of the above correction document, and inputting it into the RAG module and loading it.;

[0014] According to one embodiment, a device for providing a configuration document correction for improving chatbot performance is provided, which improves the conversational performance of a RAG chatbot that performs document-based question answering by combining a technology for outputting an appropriate linguistic response to a query prompt using a Large Language Model (LLM) and a technology for refining a query prompt by extracting a document-based question context through Retrieval Augmented Generation (RAG), and which includes a correction extraction module, a document / conversation database search / display module, a document / conversation database management module, a purpose-specific list display module, and a purpose-specific list item activation module, wherein the correction extraction module determines whether a pair of user feedback and a corresponding chatbot response includes an intention and content for correcting a previous answer of the chatbot, and if it is determined that there is an intention for correction, extracts new correction content from the feedback and response, adds the correction content to an existing correction document related to the conversation if there is one, and if not, creates a new correction document, and displays the purpose-specific list. The module provides a list of configuration documents, correction documents, and question-and-answer conversations related thereto, grouped by purpose, according to user access rights or sharing scope, and the purpose-specific list item activation module activates and displays configuration documents, correction documents, and conversation items corresponding to a specific purpose tag selected by the user, and deactivates and hides other items, thereby selectively providing only relevant information, and the document / conversation database search / display module searches configuration documents, correction documents, and conversations according to a purpose tag, search word, or search phrase entered by the user, and enables display according to various sorting criteria such as name order, frequency of use order, and most recent order, and the document / conversation database management module manages configuration documents,It may be characterized by managing the creation or update of correction documents and conversation records by storing them in a database.

[0015] According to embodiments, situations in which a chatbot's response requires supplementation or correction, i.e., adjustment, can be detected through analysis of a pair of user feedback and chatbot responses or a separate request to LLM, and the adjustment content can be extracted through this, and the RAG module configuration document of the chatbot server can be adjusted based on the adjustment content, thereby improving the response performance of the chatbot.

[0016] FIG. 1 is a block diagram illustrating an improved Retrieval Augmented Generation (RAG) chatbot system according to one embodiment.

[0017] FIG. 2 is a block diagram illustrating a correction providing device of an improved RAG chatbot system according to one embodiment.

[0018] FIG. 3 is a flowchart illustrating a correction provision method of an improved RAG chatbot system according to one embodiment.

[0019] FIG. 4 is a drawing showing an example of a correction provision method and a correction device of an improved RAG chatbot system according to one embodiment, showing confirmation of a correction situation and extraction of correction content during a question-and-answer conversation.

[0020] FIG. 5 is a drawing showing an example of a correction provision method and a correction device of an improved RAG chatbot system according to one embodiment, showing correction status confirmation and correction content extraction initiated by operating a correction button during a question-and-answer conversation.

[0021] FIG. 6 is a diagram showing a conceptual diagram of three types of configuration documents and corresponding dialogue and correction documents according to one embodiment, a diagram showing a correspondence relationship between configuration documents and correction documents mediated by dialogue, and an example of a correspondence table showing the same.

[0022] FIG. 7 is a drawing showing an example of a correction providing method and a correction device of an improved RAG chatbot system according to one embodiment, which provides a screen for searching a list of conversations, configuration documents, and correction documents grouped around configuration documents for the purpose of helping a user confirm and edit correction contents during a question-and-answer conversation.

[0023] FIG. 8 is a drawing showing an example of a correction providing method and a correction device of an improved RAG chatbot system according to one embodiment, in which conversations, configuration documents, and correction documents are grouped by type and list items are activated by selection purpose to provide a searchable screen for the purpose of helping a user confirm and edit correction contents during a question-and-answer conversation.

[0024] FIG. 9 is a drawing showing an example of an implementation of a correction providing method and a correction device of an improved RAG chatbot system according to one embodiment, wherein when a configuration document is selected through a list of configuration documents by purpose, an extended view user interface screen is provided that allows related conversations and correction documents to be checked together.

[0025] FIG. 10 is a drawing showing an example of an implementation of a correction providing method and a correction device of an improved RAG chatbot system according to one embodiment, which provides an extended view user interface screen that allows related configuration documents and correction documents to be checked together when selected through a purpose-specific dialogue list.

[0026] FIG. 11 is a block diagram illustrating a computing device according to one embodiment.

[0027] Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted for clarity of description, and similar parts are designated with similar reference numerals throughout the specification.

[0028] Throughout the specification and claims, whenever a part is said to "include" a certain component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise stated.

[0029] Additionally, terms such as “part,” “unit,” and “module” described in the specification may mean a unit capable of processing at least one function or operation described in the specification, which may be implemented by hardware, software, or a combination of hardware and software.

[0030] FIG. 1 is a block diagram illustrating an improved augmented search generation (RAG) chatbot system according to one embodiment.

[0031] Referring to FIG. 1, an improved RAG chatbot system (1) according to one embodiment may include a user (10), a chatbot interface (20), a correction providing device (30), a RAG module (42) constituting a RAG chatbot server (40), and a large language model (LLM) (44).

[0032] FIG. 2 is a block diagram illustrating a correction providing device of an improved RAG chatbot system according to one embodiment.

[0033] As with using the RAG chatbot, before starting a question-and-answer conversation, as a preparatory step, the user (10) inputs a document containing knowledge about the content to be asked into the RAG module (42) of the RAG chatbot server (40) as a configuration document so that the question context can be extracted for each question. This document can be input through various means, such as uploading the document file through the chatbot interface (20) or providing the URL of the document file.

[0034] Alternatively, the user (10) may input one of the configuration documents stored in the existing chatbot system as a configuration document of the RAG module (42) by selecting one from the list of documents / conversations provided by the document / conversation database search / display module (306) of the correction provision device (30) through the chatbot interface (20).

[0035] The user (10) can conduct a question-and-answer format conversation with the chatbot in the same way as when using the existing RAG chatbot before improvement, and unlike the RAG chatbot system, this conversation is always monitored by the correction provision device (30) to determine whether a situation requires correction or can be utilized to extract correction content.

[0036] The chatbot interface (20) guides the user to input a document containing related knowledge into the RAG module (42) of the RAG chatbot server (40) before the user starts a conversation on a certain topic for the first time, or to select an appropriate document from a list of loadable documents stored in an existing database through a purpose-specific list provided by the document / conversation database search / display module (306) of the correction provision device (30), or to designate a specific document accessible through a shortened address or URL link, so that the document is loaded as a configuration document of the RAG module, and the RAG chatbot server (40) is ready to start a question-and-answer conversation.

[0037] In addition, the chatbot interface (20) receives a question from a user (10) while a question-and-answer conversation is in progress, transmits the question to the correction providing device (30), and the correction providing device analyzes whether the question is user feedback for the purpose of correction, and simultaneously transmits the question to the RAG module (42) of the RAG chatbot server (40). The RAG module (42) vectorizes the question using the encoding method used by the LLM of the chatbot server, and calculates vector similarity (e.g., cosine similarity) with the vectors (i.e., vector databases) of the constituent document fragments loaded into the RAG module using the same encoding method, thereby deriving the constituent document fragment most similar to the question as a question context, and then configures the context and the question together as a query prompt to be input into the LLM (44).

[0038] The chatbot response generated by the LLM (44) of the RAG chatbot server (40) for the query prompt generated in this way is provided to the user (10) through the chatbot interface (20) via the correction provision device (30) in reverse order.

[0039] Additionally, the chatbot interface (20) can receive user feedback for correction of a previous answer from the user (10) and input it into the RAG chatbot server (40) via the correction providing device (30). A corresponding response from the chatbot server (40) can also be provided to the user (10) via the chatbot interface (20) via the correction providing device (30).

[0040] The chatbot correction provision device (30) is located on the path of questions and answers exchanged between the chatbot interface (20) and the RAG chatbot server (40), and recognizes and analyzes user feedback on previous chatbot responses and their corresponding chatbot responses, extracts content requiring correction as a correction document, and reflects this in the RAG module configuration document of the chatbot server, thereby improving the quality of the chatbot's responses. Such improvement can reduce inaccurate or hallucinatory content that may occur in the response of the LLM (44), increase the reliability and accuracy of the response, and further provide a secondary effect of facilitating the transfer of knowledge by documenting the tacit knowledge or know-how of the user (10) during the question-and-answer dialogue between the user (10) and the chatbot.

[0041] Descriptions of the RAG chatbot server (40) and its components, the RAG module (42) and LLM (44) shown in Fig. 1, have already been mentioned in the “Technology forming the background of the invention” and are therefore omitted in this paragraph.

[0042] FIG. 2 is a block diagram illustrating a correction providing device of an improved RAG chatbot system according to one embodiment.

[0043] Referring to FIG. 2, a correction providing device (30) of an improved RAG chatbot system (1) according to one embodiment is a device added between a RAG chatbot server (40) and a chatbot interface that performs a response to a question input based on a RAG module (42) and an LLM (44) to improve the answer quality of the chatbot, and may include a correction extraction module (301), a purpose-specific list display module (302), a purpose-specific list item activation module (304), a document / conversation database search / exhibition module (306), and a document / conversation database management module (308).

[0044] Before starting a conversation through the chatbot interface (20), the user (10) prepares to start a question-and-answer conversation by inputting a configuration document in various ways, such as directly uploading a configuration document so that the question context extraction function can operate in the RAG module (42) of the RAG chatbot server (40), selecting a desired document from the document / conversation list provided by the document / conversation database search / display module (306) of the correction provision device (30), or specifying and notifying an accessible document through a link such as a shortened address or URL.

[0045] Here, user feedback for correction purposes may include, for example, correcting previous chatbot responses, training the chatbot, or providing new information that was not previously known through the chatbot interface (20).

[0046] The intelligent elements that appear in such user feedback, such as correcting, supplementing, educating, or conveying new information to the chatbot's previous responses, are extracted as correction content together with the chatbot response that recognizes or confirms that a correction is needed, and are used to correct the RAG module configuration document of the RAG chatbot server (40). Accordingly, if there is no existing correction document related to the conversation, it is created as the first correction document derived from the conversation, and if it already exists, it is added to the existing correction document, and the correspondence between the conversation and the related correction document and the configuration document can be recorded in the configuration document-conversation-correction document correspondence table (62 in FIG. 6) by the document / conversation database management module (308) of the correction providing device (30).

[0047] The newly generated correction document can be used to generate an improved configuration document by correcting the configuration document of the existing RAG module (42) for generating query prompts. Updating the configuration of the chatbot server's RAG module with this improved configuration document can prevent the same type of inaccurate or incomplete responses, or hallucinations, that occurred in conversations on the same or similar topics through more sophisticated prompt generation, thereby improving the chatbot's response performance.

[0048] In the RAG chatbot, a document-based question-and-answer chatbot, an improved configuration document created by incorporating corrections into a specific configuration document can be exposed or shared with other users with sharing rights according to a specified purpose tag or explanatory note through a purpose-specific list provided by the document / conversation database search / display module (306) of the correction provider (30). Accordingly, the prompt improvement effect of the improved configuration document, which improves the chatbot's response performance, can be shared and utilized by other users with the same or similar purposes.

[0049] A chatbot response refers to a response output by the LLM (44) of the RAG chatbot server in the form of acceptance or rejection when user feedback occurs as above.

[0050] A chatbot response that responds positively to user feedback is a response that accepts a request for supplementation or correction of the feedback content, and serves as a criterion for the correction provision device (30) to determine that a correction situation is required, and at the same time, is used as a criterion for extracting correction content and creating or updating a correction document.

[0051] The correction extraction module (301) can determine whether user feedback and chatbot responses have the intention of correcting the chatbot's responses.

[0052] If it is determined that the user feedback and chatbot response contain the intention and content to correct the previous response of the chatbot, the correction content is extracted from the feedback and response, and the RAG module (42) of the chatbot server (40) can create an initial correction document to correct the existing configuration document used for question context extraction, or can add it to the existing correction document.

[0053] In some embodiments, the correction extraction module (301) may add the correction content extracted through the conversation to an existing related correction document through the document / conversation database management module (308), or create a new correction document if the correction document does not exist.

[0054] Additionally, in some embodiments, the correction extraction module (301) may determine whether a first phrase corresponding to the user feedback and a second phrase corresponding to the chatbot response include a predetermined type of phrase to determine whether the user feedback and the corresponding chatbot response include a correction intent.

[0055] For example, as in FIG. 4, if the first phrase includes a type of phrase indicating a need for correction or supplementation, such as “You are wrong,” “You are incorrect,” or “Your answer is insufficient,” and the second phrase includes a type of phrase accommodating a need for correction or supplementation, such as “You are correct,” or “I apologize,” the correction extraction module (301) can extract new correction content from user feedback generated between the first and second phrases. That is, it is possible to remove ceremonial phrases from the corpus accompanying the first phrase and the corpus accompanying the second phrase and derive the correction content.

[0056] In some embodiments, a correction button (e.g., including the text “CORRECT”) is displayed in a clickable state on the dialogue screen of the chatbot interface (20), and the correction extraction module (301) can determine whether the correction button is clicked and whether a third phrase included in the response corpus is a predefined phrase that means that correction or supplementation is necessary to determine whether the chatbot response is corrected.

[0057] For example, as shown in FIG. 5, if the correction button is pressed and the chatbot response corpus is determined to include a third phrase such as “You are correct” or “I apologize…”, the correction extraction module (301) of the correction providing device (30) can extract new correction content after removing the ceremonial phrase from the corpus entered as user feedback after the correction button is clicked and the corpus accompanying the third phrase in the chatbot response.

[0058] The document / conversation database management module (308) of the correction provision device (30) can transmit a document selected from the document list provided by the document / conversation database search / display module (306) through the chatbot interface (20) to the RAG module (42) of the RAG chatbot server (40) before the user starts a question-and-answer conversation, thereby loading the document as a configuration document of the RAG module (42).

[0059] In addition, the document / conversation database management module (308) can create an improved configuration document by fusing the correction document selected by the user and the existing configuration document of the RAG module (42) linked thereto, and input this into the RAG module (42) of the RAG chatbot server (40) to load it into a new vector database.

[0060] The configuration document for the new RAG module, created by fusing with the correction document, is vectorized when input to the RAG module and acts as a new vector database used to extract question context for user questions.

[0061] In this way, by reflecting the correction in the RAG module configuration document used in the chatbot server (40), the response quality of the chatbot can be improved through more sophisticated query prompt generation, and the hallucination phenomenon that may occur in the LLM (44) can be reduced.

[0062] In some embodiments, where computing resource constraints are not significant, the correction extraction module (301) can determine whether user feedback and chatbot responses have correction intent during a real-time chat with the chatbot.

[0063] If the existing chatbot response requires correction, the document / conversation database management module (308) creates an improved configuration document by fusing the newly created or updated correction document from the conversation with the configuration document of the existing RAG module (42), and inputs this into the RAG module (42) instead of the existing configuration document, thereby allowing the correction content to be reflected in the chatbot response in real time.

[0064] In some embodiments, when computing resources are limited, the correction document generation module (301) may review the chat history after the real-time chat with the chatbot ends to determine whether the pair of user feedback and chatbot response has a correction intent.

[0065] If there is a situation requiring correction, the correction contents in that situation can be integrated into a single correction document, and accordingly, the document / conversation database management module (308) can create an improved configuration document by fusing the correction document with the configuration document of the existing RAG module (42) for a specific conversation, store it in the document database, and then input it into the RAG module (42) to replace the existing configuration document, thereby allowing the correction contents for the chatbot response to be reflected in non-real time.

[0066] The correction providing device (30) includes a correction extraction module (301), through which a correction document newly created or updated from a pair of user feedback and chatbot responses identified as a correction situation during a specific question-and-answer conversation and cross-reference information between the associated conversation can be maintained, and the correction document can be added to or updated in a document database to be managed.

[0067] In addition, the correction providing device (30) includes a document / conversation database management module (308), and when newly extracted content from a correction document is associated with and merged with a specific existing configuration document according to the selection of the chatbot configuration document owner, the correction providing device can perform improvement version management of the configuration document by adding automatically increased improvement version number information to the title of the configuration document or updating a separate improvement version attribute value.

[0068] In addition, the correction providing device (30) includes a document / conversation database search / display module (306), through which each user is provided with a display function to view and use configuration documents and question-and-answer conversation records loaded and used in the RAG chatbot within the scope of sharing or access rights, and the display list can be sorted and displayed in various ways such as by purpose, title name, most recent use, frequency of use, etc., according to the user's selection.

[0069] The document / conversation database search / display module (306) also supports an extended view function, for example, as shown in FIG. 9, it can provide a screen that details the associated conversations and correction documents for each document in the list of constituent documents, grouped by various sorting criteria, or as shown in FIG. 10, it can provide a screen that details the constituent documents and derived correction documents that provided the question context for each question-answer conversation in the list of constituent documents.

[0070] In addition, the document / conversation database search / display module (306) provides an interface that allows searching for documents or conversation records by inputting various conditions such as purpose, title name, author ID, and keyword, and the search results can be provided in the form of a list sorted according to various sorting criteria.

[0071] The correction providing device (30) may further include a purpose-specific list display module (302) that displays a list of ready-to-use configuration documents and correction documents stored in a document database, and a list of question-and-answer dialogue records related to each document, along with their respective purposes, and a purpose-specific item activation module (304) that activates items corresponding to the purposes entered by the user in the lists of purpose-specific documents, correction documents, and question-and-answer dialogue records, and deactivates items that are different from the purposes entered by the user. In addition, the correction providing device (30) includes a purpose-specific list display module (302) and a purpose-specific list item activation module (304), through which keyword searches for purpose tags or purpose description notes, or purpose-specific semantic searches that find similar purpose tags or similar purpose description notes using a separate LLM, are possible.

[0072] Through the purpose-specific list display module (302), as shown in FIG. 6, the entire composition document, correction document list, and conversation history list can be provided to the user according to the relationship between the conversation derived from the composition document and the correction document derived from the conversation, and the sort order of the detailed items within each item is based on the purpose order selected by the user or a criterion selected from among various predefined sorting options. Through the purpose-specific list item activation module (304), as shown in FIG. 7 or FIG. 8, only the documents or conversation items searched for by keywords are activated and displayed based on a part of the selected purpose tag or purpose description memo, or further, it is possible to provide filtered results by activating only the documents or conversations that contain tags or descriptions similar to the purpose tags or descriptions linked to the existing documents or conversations using a separate LLM.

[0073] The functions provided by the purpose-specific list display module (302) and the purpose-specific list item activation module (304) of the correction provision device (30) that improves the performance of the RAG chatbot help users who ask questions about things they are curious about or intend to explore knowledge through the chatbot with their own purpose or intention of 'purpose' to find purpose tags, keywords related to the purpose, or even clearer linguistic expressions of ambiguous purposes through the use of a separate LLM, thereby making it easy to find conversations or documents with purposes similar to or related to the originally intended purpose, and correction documents that are concentrated with elements with a high probability of insight or error, thereby making it easy to discover, maintain, or expand the focus of knowledge exploration.

[0074] In addition, users can discover conversations similar to their own conversation intentions through a list of conversations by purpose and activation by purpose. If there is a correction document that supplements the limitations, insufficiencies, or errors in the documents referenced during these conversations, the correction document can provide useful stimulation for the latest insights or intellectual challenges not found in the existing documents. Furthermore, the ease of navigating these 'purpose'-centered documents, conversations, and correction documents, and the ability to share conversations and correction documents that reflect up-to-date information regardless of the aging of the existing learning of the LLM used over time, help to maintain motivation for exploration and learning through this improved RAG chatbot that allows corrections to the component documents during the conversation and leaves a record of such corrections.

[0075] The correction provision device (30) is an additional function that allows the user to extract intangible knowledge implicitly held by the user by converting it into a correction document through a conversation with a chatbot. Through this, even if the content of the official document for handover prepared by the predecessor during the work handover process is insufficient for the successor to understand and master the work, the document can be input into the RAG module, and the successor's questions about the handover work and answers through the RAG chatbot's explanations can be answered. In addition, the document for handover itself can also be corrected. Therefore, work knowledge information that was not clearly conveyed to the successor, was insufficient, or was insufficient during the work handover process between the predecessor and the successor can be quickly discovered and easily supplemented, thereby reducing unnecessary waste of time and enabling effective knowledge transfer.

[0076] In addition to the functions described so far, the correction providing device (30) may also be specialized and implement additional functions to support two-way or integrated knowledge information exploration for learning, education and work problem solving of the user (10) in a correction situation during a question-and-answer conversation process using the RAG chatbot.

[0077] As the difficulty and scale of question solving and task performance increase, the need for integrated and interactive utilization of knowledge information in related fields is increasing.

[0078] For example, in a correction situation where the chatbot acknowledges feedback that the chatbot's previous answer is not wrong but is not sufficient for the user's question, LLM can be used separately to suggest to the user higher-order concepts that can be meta-searched or lower-order related concepts that can be analytically searched.

[0079] In other words, metasearch can help users find the information necessary to integrate newly acquired knowledge or information into a larger existing knowledge system, or naturally lead to analytical searches that seek out more detailed components and systematic structure information to identify and supplement any gaps or inadequacies in their current understanding. This is because as the scale and difficulty of question-and-answer-based interactive knowledge information exploration increases, the need for such interactive or integrated searches becomes more frequent. This search support function can help users overcome individual linguistic fluency limitations as they integrate newly acquired knowledge information into a larger or higher-level substantive body of knowledge, or systematize and divide it into lower-level equivalent components for easier understanding.

[0080] The correction provision device (30) provides a function for assigning purpose tags and writing explanatory notes for documents or conversations stored through the document / conversation database management module (308). In this process, a separate LLM question-and-answer function is utilized to supplement the user's linguistic expression limitations, and by suggesting upper or lower concept words related to purpose tags appropriate for the content of the document or conversation, it can be utilized for purpose tagging and content segmentation of stored documents / conversations, thereby facilitating the systematic management and utilization of documents and conversations.

[0081] Additionally, when a user searches for documents and conversations by purpose using the document / conversation database search / exhibition module (306), this module can be implemented in a way that supports more systematic exploration of related documents and conversations by suggesting appropriate purpose vocabulary and purpose descriptions that match the user's intention through a separate LLM.

[0082] The vector database provided in the RAG module (42) of the RAG chatbot server (40) can apply correction contents of correction documents created or updated during the question-and-answer dialogue process. This process is accomplished by adding or merging correction contents into documents constituting the corresponding vector database.

[0083] The chatbot provider (30) references and compares interrelated conversations, configuration documents, and correction documents using an activated purpose-specific list, and compiles them. Based on the results of this compilation, the device can generate and provide error notes or summary notes to support the user's education, learning, or task performance.

[0084] For example, the RAG module (42) of the RAG chatbot server (40) utilizes configuration documents loaded into a vector database to provide question context. Users can browse configuration documents grouped by purpose in the document database, effectively comparing and selecting configuration documents suited to specific question resolution or task performance purposes, and easily identify documents whose content is compiled by purpose.

[0085] The correction provision device (30) uses a configuration document-conversation-correction document correspondence table (Table 62 of FIG. 6) managed by the document / conversation database management module (308) to link conversations related to each configuration document with one-to-many (1:N) cross-references. Furthermore, each conversation is linked one-to-one (1:1) to a correction document derived from the conversation. Through this linking, a user can effectively compare and review conversations in a list of configuration documents, correction documents, and conversations classified by purpose (as exemplified in FIGS. 7, 8, 9, and 10).

[0086] Additionally, users can group and compare conversations, correction documents, and configuration documents with similar purpose tags through a purpose-specific document / conversation list. An example implementation of this feature is illustrated in Figures 8, 9, and 10.

[0087] The user (10) can easily identify the most recently useful configuration documents by selecting sorting by most frequently loaded configuration documents based on time periods such as the last 3 days or a week from the list of configuration documents by purpose provided through the chatbot interface (20).

[0088] According to this embodiment, an improved version of the configuration document can be created by adding or integrating the content of a correction document derived from the conversation to the configuration document that provided context for each question in the conversation. This improved version of the configuration document is provided to users seeking similar conversations, thereby reducing erroneous responses, such as insufficient answers or hallucinations, and improving the quality of the chatbot's responses.

[0089] FIG. 3 is a flowchart illustrating a method for providing configuration document correction for improving RAG chatbot performance according to one embodiment.

[0090] Referring to FIG. 3, a method for providing a configuration document correction for improving the performance of a RAG chatbot according to an embodiment comprises: a question-and-answer dialogue preparation step (S500) in which a user (10) inputs a document to be used for extracting a question context for each question into a RAG module of a RAG chatbot server as a configuration document through a chatbot interface (20), and this document is vectorized and loaded into a vector database of the RAG module; a step (S501) in which a user inputs a question and receives a chatbot answer generated by an LLM in response to the question; a step (S502) in which a user inputs feedback on a previous answer of the chatbot through the chatbot interface; a step (S503) in which the user receives a chatbot response corresponding to the feedback from the chatbot interface; a step (S504) in which a pair of user feedback and chatbot response is analyzed to determine whether a correction situation exists; a step (S505) in which, if a correction situation is determined, new correction content is extracted and, if an existing correction document exists, is added to the existing correction document, or if no document exists, a new correction document is created; It may include a step (S506) of integrating the extracted correction content into the existing dialogue configuration document to create an improved version configuration document; and a step (S507) of inputting the improved version configuration document into the RAG module to vectorize it and loading it into a vector database to improve the chatbot performance.

[0091] Additional details of the chatbot provision method can be found in the descriptions referring to FIGS. 4, 5, and 6 of this specification. Therefore, redundant descriptions are omitted herein. FIG. 4 is a diagram illustrating an implementation example of a configuration document correction method and device for improving chatbot performance.

[0092] According to FIG. 4, the correction extraction module (301) monitors pairs of repeated user questions and chatbot responses in the chat window (50) to determine whether each question-answer pair corresponds to a pair of user feedback with the intention of correcting the chatbot's previous response and a chatbot response affirming the feedback with the intention of correcting. To this end, the module examines whether the feedback and response each contain a first phrase and a second phrase of a type that signify a predetermined correction situation.

[0093] For example, if the first phrase included in the dialogue (501) corresponding to the user feedback is “You are wrong.”, which is a type of phrase indicating that the previous chatbot answer is wrong and should be corrected, and the second phrase in the dialogue (503) corresponding to the chatbot response is “You are correct.”, which is a type of phrase indicating that the correction is approved, then this is determined to be a correction situation, and the correction content is extracted as follows. That is, if the dialogue (502), which is the remaining part of the user feedback, contains specific correction content for correcting the previous answer of the chatbot, the correction content can be extracted from that part. Conversely, if the dialogue (502) is merely an objection or a question, the dialogue (504), which is the remaining part of the chatbot response, must be included so that the specific correction content can be confirmed and extracted.

[0094] If it is determined that the second phrase corresponding to the chatbot response in the conversation (503) includes the phrase “You are correct.” which acknowledges that the chatbot’s previous answer needs to be corrected, then from the conversations (501) and (503), which are portions of the user feedback and the chatbot response that use phrases indicating a correction situation, it can be determined that the user feedback and chatbot response pair includes correction content, and new correction content is extracted from the conversation (502), which is the remaining portion of the user feedback, or if the conversation (502) is merely an objection or question, the content correcting the chatbot’s previous answer is extracted by combining it with the conversation (504), which is the remaining portion of the chatbot response.

[0095] The correction content extracted from the conversation is created as a new correction document by the document / conversation database management module (308) if it is confirmed that there is no existing correction document related to the conversation. If there is an existing correction document, the correction content is added to the document. At this time, the purpose tag of the correction document is set to be the same as the existing configuration document related to the conversation. This correction document can be merged with the existing configuration document to create an improved version of the configuration document, or it can contribute to improving the contents of other correction documents or configuration documents with similar purposes.

[0096] The configuration document prepared in the RAG module is input in the conversation preparation stage when starting a new question-and-answer conversation or resuming an interrupted conversation, but in one embodiment of the method for providing configuration document correction for improving chatbot performance, the configuration document prepared in the existing RAG module can be replaced with an improved version configuration document generated by fusing the correction document and the configuration document related to the conversation during a real-time conversation or in the conversation preparation stage.

[0097] During a real-time conversation, the configuration document is replaced by the correction providing device (30), if there is a correction document related to the ongoing conversation, fusing the existing usage configuration document with the correction document to create an improved version configuration document, and replacing the configuration document of the RAG module with this improved version configuration document while the conversation is paused. In other words, the question input in the ongoing conversation is temporarily paused, the improved version configuration document is loaded into the RAG module, and then the question input is resumed to continue the conversation.

[0098] When replacing a previously conducted conversation with an improved version of a configuration document in the step of resuming a previously conducted conversation or preparing a new conversation using the same configuration document, the document / conversation database management module (308) checks whether there is a correction document associated with the selected existing configuration document, then merges the selected correction document with the configuration document to create an improved version of the configuration document as a separate document, loads the configuration document of the RAG module by replacing it with this improved version, and prepares to start the conversation.

[0099] Also, for example, if the first phrase corresponding to the user feedback in the conversation (505) is determined to include the phrase “You are incorrect.” which expresses that the previous chatbot answer is incorrect and needs to be corrected, and if corrections are made to correct the previous answer of the chatbot in the remaining part of the user feedback, which is the conversation (506), or if the conversation (506) is merely a question or inquiry, the specific corrections can be extracted by including the remaining part of the chatbot response, which is the conversation (508).

[0100] In the dialogue (507) which is the chatbot's response to the user feedback indicating such intention for correction, the second phrase "I apologize." is a type of phrase that acknowledges that the LLM (44) of the chatbot server was in a situation requiring correction, so the pair of the dialogue (505) which is part of the user feedback and the dialogue (507) which is part of the chatbot response serves as a criterion for determining that it is a correction situation.

[0101] In this correction situation, the correction content can be extracted from the dialogue (506), which is the part of the user feedback excluding the ceremonial phrases, or if the dialogue (506) is a simple objection, the correction content can be extracted from the dialogue (508), which is the remaining part of the chatbot response.

[0102] The new correction content extracted in this way is added to the content of an existing correction document related to the conversation by the document / conversation database management module (308) if there is one, and if there is no existing correction document, it is created as a new correction document and registered in the document / conversation database.

[0103] The corresponding correction document is stored with the same purpose tag as the existing configuration document loaded into the RAG module (42) related to the above conversation, and is subsequently used to correct the configuration document associated with the conversation, thereby generating an improved version of the configuration document. Furthermore, it may also contribute to improving other correction documents or configuration documents with similar purposes.

[0104] The improved version configuration document created in this way can be selected by a user who wants to start a new conversation for a similar purpose and input into the RAG module (42) of the chatbot server, thereby providing more improved chatbot response performance.

[0105] In addition, the correction document and the new improved version configuration document can be shared so that other users within the sharing scope of the conversation and configuration document can use them through the document / conversation database search / display module (306) of the correction providing device (30), and can also be utilized for corrections to improve the contents of other similar correction documents or configuration documents.

[0106] When sharing these purpose-specific correction documents, improved version configuration documents, and conversations, the reliable source or provider information of the document is registered along with the shared document or conversation in the form of a blockchain, GitHub, or other tamper-proof, credible internet ledger or distribution record, thereby ensuring the safety of use and promoting the safe use of shared documents and conversations.

[0107] Also, for example, if the first phrase corresponding to the user feedback in the conversation (509) is determined to include the phrase “Your answer is insufficient.”, which expresses that the previous chatbot answer is insufficient or needs to be improved, and the remaining part of the user feedback, the conversation (510), contains corrections to correct the previous answer of the chatbot and can be extracted, or if the conversation (510) is merely an objection or question, the corrections may be extracted only if the remaining part of the chatbot response, the conversation (512).

[0108] If the second phrase corresponding to the chatbot's response in the conversation (511) is determined to include phrases such as "You are correct." or "Thanks for your additional information." that acknowledge the situation requiring correction, the conversation (509) and the conversation (511), which are pairs of user feedback and chatbot response, are considered to indicate a correction situation, and the correction content can be extracted from the conversation (510), which is the remainder of the user feedback, or if the conversation (510) is simply raising an issue, the correction content can be extracted by including the conversation (512).

[0109] The correction contents extracted by the correction extraction module (301) of the correction providing device (30) are processed by the document / dialogue database management module (308) of the correction providing device (30). If the correction contents are extracted for the first time in the corresponding conversation, a new correction document related to the corresponding conversation is created and stored in the document / dialogue database, and the corresponding association information is recorded in the existing configuration document-dialogue-correction document correspondence database table (Table 62 of FIG. 6). If the correction contents are not extracted for the first time in the corresponding conversation, the correction contents are added to the existing correction document. At this time, the purpose tag of the correction document is stored with the same purpose tag as the configuration document used in the corresponding conversation. The correction document thus created or updated can be used to create a corrected improved version configuration document by merging with the configuration document used in the corresponding conversation, or to improve a correction document or configuration document of similar purpose used in a separate conversation.

[0110] A configuration document loaded into an existing RAG module can be used to search for a correction document related to the configuration document through the document / conversation database search / display device of the correction provision device (30) during a real-time situation in which the improved RAG chatbot user is provided with a question context and selects a document containing knowledge related to the question to be asked and conducts a question-and-answer conversation, or in the preparation stage for resuming an existing conversation or starting a completely new conversation after the question-and-answer conversation has ended (see FIGS. 7 and 8), create a new improved configuration document by adding the contents of the correction document, and newly input it into the RAG module of the corresponding conversation. Alternatively, the improved configuration document can be separately created as a new version of the configuration document through the document / conversation database management module (308), registered in the document database, and then newly input the new version of the configuration document into the RAG module of the corresponding conversation.

[0111] If computing resources are not significantly constrained, corrections and subsequent corrections to the configuration document loaded into the RAG module of the chatbot server can be performed immediately during a question-and-answer conversation with the RAG chatbot, which is a content exploration of the configuration document, as corrections are extracted in real time between each question-and-answer pair.

[0112] For example, the extraction of correction content, creation or update of a correction document, and application of correction to update an existing loaded vector database of a chatbot server starting from a pair of a conversation (501) which is user feedback to correct a previous answer of a chatbot and a conversation (503) which is a chatbot response are performed in real time, and subsequently, the extraction of correction content, creation of a correction document, and application of new correction for a pair of a conversation (505) which is user feedback to correct another previous answer and a conversation (507) which is a chatbot response can also be continuously performed in real time.

[0113] On the other hand, in cases where computing resources are limited, corrections and subsequent corrections can be performed in batches, non-real time, after the end of the chat with the RAG chatbot, which is a content exploration of the loaded configuration document.

[0114] For example, a series of procedures, starting from a user feedback conversation (501) and a chatbot response conversation (503) pair to correct a previous answer of a chatbot, extracting corrections, initially creating or updating a correction document, applying new corrections to an existing configuration document, and then extracting additional corrections, updating a correction document, and applying additional corrections to a configuration document with the corrections reflected, starting from a feedback conversation (505) and a chatbot response conversation (507) pair to correct another answer, can be performed in batches, non-real time, at a stage of reviewing the chat history after the real-time chat is terminated. An improved version of a configuration document improved through such batch corrections is stored in a document / conversation database by a document / conversation database management module (308), and can be loaded and used in a conversation preparation stage of the RAG module (42) at the user's choice when resuming the current conversation or starting a new conversation for a similar purpose.

[0115] In this way, when the configuration document loaded in vectorized form into the RAG module (42) is replaced with a corrected and improved version of the configuration document, the query context extracted through vector matching of the user's question based on the configuration document is improved, thereby improving the quality of the query prompt input into the LLM. Accordingly, incorrect answers or hallucinations in the response of the same LLM (44) are reduced, and insufficient answer content is improved, thereby improving the overall response quality of the chatbot.

[0116] FIG. 5 is a drawing showing an implementation example of a method and device for providing configuration document correction for improving chatbot performance according to one embodiment.

[0117] Referring to FIG. 5, the correction document generation module (301) can determine whether a correction button (e.g., a 'CORRECT' button) (521) displayed on a screen (52) on which a conversation is in progress is pressed to determine whether user feedback and chatbot response in the chat window (52) have the intention of correcting the chatbot's previous answer, and can determine whether a third phrase corresponding to the chatbot's response includes a type of phrase indicating a predetermined correction situation.

[0118] For example, if it is determined that the 'CORRECT' button in the correction button position (521) has been pressed as a user feedback for the chatbot's previous answer, and if it is determined that the third phrase corresponding to the chatbot's response in the dialogue (523) includes the phrase "You are correct." which confirms that a correction situation has occurred, the correction content can be extracted from the user feedback that occurs between the pressing of the 'CORRECT' button in the button position (521) and the dialogue (523), which is a pair of user feedback and chatbot response that can confirm that correction is necessary. Alternatively, if the dialogue (522), which is the user feedback, simply raises an objection to the previous chatbot answer, the correction content can be extracted by including the dialogue (524), which is the remaining response of the chatbot.

[0119] If there is no existing correction document associated with this conversation, the correction content extracted by the correction extraction module (301) is created as a new correction document in the document / conversation database as the first correction content extraction and registered through the document / conversation database management module (308). At this time, the purpose tag is set to be the same as the existing configuration document associated with the conversation.

[0120] The corresponding correction document is associated with the corresponding component document through the conversation from which it was derived, and this relationship is registered in the component document-conversation-correction document correspondence table in the document / conversation database. If the correction is not the first extracted, the content is added to the existing correction document for the corresponding conversation. The saved correction document can later be merged with the associated existing component document to create an improved version of the component document, or it can be used to improve the content of other correction documents or component documents with similar purposes.

[0121] Instead of using the configuration document loaded into the existing RAG module as is in a real-time question-and-answer conversation situation or a situation where the same conversation is to be resumed after the conversation ends or a new question-and-answer conversation of a similar purpose is to be started, the related correction document is checked by referring to the purpose-specific configuration document-conversation-correction document list (63 in FIG. 7, 64 in FIG. 8) provided through the purpose-specific list display module (302) or the purpose-specific list item activation module (304) of the correction provision device (30), and an improved version configuration document reflecting some or all of the correction contents is created and replaced and loaded into the RAG module (42), thereby improving the conversation performance of the chatbot.

[0122] Alternatively, you can improve the chatbot's conversational performance by registering the new improved version configuration document as a separate configuration document in the document / conversation database and using that document instead of the existing configuration document when loading the RAG module to proceed, resume, or start a similar conversation.

[0123] In addition, for example, if it is determined that the 'CORRECT' button in the correction button position (525) has been pressed, and information including correction content that complements or supplements the shortcomings of the chatbot's previous answer is generated in the conversation (526), ​​and if it is determined that the third phrase included in the chatbot's response in the conversation (527) includes "I apologize." that confirms that correction is necessary, the correction content can be extracted from the user feedback that occurs between the 'CORRECT' button in the correction button position (525) and the conversation (527). Alternatively, if the conversation (526), ​​which is the user feedback, is simply an objection to the chatbot's previous answer, the correction content can be extracted by including the conversation (528), which is the remaining response of the chatbot.

[0124] If the extracted correction content is the first correction content for a document loaded into an existing chatbot server, the correction extraction module (301) of the correction provision device (30) creates a new correction document in the document database and stores it by cross-reference with an existing configuration document associated with the conversation. If it is not the first correction content, it is added to an existing correction document, and at this time, the purpose tag of the correction document is set to be the same as the purpose tag of the configuration document loaded into the existing RAG module (42). Thereafter, the correction document can be used to create a newly corrected improved version configuration document by improving a related configuration document in the same conversation, or to improve another correction document or configuration document with a similar purpose tag.

[0125] When a user is having a conversation in real time and a correction document is derived from this conversation, or when the user wants to resume the conversation after the conversation ends or start a new conversation by inputting the same document as the configuration document used in this conversation into the chatbot server, the document / conversation database management module (308) of the correction providing device (30) checks whether there is a correction document derived from another conversation using the same configuration document through the configuration document-conversation-correction document correspondence table (Figure 62 of FIG. 6), and if there is, it merges the correction document into the configuration document to create a new improved version configuration document, and then inputs this into the RAG module of the chatbot server, thereby improving the chatbot performance. In addition, the document / conversation database management module (308) can create the improved version configuration document as a separate document, register it in the document database, and input this new configuration document into the RAG module (42), thereby improving the answer quality of the chatbot.

[0126] If computing resources are not too limited, the configuration document can be reloaded in real time from the RAG module of the chatbot server whenever corrections are extracted and correction documents are created or updated.

[0127] For example, a correction situation is confirmed through a pair of user feedback 1 and chatbot response 1, the correction content is extracted and documented as a correction document, the content of the correction document is merged with an existing configuration document to create an improved configuration document, and this improved configuration document is reloaded in real time into the RAG module of the chatbot server. Next, correction content is also extracted from a pair of user feedback 2 and chatbot response 2 and documented as a correction document, the correction content is additionally merged with the previously improved configuration document to create another improved configuration document, and this document is reloaded in real time into the RAG module, and this process can be continuously performed.

[0128] In contrast, in cases where computing resources are limited, the process of checking the correction status through the pair of user feedback 1 and chatbot response 1, extracting the correction contents and documenting them as a correction document, then merging the correction contents with the existing configuration document to create an improved configuration document, and reloading this document into the RAG module of the chatbot server, and the process of extracting the correction contents through the pair of user feedback 2 and chatbot response 2, documenting the correction, improving the existing configuration document, and reloading the document are not performed in real time, but are processed in batches at the stage of reviewing the chat history after the real-time chat is terminated. In this case, the configuration document can be performed in a non-real-time batch manner by reloading the final improved configuration document with all corrections reflected only once, without reloading the configuration document in the middle for each correction.

[0129] Accordingly, configuration document correction is applied to the RAG module of the chatbot server, which can contribute to improving the response quality of the chatbot server (40) in the future and reducing the hallucination phenomenon that may occur in the LLM (44).

[0130] FIG. 6 shows a configuration diagram (60) explaining the correspondence between three types of configuration documents and conversations and correction documents in a method and device for improving chatbot performance according to the present embodiment, an explanatory diagram (62) showing the relationship between multiple conversations using one configuration document and correction documents derived from some of them, and an implementation example (61) of a configuration document-conversation-correction document correspondence table.

[0131] A question-and-answer conversation can be started or resumed by inputting a single configuration document into the chatbot server, and a correction document can be derived during the conversation. Therefore, the document / conversation database management module (308) of the correction provision device (30) manages the correspondence between the configuration document and the conversation, or between the conversation and the correction document, by recording it in the configuration document-conversation-correction document correspondence table (Figure 62 of FIG. 6) when a conversation or correction document is created.

[0132] Referring to FIG. 6, the relationship between the three types of configuration documents used in this specification and the correction document (60) can be understood. Due to the nature of the RAG chatbot, in order for the chatbot to operate, a document for extracting a question context for each user question must be loaded into the RAG module of the chatbot server, and this document is referred to as a 'configuration document' in this application. The configuration document may be input into the RAG module immediately, or may be stored in advance for later input into the RAG module (42). These are collectively referred to as 'input configuration documents'. In addition, some or all of the input configuration documents may be loaded into the RAG module (42) where each conversation with the chatbot is conducted, or may have a loading history, and these are collectively referred to as 'configuration documents used in the conversation'.

[0133] The composition documents used in a conversation are a subset of the input composition documents, some of which can be used to extract corrections during the conversation process, thereby deriving a corresponding correction document for the conversation. These composition documents are referred to as "conversation documents from which correction documents are derived."

[0134] The conceptual diagram (61) of Fig. 6 illustrates how a single component document, from which two correction documents are derived, is related to other conversations and correction documents. For example, component document D601 has a purpose tag of 'T1' and is used as the same component document in multiple conversations (H401, H402, H405, ..., Hnnn), so that it can be utilized to extract the context of user questions in each conversation. Conversations associated with this component document have a one-to-many (1:n) correspondence.

[0135] Each conversation (H401, H402, H405, ..., Hnnn) has its own conversation purpose tag (U1, U2, U3, ..., Un) and is stored in the conversation database table. Some of these conversations produce correction documents, and the relationship is presented in the conceptual diagram (61) of Fig. 6. Even if the constituent document D601 is used in multiple conversations, none of the conversations may produce correction documents, or only some or one conversation may produce correction documents. In this case, the correction documents produced (C902, C901) correspond 1:1 to the corresponding conversations (H401, H405).

[0136] As described in the configuration document-conversation-correction document correspondence storage table (62) of FIG. 6, by recording the relationship between the configuration document and the conversation and correction document through the correspondence storage table (62 of FIG. 6) in the database, not only can the configuration document that is the target of correction be easily found from the correction document, but also the conversations and correction documents related to a specific configuration document can be efficiently searched.

[0137] Additionally, for a specific configuration document, the document / conversation database search / display module (306) can be used to group and view, cross-reference, and compare related correction documents based on the configuration document-conversation-correction document correspondence table (62), and optionally compile and edit the contents. This can support upgrading the configuration document, extracting and synthesizing related tacit knowledge, creating new derived knowledge documents, and systematically transferring knowledge.

[0138] In this way, even if the correction document and the configuration document of the RAG module (42) are stored separately, by ensuring interconnectivity, the chatbot server can immediately select and merge the latest performance-guaranteed correction configuration document each time a question-and-answer conversation is started or resumed, generate an improved version configuration document, and input this into the RAG module (42), thereby improving the performance of the chatbot in real time. The correction providing method and device for improving the response performance of a chatbot proposed in this patent is a technology for improving performance by correcting the configuration document input into the RAG module (42), and is a method of generating a correction document based on the identification and extraction of correction contents for individual chatbot responses during a question-and-answer conversation, and reflecting this correction contents into the configuration document.

[0139] A user (10) can receive a list of documents or conversations grouped based on purpose tags, sorted by various criteria such as most recently used, created, loaded, or accessed, through a chatbot interface (10).

[0140] Users can also browse selected documents or conversations by purpose, and browse a list of existing Q&A conversations tagged with similar purpose tags to determine if there are any similar questions. Furthermore, they can select the document containing the most relevant knowledge by browsing a list of existing RAG chatbot documentation tagged with similar purpose tags.

[0141] However, since multiple conversation records may exist for the same configuration document, and the purpose tag assigned to the configuration document and the purpose tag assigned to the RAG chatbot conversation utilizing it may be based on different intents, the purpose tags of the configuration document and the corresponding conversation record may not match.

[0142] FIG. 7 is a drawing showing an example of an implementation of a method and device for providing configuration document correction for improving chatbot performance according to one embodiment, in which sorting criteria such as configuration documents, correction documents, and conversation records are set in order of purpose tags through a user interface (20), thereby showing an example of a comprehensively sorted list of conversations and correction documents related to configuration documents.

[0143] Referring to FIG. 7, the configuration documents, correction documents, and conversations stored in the document and conversation database can be sorted not only based on the configuration documents by purpose, but also by the correction documents by purpose or the conversations by purpose and provided in the form of a list.

[0144] Specifically, in the implementation of the comprehensive list of purpose-specific composition documents, correction documents, and conversations exemplified in FIG. 7, the existing composition documents (D601, D606, D604, D603, D605, D602) are sorted in the order of the purpose tag (T1, T2, T3, T4, T5) of each document in the first column, and then the sorting criteria are sequentially applied in the order of document title, version number, or registration time (e.g., D601, D602, D606, ..., D602), so that the composition documents are sorted in a unique order. Since multiple conversations can be linked to each composition document, the conversations corresponding to each composition document form a 1:N relationship, and are sorted in a unique order by sequentially applying the sorting criteria such as the purpose tag of the conversation (U1 to U6), conversation title, version number, or registration time (H401 to H406).

[0145] Since the correction documents are derived from one of the conversations, they can be uniquely sorted by association with that conversation. Furthermore, the correction documents can be sorted by placing them in the first column, with their associated constituent documents and derived conversations placed in subsequent columns. For example, the correction documents can be sorted by purpose tag, document title, version number, or registration time, with the conversations from which the correction documents are derived placed in the next column, and the constituent documents associated with the conversation placed in the last column. Conversely, the conversations can be sorted by purpose tag, title, version number, or registration time, with the constituent documents associated with the conversation placed in the next column, and the correction documents derived from the conversation placed in the last column. A more specific implementation of this is illustrated in Figures (65) and (66) of Figure 8.

[0146] In the comprehensive list by purpose based on the composition document, the existing conversations (H401, H402, H403, ..., H406) each have a purpose tag (U1, U2, U3, U4, U5, U6), and the corresponding composition documents in this order are composition documents (D601, D601, D601, D604, D605, D602) each have a purpose tag (T1, T1, T1, T2, T4, T5). The list shows that some conversations have correction documents produced, and some conversations do not have correction documents. For example, the composition document D601 of the conversation (H401) has a correction document C902, the composition document D604 of the conversation (H403) has a correction document C908, the composition document D605 of the conversation (H404) has a correction document C906, and the composition document D601 of the conversation (H405) has a correction document C901.

[0147] Meanwhile, component document D606, which is registered as a component document but does not yet have any associated conversation records, shows that it does not have a correction document that can only be produced during the conversation. The same goes for component document D603. Finally, conversation H406 uses component document D602, which has component document purpose tag T5 together with conversation purpose tag U6, and the list confirms that correction document C904, which was produced from this conversation, is related to component document D602 via conversation H406.

[0148] As shown in the implementation examples of the exhibition list of Fig. 8 (64, 65, 66), in this way, the configuration documents, conversations, or correction documents can be displayed in an arranged manner so that they are placed in the front row and serve as the basis for comprehensive sorting.

[0149] When documents are the sorting criteria, as shown in the example implementation diagram (64) of the list in Fig. 8, the documents can be sorted in a unique order by sequentially applying the sorting criteria in the order of the document's purpose tag, document title, version number, or registration time in the first column, and conversations can be arranged in the next column, but the conversations can also be sorted in a unique order by applying the sorting criteria in the order of the conversation's purpose tag, conversation title, version number, or registration time. At this time, if there are correction documents derived from the conversation, those correction documents are also sorted according to the correspondence relationship.

[0150] As shown in the example of list implementation in Figure 8 (65), if the correction document is the basis for the list sorting in the first column, since the correction document and the conversation have a 0:1 or 1:1 correspondence, if a correction document exists, the conversation corresponding to the correction document and the constituent documents of the conversation are sequentially sorted. After that, the first column is left blank and the conversation and the constituent documents of the conversation are sorted in that order, and lastly, the constituent documents that have no related correction document or conversation and have been input to the chatbot server by the user and stored in the database can be listed in a unique order by applying the sorting criteria in the order of the document's purpose tag, document title, version number, or registration time.

[0151] As shown in the example of list implementation in Figure 8 (66), when a conversation is the sorting criterion, the first column can be sequentially sorted by the purpose tag of the conversation, the conversation title, the version number, or the registration time to sort the conversations in a unique order. If the item is set as a constituent document, the sorting criterion can be sequentially applied by the purpose tag of the constituent document, the document title, the version number, or the registration time to sort them in a unique order. In this case, the correction documents derived from the conversation are linked to the conversation and sorted together.

[0152] Since correction documents and conversations have a 0:1 or 1:1 correspondence, they can be sorted in the same order: correction document - conversation - component document. Furthermore, to maintain a unique sort order, correction documents that correct component documents are sorted by document purpose tag, title, and registration order, while conversations are sorted by conversation purpose tag, title, and registration order.

[0153] That is, the document / conversation database management module (308) always assigns a purpose tag to documents or conversations stored in the document / conversation database and manages the corresponding purpose information together with a memo.

[0154] When a correction document is first generated through the correction extraction module (301) of the correction provider, it is assigned the same purpose tag as the associated existing configuration document, since there is one. This correction document is linked to the configuration document in a many-to-one (N:1) relationship through a question-and-answer dialogue associated with the document.

[0155] Accordingly, as shown in the example of Figure 61 of Figure 6, the correspondence between conversations using a specific configuration document and correction documents derived therefrom can be managed by the document / conversation database management module (308) of the correction providing device (30) in the same manner as the configuration document-conversation-correction document correspondence storage table (66) of Figure 8.

[0156] When a new question-and-answer conversation begins or an existing one resumes, a specific correction document can be merged with the chatbot server configuration document used in the conversation, either in real-time or non-real-time, depending on computing resource constraints, to generate a revised, improved configuration document. By replacing this revised configuration document with the configuration document loaded into the RAG module, the chatbot's response quality can be improved in the current conversation or in other conversations with similar purposes.

[0157] Referring to FIG. 8, the purpose-specific list item activation module (304) activates and displays only items corresponding to the purpose tag of the configuration document or conversation selected by the user, and deactivates and hides unselected items, thereby allowing the user to select and view only the configuration documents, conversations, and correction documents corresponding to the desired purpose.

[0158] For example, as shown in Figure 64 of FIG. 8, if the purpose tags of the configuration documents searched by the user are T1, T4, and T5, the corresponding correction documents (C901, C902, C904, and C906), conversations (H401, H402, H404, H405, and H406), and existing configuration documents (T1: D601, D606; T4: D605; T5: D602) are activated and displayed on the screen. On the other hand, the correction document (C908) with the purpose tags T2 and T3, the related configuration documents (D603, D604), and the conversation H403 derived from the correction document C908 are deactivated and not displayed on the screen.

[0159] Accordingly, the user (10) can selectively check only related configuration documents, correction documents, and related conversations based on the specified purpose tag through the chatbot interface (20).

[0160] Similarly, as shown in Figure 66 of FIG. 8, if the purpose tags specified by the user when searching by conversation criteria are U1, U2, U4, U5, U6 excluding U3, the corresponding existing conversation records (H401, H402, H404, H405, H406) are activated, and the conversation (H403) with a different purpose tag is deactivated. At the same time, by utilizing the correspondence table between the composition document-conversation-correction document, the composition documents (D601, D602, D605, D606) and the derived correction documents (C901, C902, C904, C906) used in the conversations (H401, H402, H404, H405, H406) that meet the user search conditions are also displayed, and the composition document (D604) and correction document (C908) linked to the conversation (H403) that does not meet the search conditions are deactivated and not displayed. Of course, configuration documents (D603, D606) that are not connected to any conversation are disabled and not displayed on the screen.

[0161] In this regard, the chatbot's correction provision device (30) can generate and provide a summary note or an error note useful for similar purposes, such as business handover, education, and learning support, from the correction documents activated by the user through search. That is, the chatbot provision device (30) can review the contents of each correction document (C901, C902, C904, C906) activated from the results filtered by the purpose of the configuration document or the purpose of the conversation, select correction contents containing useful information not included in the existing configuration document, and provide some or all of them to the user in the form of a summary note or an error note.

[0162] FIG. 9 is a drawing showing an example of an implementation of an extended function of a configuration document list view provided by a document / conversation database search / exhibition module of a correction provision device according to one embodiment through a user interface (20).

[0163] Referring to FIG. 9, the document / conversation database search / display module (306) can display the constituent documents in a list format by providing various sorting criteria such as order of purpose, order of name, order of creation, order of most recently used, etc.

[0164] Documents used in conversations within a configuration document are indicated by a filled, right-pointing triangle to the left of the list item, while documents not used in conversations are indicated by an empty, right-pointing triangle. Clicking a filled triangle expands the sub-details pane, listing conversations that used that configuration document, along with their purpose tags and descriptions.

[0165] Figure 9 illustrates an example screen implementation that displays a detailed, expandable list of chatbot configuration documents. Among the conversations listed in the sub-details section of the configuration document list, conversation items that derive correction documents have a star inside the right-pointing triangle button to the left of the conversation item, visually distinguishing them as having correction documents. If a star is inside the triangle to the left of a configuration document item, it indicates that there is a conversation related to that configuration document that derives a correction document, and is utilized as a visual feature to distinguish it from configuration document items that do not have a star.

[0166] Clicking the filled, right-pointing triangle button to the right of a specific document entry in the list of document entries will change the button to a downward-pointing triangle, opening a sub-details pane that displays not only the relevant conversation but also any corrections derived from that conversation. A star inside the right-pointing triangle button before the conversation title and purpose tag in this sub-details pane indicates that a corrections document has been derived from that conversation, visually distinguishing it from conversation entries without corrections.

[0167] By default, the triangle button appears as a right-pointing triangle when not clicked, indicating that the configuration document's usage description note is not expanded. When clicked, the button changes to a downward-pointing triangle, opening another sub-details section that also displays the configuration document's usage description note. Clicking it again reverts it to its default right-pointing triangle.

[0168] FIG. 10 is a drawing showing an example of a user interface implementation of an extended conversation list view function provided by a document / conversation database search / exhibition module of a correction provision device according to one embodiment.

[0169] Referring to FIG. 10, the method and device for providing configuration document correction for improving chatbot performance according to embodiments may list conversations using various sorting criteria such as order of purpose, order of name, order of creation, order of most recent use, etc. Items from which correction documents are derived among conversation items are displayed as having a star mark inside a filled triangle on the left side of the list, and items from which correction documents are not derived are implemented to be visually distinguished as having an unstar mark inside the left triangle. If the left triangle is right-facing, when the user clicks it, a lower-level detailed information box expands, and the right-facing triangle on the left changes to a downward-facing triangle. At this time, the lower-level detailed information box may list items including configuration documents related to the conversation, and if correction documents are derived, the correction documents, along with each purpose tag and purpose description.

[0170] FIG. 11 is a block diagram illustrating a computing device according to one embodiment.

[0171] Referring to FIG. 11, a method and device for providing configuration document correction for improving chatbot performance according to embodiments can be implemented using a computing device (80).

[0172] A computing device (800) may include at least one of a processor (801), a memory (802), a storage device (803), a display device (804) communicating via a bus (809), a network interface device (805) providing access to a network (70) for communication with other entities, and an input / output interface device (806) providing a user input interface or a user output interface.

[0173] Of course, the computer device (80) may additionally include any electronic device necessary to implement the technical ideas described in this specification, although not shown in FIG. 11.

[0174] The processor (801) may be implemented in various types such as an Application Processor (AP), a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), a Neural Processing Unit (NPU), etc., and may be any electronic device that executes a program or command stored in a memory (802) or a storage device (803). In particular, the processor (801) may be configured to implement the functions or methods described above with respect to FIGS. 1 to 10, and in relation to the method and device for providing a configuration document correction for improving chatbot performance according to embodiments of the present invention, artificial intelligence-specific operations may be processed on a GPU or NPU.

[0175] The memory (802) and the storage device (803) may include various types of volatile or non-volatile storage media. For example, the memory (802) may include a read-only memory (ROM) or a random access memory (RAM), and the memory (802) may be located inside or outside the processor (801) and may be connected to the processor (801) through various known means. Meanwhile, examples of the storage device (803) include a hard disk drive (HDD) or a solid state drive (SSD), and the scope of the present invention is not limited to the elements listed above for explanation.

[0176] The method and device for providing configuration document correction for improving chatbot performance according to embodiments may be implemented as a program or software running on a computing device (80), and such program or software may be stored on a computer-readable medium.

[0177] Meanwhile, the method and device for providing configuration document correction for improving chatbot performance according to embodiments may be implemented using hardware of a computing device (80), or may be implemented as separate hardware that can be electrically connected to the computing device (80).

[0178] According to the embodiments described so far, the chatbot configuration document loaded into the RAG module of the RAG chatbot server can be corrected through user feedback and chatbot responses, and the response quality of the chatbot can be improved.

[0179] Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art to which the present invention pertains using the basic concept of the present invention defined in the following claims also fall within the scope of the present invention.

Claims

1. A method for providing a configuration document correction to improve the conversational performance of a RAG chatbot that performs document-based question answering by combining a technology for generating natural language responses to a query prompt using a Large Language Model (LLM) and a technology for refining the prompt by extracting document-based question context through Retrieval Augmented Generation (RAG). A step of preparing the RAG module of the RAG chatbot server to operate in response to a user question, wherein the step of allowing the user to select a document containing knowledge that can be referenced or answers a question through a chatbot interface and inputting it as a configuration document into the RAG module so that the question context can be extracted through vector matching of each user question, and loading the input configuration document into the vector database of the RAG module; A step of entering a question into the RAG chatbot server through the above chatbot, receiving an answer generated through LLM, and repeating the process of entering a question and receiving an answer if there is no problem with the answer; A step in which a user determines that a received chatbot response requires correction or supplementation and inputs user feedback regarding the appropriateness of the response through the chatbot interface; A step of receiving a chatbot response corresponding to the user feedback from the chatbot; A step of determining whether the above user feedback and chatbot response include an intention to correct a previous chatbot response; If it is determined that there is an intention to correct, a step of extracting new correction content from the user feedback and chatbot response; If the extracted correction content is the first correction content derived from the conversation, a step of creating a new correction document, otherwise adding it to the existing correction document; and A step of generating an improved version configuration document that corrects an existing configuration document by reflecting the contents of the above correction document, and loading it by inputting it into the RAG module. A method for providing configuration document corrections to improve chatbot performance.

2. In paragraph 1, The step of determining whether the above user feedback and chatbot response have the intention of correcting the previous answer of the chatbot is as follows: A step of determining whether a first phrase included in user feedback and a second phrase included in a chatbot response correspond to one of the predefined types of phrases indicating a correction situation, The step of extracting new corrections from the above user feedback and chatbot responses is as follows: If it is determined that the first phrase and the second phrase correspond to predefined phrases indicating a correction situation, a step of extracting new correction content from a user feedback corpus including the first phrase and a chatbot response corpus including the second phrase is included. A method for providing configuration document corrections to improve chatbot performance.

3. In paragraph 1, A step of displaying a correction button on a screen where a conversation with the chatbot is in progress, and determining whether the user feedback and chatbot response have the intention of correcting the previous answer of the chatbot, A step for determining whether the correction button has been pressed, Including a step of determining whether the third phrase included in the chatbot response corresponds to a predefined type of phrase indicating a correction situation, The step of extracting new corrections from the above user feedback and chatbot responses is as follows: If it is determined that the correction button has been pressed and the third phrase corresponds to a predefined phrase indicating a correction situation, a step of extracting new correction content from the user feedback and chatbot response that occurred between the pressing of the correction button and the chatbot response including the third phrase is included. A method for providing configuration document corrections to improve chatbot performance.

4. In paragraph 1, The step of determining whether the above user feedback and chatbot response have the intention of correcting the previous answer of the chatbot is as follows: A step of determining whether the user feedback and chatbot response have a corrective intent during a real-time chat with the chatbot, A method for providing configuration document corrections to improve chatbot performance.

5. In paragraph 1, The step of determining whether the above user feedback and chatbot response have the intention of correcting the previous answer of the chatbot is as follows: A step of reviewing the chat history after the real-time chat with the chatbot ends to determine whether the user feedback and chatbot response have a corrective intent. A method for providing configuration document corrections to improve chatbot performance.

6. In paragraph 1, Share the created or updated correction document or the improved version configuration document that reflects the contents of the correction document, Further comprising a step of providing a user who is permitted to share the document to view and utilize it and to load the shared improved version configuration document by inputting it into the RAG module that he or she uses. A method for providing configuration document corrections to improve chatbot performance.

7. In paragraph 1, The step where the user inputs user feedback on the appropriateness of the chatbot's response to the RAG chatbot server through the chatbot's user interface when the user determines that the chatbot's response requires correction or supplementation is as follows: Further comprising a step of inputting user feedback based on what the user has verified through other means such as a separate large-scale language model (LLM), an Internet search service, or a book, A method for providing configuration document corrections to improve chatbot performance.

8. In paragraph 1, The step of determining whether there is an intention to correct the previous answer of the chatbot from the pair of the user feedback and the chatbot response and extracting the correction content is as follows. A step of inputting the pair of the user feedback and the chatbot response into a separate large-scale language model (LLM) to determine whether a situation requires correction, and if a situation is determined to require correction, extracting the content to be corrected from the pair of the feedback and the chatbot response. A method for providing configuration document corrections to improve chatbot performance.

9. In paragraph 1, In order to perform a question-and-answer conversation, the configuration document that is vectorized and loaded into the above vector database and the correction document that is created or updated during the question-and-answer conversation are managed together with a purpose tag and a purpose description note added to each. The steps to correct the configuration document used for the above conversation are: A step of searching the document database for existing correction documents that are searched based on the purpose tagged in the above configuration document or that LLM determines have tags similar to the above purpose; and A step of generating an improved version configuration document by merging the contents of a correction document selected by the user from among the searched correction documents into the configuration document, A method for providing configuration document corrections for improving chatbot performance.

10. In paragraph 9, A step for the user to browse and display a list of purpose-specific documents and conversations to select a specific purpose; and Through the list of documents by purpose above, configuration documents and correction documents corresponding to the purpose entered by the user are displayed as activated, and configuration documents and correction documents that are different from the purpose entered are displayed as deactivated. It further includes a step of activating and displaying the question and answer conversation records corresponding to the purpose entered through the above purpose-specific conversation list, and deactivating and displaying the question and answer conversation records different from the entered purpose. A method for providing configuration document corrections for improving chatbot performance.

11. In paragraph 10, Based on the above activated correction document, the user submits the contents of his / her correction document or shared correction document to LLM, If it is determined that it is appropriate to consolidate highly interrelated correction documents into one content under one title based on the linguistic reasoning and judgment functions of the above LLM, It further includes a step of selecting and collating the above-mentioned correction documents, summarizing and organizing them according to the title and composition order suggested by LLM, and deriving them into new knowledge documents such as summary notes or study support notes. A method for providing configuration document corrections to improve chatbot performance.

12. A device for providing a configuration document correction to improve the conversational performance of a RAG chatbot that performs document-based question answering by combining a technology for outputting an appropriate linguistic response to a question prompt using a Large Language Model (LLM) and a technology for refining a question prompt by extracting a document-based question context through Retrieval Augmented Generation (RAG). Correction Extraction Module, Document / Conversation database search / exhibition module, Document / Conversation database management module, List display module by purpose, and Includes a module to activate list items by purpose, The above correction extraction module determines whether a pair of user feedback and a corresponding chatbot response includes the intention and content to correct the previous answer of the chatbot, If it is determined that there is an intention to correct, new corrections are extracted from the above feedback and responses, and if there is an existing correction document related to the conversation, the corrections are added to that document, and if not, a new correction document is created. The above purpose-specific list display module provides a list of configuration documents, correction documents, and question-and-answer conversations related to them grouped by purpose, based on user access rights or sharing scope. The above purpose-specific list item activation module selectively provides only relevant information by activating and displaying configuration documents, correction documents, and conversation items corresponding to specific purpose tags selected by the user, and deactivating and hiding other items. The above document / conversation database search / display module searches for composition documents, correction documents, and conversations based on the purpose tags, search words, or search phrases entered by the user, and enables display based on various sorting criteria such as name order, frequency of use order, and most recent order. The above document / conversation database management module is characterized in that it manages the creation or update of configuration documents, correction documents, and conversation records by storing them in a database. A device that provides configuration document correction for improving chatbot performance.

13. In paragraph 12, Determining whether the above user feedback and the above chatbot response have the intention of correcting the previous answer of the chatbot is as follows: Including determining whether the first phrase included in the user feedback and the second phrase included in the chatbot response are included in a phrase type predetermined to mean a correction situation, Extracting new corrections from the above user feedback and chatbot responses, If the above first and second phrases are judged to be included in the phrase type indicating a correction situation, Extracting new correction content from a pair of user feedback including the first phrase and a chatbot response including the second phrase, adding it to an existing correction document related to the conversation if there is one, or creating a new correction document if there is none. A device that provides configuration document correction for improving chatbot performance.

14. In paragraph 12, On the screen where the conversation with the above chatbot is in progress, a button indicating the start of the correction operation is displayed, Determining whether the above user feedback and the above chatbot response have the intention of correcting the previous answer of the chatbot is as follows: Determine whether the above button has been pressed, When the above button is pressed, it determines whether the user's feedback has been entered thereafter, If user feedback is entered, Including determining whether the third phrase included in the response of the above chatbot corresponds to one of the predefined phrase types that signify a correction situation; If the above third phrase is judged to mean a correction situation, New corrections are extracted based on user feedback and chatbot responses from the time the button is pressed to the chatbot response containing the third phrase. If there is an existing correction document related to the conversation, add the correction to that document, If there is no existing correction document, this includes creating a new correction document. A device that provides configuration document correction for improving chatbot performance.

15. In paragraph 12, Determining whether the above user feedback and the above chatbot response have the intention of correcting the previous answer of the chatbot is as follows: During a real-time chat with the chatbot, determine whether the user feedback and the chatbot response are intended to correct the chatbot's response, or After the real-time chat with the chatbot ends, reviewing the chat history to determine whether the pair of the user feedback and the chatbot response has the intention of correcting the previous answer of the chatbot. A device that provides configuration document correction for improving chatbot performance.

Images