Method and system for generating answer using feedback of user

Abstract

To provide a method and system for generating an answer using feedback of a user.SOLUTION: A method includes: generating answer candidates using a retriever of RAG (Retrieval Augmented Generation) for a first question from a user; generating a first answer to the first question from the user based on the answer candidates; providing a user interface to the user, the user interface including a function that allows the user to input a first feedback to the answer candidates; storing, in a storage, the first question, the first feedback input through the user interface, and information on answer candidates corresponding to the first feedback, in association with each other; selecting answer candidates to a second question from the user or a new user, using the first feedback and the information on the answer candidates corresponding to the first feedback stored in the storage; and generating a second answer to the second question using the selected candidates.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The following description relates to a method and system for generating answers using user feedback.

Background Art

[0002] Large Language Models (LLMs) are a type of artificial intelligence trained on a collection of large-scale text data to generate responses similar to those of humans for natural language input, and are language models composed of artificial neural networks with a huge number of parameters (usually billions of weights or more). Such LLMs can learn from a significant amount of unlabeled text using self-supervised learning or semi-self-supervised learning.

[0003] RAG (Retrieval-Augmented Generation) is a technique used to complement the hallucination phenomenon of LLMs and knowledge that has not been learned. LLMs have the ability to grasp general knowledge and context to generate responses but cannot answer except for the knowledge learned during the pre-training process. Even if it is learned content, the answers generated by retrieving the stored knowledge show a significant hallucination phenomenon.

[0004] RAG is a method of pre-inserting knowledge related to the prompts of LLMs to improve this and making them answer based on this knowledge. Various LLMs have achieved considerable performance improvement through such a method.

[0005] However, RAG depends to a considerable extent on the performance of the retriever to answer questions. Naturally, the information necessary to answer the user's questions must be provided in advance. The problem of searching for relevant documents for answers from a wide range of documents or searching for answers has already been Open Domain Question Answering (ODQA) This is commonly known as the "Question Answering" problem. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Korean Patent No. 10-2551531 Summary of the Invention [Problem to be solved by the invention]

[0007] A method and system for generating answers utilizing user feedback is provided. [Means for solving the problem]

[0008] A method for generating an answer for a computer device including at least one processor, comprising: generating, by at least one processor, answer candidates for a first question of a user using a retriever of Retrieval Augmented Generation (RAG); generating, by the at least one processor, a first answer to the first question of the user based on the answer candidates; and displaying, by the at least one processor, a user interface including a function for allowing the user to input a first feedback for the answer candidates. providing the user with the first question; storing, by the at least one processor, information related to the first question, the first feedback input through the user interface, and answer candidates corresponding to the first feedback in a storage device in a coordinated manner; and and selecting an answer candidate for a second question of the user or a new user by using information on the first feedback stored in the storage and the answer candidate corresponding to the first feedback, and Provided is an answer generation method including a step of generating a second answer to the second question using a supplement.

[0009] According to one embodiment, the storing step includes storing information regarding the first feedback and answer candidates corresponding to the first feedback in association with a sentence embedding value of the first question, and the step of generating the second answer uses the first feedback and the information regarding the answer candidates corresponding to the first feedback retrieved from the storage through a sentence embedding value of the second question to add an answer candidate corresponding to the first feedback to a list of answer candidates generated for the second question or exclude the answer candidate corresponding to the first feedback from the list to select an answer candidate for the second question.

[0010] According to another aspect, the step of generating the second answer includes, when the first feedback associated with the question retrieved from the storage is positive feedback, adding an answer candidate corresponding to the first feedback to a list of answer candidates generated for the second question; and when the first feedback associated with the question retrieved from the storage is negative feedback, excluding an answer candidate corresponding to the first feedback from the list of answer candidates generated for the second question.

[0011] Also, according to another aspect, the information about the answer candidate corresponding to the first feedback includes information about the position where the first feedback occurred in the answer candidate, and the step of selecting an answer candidate for the second question includes a new answer candidate including the content of the position. It can be characterized by generating a supplement and adding it to the list, or excluding an answer candidate including the content at the position from the list.

[0012] Also, according to another aspect, the position can include the page of the document corresponding to the answer candidate or the position of the content selected by the user in the answer candidate.

[0013] Also, according to another aspect, the user interface is generated for the first question a function for displaying a list of generated answer candidates, a function for displaying the content of the answer candidate selected in the list, and inputting first feedback for the displayed answer candidate It can be characterized by including a function for doing so.

[0014] Also, according to another aspect, the step of generating the first answer includes generating a prompt based on the answer candidate; and inputting the generated prompt into a large language model (LLM) to generate the first answer. It can be characterized by including these steps.

[0015] According to another aspect, the user interface further includes a function for the user to input second feedback on the first answer, and the answer generation method includes, by at least one processor, when negative feedback is input as the second feedback through the user interface, generating a new prompt based on the negative feedback; and further including, by the at least one processor, inputting the new prompt into the LLM to generate a new first answer. at least It can be further included.

[0016] Also, according to another aspect, in the step of storing, when positive feedback is input as the second feedback through the user interface, the answer corresponding to the positive feedback is further associated with the first question and stored in the storage, and the step of generating the second answer can be characterized by further using the answer stored in further association with the first question to select a candidate answer for the second question.

[0017] There is provided a computer program stored in a computer-readable recording medium coupled to a computer device for causing the computer device to execute the method.

[0018] There is provided a computer-readable recording medium on which a program for causing a computer device to execute the method is recorded.

[0019] Including at least one processor, the at least one processor is embodied to execute instructions readable by a computer device, generates candidate answers using a RAG retriever for a user's first question, generates a first answer for the user's first question based on the candidate answers, provides a user interface to the user including a function for inputting first feedback on the candidate answers, stores information related to the first question, the first feedback input through the user interface, and the candidate answers corresponding to the first feedback in association in a storage, and for a second question of the previous user or a new user, selects candidate answers for the second question using the information related to the first feedback stored in the storage and the candidate answers corresponding to the first feedback, and provides a computer device characterized by generating a second answer for the second question using the selected candidate answers.

Advantages of the Invention

[0020] A method and system for generating an answer by utilizing user feedback can be provided.

Brief Description of Drawings

[0021]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0022] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings.

[0023] The response generation system according to an embodiment of the present invention can be implemented by at least one computer device. At this time, a computer program according to an embodiment of the present invention can be installed and driven in the computer device that implements the response generation system, and the computer device can perform the response generation method according to an embodiment of the present invention under the control of the driven computer program. The above-mentioned computer program can be stored in a computer-readable recording medium that is combined with the computer device to cause the computer to execute the response generation method. body. body.

[0024] FIG. 1 is a drawing illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 shows an example including a plurality of electronic devices (110, 120, 130, 140), a plurality of servers (150, 160), and a network (170). Such FIG. 1 is an example for the description of the invention, and the number of electronic devices and the number of servers are not limited to those in FIG. 1.

[0025] The plurality of electronic devices (110, 120, 130, 140) can be fixed terminals or mobile terminals implemented by a computer system. Examples of the plurality of electronic devices (110, 120, 130, 140) include a smart phone, a mobile phone, a navigation device, a computer, a notebook computer, a digital broadcast terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a tablet PC, a game console, a wearable device, an IoT (internet of things) device, a VR (virtual reality) device, an AR (augmented reality) device, etc. As an example, in FIG. 1, the electronic device (110) is shown in the shape of a smart phone, but in the embodiment of the present invention, the electronic device (110) is substantially any one of various physical computer systems that can communicate with other electronic devices (120, 130, 140) and / or servers (150, 160) through the network (170) using a wireless or wired communication method. The communication method is not limited, and it utilizes a communication network that the network (170) can include (for example, a mobile communication network, a wired internet, a wireless internet, a broadcast network, a satellite network, etc.).

[0026] Not only communication methods, but also short-range wireless communication between devices can also be included. For example, the network (170) can include any one or more of networks such as PAN (personal area network), LAN (local area network), CAN (campus area network), MAN (metropolitan area network), WAN (wide area network), BBN (broadband network), and the Internet. In addition, the network (170) can include any one or more of network topologies including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or hierarchical network, etc., but is not limited thereto. Each of the servers (150, 160) can communicate with a plurality of electronic devices (110, 120, 130, 140) through the network (170) to provide commands, codes, files, contents, services, etc., and can be embodied as a computer device or a plurality of computer devices. For example, the server (150) can be a system that provides a first service to a plurality of electronic devices (110, 120, 130, 140) connected through the network (170), and the server (160) can also be a system that provides a second service to a plurality of electronic devices (110, 120, 130, 140) connected through the network (170). As a more specific example, the server (150) provides, through an application as a computer program stored and driven in a plurality of electronic devices (110, 120, 130, 140), the service targeted by the application (for example, a search service, etc.) as the first service.

[0027] Each of the servers (150, 160) can communicate with a plurality of electronic devices (110, 120, 130, 140) through the network (170) to provide commands, codes, files, contents, services, etc., and can be embodied as a computer device or a plurality of computer devices. For example, the server (150) can be a system that provides a first service to a plurality of electronic devices (110, 120, 130, 140) connected through the network (170), and the server (160) can also be a system that provides a second service to a plurality of electronic devices (110, 120, 130, 140) connected through the network (170). Each of the servers (150, 160) can communicate with a plurality of electronic devices (110, 120, 130, 140) through the network (170) to provide commands, codes, files, contents, services, etc., and can be embodied as a computer device or a plurality of computer devices. For example, the server (150) can be a system that provides a first service to a plurality of electronic devices (110, 120, 130, 140) connected through the network (170), and the server (160) can also be a system that provides a second service to a plurality of electronic devices (110, 120, 130, 140) connected through the network (170). Each of the servers (150, 160) can communicate with a plurality of electronic devices (110, 120, 130, 140) through the network (170) to provide commands, codes, files, contents, services, etc., and can be embodied as a computer device or a plurality of computer devices. For example, the server (150) can be a system that provides a first service to a plurality of electronic devices (110, 120, 130, 140) connected through the network (170), and the server (160) can also be a system that provides a second service to a plurality of electronic devices (110, 120, 130, 140) connected through the network (170). As a more specific example, the server (150) provides, through an application as a computer program stored and driven in a plurality of electronic devices (110, 120, 130, 140), the service targeted by the application (for example, a search service, etc.) as the first service. As a more specific example, the server (150) provides, through an application as a computer program stored and driven in a plurality of electronic devices (110, 120, 130, 140), the service targeted by the application (for example, a search service, etc.) as the first service. It can be provided to a plurality of electronic devices (110, 120, 130, 140). As another example, the server (160) stores files for installing and driving the above-described applications Provide, as a second service, a service that distributes to a plurality of electronic devices (110, 120, 130, 140) It can be done.

[0028] FIG. 2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. Each of the plurality of electronic devices (110, 120, 130, 140) and each of the servers (150, 160) described above can be implemented by the computer device (200) illustrated through FIG. 2.

[0029] Such a computer device (200) can include, as shown in FIG. 2, a memory (210), a processor (220), a communication interface (230), and an input / output interface (240). The memo ry (210) can include, as a computer-readable recording medium, a RAM (random access memory), a ROM (read only memory), and a permanent mass storage device such as a disk drive. Here, non-volatile mass storage devices such as ROM and disk drive are distinguished from the memory (210) as a separate permanent storage device for the computer It can also be included in the device (200). Further, an operation system and at least one program code can be stored in the memory (210). Such software components can be loaded from a computer-readable recording medium separate from the memory (210) into the memory (210). Such a separate computer-readable recording medium can include computer-readable recording media such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, and the like. In other embodiments, the software components can also be loaded into the memory (210) through a communication interface (230) that is not a computer-readable recording medium. For example, the software components can be based on a computer program installed by files received through the network (170) and loaded into the memory (210) of the computer device (200). It can be loaded into the memory (210) of the computer device (200) based on a computer program installed by files received through the network (170).

[0030] The processor (220) can be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions can be provided to the processor (220) by the memory (210) or the communication interface (230). For example, the processor (220) can be configured to execute instructions received according to program code stored in a recording device such as the memory (210). The instructions can be provided to the processor (220) by the memory (210) or the communication interface (230). For example, the processor (220) can be configured to execute instructions received according to program code stored in a recording device such as the memory (210). The instructions can be provided to the processor (220) by the memory (210) or the communication interface (230). For example, the processor (220) can be configured to execute instructions received according to program code stored in a recording device such as the memory (210). The instructions can be provided to the processor (220) by the memory (210) or the communication interface (230). For example, the processor (220) can be configured to execute instructions received according to program code stored in a recording device such as the memory (210). The instructions can be provided to the processor (220) by the memory (210) or the communication interface (230). For example, the processor (220) can be configured to execute instructions received according to program code stored in a recording device such as the memory (210).

[0031] The communication interface (230) can provide a function for the computer device (200) to communicate with other devices (for example, the aforementioned storage devices) through the network (170). As an example, the processor (220) of the computer device (200) can be a recording device such as the memory (210). The communication interface (230) can provide a function for the computer device (200) to communicate with other devices (for example, the aforementioned storage devices) through the network (170). As an example, the processor (220) of the computer device (200) can be a recording device such as the memory (210). Requests, instructions, data, files, etc. generated according to the stored program code can be transmitted to other devices through the network (170) under the control of the communication interface (230). Conversely, signals, instructions, data, files, etc. from other devices can be received by the computer device (200) through the communication interface (230) of the computer device (200) via the network (170). Signals, instructions, data etc. received through the communication interface (230) can be transmitted to the processor (220) and the memory (210), and files etc. can be stored in a storage medium (the above-mentioned permanent storage device) that the computer device (200) may further include.

[0032] The input / output interface (240) can be means for interfacing with the input / output device (250). For example, the input device can include devices such as a microphone, keyboard, or mouse, and the output device can include devices such as a display and speaker. As another example, the input / output interface (240) can also be means for interfacing with a device in which functions for input and output are integrated into one, such as a touch screen. The input / output device (250) can also be configured as one device with the computer device (200). re. The input / output device (250) can also be configured as one device with the computer device (200).

[0033] Also, in other embodiments, the computer device (200) can include fewer or more components than the components in FIG. 2. However, there is no need to clearly illustrate most of the conventional components. For example, the computer device (200) can be embodied to include at least a part of the above-mentioned input / output device (250), or can further include other components such as a transceiver , a database, etc.

[0034] FIG. 3 is a drawing illustrating an example of a general form of a response generation system in an embodiment of the present invention. is as follows. The answer generation system (Answer Generation System, 300) according to the embodiment of FIG. 3 can include a retriever (Retriever, 310), a summarizer (Summarizer, 320), a large language model (LLM, 330), and a storage (Storage, 340) of RAG (Retrieval Augmented Generation), and can use the user's feedback to improve the performance of the retriever (310). This can be done.

[0035] When an input is input to the retriever (310) according to the user's question, the retriever (310) can search for answer candidates for the user's question based on the input. At this time, the search method can be a sparse method that uses keyword search to find answer candidates and an ensemble of inverted index vector search or cross-encoder methods to find answer candidates in a dense method. At this time, the answer candidates retrieved by the retriever (310) are called clues (CLUE). If all the retrieved answer candidates are to be input to the LLM (330) as prompts, it may exceed the maximum token length limit. To solve this problem, the summarizer (320) can receive the clue as an input and summarize the clue centered on the content related to the answer. At this time, the summarization can use a generation summarization method or an extraction summarization method based on the LLM. The LLM for summarization may be the same as or different from the LLM (320) in FIG. 3. The summarizer (320) creates a prompt using the summarized clue and inputs it to the LLM (320).

[0036] After generating the final answer through the LLM (320), it can be provided to the user. Then, the summarizer (320) can receive the clue as an input and summarize the clue centered on the content related to the answer. At this time, the summarization can use a generation summarization method or an extraction summarization method based on the LLM. The LLM for summarization may be the same as or different from the LLM (320) in FIG. 3. The summarizer (320) creates a prompt using the summarized clue and inputs it to the LLM (320). Then, the summarizer (320) can receive the clue as an input and summarize the clue centered on the content related to the answer. At this time, the summarization can use a generation summarization method or an extraction summarization method based on the LLM. The LLM for summarization may be the same as or different from the LLM (320) in FIG. 3. The summarizer (320) creates a prompt using the summarized clue and inputs it to the LLM (320). After generating the final answer through the LLM (320), it can be provided to the user.

[0037] At this time, the answer generation system (300) can receive two types of feedback from the user: clue feedback and answer feedback as inputs.

[0038] The clue feedback may be feedback on the clues (answer candidates) used in generating the prompt. As an example, the answer generation system (300) can allow the user to check the content of each clue through the list of clues used in generating the answer, and provide the user with a function to select positive feedback or negative feedback for each clue. At this time the answer generation system (300) can confirm the positive feedback or negative feedback for each clue selected by the user as the clue feedback for the corresponding clue. Answer feedback can include feedback on the answers generated by the LLM (320). As an example, although there is no problem with the clue itself, the LLM (320) may generate an answer unrelated to the clue. At this time, the answer generation system (300) can provide a function to check the content of each clue through the above-mentioned list of clues, and provide the user with a function to input answer feedback when the answer is incorrect even though the clue is correct.

[0039] The answer generation system (300) can save the user's feedback including clue feedback and / or answer feedback in the storage (340), and use the feedback stored in the storage (340) for new questions from the user and / or new users. At this time, the answer generation system (300) can provide a function to check the content of each clue through the above-mentioned list of clues, and provide the user with a function to input answer feedback when the answer is incorrect even though the clue is correct. The answer generation system (300) can save the user's feedback including clue feedback and / or answer feedback in the storage (340), and use the feedback stored in the storage (340) for new questions from the user and / or new users.

[0040] The answer generation system (300) can save the user's feedback including clue feedback and / or answer feedback in the storage (340), and use the feedback stored in the storage (340) for new questions from the user and / or new users. can generate an answer. As an example, the answer generation system (300) controls the retriever (310) to select more accurate clues by using clue feedback and can control the summarizer (320) to generate a more accurate prompt by using answer feedback.

[0041] FIG. 4 is a drawing illustrating an example of a user interface for receiving clue feedback from a user in one embodiment of the present invention. FIG. 4 shows an example of a user interface (400) that the answer generation system (300) can provide to the user. The user interface (400) includes a region (410) for presenting a question 1 and an answer 1 provided in response to the user's question 1, a region (the region of the dotted box (420)) for displaying a list of clues used to generate the answer 1, and a region (430) for displaying the content of the clue selected from the list of clues. Further, the user interface (400) can include functions (for example, a button (440) and a button (450)) for receiving positive feedback and / or negative feedback on the selected clue (clue 1 in the embodiment of FIG. 4). (400) includes a region (410) for presenting a question 1 and an answer 1 provided in response to the user's question 1, a region (the region of the dotted box (420)) for displaying a list of clues used to generate the answer 1, and a region (430) for displaying the content of the clue selected from the list of clues. Further, the user interface (400) can include functions (for example, a button (440) and a button (450)) for receiving positive feedback and / or negative feedback on the selected clue (clue 1 in the embodiment of FIG. 4). (400) includes a region (410) for presenting a question 1 and an answer 1 provided in response to the user's question 1, a region (the region of the dotted box (420)) for displaying a list of clues used to generate the answer 1, and a region (430) for displaying the content of the clue selected from the list of clues. Further, the user interface (400) can include functions (for example, a button (440) and a button (450)) for receiving positive feedback and / or negative feedback on the selected clue (clue 1 in the embodiment of FIG. 4). (400) includes a region (410) for presenting a question 1 and an answer 1 provided in response to the user's question 1, a region (the region of the dotted box (420)) for displaying a list of clues used to generate the answer 1, and a region (430) for displaying the content of the clue selected from the list of clues. Further, the user interface (400) can include functions (for example, a button (440) and a button (450)) for receiving positive feedback and / or negative feedback on the selected clue (clue 1 in the embodiment of FIG. 4). (400) includes a region (410) for presenting a question 1 and an answer 1 provided in response to the user's question 1, a region (the region of the dotted box (420)) for displaying a list of clues used to generate the answer 1, and a region (430) for displaying the content of the clue selected from the list of clues. Further, the user interface (400) can include functions (for example, a button (440) and a button (450)) for receiving positive feedback and / or negative feedback on the selected clue (clue 1 in the embodiment of FIG. 4).

[0042] For example, the user can check the content of each clue from the list of clues used to generate the answer 1 to confirm whether the content of the clue is appropriate. At this time, the user can select the button (440) to input positive feedback or select the button (450) to input negative feedback for the currently displayed clue. (450) to input negative feedback for the currently displayed clue.

[0043] At this time, the answer generation system (300) senses the user's question (question 1 in the embodiment of FIG. 4) ​Store the tense embedding value, the information for the selected crew (crew 1 in the example of FIG. 4), and the user's crew feedback in storage (340). This can be done. According to the embodiment, the user interface (400) can also provide a function that can receive positive feedback and / or negative feedback from the user for each page of the crew or for some content (for example, specific text) selected by the user in the content including the crew content. In this case, the answer generation system (300) can further store in the storage (340) the information about the crew and the information regarding the position of the crew (for example, document information, page information, and / or selected content information). in the storage (340).

[0044] When a new question is input from the user or a new user later, the retriever (310) can search the storage (340) for crew feedback related to similar questions in the storage (340) through the sentence embedding value of the new question. At this time, the retriever (310) can select a more accurate crew by adding the crew associated with the positive feedback retrieved from the storage (340) to the list of crews or removing the crew associated with the negative feedback from the list of crews.

[0045] FIG. 5 is a drawing illustrating an example of a user interface for inputting answer feedback from a user in an embodiment of the present invention. FIG. 5 shows an example of a user interface (500) that the answer generation system (300) can provide to the user. The user interface (500) can include an area where question 1 input by the user and answer 1 generated for question 1 are displayed. At this time, the user interface (500) provides a function that can select positive or negative for answer 1, such as a dotted box (510). This can be done. This can be done. At this time, if the user selects "negative", the system can either select specific content from the content prepared in advance as the user's response feedback or provide a user interface (520) for the user to input new reasons.

[0046] The answer generation system (300) can construct a new prompt based on the prompt used to generate Answer 1 and the response feedback input through the user interface (520). Also, the answer generation system (300) can input the new prompt into the LLM (330) to generate a new Answer 2 and provide it to the user. At this time, for the new Answer 2, the user can also select "positive" or "negative". If "negative" is selected, the process of receiving response feedback again to generate other new answers can be repeated. Alternatively, the answer generation system (300) can receive the answer desired by the user as feedback.

[0047] If the user selects "positive", the answer generation system (300) can store the sentence embedding value of the user's question and the selected positive answer in the storage (340).

[0048] After that, when a new question is input by the user or a new user, the answer generation system (300) can search the storage (340) for response feedback related to similar questions through the sentence embedding value of the new question. At this time, the retriever (310) includes the answers related to the response feedback retrieved from the storage (340) in a clue list, enabling the summarizer (320) to generate a prompt, thereby generating a more accurate answer.

[0049] FIG. 6 is a flowchart showing an example of a response generation method according to an embodiment of the present invention. This The response generation method according to the embodiment can be executed by a computer device (200) that implements the above-described response generation system (300). At this time, the processor (220) of the computer device (200) can be implemented to execute control instructions according to the code of the operating system included in the memory (210) or the code of at least one computer program. Here, the processor (220) can control the computer device (200) so that the computer device (200) executes steps (610 to 660) included in the method of FIG. 6 according to the control instructions provided by the code stored in the computer device (200).

[0050] In step (610), the computer device (200) can generate answer candidates for the user's first question using the RAG retriever. The retriever can correspond to the retriever (310) described above through FIG. 3.

[0051] In step (620), the computer device (200) can generate a first answer to the user's first question based on the answer candidates. As an example, the computer device (200) can generate a prompt based on the answer candidates, and input the generated prompt into a large language model (LLM) to generate a first answer.

[0052] In stage (630), the computer device (200) can provide the user with a user interface including a function that allows the user to input first feedback on the answer candidates. As an example, the user interface can be a list of answer candidates generated for the first question Including a function for displaying a prompt, a function for displaying the content of a selected answer candidate from a list, and a function for inputting first feedback for the displayed answer candidate is possible.

[0053] Also, according to an embodiment, the user interface can further include a function for the user to input second feedback for the first answer. In this case, when negative feedback is input as the second feedback through the user interface, the computer device (200) can generate a new prompt based on the negative feedback, input the new prompt into the LLM, and generate a new first answer. When the user inputs negative feedback for the new first answer as well, the computer device (200) generates a new prompt based on the negative feedback again, and the new first answer can be generated.

[0054] Here, the first feedback can correspond to the aforementioned clue feedback, and the second feedback can correspond to the answer feedback respectively.

[0055] In step (640), the computer device (200) can store in cooperation the first question, the first feedback input through the user interface, and information regarding the answer candidate corresponding to the first feedback in a storage. Here, the storage can correspond to the aforementioned storage (340). As an example, the computer device (200) can store in cooperation the first feedback and information regarding the answer candidate corresponding to the first feedback with the sentence embedding value of the first question. Also, positive feedback as the second feedback through the user interface can be stored in cooperation with the sentence embedding value of the first question and information regarding the first feedback and the answer candidate corresponding to the first feedback.

[0056] can be stored in cooperation with the sentence embedding value of the first question and information regarding the first feedback and the answer candidate corresponding to the first feedback. If a positive feedback is entered, the answer corresponding to the positive feedback can be further associated with the first question and saved in storage.

[0057] In step (650), the computer device (200) may select answer candidates for the second question of the user or the new user by using the first feedback stored in the storage and information on the answer candidates corresponding to the first feedback. As an example, the computer device (200) may select answer candidates for the second question of the user or the new user by using the sentence embedding value of the sentence embedding value of the second question. Respond to first-stage feedback and first-stage feedback linked to questions searched in the repository Using information about the answer candidates that correspond to the first feedback, add the answer candidates that correspond to the first feedback to the list of answer candidates generated for the second question, or select the answer candidates that correspond to the first feedback from the list. In a more specific example, when the first feedback associated with the question searched in the storage is a positive feedback, the computer device 200 may select an answer candidate for the second question by excluding an answer candidate corresponding to the first feedback. The corresponding answer candidate can be added to a list of answer candidates generated for the second question. As another example, the computer device (200) can remove the answer candidate corresponding to the first feedback from the list of answer candidates generated for the second question when the first feedback associated with the question retrieved in the storage is negative feedback.

[0058] Meanwhile, the information about the answer candidate corresponding to the first feedback may include information about the location where the first feedback occurs in the answer candidate. The location where the query occurred may include a page of a document corresponding to the answer candidate or a location of the content selected by the user in the answer candidate. In this case, the computer device (200) may A new answer candidate including the content can be generated and added to the list of answer candidates, or an answer candidate including the content at the above position can be excluded from the list of answer candidates.

[0059] Also, when an answer corresponding to the positive feedback as the second feedback is further stored in the storage in association with the first question, the computer device (200) is further associated with the first question The stored answer can be further used to select an answer candidate for the second question. For example, the computer device (200) can add the answer corresponding to the positive feedback to the list of answer candidates generated for the second question.

[0060] In step (660), the computer device (200) can generate a second answer to the second question by using the selected answer candidate. In this way, the computer device (200) can generate an answer through a more accurate answer candidate by using the user's clue feedback (first feedback) and / or answer feedback (second feedback). Also, the computer device (200) can generate a new prompt through the answer feedback (second feedback) to provide a more suitable answer to the user.

[0061] As described above, according to the embodiments of the present invention, an answer generation method and system that utilize user feedback can be provided.

[0062] ​​The system or apparatus described above can be implemented by hardware components, or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments can be, for example, a processor, a controller, an ALU (arithmetic logic unit), a digital signal processor, a microcomputer, an FPGA (field programmable gate array), a PLU (programmable logic unit), a microprocessor, or can execute and respond to an instruction like any other device, one or more general-purpose computers or special-purpose computers can be utilized for implementation. The processing device can execute an operating system (OS) and one or more software applications executed on the operating system and can also access, store, operate on, process, and generate data in response to the execution of the software. For the sake of convenience of understanding, although the processing device may be described as if one is used, those with ordinary knowledge in the art can know that the processing device can include multiple processing elements and / or multiple types of processing elements. For example, the processing device can include multiple processors or one processor and one controller. Also, other processing configurations such as a parallel processor are possible.

[0063] Software can include a computer program, code, instruction, or a combination of one or more of these, and can operate as desired The processing device can be configured or commanded to process the device independently or collectively. Software and / or data can be interpreted by the processing device or To provide instructions or data to the processing device, certain types of machines, components , physical devices, virtual equipment, computer storage media or devices can be embodied Software can also be distributed on a computer system connected by a network and stored or executed in a distributed manner. Software and data can be stored on one or more computer-readable recording media.

[0064] The method according to this embodiment can be embodied in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium can include program instructions, data files, data structures, etc. alone or in combination. The medium may continuously store a computer-executable program or temporarily store it for execution or download. Also, the medium can be various recording or storage means in the form of a single or several pieces of hardware combined, but is not limited to the medium directly connected to a computer system and may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program instruction words including ROM, RAM, flash memory, etc. Also, as examples of other media, application stores that distribute applications and recording or storage media managed by sites, servers, etc. that supply or distribute various other software can be mentioned. Examples of program instructions include not only machine language codes identical to those made by a compiler but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0065] As described above, the embodiments have been described with reference to limited embodiments and drawings, but those with ordinary knowledge in the relevant technical field can make various modifications and variations from the above description. For example, the described technology may be executed in an order different from the described method, and / or the components of the described system, structure, device, circuit, etc. may be combined or combined in a form different from the described method, or replaced or exchanged with other components or equivalents, and appropriate results can still be achieved.

[0066] Therefore, other embodiments, other examples and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. 1. A method of answer generation for a computing device including at least one processor, comprising: generating, by the at least one processor, answer candidates for a first question of a user using a retriever of Retrieval Augmented Generation (RAG); generating, by the at least one processor, a first answer to the user's first question based on the candidate answers; providing, by the at least one processor, a user interface to the user, the user interface including a function that allows the user to input a first feedback for the answer candidate; storing, by the at least one processor, information regarding the first question, the first feedback input through the user interface, and answer candidates corresponding to the first feedback in a storage in association with each other; The at least one processor selects answer candidates for a second question of the user or a new user by using information about the first feedback stored in the storage and the answer candidates corresponding to the first feedback, and generating a second answer to the second question using the selected answer candidates; Answer generation method.

2. 2. The method of claim 1, The storing step includes: storing information about the first feedback and candidate answers corresponding to the first feedback in association with sentence embedding values ​​of the first question; The step of generating a second answer includes: The second question is associated with the query retrieved from the storage through the sentence embedding value of the second question. using information about the first feedback and the candidate answers corresponding to the first feedback to add the candidate answers corresponding to the first feedback to a list of candidate answers generated for the second question, or to add the candidate answers corresponding to the first feedback from the list of candidate answers generated for the second question; and selecting an answer candidate for the second question by excluding the candidate. A method for generating answers.

3. 3. The method of claim 2, The step of generating the second answer includes: The first feedback associated with the question retrieved in the storage is a positive feedback. if the first feedback is a question, adding a candidate answer corresponding to the first feedback to a list of candidate answers generated for the second question; and The first feedback associated with the question retrieved in the storage is negative feedback. removing the candidate answer corresponding to the first feedback from a list of candidate answers generated for the second question if the first feedback is a candidate answer corresponding to the second question; A method for generating an answer comprising:

4. 3. The method of claim 2, The information on the answer candidate corresponding to the first feedback is information regarding a location where the first feedback occurred; The step of selecting answer candidates to the second question includes: An answer generating method, comprising: generating a new answer candidate including the content of the location and adding it to the list; or removing an answer candidate including the content of the location from the list.

5. 5. The method of claim 4, The answer generating method, wherein the location includes a page of a document corresponding to the answer candidate or a location of content selected by the user in the answer candidate.

6. 10. The method of claim 1 , The user interface displays a list of candidate answers generated for the first question. a function for displaying a content of an answer candidate selected from the list, a function for displaying a content of an answer candidate selected from the list, and a function for inputting a first feedback for the displayed answer candidate. Draft generation method.

7. 10. The method of claim 1 , The step of generating a first answer includes: generating a prompt based on the candidate answers; and inputting the generated prompt into a Large Language Model (LLM) to generate the first answer; An answer generation method comprising:

8. 8. The method of claim 7, the user interface further includes a function for receiving, from the user, a second feedback on the first answer; generating, by the at least one processor, if negative feedback is input as the second feedback via the user interface, a new prompt based on the negative feedback; and inputting, by said at least one processor, said new prompt into said LLM to generate a new first answer. The answer generation method further includes:

9. 8. The method of claim 7, The storing step includes: and providing positive feedback as the second feedback via the user interface. If a positive feedback is input, the answer corresponding to the first question is further and storing the data in the storage in cooperation with the data. The step of generating a second answer includes: and selecting a candidate answer to the second question by further utilizing the stored answers in further cooperation with the first question. The answer generation method is characterized by:

10. At least one computer readable instruction set is embodied to execute the instructions. A processor is included. by said at least one processor, Generate answer candidates for the user's first question using a RAG (Retrieval Augmented Generation) retriever, generating a first answer to a first question of the user based on the candidate answers; A function for allowing the user to input a first feedback for the answer candidate is included. providing a user interface to said user; The first question, the first feedback input through the user interface, and information about the answer candidates corresponding to the first feedback are stored in a storage device in a linked manner. Remains, For a second question of the user or a new user, the first feedback stored in the storage and information about the answer candidates corresponding to the first feedback are used to selecting answer candidates to the second question, and generating a second answer to the second question using the selected answer candidates. A computer device comprising:

Images