Answer generation method and system

Abstract

The present invention relates to an answer generation method and system. More specifically, the present invention relates to an answer generation method and system, which enable an answer to a user query to be generated by retrieving external knowledge.

Description

Method and system for generating answers

[0001] The present invention relates to a method and system for generating answers. More specifically, the present invention relates to a method and system for generating answers to user queries by searching external knowledge.

[0002] The dictionary definition of artificial intelligence is "the technology that embodies human learning, reasoning, perception, and natural language comprehension abilities through computer programs." This type of artificial intelligence has made remarkable progress thanks to deep learning.

[0003] In particular, thanks to the development of artificial intelligence, various language models have been developed, and these language models have reached the level of not only recognizing text and understanding its meaning, but also extracting and classifying information from data containing large amounts of text, such as documents, and even directly generating text.

[0004] These language models are actively used in various fields, such as search engines, document writing (e.g., resume writing, report writing, posting writing, etc.), free conversation on various topics, data parsing from a given text (e.g., data summarization, classification, etc.), providing expert knowledge, programming, and converting a given sentence into sentences in an appropriate style, etc., and they exist in various fields that can perform text-based tasks.

[0005] In this regard, large language models (LLMs) have recently emerged, capable of understanding and generating human language through pre-training on massive amounts of text data. Unlike traditional manually built chatbots, which only offer limited responses, LLMs demonstrate the technological prowess to communicate naturally, almost human-like, and provide fast, accurate information, revolutionizing the artificial intelligence market.

[0006] However, large-scale language models tend to generate inaccurate or outdated information, especially in domain-specific tasks, because they rely solely on the knowledge contained in their parameters.

[0007] To overcome these limitations, Retrieval Augmented Generation (RAG), a technique that integrates external knowledge, is attracting attention as a promising approach. While RAG can improve the accuracy of content generated by large-scale language models without retraining, effectively retrieving external knowledge remains challenging due to potential noise and computational costs.

[0008] To address this, conventional information retrieval systems utilize a two-step approach: quickly retrieving a small number of candidate documents from a large number of documents using an efficient retrieval engine (step 1), and then applying a reranking mechanism to the candidate documents using a sophisticated reranker (step 2).

[0009] However, these conventional information retrieval systems (or models) can only perform single-granularity ranking at a time and have limited context size, making it difficult to directly apply them to actual RAG situations due to effectiveness and efficiency issues.

[0010] The present invention provides a method and system for generating answers that can simultaneously improve search efficiency and accuracy of content generation.

[0011] More specifically, the present invention provides a method and system for generating answers that can simultaneously improve search efficiency and accuracy of content generation by simultaneously performing document reranking and paragraph extraction necessary for content generation.

[0012] In particular, the present invention provides a method and system for generating answers that can simultaneously perform relevance judgments on documents and paragraphs.

[0013] In addition, the present invention provides a method and system for generating an answer that can simultaneously perform searching for a document related to a query and extracting a paragraph related to the query from the searched document.

[0014] Furthermore, the present invention provides a method and system for generating an answer that can efficiently search documents and paragraphs related to a query, thereby generating and providing a more accurate and relevant answer to the query.

[0015] In addition, the present invention provides a method and system for generating answers that can provide a user-customized environment.

[0016] More specifically, the present invention provides a method and system for generating answers that can provide accurate and high-quality answers to user inquiries while providing a service environment optimized for the user.

[0017] Furthermore, the present invention provides a method and system for generating answers that provide customized information to users and enable users to efficiently search for accurate information.

[0018] In order to solve the problem discussed above, a computerized answer generation method according to the present invention may include the steps of: receiving a user query from a user terminal; specifying a plurality of candidate documents related to the user query based on preset criteria; determining a relevance to the user query for each passage included in each of the plurality of candidate documents and distinguished based on preset criteria; specifying a related document related to the user query among the plurality of candidate documents based on the relevance determined for each paragraph included in each of the user query and the plurality of candidate documents; and generating an answer to the user query using the related document.

[0019] In an embodiment, the method further includes a step of configuring input data so that a specific paragraph among paragraphs included in each of the user query and the plurality of candidate documents forms a pair, and a step of processing the input data as an input of an encoder to specify the related document associated with the user query among the plurality of candidate documents, and in the step of determining the association with the user query, the association with the user query can be determined for each paragraph included in each of the plurality of candidate documents using an embedding vector corresponding to the input data obtained from the encoder.

[0020] In an embodiment, if there are multiple specific paragraphs, the input data may include multiple specific paragraphs.

[0021] In an embodiment, the encoder may perform an interaction between the user query included in the input data and the specific paragraph to obtain an embedding vector corresponding to the input data.

[0022] In an embodiment, when the input data includes the plurality of specific paragraphs, a plurality of embedding vectors corresponding to the input data can be obtained from the encoder.

[0023] In an embodiment, the step of determining the relevance with the user query may include the step of combining the plurality of embedding vectors into one to create a combined embedding vector, the step of processing the combined embedding vector as an input to a decoder, and the step of determining the relevance with the user query for each of the plurality of candidate documents using the combined embedding vector in the decoder.

[0024] In an embodiment, the combined embedding vector may represent the overall semantic association between the user query and the paragraphs included in each of the plurality of candidate documents.

[0025] In an embodiment, the decoder may generate an association word token for each of the plurality of candidate documents based on the association determined using the combined embedding vector.

[0026] In an embodiment, the related word token is a token indicating whether there is a relationship between the user query and the document, and may include a first related word token or a second related word token depending on whether there is a relationship between the user query and the document.

[0027] In an embodiment, the decoder may be configured to predict an associated word token for each of the plurality of candidate documents.

[0028] In an embodiment, a document including the first related word token among the plurality of candidate documents can be specified as the related document associated with the user query.

[0029] In an embodiment, when the input data includes a plurality of specific paragraphs, the step of obtaining a plurality of embedding vectors corresponding to the input data from the encoder and determining the relevance with the user query may include a step of processing the plurality of embedding vectors as inputs to a ranker layer, and a step of determining the relevance with the user query for each paragraph included in each of the plurality of candidate documents using the plurality of embedding vectors in the ranker layer.

[0030] In an embodiment, the ranker layer may generate a paragraph relevance score for each paragraph included in each of the plurality of candidate documents based on the relevance determined using the plurality of embedding vectors.

[0031] In an embodiment, the paragraph relevance score is a score indicating whether there is a relevance between each of the plurality of paragraphs and the user query, and may include a first score or a second score depending on whether there is a relevance between each of the plurality of paragraphs and the user query.

[0032] In an embodiment, the ranker layer may be trained to predict the relative similarity between paragraphs included in each of the plurality of candidate documents using a paragraph-to-paragraph attention mechanism for the plurality of embedding vectors.

[0033] In the embodiment, in the step of specifying, in order to generate an answer to the user query, a relevant paragraph related to the user query may be specified among paragraphs included in each of the plurality of candidate documents.

[0034] In the embodiment, in the step of specifying the related paragraph, a paragraph that satisfies a preset criterion among paragraphs included in each of the plurality of candidate documents is specified as the related paragraph, and the related paragraph may correspond to a paragraph included in the related document associated with the user query.

[0035] In an embodiment, in the step of generating the answer, an answer to the user query can be generated using at least one of the related document and the related paragraph.

[0036] A system for generating an answer according to the present invention, comprising a memory configured to store executable instructions and one or more processors configured to perform an operation by executing one or more instructions, receives a user query from a user terminal, specifies a plurality of candidate documents related to the user query based on preset criteria, determines a relevance to the user query for each passage included in each of the plurality of candidate documents and distinguished according to preset criteria, and specifies a related document related to the user query among the plurality of candidate documents based on the relevance determined for each paragraph included in each of the plurality of candidate documents and the user query, and generates an answer to the user query using the related document.

[0037] A computerized answer generation method according to the present invention may include the steps of: receiving a user query from a user terminal; specifying a plurality of candidate documents related to the user query based on preset criteria; specifying a related document related to the user query among the plurality of candidate documents based on a correlation between the user query and the user query determined for each paragraph included in each of the plurality of candidate documents; generating an answer to the user query using the related document; and providing the answer to the user terminal together with information on the related document used to generate the answer.

[0038] In an embodiment, in the step of providing the answer together with information about the related document, based on the user's history information, a related document related to the user's query among the plurality of candidate documents may be specified, and the answer generated using the specified related document and information about the related document used to generate the answer may be provided.

[0039] In the embodiment, in the step of generating the answer, a related paragraph to be used for generating the answer is specified among paragraphs included in each of the plurality of candidate documents, which is related to the user query, and at least one of the related document and the related paragraph is used to generate the answer.

[0040] In an embodiment, in the step of providing the answer together with information about the related document, information about the related document may be displayed on the user terminal so that the related document used to generate the answer can be confirmed.

[0041] In an embodiment, the information about the associated document may include at least one of the associated paragraph, the name of the associated document, and the storage location of the associated document.

[0042] In an embodiment, the specified storage may store at least one of the user query, an answer to the user query, a related document associated with the user query, and information about the related document.

[0043] In an embodiment, the method may further include a step of providing a plurality of recommended queries to the user terminal based on preset query recommendation criteria.

[0044] In an embodiment, in the step of providing the plurality of recommended queries, the plurality of recommended queries and the associated documents associated with each of the plurality of recommended queries may be provided to the user terminal, and each of the plurality of recommended queries may be displayed on the user terminal with a different visual appearance.

[0045] In an embodiment, the plurality of recommendation queries may include at least one of a first recommendation query having a first visual appearance and a second recommendation query having a second visual appearance different from the first visual appearance among the different visual appearances, and the associated document associated with each of the plurality of recommendation queries may include at least one of a first associated document associated with the first recommendation query and a second associated document associated with the second recommendation query.

[0046] In an embodiment, the associated document associated with each of the plurality of recommended queries is displayed on the user terminal with the same visual appearance as the visual appearance of each of the plurality of recommended queries so that the association with each of the plurality of recommended queries can be identified, and the first recommended query and the first associated document associated with the first recommended query can be displayed on the user terminal with the first visual appearance, and the second recommended query and the second associated document associated with the second recommended query can be displayed on the user terminal with the second visual appearance.

[0047] In an embodiment, when a specific recommendation query among the plurality of recommendation queries is selected from the user terminal, the user terminal may be provided with an answer to the specific recommendation query along with information about a specific related document used to generate an answer to the selected specific recommendation query among the related documents associated with each of the plurality of recommendation queries.

[0048] In an embodiment, information about a specific related document may be highlighted on the user terminal so that the user can confirm the specific related document used to generate an answer to a specific recommendation query.

[0049] In an embodiment, the information about the specific associated document may include at least one of a specific associated paragraph included in the specific associated document that is associated with the specific recommended query and used to generate an answer to the specific recommended query, a name of the specific associated document, and a storage location of the specific associated document.

[0050] In an embodiment, the method may further include a step of providing a plurality of recommended documents to the user terminal based on preset document recommendation criteria.

[0051] In an embodiment, in the step of providing the plurality of recommended documents, at least one of chat history information related to each of the plurality of recommended documents and information about each of the plurality of recommended documents may be provided to the user terminal.

[0052] In an embodiment, the chat history information may include at least one of a user query associated with each of the plurality of recommended documents and an answer generated using each of the plurality of recommended documents.

[0053] In a system including a memory configured to store executable instructions, and one or more processors configured to perform an operation by executing one or more instructions, the system may receive a user query from a user terminal, specify a plurality of candidate documents related to the user query based on preset criteria, specify a related document related to the user query among the plurality of candidate documents based on a correlation between the user query and the user query determined for each paragraph included in each of the plurality of candidate documents, generate an answer to the user query using the related document, and provide the answer to the user terminal together with information about the related document used to generate the answer.

[0054] A program according to the present invention is a program that is executed by one or more processes in an electronic device and can be stored in a computer-readable recording medium, and may include commands for performing the steps of: receiving a user query from a user terminal; specifying a plurality of candidate documents related to the user query based on preset criteria; specifying a related document related to the user query among the plurality of candidate documents based on a correlation between the user query and the user query determined for each paragraph included in each of the plurality of candidate documents; generating an answer to the user query using the related document; and providing the answer to the user terminal together with information on the related document used to generate the answer.

[0055] As discussed above, the answer generation method and system according to the present invention configures input data consisting of pairs of paragraphs included in a query and candidate documents, and independently encodes the input data to generate semantic representations, thereby enabling parallel processing of a large number of input documents. In other words, the present invention can optimize the semantic representation between paragraphs by encoding multiple paragraphs within an input document in parallel without separate sequential processing, thereby improving the input processing speed of the system and ensuring real-time responsiveness to large data sets.

[0056] Furthermore, the answer generation method and system according to the present invention integrates encoded semantic representations and uses the integrated semantic representations to predict the relevance of the entire document to the query through related word tokens, thereby accurately determining the relevance of the entire document. This enables the present invention to perform accurate document-level rankings that reflect the integrated meaning of multiple paragraphs, thereby improving document retrieval performance compared to conventional information retrieval systems.

[0057] Furthermore, according to the answer generation method and system according to the present invention, by applying an inter-paragraph attention mechanism to encoded semantic expressions to learn relative similarity between paragraphs, it is possible to distinguish semantic differences between multiple paragraphs and improve the precision and ranking consistency of paragraph-level ranking.

[0058] Furthermore, the answer generation method and system according to the present invention trains a model to simultaneously perform document reranking and paragraph extraction necessary for answer generation, thereby ensuring a balanced quality between document and paragraph ranking. This achieves highly reliable ranking results at both the document and paragraph levels, and maintains consistent performance across various search and answer generation systems.

[0059] Furthermore, the answer generation method and system according to the present invention can train a model to simultaneously determine the relevance between a query and an entire document and between a query and each of multiple paragraphs. In other words, the present invention can reduce the influence of noisy contexts among externally retrieved information by optimizing and ranking documents and paragraphs simultaneously, thereby improving the accuracy and reliability of LLM responses generated based on this. Even in a real-time RAG system environment where large-scale information retrieval occurs, rapid and stable search and response processing is possible, errors that may occur in LLM responses can be prevented, and user reliability can be enhanced.

[0060] Furthermore, the answer generation method and system according to the present invention can reduce unnecessary computation and optimize computational resource usage through a multi-granular ranking structure via a single inference path. This can reduce server infrastructure costs and improve the operational efficiency of large-scale AI search systems.

[0061] In addition, according to the method and system for generating an answer according to the present invention, by providing information on a related document related to a user's query and an answer generated using the related document together, the user can intuitively recognize the information he or she needs and understand it more quickly.

[0062] Furthermore, according to the answer generation method and system according to the present invention, by visually providing information about documents utilized in the process of generating answers, the user can intuitively confirm the source and reliability of the answer, quickly and accurately receive the information he or she needs, and maximize the efficiency of searching related documents.

[0063] Furthermore, the answer generation method and system according to the present invention can select relevant documents related to a user's query based on user history information and provide answers to the user's query based on these documents. In other words, the present invention provides users with customized information that reflects their preferences, interests, LLM usage patterns, etc., thereby significantly improving the personalized performance of the answer generation service and the quality of interaction with the user.

[0064] Figure 1 is a conceptual diagram for explaining an answer generation system according to the present invention.

[0065] Figure 2 is a flowchart for explaining a method for generating an answer according to the present invention.

[0066] Figures 3a and 3b are conceptual diagrams for explaining a learning method of an information retrieval model according to the present invention.

[0067] FIG. 4, FIG. 5, FIG. 6, and FIG. 7a and FIG. 7b are conceptual diagrams for explaining a method for generating an answer according to the present invention.

[0068] Figures 8 and 9 are formulas related to a method for generating an answer according to the present invention.

[0069] Figures 10, 11 and 12 are tables and graphs showing one example of learning results of an information retrieval model learned using a learning method according to the present invention.

[0070] Figure 13 is a flowchart illustrating a method for generating an answer according to the present invention.

[0071] Figures 14a, 14b, 15a, 15b, 16a, 16b and 17 are conceptual diagrams for explaining a method for generating an answer according to the present invention.

[0072] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0073] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0074] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0075] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0076] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0077] The present invention relates to a method and system for generating answers. The answer generation system according to the present invention performs answer generation based on generative artificial intelligence (Generative AI) or a Foundation model, and may also be referred to as an answer generation platform based on a super-large Foundation model. However, the "super-large Foundation model" may also be referred to as a generative model, a Foundation model, or a Large Language Model (LLM).

[0078] The present invention may be a system that more efficiently improves a Retrieval Augmented Generation (RAG) system that provides appropriate answers to user queries by integrating external knowledge.

[0079] In one embodiment, a conventional RAG system may comprise a document retriever, a reranker module, and a decoder for generating a final response. When a user query is input, a conventional RAG system utilizes a document retriever to retrieve the top N candidate documents relevant to the query from a large document corpus (or a large knowledge base). For example, the document retriever may include a BM25 or Dense Retriever. The reranker performs independent model inference in each component to rank candidate documents and paragraphs, and may further extract and reorder paragraphs with high relevance within the reranked documents. The decoder inputs the reranked (or selected) documents or paragraphs into an answer generation model, which then generates a final answer (or response).

[0080] In this way, while the conventional RAG system (100) requires independent model inference in each component to rank documents and paragraphs, the present invention proposes an information retrieval model (or search model, document retrieval model, etc.) that can rank documents and paragraphs simultaneously with a single inference by utilizing the Passage Fusion-in-Decoder (PFiD) structure.

[0081] The answer generation system according to the present invention may be a system that efficiently searches (or extracts) documents and paragraphs related to (or associated with) a query (or query) and generates and provides a more accurate and relevant answer to the query. To this end, the answer generation system according to the present invention includes an information retrieval model (or retrieval model, document retrieval model, etc.) that simultaneously searches for documents related to the query and extracts paragraphs related to the query from the searched documents. The present invention aims to simultaneously improve search efficiency and content generation accuracy.

[0082] Hereinafter, we will look into it in more detail with the attached drawings. Fig. 1 is a conceptual diagram for explaining an answer generation system according to the present invention. Fig. 2 is a flowchart for explaining an answer generation method according to the present invention, and Figs. 3a and 3b are conceptual diagrams for explaining a learning method of an information retrieval model according to the present invention. Figs. 4, 5, 6, and 7a and 7b are conceptual diagrams for explaining an answer generation method according to the present invention, and Figs. 8 and 9 are formulas related to the answer generation method according to the present invention. Furthermore, Figs. 10, 11, and 12 are tables and graphs showing an example of a learning result of an information retrieval model learned using a learning method according to the present invention.

[0083] Meanwhile, as illustrated in FIG. 1, the answer generation system (100) according to the present invention may include at least one of an input unit (110), an output unit (120), a communication unit (130), a storage unit (140), a document search unit (150), an information search model (160), an answer generation unit (170), and a control unit (180). However, the components of the answer generation system (100) according to the present invention are not limited thereto, and may further include various hardware components that perform roles identical or similar to the functions described in this specification.

[0084] The answer generation system (100) according to the present invention may include at least one processor and at least one memory containing computer program code. In this case, the memory may function as the storage unit (140) described above. In the present invention, the memory and the program code may cooperate with the processor to perform a series of processes described below.

[0085] Although not shown, the answer generation system (100) according to the present invention may include one or more processors, which may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., a digital signal processor, a tensor processing unit (TPU), a graphics processing unit (GPU), a neural network processing unit (NPU), an application-specific integrated circuit, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a quantum processing unit (or quantum processor, QPU), etc.). The one or more processors may be configured to execute instructions stored (or included) in the storage unit (140), computer-readable instructions, and / or other instructions described herein. The answer generation method and system according to the present invention may allow the memory and at least one processor to cooperate to perform data processing as described below. The processor may perform a series of operations and data processing using data and information stored in the memory. At this time, the memory may be a component of the storage unit (140).

[0086] Furthermore, the answer generation system (100) according to the present invention can perform data processing and calculation processes utilizing quantum gates, quantum entanglement, and quantum superposition states, taking into account implementation in a quantum computer environment. For example, the present invention can perform parallel operations based on qubits, and such quantum operations can operate complementarily with existing classical computers.

[0087] These quantum computers may include qubit-based parallel computing and high-speed data processing devices leveraging quantum entanglement, enabling hardware-based computational optimization using FPGAs and ASICs. Furthermore, quantum computers can utilize quantum processors capable of qubit-based parallel computing, and data processing efficiency can be improved through hybrid architectures with existing classical computers.

[0088] Meanwhile, the input unit (110) may be configured as a means for data input and may be configured in various types. For example, the input unit (110) may be configured to receive user input. The input unit (110) may be configured to receive user input from a user terminal (10). Here, “receiving input” may mean receiving an input signal (or selection signal) corresponding to the user’s input based on the input being made by the user through the input unit configuration provided in the user terminal (10).

[0089] Here, the user terminal (10) may include at least one of a mobile phone, a smart phone, a notebook computer, a laptop computer, a slate PC, a tablet PC, an ultrabook, a desktop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, and a wearable device (e.g., a smartwatch, a smart glass, a head mounted display (HMD)).

[0090] In addition, in the present invention, the input unit (110) does not necessarily mean a hardware means, and can be understood as a passage for receiving input from a user.

[0091] The input unit (110) may also be referred to as a user interface module. The input unit (110) may include a touch screen, computer mouse, keyboard, keypad, touch pad, trackball, joystick, voice recognition module, or other similar device. However, the present invention does not limit the type of input unit (110).

[0092] Here, the user input may include documents, text, images (or videos), voice, etc. In this case, the answer generation system (100) may further include a module that converts voice into text.

[0093] Next, the output unit (120) can output information through an output unit configuration (e.g., a display unit, a touch screen, a speaker, etc.) provided in a user terminal (10) linked to the answer generation system (100) according to the present invention. For example, the output unit (120) can output a page (or service page, 1000) linked to the answer generation system (100) according to the present invention to the display unit of the user terminal (10). In addition, the output unit (120) does not necessarily mean a hardware means, and can be understood as a passage for outputting results to the user.

[0094] Next, the communication unit (130) may be connected to a user terminal (10), a server (e.g., a central server, an external server, etc.), a device, and at least one network through a wireless or wired network, and may be configured to receive or transmit overall data and information necessary for the operation of the answer generation system (100) according to the present invention.

[0095] The communication unit (130) can support various communication methods depending on the communication standards of the communicating device.

[0096] For example, the communication unit (130) may be configured to communicate with a communication target using at least one of WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

[0097] Next, the storage unit (140, or memory) serves to store various data related to the present invention, and may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of the one or more processors.

[0098] The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage devices. In some examples, the storage (140) may be implemented using a single physical device (e.g., a single optical, magnetic, organic, or other memory or disk storage device), while in other examples, the storage (140) may be implemented using two or more physical devices.

[0099] The storage unit (140) may include computer-readable instructions and additional data. The storage unit (140) may include storage necessary to perform at least some of the methods, scenarios, and techniques described herein and / or at least some of the functions of the devices and networks.

[0100] Furthermore, at least a portion of the storage unit (140) may be a cloud storage or a cloud server. The storage unit (140) may store data corresponding to user input received from the input unit (110) and at least a portion of a learning data set (or learning data, 200). For example, the learning data set (200) may be configured to include a plurality of queries, a plurality of documents, and paragraphs included in each of the plurality of documents.

[0101] That is, it can be understood that the storage unit (140) is sufficient as a space in which information necessary for the operation of the answer generation system (100) according to the present invention is stored, and there are no restrictions on the physical space.

[0102] Furthermore, the storage unit (140) may store a computer program including computer program instructions. Furthermore, the storage unit (140) may store a computer program including computer program instructions that control the operation of the system (100) or control the operation of the control unit (180) when loaded onto the processor of the system (100).

[0103] Next, the document search unit (150, or document searcher) may be configured to sort documents or paragraphs based on their relevance to the query. The document search unit (150) may filter at least one candidate document related to the query by utilizing (or using, utilizing, etc.) an artificial intelligence model or algorithm. In this case, it may also be understood that the document search unit (150) includes (or stores) the artificial intelligence model or algorithm.

[0104] For example, the artificial intelligence model or algorithm used in the document retrieval unit (150) may include at least one of Best Matching 25 (BM25) and Dense Passage Retrieval (DPR). BM25 may be an algorithm that calculates a relevance score of a document based on the word frequency (TF), inverse document frequency (IDF), document length, etc. in the query and the document. In addition, DPR is a deep learning-based model that embeds a query with one encoder and a document with another encoder, and then measures relevance by the vector similarity (e.g., inner product) between the vectors embedded in each of the two encoders. This DRP uses a bi-encoder structure based on a language model (e.g., BERT, RoBERTa, etc.). However, the artificial intelligence model or algorithm used in the document retrieval unit (150) is not necessarily limited thereto, and any artificial intelligence model or algorithm that performs a role identical or similar to the function of the document retrieval unit (150) may be used.

[0105] Next, the information retrieval model (160) can be configured to simultaneously retrieve documents related to the query and extract paragraphs related to the query within the retrieved documents. That is, the information retrieval model (160) can simultaneously perform document reranking and paragraph extraction necessary for content generation. In the present invention, the information retrieval model (160) may also be referred to as a "Passage Fusion-in-Decoder (PFiD)."

[0106] Here, document reranking may mean a process of sorting (or adjusting) the rank (priority) of multiple candidate documents retrieved in the initial search process in relation to a user query (e.g., multiple candidate documents initially retrieved through the document retrieval unit (150)) based on their relevance to the user query.

[0107] For example, when a plurality of candidate documents related to a query are retrieved through the document retrieval unit (150), the information retrieval model (160) can retrieve at least one related document related to the query among the plurality of candidate documents, and simultaneously extract a paragraph related to the query among paragraphs included in the related documents.

[0108] This information retrieval model (160) may be configured to include at least one of an encoder (161), a decoder (162), and a ranker layer (163) (see FIGS. 3A and 3B ). Furthermore, in the present invention, the information retrieval model (160) may be a model trained using a training data set (200) stored in a storage unit (140, or memory). A more detailed description of the training method of the information retrieval model (160) will be provided below.

[0109] Next, the answer generation unit (170) may be configured to perform the role of generating an answer to a query. The answer generation unit (170) may generate an answer to a query using related documents and paragraphs associated with the query specified based on the output results of the information retrieval model (160).

[0110] For example, the answer generation unit (170) may include at least one of a generative model, a foundation model, and a large-scale language model to generate an answer to a query. Alternatively, the answer generation unit (170) itself may be understood to correspond to at least one of a generative model (or a generative AI model), a foundation model, and a large-scale language model.

[0111] Next, the control unit (180) may perform a role of controlling the overall operation of the answer generation system (100) related to the present invention. The control unit (180) may process signals, data, information, etc. input or output through the components of the answer generation system (100) discussed above, or perform a series of data processing to provide or process appropriate information and functions to the user. The control unit (180) may be physically implemented by the processor discussed above.

[0112] Meanwhile, the present invention provides a method and system for generating answers that can simultaneously improve search efficiency and content creation accuracy. More specifically, the present invention provides an information retrieval model that simultaneously performs document reranking and paragraph extraction necessary for content creation, thereby improving search efficiency and content creation accuracy. Below, the learning method for the information retrieval model will be examined in more detail.

[0113] As discussed above, the information retrieval model (160) according to the present invention can be trained using a learning data set (200).

[0114] In this regard, the present invention can construct (or generate) a learning data set (200) for learning an information retrieval model (160). The control unit (180) can construct a learning data set (200) that enables the information retrieval model (160) to simultaneously learn the relevance of document units (or levels, levels, etc.) and paragraph units (or levels, levels, etc.). More specifically, the control unit (180) can construct a learning data set (200) that enables the information retrieval model (160) to simultaneously perform relevance determination for the entire document and relevance determination for each paragraph included in the document through a single learning session. In particular, the present invention can construct a learning data set (200) by setting a document including at least one related paragraph (or positive paragraph) as a related document (or positive document) (see (a) of FIG. 10).

[0115] The training data set (200) may be configured to include at least one of a query, a document, a label (or label) for the entire document, and a label for each paragraph contained in the document. The label for the entire document may be assigned based on whether the document is relevant to the query. Additionally, the label for each paragraph contained in the document may be assigned based on whether each paragraph is relevant to the query.

[0116] Here, the label for a document may include multiple different labels. For example, the label for a document may include a first label (“1 (True)”) or a second label (“0 (False)”) depending on whether a paragraph contained in the document is relevant to the query.

[0117] Additionally, labels for paragraphs (i.e., paragraph-specific labels) may include multiple different labels. For example, labels for paragraphs may include a first label ("1") or a second label ("0"), depending on whether each paragraph in the document is relevant to the query.

[0118] In one embodiment, if there is a related paragraph (e.g., a paragraph having a passage relevance score of “1” or a paragraph having a paragraph label of “1”) among paragraphs included in a plurality of documents that is related to a query, the control unit (180) may perform labeling (or labeling) on ​​a document including the related paragraph so that the document is assigned a first label among the plurality of labels. A document assigned a first label may also be referred to as a “related document” or a “positive document.”

[0119] Alternatively, if there is no paragraph related to the query among paragraphs included in a specific document (e.g., if the paragraph relevance scores of the paragraphs are all “0” or if the labels of the paragraphs are all “0”), the control unit (180) may perform labeling on the document so that the document is assigned a second label among multiple labels. A document assigned a second label may also be named a “non-related document” or a “negative document.”

[0120] That is, the control unit (180) can perform labeling on a document so that a first label among a plurality of labels is assigned to a document that includes at least one associated paragraph related to the query (e.g., document A: paragraph 1 (0), paragraph 2 (1), paragraph 3 (0)). On the other hand, the control unit (180) can perform labeling on a document so that a second label among a plurality of labels is assigned to a document that does not include any associated paragraph related to the query (e.g., document B: paragraph 1 (0), paragraph 2 (0), paragraph 3 (0)). Through this, for one document, a document-level association label and a paragraph-level association label for each paragraph within the document are assigned together, so that a learning data set (200) capable of simultaneously learning document-level association and paragraph-level association can be constructed.

[0121] In this way, the present invention constructs a learning environment in which an information retrieval model (160) can simultaneously perform relevance judgments for the entire document and relevance judgments for each paragraph in a single learning session by configuring a single data set to include labels for the entire document and labels for each paragraph included in the document.

[0122] Meanwhile, the control unit (180) can perform learning (e.g., supervised learning) on ​​an information retrieval model using the constructed learning data set (200).

[0123] Given a query and multiple candidate documents (or sets of paragraphs), the goal of the information retrieval model (160) is to extract k most relevant documents among the multiple candidate documents that are most relevant to the query (see (a) and (b) of FIG. 8). For example, assume that a query included in a learning data set (200) and a set of k paragraphs included in at least one candidate document are given (see (c) of FIG. 8). The control unit (180) can configure (or generate) input data of the information retrieval model (160) that includes the query and the k paragraphs.

[0124] Specifically, the control unit (180) may configure input data that pairs specific paragraphs included in a query and multiple candidate documents. The control unit (180) may configure the input data such that specific paragraphs included in each of the query and multiple candidate documents form a pair. In this case, if there are multiple specific paragraphs, the input data may include multiple (or multiple) specific paragraphs.

[0125] In one embodiment, as illustrated in FIGS. 3A and 3B , the control unit (180) may configure input data (first input data, 301) that pairs a query (300) with a first paragraph (301a) among paragraphs included in a plurality of candidate documents. In addition, the control unit (180) may configure input data (second input data, 302) that pairs a query (300) with a second paragraph (302a) among paragraphs included in a plurality of candidate documents. In addition, the control unit (180) may configure input data (third input data, 303) that pairs a query (300) with a third paragraph (303a) among paragraphs included in a plurality of candidate documents.

[0126] Here, the size (or length) of paragraphs included in the input data can be distinguished according to preset criteria. The preset criteria can be set in various ways, taking into account the efficiency of the information retrieval model (160) (e.g., considering a stable size that can be input to the encoder (161). For example, i) the number of tokens included in one paragraph can be set to 256 tokens or 522 tokens, or ii) a specific paragraph can be distinguished by 256 tokens, and another paragraph can be distinguished in a direction that overlaps some tokens of the specific paragraph. It goes without saying that such preset criteria can be set in various ways by the administrator (or user) of the answer generation system (100) or the answer generation system (100) itself.

[0127] The control unit (180) may process the input data (301, 302, 303) as input to the encoder (Encoder, 161) of the information retrieval model (160) to specify a related document related to the query among a plurality of candidate documents. The encoder (161) outputs an embedding for the kth paragraph (see (d) of FIG. 8), where L may represent the maximum token length and d may represent the dimension of the hidden state. More specifically, the encoder (161) may be configured to perform an interaction between the query (300) included in the input data (301, 302, 303) and specific paragraphs (301a, 302a, 303a) to generate an embedding vector (311, 312, 313) corresponding to the input data (301, 302, 303). In the present invention, the encoder (161) may also be referred to as a “cross-encoder.” Furthermore, the embedding vector corresponding to the output of the encoder (161) in the present invention may also be referred to as a “hidden vector,” a “hidden state vector,” a “feature vector,” a “semantic representation,” or a “semantic representation vector.”

[0128] Here, the embedding vectors (311, 312, 313) can indicate how semantically related the query (300) and the specific paragraphs (301a, 302a, 303a) are based on the similarity of semantic expressions between the query (300) and the specific paragraphs (301a, 302a, 303a). The embedding vectors (311, 312, 313) are information that numerically expresses what semantic relationship (or semantic relationship, semantic association, etc.) the query (300) and the paragraphs (301a, 302a, 303a) have with each other, and can be understood as an n-dimensional vector that compressively expresses (or summarizes) the similarity between the query and the paragraph, the related topic, the likelihood that the paragraph will provide an answer, the degree to which the information required in the question matches the paragraph, the semantic distance, etc. That is, it can be a vector that includes how meaningful the paragraphs (301a, 302a, 303a) paired with the query (300) are for the query (300).

[0129] In one embodiment, let us assume that the query contained in the input data is “What does PFID stand for?” and the paragraph is “PFID stands for Passage Fusion-in-Decoder.” When the input data is input to the encoder (161), the encoder (161) can analyze the semantic association and / or semantic features of the input data to generate an embedding vector exhibiting high association (high association embedding vector).

[0130] In another embodiment, assume that the query contained in the input data is “What does PFID stand for?” and the paragraph is “RAG stands for Retrieval-Augmented Generation, which is a natural language processing (NLP) technology that combines information retrieval and generation models.” When the input data is input to the encoder (161), the encoder (161) can analyze the semantic association and / or semantic features of the input data to generate a vector indicating low association (low association embedding vector).

[0131] In this way, even if the query is the same, the vector results generated (or output) may differ depending on the paragraph paired with the query.

[0132] The control unit (180) inputs a plurality of input data (301, 302, 303) each including a query (300) and a plurality of specific paragraphs (301a, 302a, 303a) to the encoder (161), and can obtain an embedding vector (311, 312, 313) corresponding to each of the plurality of input data (301, 302, 303) from the encoder (161).

[0133] And, the control unit (180) can use a plurality of embedding vectors (311, 312, 313) obtained from the encoder (161) to determine the relevance with the query (300) for each paragraph (301a, 302a, 303a) included in each of a plurality of candidate documents and distinguished according to preset criteria.

[0134] In this regard, the information retrieval model (160) of the present invention can be trained to simultaneously perform document reranking and paragraph extraction necessary for answer (or content) generation. More specifically, the information retrieval model (160) can be trained to simultaneously determine the relevance between a query and the entire document and the relevance between the query and each of multiple paragraphs.

[0135] First, the control unit (180) can process multiple embedding vectors (311, 312, 313) as inputs to the decoder (Decoder, 162) of the information retrieval model (160) to determine the correlation between multiple candidate documents and the query (300).

[0136] At this time, the control unit (180) can combine (or concat) multiple embedding vectors (311, 312, 313) for each of multiple candidate documents to determine the relevance with the query (300), generate a combined embedding vector (320), and input the combined embedding vector (320) to the decoder (162) (see (f) of FIG. 8).

[0137] When the control unit (180) independently (or individually) completes encoding of input data (e.g., multiple input data (301, 302, 303)) including paragraphs included in each of the query and multiple candidate documents, the control unit (180) can combine the encoding results of paragraphs belonging to the same candidate document into one vector (or sequence) to generate a combined vector per document.

[0138] More specifically, the control unit (180) uses the encoder (161) to obtain an embedding vector (311, 312, 313) for each of input data (e.g., multiple input data (301, 302, 303)) including paragraphs included in each of the query and multiple candidate documents, and sequentially combines the embedding vectors belonging to a specific candidate document to generate a combined embedding vector (320) corresponding to the specific document. Thereafter, the combined embedding vector (320) is used as an input of the decoder (162), and enables determination of relevance with the query and comparison between paragraphs for each document.

[0139] At this time, paragraphs contained in each of the multiple candidate documents are processed independently, and vector combinations can be performed for each candidate document. Through this, the present invention can improve document-level ranking accuracy and efficiency by preventing paragraph information from being mixed between documents. For example, the control unit (180) can independently and repeatedly perform vector combination and decoding processes on a document-by-document basis for each candidate document, and process the encoding results of paragraphs across different documents to prevent mixing. Through this, the semantic expression is preserved for each document, and the interrelationships between paragraphs are considered only within the document, thereby improving the accuracy of document-by-document relevance determination.

[0140] This combined embedding vector (320) can represent the entire semantic association between the query (300) and the paragraphs (301a, 302a, 303a) included in each of the plurality of candidate documents. The combined embedding vector (320) can be compressed data of all relationships between the paragraphs (301a, 302a, 303a) included in each of the plurality of candidate documents and the query (300). The process of combining the embedding vectors can be a process of connecting all paragraphs in the document into a single long sequence of what meaning they have for the query so that the decoder (162) can comprehensively determine whether the entire document is related to the query. In other words, the combined embedding vector (320) is information that comprehensively reflects the relationship between all paragraphs included in the document and the query, and the decoder (162) can be trained to make a final judgment on the association between the entire document and the query based on this information. In the present invention, the combined embedding vector (320) may also be referred to as “combined embedding vector sequence” or “combined vector sequence.”

[0141] The decoder (162) of the information retrieval model (160) can be trained to determine the relevance of each of a plurality of candidate documents to a query (300) using the combined embedding vector (320).

[0142] For example, the decoder (162) can recognize the combined embedding vector (320) as a summary of the document, and analyze the association between paragraphs, the likelihood of answering a query generated by a combination of paragraphs, and the hidden dependency between paragraphs through an attention operation through the combined embedding vector (320). Finally, the decoder (162) can be trained to classify whether a specific document is a related document related to a query or not.

[0143] That is, the decoder (162) can recognize the combined embedding vector (320) as a sequence-to-label problem and softly learn which paragraphs contribute significantly to document relevance using an attention mechanism. Here, attention can be a process of automatically extracting which paragraphs among multiple paragraphs included in a document are more important. Consequently, the output of the decoder (162) can also be understood as a collective judgment result on the overall consistency between the document and the query. That is, the decoder (162) can be trained to give high attention to relevant paragraph vectors and ignore irrelevant paragraph vectors.

[0144] In this way, in the present invention, the decoder (162) can be trained to receive a combined vector (320) as input and generate document-level related word tokens (e.g., True or False, 330).

[0145] Here, the relevance word token (330) may mean a token indicating whether there is a relevance between the query (300) and the document. The relevance word token (330) may be configured to include a first relevance word token (e.g., “True”) or a second relevance word token (e.g., “False”) depending on whether there is a relevance between the query (300) and the document. The first relevance word token (e.g., “”) may indicate that the document is a related document that is related to the query (300), and the second relevance word token (e.g., “”) may indicate that the document is a document that is not related to the query (300).

[0146] Furthermore, a loss function (or first loss function) for learning an information retrieval model (160, or decoder (162)) that predicts (or generates) relevant word tokens (330) for each of a plurality of candidate documents may be represented as in (g) of FIG. 8. This first loss function may be a loss function for learning and evaluating how accurately the decoder (162) generates a correct token sequence (i.e., correct relevant word tokens) based on information integrated (or fused) between paragraphs within a document.

[0147] This first loss function can be understood as a loss function for reducing the loss of generating (or predicting) related word tokens for determining relatedness for the entire document (or a set of all paragraphs included in the document). The first loss function is used in a learning process for determining relatedness at the document level, and the decoder (162) can be trained to minimize (or reduce) the difference (or error) between the related word tokens predicted (or generated) for the document (or generated related word tokens, predicted related word tokens, etc.) and the correct related word tokens (or target related word tokens) through the first loss function.

[0148] That is, the control unit (180) can calculate the loss between the correct related word token for the document and the generated related word token generated by the decoder (162) using the loss function shown in (g) of FIG. 8, and perform learning on the decoder (162) so that the loss is reduced.

[0149] In this way, the control unit (180) trains the decoder (162) of the information retrieval model (160) in a direction of reducing the loss of the first loss function, so that the decoder (162) can generate a correct token sequence such as “true” or “false” by utilizing paragraph information close to the correct answer among the given paragraphs.

[0150] Next, the control unit (180) may process the plurality of embedding vectors (311, 312, 313) as inputs to the ranker layer (or module, 163) of the information retrieval model (160) to determine the relevance between the paragraphs (301a, 302a, 303a) included in each of the plurality of candidate documents and the query (300). In the present invention, the ranker layer (163) may also be named as a “paragraph-to-paragraph attention layer (or module)” or a “paragraph relevance scoring layer (or module)” or a “paragraph relevance modeling layer (or module)” or a “self-attention layer (or module)”.

[0151] In the present invention, the ranker layer (163) can be trained to determine the relevance with the query (300) for each paragraph (301a, 302a, 303a) included in each of the plurality of candidate documents using a plurality of embedding vectors (311, 312, 313). More specifically, the ranker layer (163) can be trained to generate (or predict, produce, etc.) a paragraph relevance score (score, or score, 340) for each paragraph (301a, 302a, 303a) included in each of the plurality of candidate documents based on the relevance determined using a plurality of embedding vectors (311, 312, 313).

[0152] Here, the paragraph relevance score (340) may be a score indicating whether there is a relevance between the paragraphs (301a, 302a, 303a) included in each of the plurality of candidate documents and the query (300). The paragraph relevance score (340) may be configured to include a first score (e.g., “”) or a second score (e.g., “”) depending on whether there is a relevance between each of the plurality of paragraphs (301a, 302a, 303a) and the query (300). The first score (e.g., “”) may indicate that the paragraph is a related paragraph that has a relevance to the query (300), and the second score (e.g., “”) may indicate that the paragraph is a paragraph that has no relevance to the query (300).

[0153] This ranker layer (163) can be trained to predict the relative similarity between paragraphs (301a, 302a, 303a) included in each of a plurality of candidate documents by utilizing (or applying) an inter-passage attention mechanism (e.g., inter-passage attention) for a plurality of embedding vectors (311, 312, 313). Here, the relative similarity can mean a measure (or relationship) that expresses, on a relative basis, how much more relevant each paragraph is to the query compared to other paragraphs among a plurality of paragraphs given for the same query.

[0154] More specifically, the ranker layer (163) learns the relationship indicating how similar each paragraph is to each other, and through this, the information retrieval model (160) can recognize and evaluate not only the relevance of individual paragraphs, but also the relative importance or similarity between multiple paragraphs. For example, the ranker layer (163) can not only evaluate the relevance between paragraphs (301a, 302a, 303a) included in each of the multiple candidate documents and the query (300), but can also compare and learn the relative similarity and / or importance, such as which paragraph among the multiple paragraphs (301a, 302a, 303a) is more important for the query (300) or which paragraph is less important for the query. Such comparison can be performed using an attention mechanism between paragraphs.

[0155] In this regard, the control unit (180) can utilize the inter-paragraph attention mechanism discussed above to explicitly model the interaction between paragraphs (301a, 302a, 303a) included in each of the plurality of candidate documents. The control unit (180) can construct a set of input sequences by concatenating the embeddings of the first tokens for each pair (see (a) of FIG. 9). Here, the embedding of the j-th token of the i-th paragraph can be represented as in (b) of FIG. 9. The control unit (180) can also extend these tokens to express relative meaning between paragraphs through a self-attention mechanism.

[0156] The control unit (180) considers a single-layer transformer model to represent (or model) the relative relevance between paragraphs, which may be represented as in (c) of FIG. 9. For example, the ranker layer (163) is based on the self-attention structure of the transformer and may have a structure that explicitly learns the relative similarity between paragraphs through interactions between the first token hidden states of each paragraph.

[0157] Here, learnable parameters (or parameter matrices) can be represented as in (d) of Fig. 9, and inter-paragraph information can be mutually fused and exchanged through a self-attention mechanism.

[0158] Furthermore, the loss function (or second loss function) used for paragraph-to-paragraph attention learning of the information retrieval model (160, or ranker layer (163)) can be represented as in (e) of Fig. 9. Here, y can represent a paragraph-to-paragraph relevance label. This second loss function is for paragraph-to-paragraph relative relevance learning and can be understood as a loss function in a binary classification problem that predicts whether a paragraph is related to a query. The ranker layer (163) can perform paragraph-to-paragraph attention-based learning through the second loss function.

[0159] More specifically, the second loss function may be a loss function for increasing the accuracy of determining relevance at the paragraph level, and may be a loss function for allowing the ranker layer (163) to learn by clearly distinguishing which paragraphs are relevant or not for a query. In order to increase the ranking performance at the paragraph level, the control unit (180) may use the second loss function to train the ranker layer (163) in a direction that minimizes the error (or difference) between the correct label (y) of each paragraph and the predicted probability.

[0160] In order to increase the accuracy of judging the relative relevance of paragraphs for a query, the ranker layer (163) can be trained to minimize (or reduce) the difference between the paragraph relevance score predicted (or generated) for each paragraph and the paragraph relevance score corresponding to the correct answer (or target) through a second loss function that reduces the binary classification error for each paragraph (related paragraph (e.g., “”) or unrelated paragraph (e.g., “”)).

[0161] That is, the control unit (180) can calculate the loss between the paragraph relevance score predicted by the ranker layer (163) and the paragraph relevance score corresponding to the correct answer (or target) for each paragraph of the query using the second loss function shown in (e) of FIG. 9, and perform learning on the ranker layer (163) so that the loss is reduced.

[0162] In this way, the ranker layer (163) can generate a probability distribution (e.g., “- related” or “- not related”) that determines whether each paragraph included in the document is related to the query or not by performing learning so that the predicted probability for determining the relationship between the paragraph and the query becomes closer to the actual correct answer.

[0163] Meanwhile, the loss function (or third loss function, final loss function, etc.) for the final learning of the information retrieval model (160) including the encoder (161), decoder (162), and ranker layer (163) discussed above can be represented as in (f) of Fig. 9. The final loss function can also be understood as the final objective function (or target function) for learning multi-granularity ranking of the information retrieval model (160).

[0164] This final loss function may be a global loss function for simultaneously learning document relevance judgments and paragraph relevance judgments. More specifically, the final loss function may be a loss function aimed at training the information retrieval model (160) to simultaneously accurately predict "whether a document is relevant to the query" and "which paragraph is key." Here, the hyperparameters controlling the two losses (e.g., the document relevance judgment results and the paragraph relevance judgment results) may be represented as shown in (g) of Fig. 9.

[0165] That is, the information retrieval model (160) can be trained to accurately rank related documents and related paragraphs simultaneously through a final loss function. Through this learning process, the present invention can simultaneously improve search efficiency and accuracy by processing document reranking and paragraph extraction as a single integrated process, and obtain an information retrieval model (160) optimized for paragraph-based information extraction and enhanced RAG connectivity.

[0166] Meanwhile, the method and system for generating an answer including an information retrieval model (160) learned according to the learning method described above, as shown in FIG. 2, in the inference stage, can provide a more accurate and highly relevant answer to a query to a user through a process (S210) of receiving a user query from a user terminal, a process (S220) of specifying a plurality of candidate documents related to the user query based on preset criteria, a process (S230) of determining a relevance to the user query for each passage included in each of the plurality of candidate documents and distinguished according to preset criteria, a process (S240) of specifying a related document related to the user query among the plurality of candidate documents based on the relevance determined for each paragraph included in each of the user query and the plurality of candidate documents, and a process (S250) of generating an answer to the user query using the related documents.

[0167] As illustrated in FIG. 4, when a user query (400) input from a user terminal (10) through a service page (1000) is received by the control unit (180), the control unit (180) may specify a plurality of candidate documents (401, 402) related to the user query (400) based on preset criteria. Here, the preset criteria may be related to a search criterion (e.g., BM25 score) of a document search unit (150) that initially specifies documents related to the query from the entire document set (or large-scale document corpus, learning data set, etc.) based on the user query (400). Each of the plurality of documents (401, 402) may be configured to include at least one paragraph (or a plurality of paragraphs).

[0168] In addition, the control unit (180) can configure the input data so that specific paragraphs among the paragraphs included in each of the user query (400) and the plurality of candidate documents (401, 402) form a pair. In this case, if there are multiple specific paragraphs, the input data can include multiple (or multiple) specific paragraphs.

[0169] For example, the control unit (180) can configure input data (first input data, 411) that pairs a user query (400) and a first specific paragraph (411a) among paragraphs included in each of the plurality of candidate documents (401, 402). In addition, the control unit (180) can configure input data (second input data, 412) that pairs a second specific paragraph (412a) among paragraphs included in each of the plurality of candidate documents (401, 402). In addition, the control unit (180) can configure input data (Nth input data, 413) that pairs a user query (400) and an Nth specific paragraph (413a) among paragraphs included in each of the plurality of candidate documents (401, 402).

[0170] Next, as illustrated in FIG. 5, the control unit (180) may input input data (411, 412, 413) into the encoder (161) of the information retrieval model (160) to specify a related document associated with a user query (400) among a plurality of candidate documents (401, 402). The encoder (161) may perform an interaction between the user query (400) included in the input data (411, 412, 413) and a specific paragraph (411a, 412a, 413a), thereby obtaining an embedding vector corresponding to the input data (411, 412, 413). At this time, based on the fact that each of the input data (411, 412, 413) includes a plurality of specific paragraphs (411a, 412a, 413a), the embedding vectors obtained from the encoder (161) may be plural (or multiple). That is, based on the fact that the plurality of input data (411, 412, 413) are input to the encoder (161), the control unit (180) can obtain the embedding vectors (421, 422, 423) corresponding to each of the plurality of input data (411, 412, 413) from the encoder (161).

[0171] Here, the embedding vectors (421, 422, 423) can indicate how semantically related the user query (400) and the specific paragraphs (411a, 412a, 413a) are based on the similarity of semantic expressions between the user query (400) and the specific paragraphs (411a, 412a, 413a). The embedding vectors (421, 422, 423) are information that numerically expresses what semantic relationship (or semantic relationship, semantic association, etc.) the user query (400) and the specific paragraphs (411a, 412a, 413a) have with each other, and can be understood as an n-dimensional vector that compressively expresses (or summarizes) the similarity between the query and the paragraph, the related topic, the likelihood that the paragraph will provide an answer, the degree to which the information required in the question matches the paragraph, the semantic distance, etc. That is, it may be a vector that includes how significant a specific paragraph (411a, 412a, 413a) paired with a user query (400) is to the user query (400).

[0172] The control unit (180) can use a plurality of embedding vectors (421, 422, 423) obtained from the encoder (161) to determine the relevance with the user query (400) for each paragraph (411a, 412a, 413a) included in each of the plurality of candidate documents (401, 402) and distinguished according to preset criteria.

[0173] In this regard, the information retrieval model (160) of the present invention may be a model trained to simultaneously perform document reranking and paragraph extraction necessary for answer (or content) generation. More specifically, the information retrieval model (160) may be a model trained to simultaneously determine the relevance between a query and the entire document and the relevance between the query and each of multiple paragraphs.

[0174] First, the control unit (180) may input a plurality of embedding vectors (421, 422, 423) into the decoder (162) of the information retrieval model (160) in order to determine the relevance between the plurality of candidate documents (401, 402) and the user query (400). At this time, the control unit (180) may combine (or concatenate) the plurality of embedding vectors (421, 422, 423) to generate a combined embedding vector (430) in order to determine the relevance between the plurality of candidate documents (401, 402) and the user query (400), and input the combined embedding vector (430) into the decoder (162).

[0175] In this case, the control unit (180) sequentially combines the generated embedding vectors (421, 422, 423) by using the encoder (161) for each of the input data (e.g., the plurality of input data (301, 302, 303)) including paragraphs (411a, 412a, 413a) included in each of the query and the plurality of candidate documents (401, 402) by dividing them by document (or by document unit), and defines the combined embedding vector (430) as an integrated embedding vector (or semantic representation) of the candidate document. The combined embedding vector for each candidate document is processed as an input of the decoder (162), and processing can be performed to determine relevance at the document level or to compare relative relevance between paragraphs.

[0176] At this time, in the present invention, vector combining is performed only for paragraphs included in the same candidate document, and the embedding vectors of paragraphs included in different candidate documents are configured so as not to be combined with each other, thereby allowing decoding to be performed while maintaining semantic consistency for each document.

[0177] As an example, let us assume that a first embedding vector and a second embedding vector corresponding to input data including a paragraph and a user query (400) included in a first candidate document (401) among a plurality of candidate documents (401, 402) are generated through an encoder (161). The control unit (180) can combine the first embedding vector and the second embedding vector belonging to the first candidate document (401) to generate a combined embedding vector corresponding to the first candidate document (401).

[0178] As another example, let us assume that a third embedding vector and a fourth embedding vector corresponding to input data including a paragraph and a user query (400) included in a second candidate document (402) among a plurality of candidate documents (401, 402) are generated through an encoder (161). The control unit (180) can combine the third embedding vector and the fourth embedding vector belonging to the second candidate document (402) to generate a combined embedding vector corresponding to the second candidate document (402).

[0179] Thereafter, the combined embedding vector corresponding to the first candidate document (401) and the combined embedding vector corresponding to the second candidate document (402) are each processed as inputs to the decoder (162), enabling determination of relevance to the query and comparison between paragraphs for each candidate document.

[0180] This combined embedding vector (430) can represent the entire semantic association between the user query (400) and the paragraphs (411a, 412a, 413a) included in each of the plurality of candidate documents (401, 402). The combined embedding vector (430) can be data that compresses all relationships between the paragraphs (411a, 412a, 413a) included in each of the plurality of candidate documents (401, 402) and the user query (400). In other words, the combined embedding vector (430) can be understood as information that comprehensively reflects the relationships between the query and all paragraphs included in each of the plurality of candidate documents (401, 402).

[0181] As discussed above, the decoder (162) may be configured to predict a related word token (440) for each of the plurality of candidate documents (401, 402). The decoder (162) may be configured to generate a related word token (440) for each of the plurality of candidate documents (401, 402) based on the determined relatedness using the combined embedding vector (430). Here, the related word token (440) may mean a token indicating whether or not there is a relatedness between the user query (400) and the plurality of candidate documents (401, 402). The relevance word token (440) may be configured to include a first relevance word token (e.g., “True”) or a second relevance word token (e.g., “”) depending on whether a user query (400) and multiple candidate documents (401, 402) are related. The first relevance word token (e.g., “”) may indicate that the candidate document is a related document that is related to the user query (400), and the second relevance word token (e.g., “”) may indicate that the candidate document is a document that is not related to the user query (400).

[0182] Next, as illustrated in FIG. 6, the control unit (180) can process a plurality of embedding vectors (421, 422, 423) as inputs to the ranker layer (163) of the information retrieval model (160) to determine the relevance of each paragraph (411a, 412a, 413a) included in each of a plurality of candidate documents (401, 402) with a user query (400).

[0183] As discussed above, the ranker layer (163) may be a layer trained to predict the relative similarity between paragraphs (411a, 412a, 413a) included in each of a plurality of candidate documents (401, 402) by utilizing (or applying) an inter-passage attention mechanism (e.g., inter-passage attention) for a plurality of embedding vectors (421, 422, 423). Here, the relative similarity may mean a measure (or relationship) that expresses, on a relative basis, how much more relevant each paragraph is to the query compared to other paragraphs among a plurality of paragraphs given for the same query.

[0184] The ranker layer (163) may be configured to determine the relevance of each paragraph (411a, 412a, 413a) included in each of the plurality of candidate documents (401, 402) with the user query (400) using a plurality of embedding vectors (421, 422, 423). More specifically, the ranker layer (163) may generate (or predict, produce, etc.) a paragraph relevance score (score, or score, 450) for each paragraph (411a, 412a, 413a) included in each of the plurality of candidate documents (401, 402) based on the relevance determined using a plurality of embedding vectors (421, 422, 423).

[0185] Here, the paragraph relevance score (450) may be a score indicating whether there is a relevance between paragraphs (411a, 412a, 413a) included in each of the plurality of candidate documents (401, 402) and the user query (400). The paragraph relevance score (450) may be configured to include a first score (e.g., “”) or a second score (e.g., “”) depending on whether there is a relevance between each of the plurality of paragraphs (411a, 412a, 413a) and the user query (400). The first score (e.g., “”) may indicate that the paragraph is a related paragraph that has a relevance to the user query (400), and the second score (e.g., “”) may indicate that the paragraph is a paragraph that has no relevance to the user query (400).

[0186] Furthermore, as illustrated in FIG. 7a, the control unit (180) can specify a related document (460) related to the user query among the plurality of candidate documents (401, 402) based on the relevance determined for each paragraph (411a, 412a, 413a) included in each of the user query (400) and the plurality of candidate documents (401, 402).

[0187] The control unit (180) can specify a document including a first related word token among a plurality of candidate documents (401, 402) as a related document associated with the user query (400). For example, let us assume that the related word token for the first candidate document (401) among the plurality of candidate documents (401, 402) is generated as “True” and the related word token for the second candidate document (402) is generated as “False”. The control unit (180) can specify the first candidate document (401) including the first related word token among the plurality of candidate documents (401, 402) as a related document (460) associated with the user query (400).

[0188] In addition, the control unit (180) may specify a related paragraph (460a) associated with the user query (400) among paragraphs (411a, 412a, 413a) included in each of a plurality of candidate documents (401, 402) in order to generate an answer to the user query (400). The control unit (180) may specify a paragraph that satisfies a preset criterion among paragraphs (411a, 412a, 413a) included in each of a plurality of candidate documents (401, 402) as a related paragraph (460a). Here, the preset criterion may be set based on a paragraph included in a related document (460) associated with the user query (400). Therefore, the related paragraph (460a) may correspond to a paragraph included in a related document (460) associated with the user query (400).

[0189] Finally, the control unit (180) can generate an answer (470) to the user query (400) using the associated document (460) and the associated paragraph (460a) included in the associated document (460), and provide the generated answer (470) to the user terminal (10). For example, the control unit (180) can input the associated document (460) and the associated paragraph (460a) into the answer generation unit (e.g., a large-scale language model, 170), and the answer generation unit (170) can generate an answer (470) to the user query (400) using the associated document (460) and the associated paragraph (460a).

[0190] In this way, relevant paragraphs are entered as part of the input prompts for large-scale language models, and the large-scale language model references these paragraphs when generating answers. This means that key concepts, figures, definitions, and arguments from specific sentences or paragraphs within related documents can be summarized or reconstructed and reflected in the query response. Therefore, users receive an optimal, accurate, and contextually relevant answer to their query, rather than a simple summary.

[0191] Meanwhile, a user may have a user account pre-registered in the answer generation system (100) according to the present invention. In this case, the account may be created through a page (or screen) linked to the answer generation system (100). Alternatively, the account may also be created in at least one other system linked to the answer generation system (100). However, in this specification, the system in which the user account is issued is not separately distinguished, and all accounts that can utilize the various services (or functions) provided by the answer generation system (100) according to the present invention are referred to as "accounts pre-registered in the answer generation system (100)."

[0192] Accordingly, various information related to a user account can be stored in the storage unit (140) of the answer generation system (100). Here, the information related to the user account can include at least one of the user's (or user account's) history information and the user's metadata (e.g., name, gender, age, major, occupation, workplace (or company), etc.).

[0193] More specifically, a user's history information may include information related to various events occurring in the user account. For example, an event occurring in the user account may include at least one of: i) a user query input for obtaining an answer from a large-scale language model (170); ii) an input (or selection) of a document; and iii) a new (or additional) query input for an answer generated from the large-scale language model (170).

[0194] Based on these events, the user's history information may include at least one of: i) a user query entered by the user, ii) a document entered by the user (or a history of the user's document inputs), iii) content extracted from the document entered by the user (e.g., paragraphs included in the document, paragraphs associated with the user query, etc.), and iv) an answer from a large-scale language model (170) to the user query.

[0195] Accordingly, in the storage unit (140) of the answer generation system (100), at least one of a document, content extracted from the document, a user query, and an answer to the user query may be stored in association with a user account.

[0196] In this case, documents stored in the storage unit (140) linked to a user account can be stored in various ways. In the present invention, a document received according to a user's input or a space storing the user's document can also be named "user DB", "user document DB", or "document DB".

[0197] In one embodiment, the answer generation system (100) is linked to the user terminal (10) and can store at least one document stored (or built-in) in the storage (or memory or storage space or database) of the user terminal (10) in the storage (140) in association with a user account.

[0198] In another embodiment, the answer generation system (100) may be linked to an external cloud storage service (e.g., LG Cloud, Google Drive, Dropbox, etc.) that stores the user's documents. The answer generation system (100) may store the user's documents stored in the cloud storage service in the storage unit (140) by linking them to the user account.

[0199] As illustrated in FIG. 7b, when the answer generation system (100) receives a user query (710), it can specify a plurality of candidate documents related to the user query (710) based on preset criteria. Then, the answer generation system (100) can specify a related document related to the user query (710) among the plurality of candidate documents based on the correlation between the user query (710) and the user query (710) determined for each paragraph included in each of the plurality of candidate documents, and can generate an answer (720) to the user query (710) using the related document.

[0200] At this time, the plurality of candidate documents related to the user query (710) specified according to the preset criteria may be a plurality of candidate documents searched through the document search unit (150) or a plurality of candidate documents searched from the user DB. That is, the answer generation system (100) may specify a plurality of candidate documents related to the user query (710) from the user DB in order to generate an answer to the user query (710). For example, the answer generation system (100) may specify a plurality of candidate documents related to the user query (710) among documents included (or stored) in the user DB, and may generate an answer (720) to the user query (710) by using the related documents related to the user query (710) among the specified candidate documents.

[0201] Furthermore, the answer generation system (100) can provide the answer to the user terminal (10) along with information about the related document used in generating the answer. In this case, information about the related document used in the process of generating the answer (720) can be provided along with the answer (720) on the user terminal (10) where the service page (1000) is displayed.

[0202] In this case, the answer generation system (100) can provide information on related documents in various ways. For example, the answer generation system (100) can automatically identify and highlight sentences, keywords, numbers, etc. that are the basis of the actual response content within a related document or related paragraph specified through the user DB. The answer generation system (100) is included in a document related to the user query (710) and can highlight paragraphs (721, 722) related to the user query (710) on an area of ​​the service page (1000). At this time, a graphic object (721a, 722a) linked to a function that allows the user to view or move to the related document can be provided on an area of ​​the service page (1000) where the related paragraphs (721, 722) are provided.

[0203] In this way, when multiple related paragraphs are selected, the present invention highlights only the core parts that actually contributed to the answer, providing a user experience that allows the user to intuitively recognize the source and reliability of the information. In other words, by clearly showing the user "why this paragraph was selected" or "what part was reflected in the answer," the present invention enables the user to intuitively understand the basis for the model's response, thereby increasing the reliability and usability of the information.

[0204] Meanwhile, the present invention can be effectively applied and utilized in domains where the source or basis of answers to queries is important. The answer generation system (100) according to the present invention can be usefully applied in domains where clarity of source and accuracy of information are essential.

[0205] For example, in domains such as law, patents, academic papers, education and learning, and healthcare, where clarity of source and accuracy of information are crucial, the accuracy and reliability of query answers, along with the explicit basis for them, are crucial. In particular, in these domains, accurate assessment of the relevance of the documents used to generate the answer and the information contained within each paragraph of said document, is essential. More specifically, in domains such as legal documents or case law, Q&A citing medical papers, summarizing and reviewing educational content, and patent literature-based technical analysis, precise selection of relevant documents and paragraphs, along with explicit basis for the answer, are crucial. Incorrect answers can lead to legal disputes, life-threatening issues, learning errors, and distortions in technical interpretation.

[0206] Accordingly, the answer generation system (100) according to the present invention can simultaneously quantitatively evaluate related documents associated with a query and paragraphs contained within said related documents, and identify and highlight relevant paragraphs among the paragraphs contained within said related documents that serve as the basis for answer generation. This allows the user to intuitively assess the reliability of a response, and this, particularly in domain-based answer generation systems, simultaneously ensures accurate answers based on source information and verifiability of model answers.

[0207] In other words, the present invention can be optimized even in domain-specific environments that require the provision of accurate and reliable information, and thus can be expanded into various fields such as expert systems, paper writing support, medical consultation assistants, and patent analysis systems.

[0208] Meanwhile, the present invention can be implemented in various system environments. The present invention can support a flexible system architecture adaptable to various execution environments, such as on-device architectures, server-client architectures, and cloud-based hybrid architectures. This system architecture can be implemented in conjunction with at least some of the components of the answer generation system (100) illustrated in FIG. 1.

[0209] In one embodiment, the present invention can be implemented in an on-device environment. The model of the answer generation system (100) (e.g., an information retrieval model (160) and an answer generation unit (e.g., a large-scale language model, 170)) can be lightweight and installed on the user terminal (10). Accordingly, since the query preprocessing, document retrieval, reranking, and answer generation processes are all processed within the user terminal (10), operation is possible even without a network connection. Accordingly, the answer generation system (100) according to the present invention can operate stably even in environments requiring personal information security or offline environments.

[0210] In another embodiment, the present invention can be implemented through a server-client architecture. The client (or client device) collects user queries and transmits (or forwards) them to the server, and the server (200) can perform a search and reranking process utilizing a large document corpus, as well as a process of generating an answer to the query. In this case, the server may include at least one of a document search unit (150), an information retrieval model (160), and an answer generation unit (170), and the client may perform a role of receiving the final answer generated by the server. The server may utilize high-performance computing resources, such as a GPU or TPU, to enable more precise calculations. This architecture can be flexibly selected considering user requirements, device specifications, response times, etc.

[0211] In another embodiment, the present invention can be implemented in a hybrid form, combining an on-device environment with a server-client architecture. Query preprocessing and embedding extraction can be performed on the user terminal (10), document retrieval and reranking can be processed on the server, and response generation can then be performed using a lightweight model within the user terminal (10). This hybrid structure can be a flexible implementation form that can balance offline responsiveness, processing speed, and model accuracy.

[0212] In another embodiment, the server can be implemented in a single-server environment, as well as a cloud-based distributed server or edge server architecture, with search and generation models (or modules) distributed across multiple servers and operated in parallel. In a server architecture, user queries and result data can be encrypted and transmitted, and personal information protection can be enhanced through methods such as anonymization and de-identification processing, and authentication token-based access control.

[0213] Furthermore, the present invention can be extended to integrate qubit-based computational structures into on-device or server architectures, or to hybrid computational approaches with classical computing, taking into account interoperability with quantum computing environments. Such quantum-based expansion structures can contribute to simultaneously securing processing speed and accuracy for complex queries.

[0214] This structure can be flexibly selected by comprehensively considering the user's requirements, terminal processing performance, model size, real-time requirements, response delay time, power consumption, security level, etc.

[0215] Meanwhile, it can be confirmed that the information retrieval model (ex: “PFiD”, 160) learned according to the learning method described above achieves a higher recall rate than the conventional model because it sees a richer context in the document retrieval task, as illustrated in (b) of Fig. 10. In other words, the information retrieval model (ex: “PFiD”, 160) according to the present invention can achieve competitive performance while maintaining high efficiency in both the document relevance judgment task and the paragraph relevance judgment task. This can indicate that the information retrieval model (ex: “PFiD”, 160) according to the present invention shows better performance than the conventional model in the RAG scenario.

[0216] In addition, it can be confirmed that the information retrieval model (ex: “PFiD”, 160) learned according to the learning method described above maintains superior ranking performance compared to the conventional technology even under various conditions of the number of documents and the number of paragraphs, as illustrated in Fig. 11. The information retrieval model (ex: “PFiD”, 160) according to the present invention stably improves performance as the number of paragraphs increases, and can secure scalability and efficiency according to the increase in input. That is, the information retrieval model (ex: “PFiD”, 160) according to the present invention can provide accurate and sophisticated ranking results in an actual RAG environment by explicitly learning the relative similarity between paragraphs.

[0217] Furthermore, the information retrieval model (e.g., “PFiD”, 160) learned according to the learning method described above can make more accurate relevance judgments compared to conventional models by concentrating related paragraphs in higher ranks, as illustrated in Fig. 12 (the left side represents the conventional model, and the right side represents the information retrieval model (160) according to the present invention). Through this, the present invention can more easily identify correct paragraphs and improve accessibility to correct answers for user queries in actual information retrieval scenarios.

[0218] Meanwhile, the present invention provides a method and system for generating an answer that can provide a user-tailored environment. More specifically, the answer generation system according to the present invention provides an accurate and high-quality answer to a user's query while providing a service environment optimized for the user, and may include an information retrieval model learned according to the learning method described above. The answer generation system according to the present invention, as illustrated in FIG. 13, comprises, in the inference stage, a process of receiving a user query (S1310), a process of specifying a plurality of candidate documents related to the user query based on preset criteria (S1320), a process of specifying a related document related to the user query among the plurality of candidate documents based on a correlation between the user query and the user query determined for each paragraph included in each of the plurality of candidate documents (S1330), a process of generating an answer to the user query using the related documents (S1340), and a process of providing the answer together with information about the related documents used for answer generation to a user terminal (S1350), thereby enabling the user to intuitively recognize and more quickly understand the information required by the user.

[0219] In this regard, in the present invention, receiving a user query or providing various information (e.g., answers, related documents, related paragraphs, etc.) in response to a user query to the user may be performed through a service page linked to an answer generation system (100, or answer generation platform) or through a display unit of a user terminal (10). However, for convenience of explanation, the following description will be made on the assumption that the process of receiving a user query or providing various information in response to a user query is performed through a service page.

[0220] As illustrated in FIG. 14A, the control unit (180) may provide (or output) a service page (1400) linked to the answer generation system (100) to the user terminal (10). The service page (1400) may be configured to include a plurality of areas. For example, the service page (1400) may include at least one of a first area (1410) in which information and paragraphs about a document are provided, a second area (1420) in which answers to user queries are provided, and a third area (1430) in which user queries are received. In this case, the first area (1410) may also be named “a first area of ​​the display unit of the user terminal (10)” or “a document provision area,” and the second area (1420) may also be named “a second area of ​​the display unit of the user terminal (10)” or “an answer provision area.” Additionally, the third area (1430) may also be named “third area of ​​the display unit of the user terminal (10)” or “query reception area”.

[0221] The control unit (180) can receive a user query input from a user terminal (10) on which a service page (1400) is output. For example, the control unit (180) can receive a user query (1401) input through the third area (1430) of the service page (1400).

[0222] When a user query (1401) is received, the control unit (180) can specify a plurality of candidate documents related to the user query (1401) based on preset criteria. The control unit (180) can specify a plurality of candidate documents related to the user query (1401) from among a plurality of documents stored in a preset repository. Here, the preset criteria may be related to a search criterion (e.g., BM25 score) of a document search unit (150) that initially specifies (or searches) candidate documents related to the user query from a plurality of documents stored in a preset repository and / or an external server and / or website based on the user query.

[0223] In one embodiment, the plurality of documents may include documents (or files) stored in a predetermined storage (e.g., storage unit (140, or memory), DB of the answer generation system (or platform), user DB associated with a user account, pre-configured server (e.g., server of the answer generation system), etc.).

[0224] In another embodiment, the plurality of documents may include documents (or files) stored (or built-in) in a specific storage (or memory (built-in memory) or storage space or database, etc.) of a user terminal (10) linked with the answer generation system (100).

[0225] In another embodiment, the answer generation system (100) may be linked to an external cloud storage service (e.g., LG Cloud, Google Drive, Dropbox, OneDrive, etc.) that stores the user's documents (or files). The answer generation system (100) may store the user's documents stored in the cloud storage service in a pre-specified storage by linking them to the user account. In this case, the plurality of documents may include documents stored in the pre-specified storage and / or the cloud storage service.

[0226] In addition, the control unit (180) can use the information retrieval model (160) learned through the learning method described above to identify a related document related to the user query (1401) among a plurality of candidate documents. The information retrieval model (160) can identify a related document related to the user query (1401) among a plurality of candidate documents based on the correlation between the user query (1401) and the user query (1401) determined for each paragraph included in each of the plurality of candidate documents.

[0227] Alternatively, the answer generation system (100) may identify a relevant document associated with the user query (1401) among a plurality of candidate documents based on the history information of the user (or user account). Here, the user's history information may be related to the user's history of using the answer generation system (or the answer generation platform (or service)).

[0228] The user's history information may include information related to various events occurring in the user terminal (10, or user account). For example, the events occurring in the user terminal (10) may include at least one of: i) the user's query input for obtaining an answer from the answer generation system (100), ii) the user's document input (or selection), and iii) the user's new (or additional) query input for an answer generated from the answer generation system (100).

[0229] Based on these events, the user's history information may include at least one of i) a user query entered by the user, ii) metadata about the document entered by the user (e.g., the document's title, author, creation date, modification date, file size, file format, language, document version, keywords, document description (or summary), etc.), iii) information about the document entered by the user (e.g., the document's name, the document's storage location, etc.), iv) content extracted from the document entered by the user (e.g., paragraphs included in the document, paragraphs related to the user's query, etc.), v) answers to the user's query. However, the information included in the history information is not necessarily limited to this, and it is obvious that the history information may include various other information in addition to the above-mentioned.

[0230] Furthermore, the information retrieval model (160) can identify a related paragraph among paragraphs included in each of a plurality of candidate documents, which is associated with the user query (1401) and is to be used for answer generation. In this case, the answer generation unit (170) can generate an answer to the user query (1401) using at least one of the related documents and the related paragraphs. Here, the related paragraph can also be understood as a paragraph included in a related document associated with the user query (1401).

[0231] Meanwhile, the control unit (180) can provide information about an answer generated using a specific related document and an associated document used to generate the answer to the user terminal (10).

[0232] The control unit (180) can provide information about an answer generated using a specific related document and the related document used to generate the answer to the user terminal (10) where the service page is output.

[0233] For example, as illustrated in FIG. 14b, the control unit (180) may provide an answer (1421) to a user query (1401) in a second area (1420) of a service page (1400) output to a user terminal (10). The answer (1421) generated for the user query (1401) may include information (ex: “Local disk N → Folder 1 → Document 1”, “Local disk N → Folder 1 → Document 1”) about the storage location (or storage path, source, etc.) of related documents (ex: “Document 1” and “Document 5”). Additionally, the answer (1421) may include related documents (e.g., “Document 1” and “Document 5”), and the related paragraphs used to generate the answer (1421) may be highlighted (e.g., “Language Model is a model that learns patterns of natural languages ​​(e.g., Korean, English, etc.) used by humans and predicts or generates words, sentences, or documents appropriate to a given context”).

[0234] At this time, while providing a response (1421) to the user's query, the control unit (180) can provide (or display) information about the associated document to the user terminal (10) so that the associated document used in generating the response (1421) can be confirmed (i.e., so that the user can intuitively recognize (or confirm) it). Here, the information about the associated document can include at least one of the associated paragraph, the name of the associated document, and the storage location of the associated document. For example, in the first area (1410) of the service page (1400) output to the user terminal (10) or the peripheral area where the answer (1421) is provided, information about the storage location (1411, 1412) for each of a plurality of related documents (ex: “document 1” and “document 5”) and the related paragraphs (1413, 1414) used to generate the answer (1421) among the paragraphs included in each of the plurality of related documents (ex: “document 1” and “document 5”) may be highlighted.

[0235] Alternatively, the control unit (180) may overlap a graphic object (or highlighting object) corresponding to a storage space (e.g., memory, disk, drive, cloud, folder, etc.) in which the associated document is included (or stored) with the user terminal (10) in relation to the storage location (or storage space) of the associated document.

[0236] For example, in a first area (1410) of a service page (1400), a graphic object (1411c) corresponding to a first associated document (e.g., “document 1”) among a plurality of associated documents (e.g., “document 1” and “document 5”), a graphic object (1411b) corresponding to a first storage space (e.g., “folder 1”) that includes the first associated document (or where the first associated document is located), and a graphic object (1411a) corresponding to a second storage space (e.g., “local disk N”) that includes the first storage space (or where the first storage space is located) may overlap, respectively.

[0237] For another example, in the first area (1410) of the service page (1400), a graphic object (1412c) corresponding to a fifth related document (e.g., “document 5”), a graphic object (1412b) corresponding to a first storage space (e.g., “folder 5”) containing the second related document (or where the second related document is located), and a graphic object (1412a) corresponding to a second storage space (e.g., “L cloud”) containing the first storage space may overlap, respectively.

[0238] Furthermore, the control unit (180) may overlap information about associated paragraphs included in the associated document, along with the associated document used to generate the answer (1421), on the user terminal (10). In this case, the associated paragraphs may be provided around the associated document, around a graphic object corresponding to the overlapped associated document, or in a peripheral area where the answer (1421) is provided. The present invention does not limit the area (or space) where such associated paragraphs are provided to any one.

[0239] For example, in the first area (1410) of the service page (1400), a graphic object (1411c) corresponding to the first associated document may be overlapped with information about the first associated paragraph (1413) included in the first associated document (e.g., “document 1”) and used to generate the answer (1421). At this time, a graphic object (1413a) linked to a function that allows a user to view or move to the first associated document may be provided around the graphic object (1411c) corresponding to the first associated document or the overlapped first associated document or the first associated paragraph (1413).

[0240] For another example, in the first area (1410) of the service page (1400), a graphic object (1412c) corresponding to the second associated document may be overlapped with information about the second associated paragraph (1414) included in the second associated document (e.g., “document 5”) and used to generate the answer (1421). At this time, a graphic object (1414a) linked to a function that allows a user to view or move to the second associated document may be provided around the graphic object (1412c) corresponding to the second associated document or the overlapped second associated document or the second associated paragraph (1414).

[0241] Meanwhile, the answer generation system (100) according to the present invention may store documents (or files) utilized in the process of generating answers to user queries. More specifically, at least one of a user query, an answer to the user query, a related document related to the user query, and information about the related document may be stored in a pre-specified storage. For example, the pre-specified storage may store a user query (1401), an answer to the user query (1421), related documents related to the user query (1401) (e.g., “document 1” and “document 5”), storage locations (1411, 1412) of the related documents, and names (or document names, file names, file names, etc.) of the related documents as a set. Such information may be stored in a pre-specified storage in connection with a user account, and may also be understood as user history information.

[0242] In this regard, the present invention can provide a plurality of recommended queries (or a plurality of user queries, a plurality of recommended user queries, etc.) specified according to various recommendation criteria to the user terminal (10). The control unit (180) can provide a plurality of recommended queries to the user terminal (10) based on preset query recommendation criteria. Here, the preset query recommendation criteria can be set based on the user's history information. For example, the preset query recommendation criteria can include at least one of i) a query with a high user input frequency (i.e., a question frequently asked by the user), ii) a recent query of the user, iii) a word or keyword frequently used by the user (e.g., a word or keyword specified from a user query, a word or keyword specified as having high importance, etc.), iv) a user's search term criteria (e.g., a search for a user's document, file, information, etc. performed on the user terminal (10)), v) a query with a high user interest, and vi) an unanswered or incomplete query (e.g., a query previously entered by the user but not answered or stopped).

[0243] For example, as illustrated in FIG. 15a, the control unit (180) may provide a plurality of recommended queries (e.g., “user query 1”, “user query 2”, “user query 3”) to a region (fourth region (1540)) of a service page (1500) output to a user terminal (10) based on preset query recommendation criteria. In this case, the fourth region (1540) of the service page (1500) may also be named “fourth region of the display unit of the user terminal (10)”, “recommended query provision region”, “query recommendation region”, etc.

[0244] Alternatively, providing multiple recommendation queries may be understood as overlapping graphic objects (or highlighting objects, 1541, 1542, 1543) corresponding to each of the multiple recommendation queries (e.g., “user query 1”, “user query 2”, “user query 3”) on the user terminal (10).

[0245] At this time, the control unit (180) may provide the user terminal (10) with a plurality of recommended queries and associated documents associated with each of the plurality of recommended queries. For example, the control unit (180) may provide, in the first area (1510) of the service page (1500), associated documents (“Document 1”, “Document 2”, “Document 3”, “Document 4”, “Document 5”) associated with each of the plurality of recommended queries (ex: “User Query 1”, “User Query 2”, “User Query 3”). In the first area (1510) of the service page (1500), graphic objects (1541, 1542, 1543) corresponding to each of a plurality of recommended queries (e.g., “user query 1”, “user query 2”, “user query 3”) and graphic objects (or highlighting objects, 1511, 1512, 1513, 1514, 1515) corresponding to each of the associated documents (“document 1”, “document 2”, “document 3”, “document 4”, “document 5”) may overlap each other.

[0246] In the present invention, each of the plurality of recommended queries may be displayed on the user terminal (10) with a different visual appearance. For example, graphic objects (1541, 1542, 1543) corresponding to each of the plurality of recommended queries (e.g., “User Query 1”, “User Query 2”, “User Query 3”) overlapping in the fourth area (1540) of the service page (1500) may be displayed on the user terminal (10) with a different visual appearance.

[0247] More specifically, a graphic object (1541) corresponding to a first recommendation query (e.g., “user query 1”) among the plurality of recommendation queries may be configured to have a first visual appearance (e.g., a first color (red)) among different visual appearances. In addition, a graphic object (1542) corresponding to a second recommendation query (e.g., “user query 2”) among the plurality of recommendation queries may be configured to have a second visual appearance (e.g., a second color (blue)) that is different from the first visual appearance. In addition, a graphic object (1543) corresponding to a third recommendation query (e.g., “user query 3”) among the plurality of recommendation queries may be configured to have a third visual appearance (e.g., a third color (yellow)) that is different from the first visual appearance and the second visual appearance.

[0248] In this case, in the present invention, the associated documents associated with each of the plurality of recommended queries may be displayed on the user terminal (10) with the same visual appearance as the visual appearances of the plurality of recommended queries, so that the association with each of the plurality of recommended queries can be identified. A first recommended query (e.g., “user query 1”) and associated documents associated with the first recommended query (e.g., “document 1” and “document 3”) may be displayed on the user terminal (10) with the first visual appearance, a second recommended query (e.g., “user query 2”) and associated documents associated with the second recommended query (e.g., “document 2” and “document 4”) may be displayed on the user terminal (10) with the second visual appearance, and a third recommended query (e.g., “user query 3”) and associated documents associated with the third recommended query (e.g., “document 5”) may be displayed on the user terminal (10) with the third visual appearance.

[0249] In one embodiment, a graphic object (1511) corresponding to a first associated document (e.g., “document 1”) associated with a first recommended query (e.g., “user query 1”) having a first visual appearance among multiple recommended queries, and a graphic object (1513) corresponding to a third associated document (e.g., “document 3”) may overlap in the first area (1510) each with a first visual appearance identical to the visual appearance of the first recommended query.

[0250] In another embodiment, a graphic object (1512) corresponding to a second related document (e.g., “document 2”) associated with a second recommended query (e.g., “user query 2”) having a second visual appearance among multiple recommended queries and a graphic object (1514) corresponding to a fourth related document (e.g., “document 4”) may overlap the first area (1510) with a second visual appearance identical to the visual appearance of the second recommended query.

[0251] In another embodiment, a graphic object (1515) corresponding to a fifth related document (e.g., “document 5”) associated with a third recommended query (e.g., “user query 3”) having a third visual appearance among multiple recommended queries may overlap the first region (1510) with a third visual appearance that is identical to the visual appearance of the third recommended query.

[0252] Furthermore, when a specific recommended query is selected from among a plurality of recommended queries, the control unit (180) may provide the user terminal (10) with an answer to a specific recommended query along with information about a specific related document used to generate an answer to the selected specific recommended query among the related documents associated with each of the plurality of recommended queries.

[0253] As discussed above, a pre-specified repository may store a user query, a response to the user query, associated documents related to the user query, and information about the associated documents as a single set. Accordingly, a pre-specified repository may store and exist multiple sets containing a user query, a response to the user query, associated documents related to the user query, and information about the associated documents.

[0254] The control unit (180) may, based on selection of any one specific recommendation query from among a plurality of recommendation queries from the user terminal (10), specify a specific set related to (or including) the selected specific recommendation query from among a plurality of sets stored in a predetermined storage. The control unit (180) may provide the user terminal (10) with an answer to a specific recommendation query included in the specific set and information about a specific related document used to generate an answer to the specific recommendation query. Alternatively, the control unit (180) may, based on selection of any one specific recommendation query from among a plurality of recommendation queries from the user terminal (10), provide the user terminal (10) with an answer to a specific recommendation query included in (or linked, associated, connected, etc.) the specific recommendation query and information about a specific related document used to generate an answer to the specific recommendation query. Here, information about a specific associated document may include at least one of a specific associated paragraph included in the specific associated document and associated with a specific recommendation query, a name of the specific associated document, and a storage location of the specific associated document used to generate an answer to the specific recommendation query.

[0255] For example, as illustrated in FIG. 15b, the control unit (180) may provide an answer (1550) to the first recommendation query included in the graphic object (1541) corresponding to the first recommendation query (e.g., “What is reinforcement learning?”) in the second area (1520) of the service page (1500) output to the user terminal (10) based on the selection of the graphic object (1541) corresponding to the first recommendation query among the graphic objects (1541, 1542, 1543) corresponding to each of a plurality of recommendation queries (e.g., “User query 1”, “User query 2”, “User query 3”) from the user terminal (10). These answers (1550) may include information about the storage locations of related documents (e.g., “Document 1” and “Document 3”) (e.g., “L Drive → Folder 3 → Document 1”, “Local Disk C → Folder 5 → Document 3”). Additionally, the answers (1550) may include related documents (e.g., “Document 1” and “Document 3”), and may highlight related paragraphs used to generate the answers (e.g., “Reinforcement Learning (RL) is one of the artificial intelligence (AI) learning methods, and is a method of learning behavior in a way that maximizes reward while interacting with the environment”).

[0256] At this time, while providing an answer (1550) to a specific recommendation query selected by the user, the control unit (180) may display information about a specific related document on the user terminal (10) so that the user can confirm the specific related document used to generate the answer (1550) to the specific recommendation query. For example, in the first area (1510) of the service page (1500) output to the user terminal (10) or the peripheral area where the answer (1550) is provided, information about the storage locations (1521, 1522) for each of a plurality of specific related documents (e.g., “document 1” and “document 3”) and the related paragraphs (1523, 1524) used to generate the answer (1550) among the paragraphs included in each of the plurality of specific related documents (e.g., “document 1” and “document 3”) may be highlighted.

[0257] Alternatively, the control unit (180) may overlap a graphic object (or highlighting object) corresponding to a storage space (e.g., memory, disk, drive, cloud, folder, etc.) containing (or storing) a specific associated document with the storage location (or storage space) of the specific associated document used to generate an answer (1550) to a specific recommended query, on the user terminal (10).

[0258] For example, in a first area (1510) of a service page (1500), a graphic object (1521c) corresponding to a first specific associated document (e.g., “document 1”) among a plurality of specific associated documents (e.g., “document 1” and “document 3”), a graphic object (1521b) corresponding to a first storage space (e.g., “folder 3”) that includes the first specific associated document (or where the first specific associated document is located), and a graphic object (1521a) corresponding to a second storage space (e.g., “L drive”) that includes the first storage space (or where the first storage space is located) may overlap each other.

[0259] For another example, in the first area (1510) of the service page (1500), a graphic object (1522c) corresponding to a third specific associated document (e.g., “document 3”), a graphic object (1522b) corresponding to a first storage space (e.g., “folder 5”) containing the third specific associated document (or where the third specific associated document is located), and a graphic object (1522a) corresponding to a second storage space (e.g., “local disk C”) containing the first storage space may overlap, respectively.

[0260] Furthermore, the control unit (180) may overlap, on the user terminal (10), information about a specific related paragraph included in a specific related document, along with a specific related document used to generate a response (1550) to a specific recommended query. In this case, the specific related paragraph may be provided around the specific related document, around a graphic object corresponding to the overlapping specific related document, or in a peripheral area where the response (1550) is provided. The present invention does not limit the area (or space) where such a specific related paragraph is provided to any one.

[0261] For example, in the first area (1510) of the service page (1500), a graphic object (1521c) corresponding to the first specific associated document may be overlapped with information about a specific associated paragraph (1523) included in the first specific associated document (e.g., “document 1”) and used to generate an answer (1550). At this time, a graphic object (1523a) linked to a function that allows a user to view or move to the first specific associated document may be provided around the graphic object (1521c) corresponding to the first specific associated document or the overlapped first specific associated document or the specific associated paragraph (1523).

[0262] For another example, in the first area (1510) of the service page (1500), a graphic object (1522c) corresponding to the third specific associated document may be overlapped with information about a specific associated paragraph (1524) included in the third specific associated document (e.g., “document 3”) and used to generate a response (1550). At this time, a graphic object (1524a) linked to a function that allows a user to view or move to the third specific associated document may be provided around the graphic object (1522c) corresponding to the third specific associated document or the overlapped third specific associated document or the specific associated paragraph (1524).

[0263] In another embodiment of the present invention, the present invention can provide a plurality of different keyword-specific queries (or keyword-specific recommended queries) to a user terminal (10) based on preset query recommendation criteria.

[0264] For example, as illustrated in FIG. 16a, the control unit (180) can provide keyword-specific recommended queries (ex: “AI keyword query N”, “chemistry keyword query N”, “medical keyword query N”) to one area (fourth area (1640)) of the service page (1600) output to the user terminal (10) based on preset query recommendation criteria.

[0265] Alternatively, providing a keyword-specific recommendation query may also be understood as overlapping a graphic object (or highlighting object, 1641, 1642, 1643) corresponding to each keyword-specific recommendation query (ex: “AI keyword query N”, “chemistry keyword query N”, “medical keyword query N”) on the user terminal (10).

[0266] At this time, the control unit (180) may provide the user terminal (10) with the recommended queries by keyword and the related documents associated with each recommended query by keyword. For example, the control unit (180) may provide the related documents (“Document 1”, “Document 2”, “Document 3”, “Document 4”, “Document 5”) associated with each recommended query by keyword (ex: “AI Keyword Query N”, “Chemistry Keyword Query N”, “Medicine Keyword Query N”) in the first area (1610) of the service page (1600). In the first area (1610) of the service page (1500), graphic objects (1641, 1642, 1643) corresponding to each of the keyword-specific recommended queries (ex: “AI keyword query N”, “chemistry keyword query N”, “medical keyword query N”) and graphic objects (or highlighting objects, 1611, 1612, 1613, 1614, 1615) corresponding to each of the associated documents (“document 1”, “document 2”, “document 3”, “document 4”, “document 5”) may overlap each other.

[0267] In the present invention, each of the keyword-specific recommended queries may be displayed on the user terminal (10) with different visual appearances. For example, graphic objects (1641, 1642, 1643) corresponding to each of the keyword-specific recommended queries (ex: “AI keyword query N”, “chemistry keyword query N”, “medical keyword query N”) overlapped in the fourth area (1640) of the service page (1600) may be displayed on the user terminal (10) with different visual appearances.

[0268] More specifically, a graphic object (1641) corresponding to a recommendation query for a first keyword among recommendation queries by keyword (e.g., “AI keyword query N”) may be configured to have a first visual appearance (e.g., a first color (red)) among different visual appearances. In addition, a graphic object (1642) corresponding to a recommendation query for a second keyword among recommendation queries by keyword (e.g., “chemistry keyword query N”) may be configured to have a second visual appearance (e.g., a second color (blue)) different from the first visual appearance. In addition, a graphic object (1643) corresponding to a recommendation query for a third keyword among recommendation queries by keyword (e.g., “medical keyword query N”) may be configured to have a third visual appearance (e.g., a third color (yellow)) different from the first visual appearance and the second visual appearance.

[0269] In this case, in the present invention, the related documents associated with each keyword-specific recommended query can be displayed on the user terminal (10) with the same visual appearance as the visual appearance of each keyword-specific recommended query so that the association with each keyword-specific recommended query can be identified. A recommended query for a first keyword (e.g., “AI keyword query N”) and related documents (e.g., “document 1” and “document 4”) associated with the recommended query for the first keyword may be displayed on the user terminal (10) with a first visual appearance, a recommended query for a second keyword (e.g., “chemistry keyword query N”) and related documents (e.g., “document 2” and “document 4”) associated with the recommended query for the second keyword may be displayed on the user terminal (10) with a second visual appearance, and a recommended query for a third keyword (e.g., “medical keyword query N”) and related documents (e.g., “document 5”) associated with the recommended query for the third keyword may be displayed on the user terminal (10) with a third visual appearance.

[0270] In one embodiment, a graphic object (1611) corresponding to a first related document (e.g., “document 1”) and a graphic object (1614) corresponding to a fourth related document (e.g., “document 4”) associated with a recommendation query for a first keyword having a first visual appearance among keyword-specific recommendation queries may overlap each other in the first area (1610) with the same first visual appearance as the visual appearance of the recommendation query for the first keyword.

[0271] In another embodiment, a graphic object (1612) corresponding to a second related document (e.g., “document 2”) associated with a recommendation query for a second keyword (e.g., “chemistry keyword query N”) having a second visual appearance among keyword-specific recommendation queries and a graphic object (1615) corresponding to a fifth related document (e.g., “document 5”) may overlap the first area (1610) with a second visual appearance identical to the visual appearance of the recommendation query for the second keyword.

[0272] In another embodiment, a graphic object (1613) corresponding to a third related document (e.g., “document 3”) associated with a recommendation query for a third keyword (e.g., “medical keyword query N”) having a third visual appearance among keyword-specific recommendation queries may be overlapped with the first region (1610) with a third visual appearance identical to the visual appearance of the recommendation query for the third keyword.

[0273] Furthermore, when a recommendation query for a specific keyword (or a recommendation query related to a specific keyword) is selected from among the keyword-specific recommendation queries from the user terminal (10), the control unit (180) can provide the answer to the user terminal (10) together with information about a specific related document used to generate an answer to the recommendation query for the selected specific keyword among the related documents associated with each of the keyword-specific recommendation queries.

[0274] The control unit (180) may, based on selection of a recommendation query for a specific keyword from among the keyword-specific recommendation queries from the user terminal (10), specify a specific set related to the recommendation query for the selected specific keyword (or including the recommendation query for the specific keyword) from among the multiple sets stored in a predetermined storage. The control unit (180) may provide the user terminal (10) with an answer to the recommendation query for the specific keyword included in the specific set and information about a specific related document used to generate the answer. Alternatively, the control unit (180) may, based on selection of a recommendation query for a specific keyword from among the keyword-specific recommendation queries from the user terminal (10), provide the user terminal (10) with an answer to the recommendation query included in (or linked, associated, connected, etc.) the above-mentioned recommendation query and information about a specific related document used to generate the answer.

[0275] For example, as illustrated in FIG. 16b, the control unit (180) may provide, based on the selection of a graphic object (1642) corresponding to a recommendation query for a second keyword (ex: “What is Crosslinking agent in Korean? Please explain the exact definition and function as well.”) among graphic objects (1641, 1642, 1643) corresponding to each of the keyword-specific recommendation queries (ex: “AI keyword query N”, “Chemical keyword query N”, “Medical keyword query N”) from the user terminal (10), an answer (1650) to a recommendation query for a second keyword included in a graphic object (1642) corresponding to the recommendation query for the second keyword may be provided in a second area (1620) of a service page (1600) output to the user terminal (10). These answers (1650) may include information about the storage locations of related documents (e.g., “Document 2” and “Document 5”) (e.g., “C drive → Folder 10 → Document 2”, “C drive → Folder 5 → Document 5”). In addition, the answer (1650) may include related documents (e.g., “Document 2” and “Document 5”), and may highlight related paragraphs used to generate the answer (1650) (e.g., “Crosslinking agent” is translated as “crosslinking agent” in Korean. It is a substance that forms a covalent bond or physical bond between polymer chains, thereby forming a three-dimensional structure of the polymer or improving its structural stability”).

[0276] At this time, while providing an answer (1650) to a recommendation query for a specific keyword selected by the user, the control unit (180) may display information about a specific related document on the user terminal (10) so that the specific related document used to generate the answer (1650) can be confirmed. For example, in the first area (1610) of the service page (1600) output to the user terminal (10) or the peripheral area where the answer (1650) is provided, information about storage locations (1621, 1622) for each of a plurality of specific related documents (e.g., “document 2” and “document 5”) and related paragraphs (1623, 1624) used to generate the answer (1650) among paragraphs included in each of the plurality of specific related documents (e.g., “document 2” and “document 4”) may be highlighted.

[0277] Alternatively, the control unit (180) may overlap a graphic object (or highlighting object) corresponding to a storage space (e.g., memory, disk, drive, cloud, folder, etc.) containing (or storing) a specific associated document with the storage location (or storage space) of the specific associated document used to generate an answer (1650) to a recommendation query for a specific keyword, on the user terminal (10).

[0278] For example, in a first area (1610) of a service page (1600), a graphic object (1621c) corresponding to a second specific associated document (e.g., “document 2”) among a plurality of specific associated documents (e.g., “document 2” and “document 4”), a graphic object (1621b) corresponding to a first storage space (e.g., “folder 10”) that includes the second specific associated document (or where the second specific associated document is located), and a graphic object (1621a) corresponding to a second storage space (e.g., “C drive”) that includes the first storage space (or where the first storage space is located) may overlap each other.

[0279] For another example, in the first area (1610) of the service page (1600), a graphic object (1622c) corresponding to a fourth specific associated document (e.g., “document 4”), a graphic object (1622b) corresponding to a first storage space (e.g., “folder 5”) containing the fourth specific associated document (or where the fourth specific associated document is located), and a graphic object (1622a) corresponding to a second storage space (e.g., “C drive”) containing the first storage space may overlap, respectively.

[0280] Furthermore, the control unit (180) may overlap, on the user terminal (10), information about a specific related paragraph included in a specific related document, along with a specific related document used to generate a response (1650) to a recommendation query related to a specific keyword. In this case, the specific related paragraph may be provided around the specific related document, around a graphic object corresponding to the overlapping specific related document, or in a peripheral area where the response (1650) is provided. The present invention does not limit the area (or space) where such a specific related paragraph is provided to any one.

[0281] For example, in the first area (1610) of the service page (1600), a graphic object (1621c) corresponding to the second specific associated document may be overlapped with information about a specific associated paragraph (1623) included in the second specific associated document (e.g., “document 2”) and used to generate an answer (1650). At this time, a graphic object (1623a) linked to a function that allows a user to view or move to the second specific associated document may be provided around the graphic object (1621c) corresponding to the second specific associated document or the specific associated paragraph (1623) that overlaps the second specific associated document.

[0282] For another example, in the first area (1610) of the service page (1600), a graphic object (1622c) corresponding to the fifth specific associated document (e.g., “document 5”) may be overlapped with information about a specific associated paragraph (1624) used to generate a response (1650) and included in the fifth specific associated document. At this time, a graphic object (1624a) linked to a function that allows a user to view or move to the fifth specific associated document may be provided around the graphic object (1622c) corresponding to the fifth specific associated document or the overlapped fifth specific associated document or the specific associated paragraph (1624).

[0283] Meanwhile, the present invention can provide a plurality of recommended documents specified according to various document recommendation criteria to the user terminal (10). The control unit (180) can provide a plurality of recommended documents to the user terminal (10) based on preset document recommendation criteria. Here, the preset document recommendation criteria can be set based on the user's history information. For example, the preset document recommendation criteria can include at least one of i) documents with a high user usage frequency (i.e., documents frequently used by the user), ii) documents recently used by the user, iii) search term criteria by the user (e.g., searches for documents, files, information, etc., by the user on the user terminal (10), and iv) documents with a high user interest.

[0284] For example, as illustrated in FIG. 17, the control unit (180) may provide a plurality of recommended documents (e.g., “document 1”, “document 2”, “document 3”, “document 4”, “document 5”) to one area (first area (1710)) of the service page (1700) output to the user terminal (10) based on preset document recommendation criteria. Alternatively, providing a plurality of recommended documents may also be understood as overlapping graphic objects (or highlighting objects, 1711, 1712, 1713, 1714, 1715) corresponding to each of the plurality of recommended documents (e.g., “document 1”, “document 2”, “document 3”, “document 4”, “document 5”) on the user terminal (10).

[0285] The control unit (180) may provide, to the user terminal, at least one of chat history information related to each of the plurality of recommended documents and information about each of the plurality of recommended documents. When any one of the plurality of recommended documents is selected from the user terminal (10), the control unit (180) may provide, from the user terminal (10), at least one of chat history information related to the selected specific recommended document and information about the specific recommended document. The chat history information may include at least one of a user query related to each of the plurality of recommended documents, an answer generated using each of the plurality of recommended documents, and a related paragraph used to generate an answer for each of the plurality of recommended documents.

[0286] For example, the control unit (180) may provide chat history information (1721, 1722, 1723) included in (or linked, associated, linked, etc.) the selected graphic object to the second area (1720) of the service page (1700) based on the selection of a graphic object (1712) corresponding to the second recommended document (e.g., “document 2”) from among the graphic objects (1711, 1712, 1713, 1714, 1715) corresponding to each of the plurality of recommended documents (e.g., “document 1”, “document 2”, “document 3”, “document 4”, “document 5”) provided from the user terminal (10) to the second area (1720) of the service page (1700). At this time, in an area of ​​a service page (1000) where chat history (1721, 1722, 1723) is provided or in a peripheral area where chat history (1721, 1722, 1723) is provided, a graphic object (1721a, 1722a, 1723a) linked to a function that allows a user to view or move to the corresponding chat may be provided.

[0287] In addition, the control unit (180) may display information about a selected specific recommended document on the user terminal (10). For example, information about a storage location (1731) for a specific recommended document (ex: “document 2”) may be highlighted in the first area (1710) of the service page (1700) output on the user terminal (10) or in the surrounding area of ​​the specific recommended document.

[0288] Alternatively, the control unit (180) may overlap a graphic object (or highlighting object) corresponding to a storage space (e.g., memory, disk, drive, cloud, folder, etc.) in which a specific recommended document is contained (or stored) with the user terminal (10) in relation to the storage location (or storage space) of a specific recommended document.

[0289] For example, in the first area (1710) of the service page (1700), a graphic object (1731b) corresponding to a second specific recommended document (e.g., “document 2”), a graphic object (1731b) corresponding to a first storage space (e.g., “folder 10”) that includes the second specific recommended document (or where the second specific recommended document is located), and a graphic object (1731a) corresponding to a second storage space (e.g., “local disk N”) that includes the first storage space (or where the first storage space is located) may overlap each other.

[0290] As discussed above, the answer generation method and system according to the present invention configures input data consisting of pairs of paragraphs included in a query and candidate documents, and independently encodes the input data to generate semantic representations, thereby enabling parallel processing of a large number of input documents. In other words, the present invention can optimize the semantic representation between paragraphs by encoding multiple paragraphs within an input document in parallel without separate sequential processing, thereby improving the input processing speed of the system and ensuring real-time responsiveness to large data sets.

[0291] Furthermore, the answer generation method and system according to the present invention integrates encoded semantic representations and uses the integrated semantic representations to predict the relevance of the entire document to the query through related word tokens, thereby accurately determining the relevance of the entire document. This enables the present invention to perform accurate document-level rankings that reflect the integrated meaning of multiple paragraphs, thereby improving document retrieval performance compared to conventional information retrieval systems.

[0292] Furthermore, according to the answer generation method and system according to the present invention, by applying an inter-paragraph attention mechanism to encoded semantic expressions to learn relative similarity between paragraphs, it is possible to distinguish semantic differences between multiple paragraphs and improve the precision and ranking consistency of paragraph-level ranking.

[0293] Furthermore, the answer generation method and system according to the present invention trains a model to simultaneously perform document reranking and paragraph extraction necessary for answer generation, thereby ensuring a balanced quality between document and paragraph ranking. This achieves highly reliable ranking results at both the document and paragraph levels, and maintains consistent performance across various search and answer generation systems.

[0294] Furthermore, the answer generation method and system according to the present invention can train a model to simultaneously determine the relevance between a query and an entire document and between a query and each of multiple paragraphs. In other words, the present invention can reduce the influence of noisy contexts among externally retrieved information by optimizing and ranking documents and paragraphs simultaneously, thereby improving the accuracy and reliability of LLM responses generated based on this. Even in a real-time RAG system environment where large-scale information retrieval occurs, rapid and stable search and response processing is possible, errors that may occur in LLM responses can be prevented, and user reliability can be enhanced.

[0295] Furthermore, the answer generation method and system according to the present invention can reduce unnecessary computation and optimize computational resource usage through a multi-granular ranking structure via a single inference path. This can reduce server infrastructure costs and improve the operational efficiency of large-scale AI search systems.

[0296] In addition, according to the method and system for generating an answer according to the present invention, by providing information on a related document related to a user's query and an answer generated using the related document together, the user can intuitively recognize the information he or she needs and understand it more quickly.

[0297] Furthermore, according to the answer generation method and system according to the present invention, by visually providing information about documents utilized in the process of generating answers, the user can intuitively confirm the source and reliability of the answer, quickly and accurately receive the information he or she needs, and maximize the efficiency of searching related documents.

[0298] Furthermore, the answer generation method and system according to the present invention can select relevant documents related to a user's query based on user history information and provide answers to the user's query based on these documents. In other words, the present invention provides users with customized information that reflects their preferences, interests, LLM usage patterns, etc., thereby significantly improving the personalized performance of the answer generation service and the quality of interaction with the user.

[0299] Meanwhile, the present invention discussed above can be implemented based on a quantum computer. The present invention implemented based on a quantum computer may include a quantum processor and quantum memory based on qubits, and may include software and hardware interfaces optimized for quantum computing.

[0300] Quantum processors in quantum computers utilize qubits (qubits) to efficiently perform complex calculations, such as parallel computation, quantum entanglement, and quantum superposition, that classical computers' binary bits cannot perform. Quantum processors process data using quantum gates, and can provide exponential speedups for certain problems.

[0301] Meanwhile, the present invention discussed above can be implemented as a program that is executed by one or more processes on a computer and can be stored on a medium (or recording medium) that can be read by the computer.

[0302] Furthermore, the present invention discussed above can be implemented as computer-readable code or instructions on a program-recorded medium. In other words, the present invention can be provided in the form of a program.

[0303] Meanwhile, computer-readable media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disk drives (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices.

[0304] Furthermore, the computer-readable medium may include a storage device and may be a server or cloud storage device accessible via communication. In this case, the computer may download the program according to the present invention from the server or cloud storage device via wired or wireless communication.

[0305] A computer program can reach the system (100) via a variety of suitable transmission mechanisms. The transmission mechanism may be, for example, a computer-readable storage medium, a computer program product, a memory device, a recording medium such as a CD-ROM or DVD, or a product tangibly embodying a computer program. The transmission mechanism may be a signal configured to reliably transmit a computer program over air or through an electrical connection. The system (100) may propagate or transmit a computer program as a computer data signal.

[0306] Furthermore, references to 'computer-readable storage media', 'computer program product', 'a computer program tangibly embodied', or the like, or to 'controller', 'computer', 'processor', etc., should be understood to include computers having various architectures, such as single / multiprocessor architectures and sequential (Von Neumann) / parallel architectures, as well as specialized circuits such as field-programmable gate arrays (FPGAs), application-specific circuits (ASICs), signal processing devices, and other devices. References to computer programs, instructions, code, etc., should be understood to include software for programmable processors or firmware, such as the programmable contents of hardware devices, whether instructions for a processor or configuration settings for fixed-function devices, gate arrays, or programmable logic devices.

[0307] Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, i.e., a CPU (Central Processing Unit), and there is no particular limitation on its type.

[0308] Meanwhile, the above detailed description should not be construed as limiting in any respect and should be considered illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are intended to be included within the scope of the present invention.

Claims

1. A computerized method comprising: A step of receiving a user query from a user terminal; A step of specifying multiple candidate documents related to the user query based on preset criteria; A step of determining the relevance to the user query for each paragraph included in each of the plurality of candidate documents and distinguished according to preset criteria; A step of specifying a related document related to the user query among the plurality of candidate documents based on the relevance determined for each paragraph included in the user query and each of the plurality of candidate documents; and An answer generation method characterized by comprising a step of generating an answer to the user query using the above-mentioned related document.

2. In paragraph 1, A step of configuring input data so that a specific paragraph among the paragraphs included in each of the user query and the plurality of candidate documents forms a pair; and In order to specify the related document associated with the user query among the plurality of candidate documents, the step of processing the input data as input to an encoder is further included. In the step of determining the relevance to the above user query, An answer generation method characterized by comprising a step of determining the relevance with the user query for each paragraph included in each of the plurality of candidate documents using an embedding vector corresponding to the input data obtained from the encoder.

3. In paragraph 2, A method for generating an answer, characterized in that when there are multiple specific paragraphs, the input data includes multiple specific paragraphs.

4. In paragraph 2, An answer generation method characterized in that the encoder performs an interaction between the user query included in the input data and the specific paragraph to obtain an embedding vector corresponding to the input data.

5. In paragraph 3, An answer generation method characterized in that, when the input data includes the plurality of specific paragraphs, a plurality of embedding vectors corresponding to the input data are obtained from the encoder.

6. In paragraph 5, The step of determining the relevance to the above user query is: A step of combining the above plurality of embedding vectors into one to generate a combined embedding vector; A step of processing the combined embedding vector as input to a decoder; and An answer generation method characterized by comprising a step of determining the relevance of each of the plurality of candidate documents to the user query using the combined embedding vector in the decoder.

7. In paragraph 6, The above combined embedding vector is, A method for generating an answer, characterized in that it represents the entire semantic association between the user query and the paragraphs included in each of the plurality of candidate documents.

8. In paragraph 6, The above decoder, An answer generation method characterized in that an association word token is generated for each of the plurality of candidate documents based on the association determined using the combined embedding vector.

9. In paragraph 8, The above related word tokens are, A token indicating whether there is a correlation between the above user query and the above document, A method for generating an answer, characterized in that it comprises a first relevance word token or a second relevance word token depending on whether there is a relevance between the user query and the document.

10. In paragraph 8, The above decoder, An answer generation method characterized in that it is configured to predict a related word token for each of the plurality of candidate documents.

11. In paragraph 9, An answer generation method characterized in that a document including the first related word token among the plurality of candidate documents is specified as the related document associated with the user query.

12. In paragraph 2, If the input data includes multiple specific paragraphs, obtain multiple embedding vectors corresponding to the input data from the encoder, The step of determining the relevance to the above user query is: A step of processing the above plurality of embedding vectors as input to a ranker layer; and An answer generation method characterized by comprising a step of determining the relevance with the user query for each paragraph included in each of the plurality of candidate documents using the plurality of embedding vectors in the ranker layer.

13. In paragraph 12, The above ranker layer is, An answer generation method characterized in that a paragraph relevance score is generated for each paragraph included in each of the plurality of candidate documents based on the relevance determined using the plurality of embedding vectors.

14. In paragraph 9, The above paragraph relevance score is, A score indicating whether there is a correlation between each of the above multiple paragraphs and the user query, An answer generation method characterized in that it comprises a first score or a second score depending on whether there is a correlation between each of the plurality of paragraphs and the user query.

15. In paragraph 13, The above ranker layer is, An information response generation method characterized in that, for the plurality of embedding vectors, a paragraph-to-paragraph attention mechanism is used to learn to predict the relative similarity between paragraphs included in each of the plurality of candidate documents.

16. In paragraph 1, In the above specific step, An answer generation method characterized in that it further comprises a step of specifying a relevant paragraph related to the user query among paragraphs included in each of the plurality of candidate documents, in order to generate an answer to the user query.

17. In paragraph 16, In the step of specifying the above related paragraph, Among the paragraphs included in each of the above multiple candidate documents, a paragraph that satisfies the established criteria is specified as the related paragraph, The above related paragraphs are: A method for generating an answer characterized in that it corresponds to a paragraph included in the associated document associated with the user query.

18. In paragraph 17, In the step of generating the above answer, An answer generation method characterized in that an answer to the user query is generated using at least one of the above-mentioned related documents and the above-mentioned related paragraphs.

19. A system comprising a memory configured to store executable instructions, and one or more processors configured to perform operations by executing one or more instructions, The above system, Receive a user query from the user terminal, Based on the preset criteria, multiple candidate documents related to the user query are identified, Each of the above multiple candidate documents is included and, for each paragraph that is distinguished according to preset criteria, the relevance to the user query is determined, Based on the above user query and the above relevance determined for each paragraph included in each of the above multiple candidate documents, a related document related to the above user query is specified among the above multiple candidate documents, An information retrieval system characterized in that it generates an answer to the user query using the above related document.

20. A computerized method comprising: A step of receiving a user query from a user terminal; A step of specifying multiple candidate documents related to the user query based on preset criteria; A step of specifying a related document related to the user query among the plurality of candidate documents based on the relevance between the user query and the paragraphs included in each of the plurality of candidate documents; A step of generating an answer to the user query using the above-mentioned related document; and A method for generating an answer, characterized by comprising a step of providing the answer together with information about the related document used to generate the answer to the user terminal.

Images