On-device question-answering processing apparatus and method for kiosk

Abstract

The present invention relates to an on-device question-and-answer processing device and method. An on-device question-and-answer processing device according to one embodiment stores document data regarding a plurality of documents and an embedding index including pre-calculated embedding information for each of the plurality of documents, and can perform vocabulary-based search and semantic-based search based on a user query. The device can form a candidate document group based on the ranking of the vocabulary-based search results and the semantic-based search results, include a preset number of top documents with high semantic similarity in the candidate document group regardless of the fusion ranking, and then select at least one document from the candidate document group. Furthermore, the device can generate an answer using a large-scale language model based on information contained in the user query and the selected document, and determine a pre-defined action in the top document as an action corresponding to the user query. In addition, if the inference of the large-scale language model fails, it can provide an alternative answer based on information contained in the top document and manage cache entries of the response cache based on a value corresponding to the regeneration cost of the generated answer.

Description

Technology Field

[0001] The present invention relates to information retrieval and question-and-answer processing technology, and more specifically, to a question-and-answer processing device and method that provide answers corresponding to user queries in an on-device environment. Background Technology

[0002] Kiosk terminals are utilized in various locations, such as libraries, government offices, hospitals, exhibition facilities, and transportation facilities, to provide users with services including facility guidance, information search, reservations, registration, and civil complaint processing. Recently, in addition to the method where users sequentially select from a fixed menu, interactive question-and-answer (Q&A) methods, which allow users to input queries in natural language and receive corresponding responses, are being applied to kiosk terminals.

[0003] In natural language-based question-and-answer services, a search augmentation generative approach can be utilized by combining large-scale language models with document search technology to retrieve information related to user queries and generate answers based on the retrieved information. This approach has the advantage of providing answers based on separately constructed document data, rather than relying solely on information previously learned by the large-scale language model.

[0004] Meanwhile, on-device environments, such as kiosk terminals or adjacent edge servers, may have limited available computing and memory resources compared to external cloud servers. Consequently, repetitive encoding of multiple documents, complex search processing, the construction of long prompts, or iterative inference of large-scale language models can increase response delays and computational load. In particular, in kiosk services where users wait for a response directly in front of the terminal, such delays can degrade the convenience of using the service.

[0005] Furthermore, if natural language query expressions vary or documents have similar content, documents substantially related to the user's query may be omitted from search results, or documents with low relevance may be used to generate answers. If the inference of large-scale language models is not performed properly or inaccurate results are generated, the stability of the question-and-answer service may also be compromised. In addition, kiosk services that provide screen transitions or function execution along with answers require not only the accuracy of the answers but also the consistency and reliability of user interface behavior.

[0006] Therefore, there is a need for question-and-answer processing technology that can reliably provide information related to user queries while reducing the throughput required for document search and answer generation in on-device environments where computational and memory resources are limited. The problem to be solved

[0007] One objective of the present invention is to provide a question-and-answer processing device and method capable of effectively selecting documents related to a user query in an on-device environment having limited computational resources, and providing a reliable answer based thereon.

[0008] Another objective of the present invention is to address response generation failures and reduce response delays and computational load by efficiently managing the response cache. means of solving the problem

[0009] An on-device question-and-answer processing device corresponding to a user query input through a kiosk terminal according to one embodiment comprises: a storage unit that stores document data regarding a plurality of documents and an embedding index that includes pre-calculated embedding information for each of the plurality of documents; and an on-device question-and-answer processing unit that acquires a user query, performs a vocabulary-based search and a semantic-based search based on the user query, calculates a fusion ranking based on the respective rankings of the vocabulary-based search results and the semantic-based search results, forms a candidate document group based on the fusion ranking, includes a preset number of top documents in order of high semantic similarity among the semantic-based search results in the candidate document group regardless of the fusion ranking, selects at least one document from the candidate document group, and generates an answer using a large-scale language model based on the user query and information included in the selected at least one document.

[0010] The above-described on-device question-and-answer processing unit can perform the vocabulary-based search, the semantic-based search, the configuration of the candidate document group, and document selection at least one of the kiosk terminal or an adjacent edge server.

[0011] The above-described on-device question-and-answer processing unit may exclude from the above-described candidate document group a candidate document whose semantic similarity to the user query does not satisfy a preset absolute threshold among the candidate documents included in the above-described candidate document group.

[0012] The above-described on-device question-and-answer processing unit may output a preset response indicating that information related to the user query does not exist when there is no candidate document satisfying the absolute threshold.

[0013] The above-described on-device question-and-answer processing unit can determine the number of documents to be selected from the group of candidate documents based on the confidence gap between the top candidate documents when there are candidate documents that satisfy the absolute threshold.

[0014] The above-determined on-device question-and-answer processing unit may select at least one document by prioritizing candidate documents with high reliability for the user query according to the number of documents determined above, while lowering the selection priority of candidate documents that overlap in content with or are semantically similar to documents already selected.

[0015] The above document data is structured RAG data including a plurality of document entries, and each of the plurality of document entries may include search target information used for the vocabulary-based search or the semantic-based search, information body that is distinguished from the search target information and excluded from the target of the vocabulary-based search and the semantic-based search and provided to the large-scale language model, and action information corresponding to a predefined user interface operation.

[0016] The above search target information includes search term information and title information, and the on-device question-and-answer processing unit can perform the vocabulary-based search based on the search term information and perform the semantic-based search based on the embedding index including a pre-calculated embedding for the title information.

[0017] The above-described on-device question-and-answer processing unit can determine an action corresponding to the user query based on action information included in the top-level document among the selected at least one document, and output the generated answer and the determined action together.

[0018] The above-described on-device question-and-answer processing unit, when the generation of an answer using the above-described large-scale language model is not performed normally, obtains an alternative answer based on the information body of the above-described top-level document, outputs the alternative answer and the above-described determined action, and may not store the alternative answer in the response cache.

[0019] The above storage unit includes a response cache that stores a cache item containing an answer generated in response to a user query and an action associated with the answer, and the on-device question-response processing unit determines whether a cache item corresponding to the user query exists in the response cache before performing the vocabulary-based search and the semantic-based search, and if a cache item corresponding to the user query exists, outputs the answer and action included in the cache item.

[0020] The above-described on-device question-and-answer processing unit may, if there is no cache item corresponding to the user query, calculate a value corresponding to the regeneration cost of the answer generated by the large-scale language model, and store the user query, the generated answer, and the action associated with the generated answer in the response cache in conjunction with the value corresponding to the regeneration cost.

[0021] The above-described on-device question-and-answer processing unit may, when there is no cache item corresponding to the user query, calculate a value corresponding to a regeneration cost based on at least one of the length of the answer generated by the large-scale language model or the number of generated tokens, and set an ejection deferral index to defer the ejection of a new cache item based on the value corresponding to the regeneration cost.

[0022] The above-described on-device question-and-answer processing unit sets an ejection deferral index for new cache items based on a value corresponding to the cost of regenerating an answer generated by the large-scale language model, and if there is insufficient space to store the new cache items in the response cache, selects candidate cache items for ejection according to a standard ejection policy, and if the ejection deferral index of the candidate cache items for ejection is 0 or less, ejects the candidate cache items for ejection from the response cache, and if the ejection deferral index of the candidate cache items for ejection is greater than 0, reduces the ejection deferral index of the candidate cache items for ejection and retains the candidate cache items for ejection in the response cache.

[0023] An on-device question-and-answer processing method performed by an on-device question-and-answer processing device corresponding to a user query input through a kiosk terminal according to another embodiment comprises: a step of storing document data regarding a plurality of documents and an embedding index including pre-calculated embedding information for each of the plurality of documents; a step of obtaining a user query; a step of performing a vocabulary-based search and a semantic-based search based on the user query; a step of calculating a fusion rank based on the respective ranks of the vocabulary-based search results and the semantic-based search results, and configuring a candidate document group based on the fusion rank; a step of including a preset number of top documents in order of high semantic similarity among the semantic-based search results in the candidate document group regardless of the fusion rank; a step of selecting at least one document from the candidate document group; and a step of generating an answer using a large-scale language model based on the user query and information included in the selected at least one document. Effects of the invention

[0024] According to the present invention, since a group of candidate documents related to a user query can be constructed by using both vocabulary-based search results and semantic-based search results together, relevant documents can be effectively searched not only when the expression included in the query matches the expression included in the document, but also when different expressions are used.

[0025] In addition, a pre-set number of top documents with high semantic similarity among semantic-based search results can be included in the candidate document set regardless of the fusion ranking. Accordingly, even if the ranking of semantically important documents decreases during the process of fusing the rankings of vocabulary-based and semantic-based search results, it is possible to prevent those documents from being omitted from the candidate document set.

[0026] Furthermore, by comparing the semantic similarity of candidate documents with an absolute threshold, documents with low relevance can be excluded, thereby reducing the provision of documents to large-scale language models that are not substantially related to user queries. Consequently, this prevents the degradation of response quality caused by unnecessary documents and reduces the length of prompts and the amount of computation required for response generation.

[0027] In addition, the number of documents provided to a large-scale language model can be adjusted based on the confidence gap between top candidate documents. Accordingly, when the relevance of a specific document is clear, a response can be generated quickly using a small number of documents, and when multiple documents have similar relevance, sufficient information can be provided using more documents. Thus, a balance can be secured between the sufficiency of information used for response generation and the limited computational resources of an on-device environment.

[0028] Furthermore, since the relevance to the user query and the diversity among multiple documents can be considered together when selecting the final documents, the repeated provision of substantially identical or redundant documents to large-scale language models can be reduced. Accordingly, documents containing different perspectives or information can be provided within a limited input length, thereby enhancing the informativeness of the generated answers.

[0029] Furthermore, information used for searching, information used for generating answers, and action information can be managed separately from document data. Accordingly, the body of selected documents can be provided to a large-scale language model for answer generation without using the entire source text of the document as a search target. Therefore, detailed information necessary for answers provided to users can be maintained while reducing search throughput.

[0030] Furthermore, since actions corresponding to user queries can be determined using predefined action information in the top-level document, the consistency and predictability of screen transitions, function execution, or user interface control can be enhanced compared to cases where a large-scale language model randomly generates actions. Accordingly, discrepancies between the response content and the actions performed on the kiosk terminal can be reduced, and the reliability of the question-and-answer service can be improved.

[0031] In addition, if the inference of a large-scale language model fails or it cannot generate a valid answer, an alternative answer based on the information body of the top-level document can be provided. Accordingly, since users can be provided with at least minimal valid information even if temporary errors, resource shortages, or processing delays occur in the large-scale language model, the availability and fault tolerance performance of the on-device question-and-answer service can be improved.

[0032] In addition, a value corresponding to the cost of regenerating an answer can be calculated based on the length, number of characters, or number of generated tokens of the answer generated by a large-scale language model, and a cache entry evicting deferral metric can be set accordingly. As a result, answers that require a large amount of computation or processing time to regenerate after being evicted from the response cache can be retained for a relatively long period.

[0033] Furthermore, if an eviction deferral metric remains for a candidate cache entry, the entry is retained while decreasing the metric; conversely, if the deferral metric is exhausted, the corresponding cache entry can be evictioned. Accordingly, this reduces the unnecessary repeated generation of answers with high regeneration costs while preventing specific cache entries from being maintained indefinitely. Consequently, the storage space of the response cache can be utilized efficiently, and the average response time and the computational load required for large-scale language model inference can be reduced.

[0034] Furthermore, since embedding indexes are generated and stored in advance and can be utilized during on-device query-response processing, there is no need to repeatedly calculate embeddings for the entire document every time a user query is entered. In addition, by verifying the integrity of the stored embedding indexes, it is possible to prevent corrupted or inconsistent indexes from being used for searching and to enhance the stability of semantic-based search.

[0035] Consequently, the present invention can reliably select documents related to user queries and provide reliable answers and consistent actions, even in on-device environments with limited computational and memory resources, while reducing response delays and computational loads. Brief explanation of the drawing

[0036] FIG. 1 is a diagram showing the overall configuration of an on-device kiosk question-and-answer system according to one embodiment. FIG. 2 is a diagram showing the detailed configuration of an on-device question-and-answer processing unit according to one embodiment. FIG. 3 is a diagram showing the data structure of structured RAG data and the relationship of use of each piece of information according to one embodiment. FIG. 4 is a flowchart showing the overall operation of an on-device question-and-answer processing method according to one embodiment, FIG. 5 is a flowchart illustrating a hybrid context screening process according to one embodiment, FIG. 6 is a flowchart illustrating a regeneration cost-aware response cache management process according to one embodiment. Specific details for implementing the invention

[0037] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0038] In describing the embodiments of the present invention, if it is determined that a detailed description of known functions or configurations may unnecessarily obscure the essence of the invention, such detailed description will be omitted. Furthermore, the terms described below are defined considering the functions in the embodiments of the present invention, and these may vary depending on the intentions or conventions of the user or operator. Therefore, such definitions should be based on the content throughout this specification.

[0039] Additionally, each block or each step may represent a module, segment, or part of code containing one or more executable instructions for executing specific logical functions, and it should be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps described in succession may actually be performed substantially simultaneously, and the blocks or steps may also be performed in the reverse order of the corresponding functions as needed.

[0040] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. However, the embodiments of the present invention exemplified below may be modified in various different forms, and the scope of the present invention is not limited to the embodiments described below. The embodiments of the present invention are provided to more completely explain the present invention to those skilled in the art to which this invention pertains.

[0041] FIG. 1 is a diagram showing the overall configuration of an on-device kiosk question-and-answer system according to one embodiment.

[0042] Referring to FIG. 1, the on-device kiosk question-and-answer system includes a kiosk terminal (100) and may optionally further include an adjacent edge server (200) that is connected to communicate with the kiosk terminal (100).

[0043] In this specification, an on-device environment may refer to an environment in which all or part of the query-response processing is performed in a local computing environment, such as a kiosk terminal (100) or an adjacent edge server (200) that is communicably connected to the kiosk terminal (100). This on-device environment may be configured to limit available computing and memory resources and to minimize dependence on an external cloud. Accordingly, processing that requires a high amount of computation or memory usage, such as repetitive encoding of an entire document using a large-scale embedding model, complex preprocessing performed for each query, or processing of long prompts, may be limited.

[0044] Meanwhile, the on-device method in this specification is not limited to cases where all question-and-answer processing is performed only within the kiosk terminal (100). For example, part of the question-and-answer processing may be performed at the kiosk terminal (100), and the remaining part may be performed at an adjacent edge server (200).

[0045] The kiosk terminal (100) may include a user interface unit (110), an on-device question and answer processing unit (120), and a storage unit (130). The kiosk terminal (100) may be implemented as an unmanned terminal that provides guidance information to users, for example, in a library, government office, hospital, exhibition facility, or transportation facility, but its application location and use are not limited thereto.

[0046] The user interface unit (110) can receive a query from the user (10) and output an answer and an action corresponding to the input query. In one embodiment, the user interface unit (110) includes a touchscreen and can receive a text query through an input window or a virtual keyboard displayed on the touchscreen. Additionally, the user interface unit (110) can display at least one action button corresponding to switching to a specific screen, menu, or function along with the generated answer. The method of inputting the query is not limited to text input but can be extended to voice input or the use of an external input device.

[0047] The on-device question-and-answer processing unit (120) can process a query entered through the user interface unit (110) and determine an answer and action corresponding to the query. In one embodiment, the on-device question-and-answer processing unit (120) can select documents related to the query based on document data and a search index stored in the storage unit (130), and generate an answer using a Large Language Model (LLM) based on information contained in the selected documents.

[0048] Additionally, the on-device question-and-answer processing unit (120) can check for predefined user interface actions in selected documents and determine the corresponding user interface actions as actions to be provided along with the answer. Accordingly, the content of the answer can be generated using a large-scale language model, while actions used for screen transitions, etc., can be determined based on predefined information.

[0049] In one embodiment, the on-device question-and-answer processing unit (120) can check whether an answer and an action corresponding to an input query are stored in the response cache. If a corresponding cache item exists, the stored answer and action are provided through the user interface unit (110), and if a corresponding cache item does not exist, the selection of related documents and the generation of an answer can be performed. In addition, if the call of a large-scale language model is not performed normally, an alternative answer can be provided using information contained in the selected documents.

[0050] The storage unit (130) can store data used for the operation of the on-device question-and-answer processing unit (120). For example, the storage unit (130) can store at least one of structured Retrieval-Augmented Generation (RAG) data, a pre-calculated embedding index, and a response cache.

[0051] Structured search augmented generative data may include search target information used for document search, information body provided to a large-scale language model, and action information defining user interface behavior. A pre-calculated embedding index may include embedding information calculated in advance for the search target information, and a response cache may include answers and actions generated in response to user queries. The specific structure and usage method of this data will be explained in detail with reference to the drawings described below.

[0052] In one embodiment, the kiosk terminal (100) may be connected to communicate with an adjacent edge server (200). The adjacent edge server (200) may be a computing device deployed in a network environment closer to the kiosk terminal (100) than an external cloud server, and may support at least some of the functions of the on-device question-and-answer processing unit (120).

[0053] For example, at least some of the inference of a large-scale language model, document search, or management of an embedding index may be performed at an adjacent edge server (200). In another embodiment, the entire question-and-answer processing may be performed within the kiosk terminal (100), and in yet another embodiment, it may be performed distributed between the kiosk terminal (100) and the adjacent edge server (200).

[0054] In this way, the kiosk terminal (100) can provide answers and actions in response to queries in a local computing environment while reducing reliance on an external cloud. Accordingly, response delays can be reduced even in environments where computing and memory resources are limited, and the interruption of the question-response service can be prevented even if a failure occurs in a large-scale language model.

[0055] FIG. 2 is a diagram showing the detailed configuration of an on-device question-and-answer processing unit according to one embodiment.

[0056] Referring to FIG. 2, the on-device question-and-answer processing unit (120) may include a cache management unit (121), a hybrid context selection unit (122), an action decision unit (123), a prompt configuration unit (124), a large-scale language model inference unit (125), and a fault response unit (126). The storage unit (130) may store structured RAG data (131), an embedding index (132), and a response cache (133).

[0057] The cache management unit (121) can retrieve a cache item corresponding to a query entered through the user interface unit (110) from the response cache (133). If a cache item corresponding to the query exists, the cache management unit (121) can provide the answer and action included in the cache item to the user interface unit (110). If a cache item corresponding to the query does not exist, the cache management unit (121) can cause the entered query to be processed by the hybrid context selection unit (122).

[0058] Additionally, the cache management unit (121) can store user queries, newly generated answers, and actions determined in response thereto in the response cache (133). At this time, the cache management unit (121) can store information corresponding to the cost of regenerating the answer in conjunction with the cache item, and can manage whether to extract or retain the cache item based on the said information. A specific method for managing cache items will be explained with reference to the drawings described later.

[0059] The hybrid context selection unit (122) performs vocabulary-based search and semantic-based search on the input query and can select at least one document related to the query based on the results of each search. In one embodiment, the hybrid context selection unit (122) can generate a candidate document group using the search target information and embedding index (132) included in the structured RAG data (131) and select a document to be used for inference of a large-scale language model from the candidate document group.

[0060] The hybrid context selection unit (122) may include the top N documents with the highest semantic similarity among the semantic-based search results in a candidate document group regardless of the fusion ranking. Additionally, the number and composition of the final documents may be determined based on at least one of the semantic similarity of the documents, the difference in reliability between top documents, the relevance of the documents, or the diversity between documents. Specific details of the hybrid context selection process will be described later.

[0061] The action determination unit (123) can identify predefined user interface actions corresponding to documents selected by the hybrid context selection unit (122) and determine actions to be provided to the user. In one embodiment, the action determination unit (123) can identify action information included in the document with the highest ranking among the selected documents and determine it as an action for screen switching, menu navigation, or function execution.

[0062] Accordingly, the response provided to the user can be generated by a large-scale language model, whereas the action provided along with the response can be determined based on information predefined in the document. Therefore, it is possible to prevent the generation of user interface behaviors that are not predefined or are inaccurate by the large-scale language model.

[0063] The prompt configuration unit (124) can configure a prompt to be input into a large-scale language model based on the input query and documents selected by the hybrid context selection unit (122). In one embodiment, the prompt configuration unit (124) can include the information body of the selected documents in the prompt. At this time, the information used for document search among the structured RAG data (131) and the information provided to the large-scale language model can be distinguished from each other.

[0064] The large-scale language model inference unit (125) can generate an answer corresponding to an input query based on a prompt configured by the prompt configuration unit (124). The large-scale language model inference unit (125) can operate within the kiosk terminal (100), and in other embodiments, it can operate at the adjacent edge server (200) of FIG. 1 or be distributed between the kiosk terminal (100) and the adjacent edge server (200).

[0065] The fault response unit (126) can provide an alternative answer when the generation of an answer using the large-scale language model inference unit (125) is not performed normally. Here, cases where the generation of an answer is not performed normally may include a failure of connection of the large-scale language model, a response timeout, a lack of computational resources, or an error in the inference process.

[0066] In one embodiment, the fault response unit (126) may provide the information body of the document with the highest ranking among the documents selected by the hybrid context selection unit (122) as a substitute answer. Accordingly, even if the generation of an answer by a large-scale language model fails, the interruption of the question-and-answer service of the kiosk terminal (100) can be prevented.

[0067] The storage unit (130) can store structured RAG data (131), an embedding index (132), and a response cache (133). The structured RAG data (131) may include a plurality of document entries including search target information, information body, and action information, and the embedding index (132) may include pre-calculated embedding information for at least some of the search target information. The response cache (133) may include a plurality of cache items including information corresponding to user queries, generated answers, actions, and the cost of regenerating the answers.

[0068] Meanwhile, the connection relationships between the configurations illustrated in FIG. 2 represent functional or data transfer relationships between each configuration. Therefore, the operation of each configuration is not limited to being performed only in the arrangement order illustrated in FIG. 2. Additionally, at least some of the multiple configurations illustrated in FIG. 2 may be integrated into a single software module or implemented by being separated into multiple modules.

[0069] FIG. 3 is a diagram showing the data structure of structured RAG data and the relationship of use of each piece of information according to one embodiment.

[0070] Referring to FIG. 3, the structured RAG data (131) may include a plurality of document entries (300). Each document entry (300) may include search target information (310), information body (320), action information (330), and classification information (340).

[0071] Search target information (310) is information for searching document entries (300) related to a user query and may include search term information (311) and title information (312). The search term information (311) may include at least one search term pre-selected to correspond to the content of the document entry (300) or the intent of the query. For example, the search term information (311) may be composed of an array of curated search terms. The title information (312) may include a title or short sentence indicating the subject of the document entry (300) or the intent of the query.

[0072] Search target information (310) can be used for vocabulary-based search and semantic-based search. In one embodiment, search term information (311) can be used for vocabulary-based search to determine the correspondence with words or expressions included in a user query, and title information (312) can be used for semantic-based search to determine semantic similarity with the user query. However, the method of using search term information (311) and title information (312) is not limited thereto, and each may be used in at least one of vocabulary-based search and semantic-based search.

[0073] The embedding for the search target information (310) can be calculated in advance before processing the online query, and an embedding index (132) can be constructed based on the calculated embedding. Accordingly, the embedding index (132) can be constructed using only the information necessary for the search, without embedding or indexing the entire content of the document entry (300).

[0074] The information body (320) may include information used to generate guidance content or answers corresponding to the document entry (300). For example, the information body (320) may include content regarding the facility's usage hours, usage method, location, provided services, or usage conditions.

[0075] The information body (320) is distinguished from the search target information (310) and can be excluded from the construction target of the document search and embedding index (132). When a document entry (300) is selected by the hybrid context selection unit (122), the information body (320) of the corresponding document entry (300) is provided to the prompt configuration unit (124) and can be used as a context for a prompt to generate an answer corresponding to a user query. Thus, the search space and the information for generating answers can be separated by using the search target information (310) for document search and the information body (320) for answer generation.

[0076] The action information (330) may include at least one predefined user interface action corresponding to the document entry (300). The action information (330) may correspond, for example, to switching to a specific guidance screen, moving to a menu, executing a search function, displaying a reservation screen, or executing other kiosk functions.

[0077] The action determination unit (123) can determine the action to be provided to the user along with the answer by using the action information (330) of the selected document entry (300). Accordingly, even if the answer is generated by a large-scale language model, the user interface behavior can be determined based on the action information (330) predefined in the document entry (300).

[0078] Classification information (340) may indicate the type or category to which a document entry (300) belongs. Classification information (340) may be used, for example, for managing document data, sorting search results, diagnosing processing results, or calculating statistics. However, classification information (340) may be omitted depending on the embodiment.

[0079] In one embodiment, the information body (320) may be maintained without altering the content included in the original FAQ or original guidance data. Search term information (311), title information (312), action information (330), and classification information (340) may be composed of metadata linked to the original information body (320). Accordingly, the configuration of the structured RAG data (131) may not imply repeatedly preprocessing or augmenting the original data during the online query processing process.

[0080] In one embodiment, the structured RAG data (131) may be configured according to a pre-set data schema before online query processing is performed. Thus, during the online query processing, document search based on the search target information (310) can be performed without performing morphological analysis of the document text, repetitive embedding of the entire document, or expanding the user query.

[0081] In this way, by configuring the search target information (310), information body (320), and action information (330) of the document entry (300) separately from one another, the size of the data used for searching and indexing can be reduced, and the mixing of search signals and answer information can be prevented. Additionally, by linking the action information (330) to the document entry (300) in advance, the need for a large-scale language model to separately generate user interface actions can be reduced.

[0082] FIG. 4 is a flowchart showing the overall operation of an on-device question-and-answer processing method according to one embodiment.

[0083] Referring to FIG. 4, the on-device question-and-answer processing process may include offline pre-processing performed prior to online query processing and online query processing performed in response to user queries. In one embodiment, offline pre-processing may be performed by an on-device question-and-answer processing unit (120) or an adjacent edge server (200). Online query processing may be performed by the on-device question-and-answer processing unit (120), and part of the online query processing may be supported by the adjacent edge server (200).

[0084] First, the on-device question and answer processing unit (120) or the adjacent edge server (200) can configure structured RAG data (131) (S110). As described above, the structured RAG data (131) can be configured to include search target information (310), information body (320), action information (330), and classification information (340) separately for each document.

[0085] Next, the on-device question-and-answer processing unit (120) or the adjacent edge server (200) can calculate an embedding for the search target information (310) (S120). In one embodiment, an embedding can be pre-calculated for at least one of the search term information (311) and the title information (312). When multiple embedding models are used, an embedding for the search target information (310) can be calculated for each embedding model, and the calculated embedding can be stored in the embedding index (132).

[0086] Subsequently, the on-device question-and-answer processing unit (120) or the adjacent edge server (200) may generate and store an embedding index (132) based on the calculated embedding. The on-device question-and-answer processing unit (120) may verify the integrity of the embedding index (132) when loading it (S130). For example, it may check whether the identifier of the embedding model corresponding to the embedding index (132) matches the identifier of the currently used embedding model, and check whether the document identifier or document order included in the embedding index (132) matches the document identifier or document order of the structured RAG data (131). If the verification result does not match, the embedding for the search target information (310) may be recalculated and the embedding index (132) updated.

[0087] In the process of processing online queries, the on-device question-and-answer processing unit (120) can obtain a user query (S210). The user query can be obtained in the form of text, and if voice input is used, it can be obtained in the form of text corresponding to the voice recognition result.

[0088] Next, the on-device question-and-answer processing unit (120) can query the response cache (133) to check if a cache item corresponding to an input user query exists in the response cache (133) (S220). The on-device question-and-answer processing unit (120) can determine whether there is a cache hit by comparing the user query with the query information or identification information associated with the cache item (S230).

[0089] When a cache is hit, the on-device question-and-answer processing unit (120) can output the answer and action stored in the response cache (133) through the user interface unit (110) (S240). Accordingly, results for the user query can be provided without performing document search and large-scale language model inference again.

[0090] If the cache is not hit, the on-device question-and-answer processing unit (120) can perform hybrid context selection corresponding to the user query (S250). Hybrid context selection may include processing to construct candidate documents using vocabulary-based search results and semantic-based search results, and to select documents among the candidate documents to be used for generating answers from a large-scale language model. The specific hybrid context selection process is explained with reference to the drawings described later.

[0091] The on-device question-and-answer processing unit (120) can determine whether there is a document related to the user query based on the hybrid context selection result (S260). If there is no related document, the on-device question-and-answer processing unit (120) can output a response corresponding to no information (S270). The response corresponding to no information may include a pre-configured phrase indicating that the relevant information was not found.

[0092] If a relevant document exists, the on-device question-and-answer processing unit (120) can determine an action based on the top document among the selected documents (S280). Specifically, it can check the action information (330) included in the top document and determine an action for screen switching, menu movement, or function execution corresponding to the action information (330).

[0093] Next, the on-device question-and-answer processing unit (120) can configure a prompt based on the user query and the information body (320) of the selected documents, and perform large-scale language model inference using the configured prompt (S290). At this time, if multiple documents are selected, the information body (320) included in the multiple selected documents may be included in the prompt.

[0094] The on-device question-and-answer processing unit (120) can determine whether a normal response has been obtained from a large-scale language model (S300). For example, if a connection failure of the large-scale language model, a response timeout, insufficient computational resources, or an inference process error occurs, it can be determined that a normal response has not been obtained.

[0095] If a normal response is not obtained, the on-device question-and-answer processing unit (120) can obtain an alternative answer based on the information body (320) of the top-level document (S310). In one embodiment, the original answer included in the information body (320) of the top-level document can be used as an alternative answer without separate large-scale language model inference.

[0096] In one embodiment, alternative answers obtained in response to a failure of the large-scale language model may not be stored in the response cache (133). Accordingly, when the same or corresponding query is entered again after the large-scale language model is normalized, a newly generated answer using the large-scale language model can be provided.

[0097] When a normal response is obtained, the on-device question-and-answer processing unit (120) can obtain an answer generated by a large-scale language model (S320). Subsequently, the on-device question-and-answer processing unit (120) can store the corresponding user query, answer, and action in the response cache (133) based on the future regeneration cost of the generated answer (S330). Here, the regeneration cost may be determined based on a value corresponding to the length of the generated answer or the number of generated tokens, and may be linked with information for deferring the extraction of cache items. The specific response cache management process is explained with reference to the drawings described later.

[0098] Finally, the on-device question-and-answer processing unit (120) can output a normally generated answer or alternative answer and an action determined in S280 (S340). Accordingly, an answer and an action corresponding to the query can be provided to the user regardless of whether the large-scale language model is functioning normally.

[0099] Meanwhile, offline preprocessing is not limited to being performed repeatedly whenever an online query is entered, but can be performed in advance when the structured RAG data (131) is created or modified. In addition, at least some of the steps shown in FIG. 4 may be changed in order or performed in parallel depending on the embodiment.

[0100] FIG. 5 is a flowchart illustrating a hybrid context screening process according to one embodiment.

[0101] The steps illustrated in FIG. 5 can be performed by the hybrid context selection unit (122) of the on-device question-and-answer processing unit (120).

[0102] Referring to FIG. 5, the hybrid context selection unit (122) can perform a vocabulary-based search for a user query based on search term information (311) and determine the ranking of the vocabulary-based search results (S510).

[0103] Additionally, the hybrid context selection unit (122) can perform a semantic-based search for a user query based on title information (312) and an embedding index (132), and determine the ranking of the semantic-based search results (S520).

[0104] In one embodiment, the hybrid context selection unit (122) calculates a query embedding corresponding to a user query and can determine the ranking of semantic-based search results based on the semantic similarity between the query embedding and the embedding of title information (312) included in the embedding index (132).

[0105] When multiple embedding models are used, the hybrid context selection unit (122) calculates a query embedding corresponding to each embedding model for a user query and can combine and use semantic similarity or search results calculated by each embedding model.

[0106] Lexical-based search and semantic-based search can be performed sequentially or in parallel.

[0107] Next, the hybrid context selection unit (122) can calculate a fusion ranking based on the respective rankings of the vocabulary-based search results and the semantic-based search results, and can form a candidate document group based on the calculated fusion ranking (S530). In one embodiment, the hybrid context selection unit (122) can calculate a document-specific fusion value using the vocabulary-based search ranking and the semantic-based search ranking of each document, and determine the fusion ranking of the documents according to the fusion value. At this time, the specific ranking fusion method or the weight assigned to each search result may be set differently depending on the embodiment.

[0108] Subsequently, the hybrid context selection unit (122) may include the top N documents with semantic similarity in the candidate document group regardless of the fusion ranking (S540). Here, N may be a pre-set natural number. Among the top N documents with semantic similarity, documents that are already included in the candidate document group based on the fusion ranking are maintained in the candidate document group, and documents that are not included may be added to the candidate document group.

[0109] Accordingly, even if a vocabulary-based search ranking is calculated low because the user query is entered in colloquial language or expressions different from the stored search terms, it is possible to prevent documents with high semantic relevance to the user query from being omitted from the candidate document set. In other words, it is possible to go beyond simply fusing vocabulary-based and semantic-based search results and ensure that top-ranking documents from the semantic-based search results are included in the candidate document set.

[0110] Next, the hybrid context screening unit (122) can exclude candidate documents among the candidate document group whose semantic similarity does not meet an absolute threshold (S550). The absolute threshold can correspond to the minimum semantic similarity required to be recognized as a document related to the user query.

[0111] Even if a document is included in the candidate document set in S540 regardless of its convergence ranking, it may be excluded from the candidate document set in S550 if its semantic similarity does not meet an absolute threshold. Accordingly, while including top documents from semantic-based search in the candidate document set, it is possible to restrict the inclusion of documents with low substantial relevance to the user query in subsequent prompts.

[0112] The hybrid context screening unit (122) can determine whether a relevant document exists based on the document exclusion result based on an absolute threshold (S560). If there are no documents remaining in the candidate document group, the hybrid context screening unit (122) can return a result corresponding to no information (S570). The result corresponding to no information can be used in the response output step (S270) corresponding to no information of FIG. 4.

[0113] If there are relevant documents, the hybrid context selection unit (122) can determine the number of documents to be finally selected based on the reliability gap between the top candidate documents (S580). Here, the reliability gap can be determined based on the relative ratio between the reliability of the top candidate document and the reliability of the next or subsequent candidate document.

[0114] In one embodiment, the confidence gap may be a value calculated to increase as the difference between the confidence of the top candidate document and the confidence of the next or subsequent candidate document, or as the relative ratio of the confidence of the next or subsequent candidate document to the confidence of the top candidate document decreases. When the confidence gap between the top candidate document and the subsequent candidate document is large, a small number of documents may be selected, and when the confidence gap is small, a relatively large number of documents may be selected. Accordingly, the number of selected documents can be adaptively adjusted so that only a small number of documents are used for queries with clear relevance, and more documents are used for queries where multiple documents have similar relevance.

[0115] Next, the hybrid context selection unit (122) can select final documents by considering relevance and diversity based on the number of documents determined in S580 (S590). For example, the hybrid context selection unit (122) can prioritize the selection of documents that are highly relevant to the user query, while lowering the selection priority of documents that duplicate the content of already selected documents or are excessively semantically similar. Accordingly, it is possible to prevent a limited number of documents from being composed only of similar content and to select documents containing different information necessary for generating answers.

[0116] Finally, the hybrid context selection unit (122) can return a finally selected set of documents (S595). The information body (320) of the returned set of documents can be provided to the prompt configuration unit (124) and used to configure prompts for a large-scale language model. Additionally, the top document among the selected set of documents can be used for action determination by the action decision unit (123) and for obtaining alternative answers by the fault response unit (126).

[0117] In this way, the hybrid context selection unit (122) can sequentially apply an absolute threshold, a confidence gap, and relevance and diversity after ensuring the inclusion of the top semantically similar document in the candidate document group. Accordingly, prompt length and inference delay can be reduced by limiting the number of documents provided to the large-scale language model while reducing the omission of semantically related documents.

[0118] FIG. 6 is a flowchart illustrating a regeneration cost-aware response cache management process according to one embodiment.

[0119] The steps illustrated in FIG. 6 can be performed by the cache management unit (121).

[0120] Referring to FIG. 6, the cache management unit (121) can obtain an answer generated by the large-scale language model inference unit (125) (S610). At this time, the cache management unit (121) can obtain the user query, the generated answer, and the action determined by the action determination unit (123) together.

[0121] Next, the cache management unit (121) can calculate a value corresponding to the cost of regenerating the generated answer (S620). The value corresponding to the cost of regeneration may be a value for estimating the amount of computation or processing time required to regenerate the answer through a large-scale language model.

[0122] In one embodiment, a value corresponding to the regeneration cost may be calculated based on at least one of the length of the generated answer, the number of characters, the number of words, or the number of generated tokens. Generally, as the length of the answer generated by a large-scale language model increases, the number of token generation attempts and the amount of computation required to generate the answer may increase; therefore, the length of the answer or the number of generated tokens may be used as a proxy value for the regeneration cost.

[0123] The cache management unit (121) can determine whether the value corresponding to the regeneration cost is less than or equal to a preset threshold value (S630). The threshold value is a value for distinguishing answers that need to be granted an ejection deferral, and can be set to correspond to the answer length, number of characters, or number of tokens.

[0124] If the value corresponding to the regeneration cost is less than or equal to the threshold value, the cache management unit (121) can set the ejection deferral indicator applied to the cache item to be created to 0 (S640). Accordingly, a cache item containing an answer with a relatively small regeneration cost can be ejected without a separate deferral if it is selected as an ejection candidate according to the threshold ejection policy.

[0125] On the other hand, if the value corresponding to the regeneration cost exceeds the threshold value, the cache management unit (121) can set an ejection deferral indicator based on the default value and the amount exceeding the threshold (S650). For example, the cache management unit (121) can set the ejection deferral indicator by adding a value proportional to the difference between the value corresponding to the regeneration cost and the threshold value to a preset default deferral value.

[0126] The eviction deferral indicator may be a value representing the number of times or the extent to which a cache item can be retained without being immediately evictioned, even if it is selected as an eviction candidate. Therefore, answers with higher regeneration costs may be assigned a relatively higher eviction deferral indicator. However, the method of calculating the eviction deferral indicator is not limited to this and may be set in stages based on the range, ratio, or grade of the regeneration cost corresponding value.

[0127] Next, the cache management unit (121) can create a cache item by linking an ejection deferral indicator to a user query, a generated answer, and an action (S660). The cache item may include at least one of a user query or a corresponding cache key, a generated answer, an action, an ejection deferral indicator, and information regarding the storage time or usage time of the cache item.

[0128] Afterward, the cache management unit (121) can determine whether there is a cache storage space available to store the generated cache item in the response cache (133) (S670). If there is a cache storage space available, the generated cache item can be stored in the response cache (133) without extracting the existing cache item separately (S695).

[0129] If cache storage space is not secured, the cache management unit (121) may select cache items as candidates for eviction according to a standard eviction policy (S675). The standard eviction policy may be a policy that selects candidates for eviction based on the storage order of cache items, the time of recent use, the frequency of use, or a combination thereof. For example, the standard eviction policy may select the oldest cache item or the cache item that has not been used for the longest time as candidates for eviction.

[0130] The cache management unit (121) can determine whether the evacuation deferral index of the selected evacuation candidate cache item is greater than 0 (S680).

[0131] If the ejection deferral indicator of an ejection candidate cache item is not greater than 0, the cache management unit (121) can eject the corresponding ejection candidate cache item from the response cache (133) (S685).

[0132] On the other hand, if the ejection deferral index of an ejection candidate cache item is greater than 0, the cache management unit (121) can decrease the ejection deferral index of the ejection candidate cache item by 1 and move the ejection candidate cache item to a lower priority position according to the standard ejection policy and keep it in the response cache (133) (S694). For example, if the standard ejection policy is implemented based on a queue, the cache management unit (121) can decrease the ejection deferral index of the ejection candidate cache item by 1 and move the ejection candidate cache item to the rear of the queue.

[0133] Next, the cache management unit (121) can determine whether the ejection deferral index of all cache items stored in the response cache (133) is greater than 0 after reducing the ejection deferral index of the ejection candidate cache item (S690).

[0134] If the ejection deferral index of all cache items is not greater than 0, that is, if there are cache items with an ejection deferral index of 0 or less, the cache management unit (121) can return to S675 and select the next candidate cache item for ejection according to the standard ejection policy.

[0135] Accordingly, a cache item with a remaining ejection deferral indicator is retained in the response cache (133) with the ejection deferral indicator reduced, even if it is selected as an ejection candidate, and another cache item may be selected as the next ejection candidate according to the standard ejection policy.

[0136] On the other hand, if the evacuation deferral indicator of all cache items stored in the response cache (133) is greater than 0 even after reducing the evacuation deferral indicator of the candidate cache item for evacuation, the cache management unit (121) can forcibly evacuate the cache item selected according to the standard evacuation policy regardless of the evacuation deferral indicator of the cache item (S692).

[0137] For example, the cache management unit (121) can forcibly evict cache items that are to be evicted first according to the standard evicting policy. Accordingly, it is possible to prevent a situation in which all cache items stored in the response cache (133) remain in an evicting deferral state, thereby preventing space from being secured to store new cache items.

[0138] Even if a cache item with a high regeneration cost is selected as a candidate for eviction, it may not be immediately evicted and may remain in the response cache (133) as long as the eviction deferral indicator remains. However, since the eviction deferral indicator decreases each time a specific cache item is selected as a candidate for eviction, it is possible to prevent the specific cache item from remaining indefinitely.

[0139] In one embodiment, if space required for storing generated cache items is not secured even after one cache item has been extracted, the cache management unit (121) may repeat the processing of S675 to S692 until the required storage space is secured.

[0140] The cache management unit (121) can secure storage space by extracting existing cache items from S685 or S692, and then store the cache items created in S660 in the response cache (133) (S695).

[0141] Cache items stored in the response cache (133) can be used when querying the cache for a subsequent user query. If a cache item corresponding to the user query exists, the cache management unit (121) can provide the stored answer and action without calling the large-scale language model again.

[0142] In this way, the cache management unit (121) sets an ejection deferral index based on a value corresponding to the cost of regenerating the answer, and if the ejection deferral index of a candidate cache item remains, it can reduce the index and move the corresponding cache item to a lower priority to retain it. Additionally, if there are other cache items that can be ejected, it selects the next candidate for ejection according to the standard ejection policy, and if all cache items are in an ejection deferral state, it can forcibly eject the cache item according to the standard ejection policy.

[0143] Accordingly, it is possible to keep answers with high regeneration costs in the response cache (133) for a relatively long time, while continuously securing space to store new cache items through the reduction of the ejection deferral indicator and forced ejection.

[0144] In addition, unlike cache eviction methods that consider only recency or usage frequency, by using the length of the answer itself or the number of generated tokens generated by a large-scale language model as a value corresponding to the regeneration cost, the average response latency and the computational load required for on-device question-answer processing can be reduced.

[0145] The present invention has been described above with reference to its embodiments. Those skilled in the art will understand that the present invention may be embodied in modified forms without departing from its essential characteristics. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the claims, not by the foregoing description, and all variations within the scope of equivalents should be interpreted as being included in the present invention.

Claims

Claim 1 An on-device question-and-answer processing device for responding to a user query input through a kiosk terminal, comprising: a storage unit that stores document data regarding a plurality of documents and an embedding index that includes pre-calculated embedding information for each of the plurality of documents; and an on-device question-and-answer processing unit that acquires a user query, performs a vocabulary-based search and a semantic-based search based on the user query, calculates a fusion ranking based on the respective rankings of the vocabulary-based search results and the semantic-based search results, forms a candidate document group based on the fusion ranking, includes a preset number of top documents in order of high semantic similarity among the semantic-based search results in the candidate document group regardless of the fusion ranking, selects at least one document from the candidate document group, and generates an answer using a large-scale language model based on the user query and information included in the selected at least one document. Claim 2 An on-device question-and-answer processing device according to claim 1, wherein the on-device question-and-answer processing unit performs the vocabulary-based search, the semantic-based search, the configuration of the candidate document group, and document selection at least one of the kiosk terminal or an adjacent edge server. Claim 3 An on-device question-and-answer processing device according to claim 1, wherein the on-device question-and-answer processing unit excludes from the candidate document group a candidate document in which the semantic similarity to the user query among the candidate documents included in the candidate document group does not satisfy a preset absolute threshold. Claim 4 In claim 3, the on-device question-and-answer processing unit is characterized by outputting a preset response indicating that information related to the user query does not exist when there is no candidate document satisfying the absolute threshold. Claim 5 In claim 3, the on-device question-and-answer processing unit is characterized by determining the number of documents to be selected from the group of candidate documents based on the reliability gap between the top candidate documents when there are candidate documents satisfying the absolute threshold. Claim 6 In claim 5, the on-device question-and-answer processing unit is characterized by selecting at least one document by prioritizing candidate documents with high reliability for the user query according to the number of determined documents, while lowering the selection priority of candidate documents that overlap in content or are semantically similar to documents already selected. Claim 7 An on-device question-and-answer processing device according to claim 1, wherein the document data is structured RAG data comprising a plurality of document entries, and each of the plurality of document entries comprises: search target information used for the vocabulary-based search or the semantic-based search; an information body distinguished from the search target information, excluded from the target of the vocabulary-based search and the semantic-based search, and provided to the large-scale language model; and action information corresponding to a predefined user interface operation. Claim 8 An on-device question-and-answer processing device according to claim 7, wherein the search target information includes search term information and title information, and the on-device question-and-answer processing unit performs the vocabulary-based search based on the search term information and performs the semantic-based search based on the embedding index including a pre-calculated embedding for the title information. Claim 9 In claim 7, the on-device question-and-answer processing unit determines an action corresponding to the user query based on action information included in the top-level document among the selected at least one document, and outputs the generated answer and the determined action together. Claim 10 In claim 9, the on-device question-and-answer processing unit is characterized by obtaining an alternative answer based on the information body of the top-level document when the generation of an answer using the large-scale language model is not performed normally, outputting the alternative answer and the determined action, and not storing the alternative answer in the response cache. Claim 11 The on-device question-and-answer processing device according to claim 1, wherein the storage unit includes a response cache that stores a cache item including an answer generated in response to a user query and an action associated with the answer, and the on-device question-and-answer processing unit determines whether a cache item corresponding to the user query exists in the response cache before performing the vocabulary-based search and the semantic-based search, and if a cache item corresponding to the user query exists, outputs the answer and action included in the cache item. Claim 12 An on-device question-and-answer processing device according to claim 11, wherein the on-device question-and-answer processing unit calculates a value corresponding to the cost of regenerating an answer generated by the large-scale language model when there is no cache item corresponding to the user query, and stores the user query, the generated answer, and an action associated with the generated answer in the response cache in conjunction with the value corresponding to the cost of regeneration. Claim 13 An on-device question-and-answer processing device according to claim 11, wherein the on-device question-and-answer processing unit calculates a value corresponding to a regeneration cost based on at least one of the length of the answer generated by the large-scale language model or the number of generated tokens when there is no cache item corresponding to the user query, and sets an ejection deferral index for deferring the ejection of a new cache item based on the value corresponding to the regeneration cost. Claim 14 In claim 11, the on-device question-and-answer processing unit sets an ejection deferral index for a new cache item based on a value corresponding to the regeneration cost of an answer generated by the large-scale language model, and if there is insufficient space to store the new cache item in the response cache, selects a candidate cache item for ejection according to a standard ejection policy, and if the ejection deferral index of the candidate cache item for ejection is 0 or less, ejects the candidate cache item for ejection from the response cache, and if the ejection deferral index of the candidate cache item for ejection is greater than 0, reduces the ejection deferral index of the candidate cache item for ejection and retains the candidate cache item for ejection in the response cache. Claim 15 An on-device question-and-answer processing method performed by an on-device question-and-answer processing device corresponding to a user query input through a kiosk terminal, comprising: a step of storing document data regarding a plurality of documents and an embedding index including pre-calculated embedding information for each of the plurality of documents; a step of obtaining a user query; a step of performing a vocabulary-based search and a semantic-based search based on the user query; a step of calculating a fusion rank based on the respective ranks of the vocabulary-based search results and the semantic-based search results, and forming a candidate document group based on the fusion rank; a step of including a preset number of top documents in order of high semantic similarity among the semantic-based search results in the candidate document group regardless of the fusion rank; a step of selecting at least one document from the candidate document group; and a step of generating an answer using a large-scale language model based on the user query and information included in the selected at least one document.

Images