System, method, and computer program product for cache-based query and generation

Abstract

Systems, methods, and computer program products for cache-based query generation are provided. The system includes at least one processor configured to generate an output for each query of a plurality of queries by inputting at least one document for each query into a large language model, receive feedback data for each output of each query of the plurality of queries, based on the feedback data, generate at least one embedding for each output and query pair, store the plurality of embeddings in a cache separate from the at least one data storage device, compare at least one query to the plurality of embeddings to determine a matching embedding, and in response to determining the matching embedding, generate at least one output for the at least one query based on the matching embedding without querying the large language model or the at least one data storage device.

Description

Attorney Docket No. 08223-2502249 (8485WO01)SYSTEM, METHOD, AND COMPUTER PROGRAM PRODUCT FOR CACHEBASED QUERY GENERATIONCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 681 ,929, filed August 12, 2024, the disclosure of which is hereby incorporated by reference in its entirety.BACKGROUND1 . Technical Field

[0002] This disclosure relates generally to machine-learning and, in some nonlimiting embodiments or aspects, to systems, methods, and computer program products for cache-based query generation.2. Technical Considerations

[0003] In generative artificial intelligence (Al) applications, feedback aids in the learning process, allowing models to be refined based on past performance. Feedback can guide an Al model to optimize its actions and enhances its predictive accuracy over time.

[0004] Caching boosts the efficiency of a model system by storing frequently used data for quick access. This reduces data retrieval time and computational redundancy, making the overall system incorporating the AU more efficient. Besides enhancing latency and computational efficiency, cached data can also be utilized to further refine and elevate the quality of outputs over time.

[0005] Retrieval-augmented generation (RAG) methods suffer from issues of accuracy relating to the information retrieved, and from increased latency through the inclusion of additional retrieval steps (e.g., such as an increased number of API calls).SUMMARY

[0006] According to non-limiting embodiments or aspects, provided is a system comprising: at least one processor configured to: receive a plurality of queries; determine at least one document from a plurality of documents for each query of the plurality of queries, the plurality of documents stored in at least one data storage device; generate an output for each query of the plurality of queries by inputting the at least one document for each query into a large language model; receive feedback data for each output of each query of the plurality of queries; based on the feedback data, generate at least one embedding for each output and query pair of at least a subset66F1231. DOCX Page 1 of 28Attorney Docket No. 08223-2502249 (8485WO01) of outputs and corresponding queries from the plurality of queries, resulting in a plurality of embeddings; store the plurality of embeddings in a cache separate from the at least one data storage device; receive at least one query; compare the at least one query to the plurality of embeddings to determine a matching embedding; and in response to determining the matching embedding, generate at least one output for the at least one query based on the matching embedding without querying the large language model or the at least one data storage device.

[0007] In non-limiting embodiments or aspects, the feedback data comprises at least one of the following: a selection of approval, a selection of disapproval, textual input, or any combination thereof. In non-limiting embodiments or aspects, the cache comprises temporary memory. In non-limiting embodiments or aspects, the at least one processor is further configured to: receive at least one second query; compare the at least one second query to the plurality of embeddings to determine a second matching embedding; and in response to determining the second matching embedding, generate at least one output for the at least one query based on the second matching embedding by querying the large language model without querying the at least one data storage device. In non-limiting embodiments or aspects, the at least one processor is further configured to: receive at least one second query; compare the at least one second query to the plurality of embeddings to determine that the at least one second query does not match a cached embedding; and in response to determining that the at least one second query does not match a cached embedding, generate at least one output for the at least one query based on querying the at least one data storage device and querying the large language model.

[0008] In non-limiting embodiments or aspects, the at least one processor is further configured to: receive feedback data for the at least one output; and in response to determining that the feedback data for the at least one output comprises negative feedback, generate at least one new output for the at least one query based on identifying at least one document in the at least one data storage device and querying the large language model based on the at least one document. In non-limiting embodiments or aspects, the at least one processor is further configured to: receive feedback data for the at least one new output; and in response to determining that the feedback data for the at least one new output comprises negative feedback, generate at least one second new output for the at least one query based on identifying a subset66F1231. DOCX Page 2 of 28Attorney Docket No. 08223-2502249 (8485WO01) of documents other than the at least one document in the at least one data storage device and querying the large language model based on the subset of documents.

[0009] According to non-limiting embodiments or aspects, provided is a method comprising: receiving a plurality of queries; determining at least one document from a plurality of documents for each query of the plurality of queries, the plurality of documents stored in at least one data storage device; generating an output for each query of the plurality of queries by inputting the at least one document for each query into a large language model; receiving feedback data for each output of each query of the plurality of queries; based on the feedback data, generating at least one embedding for each output and query pair of at least a subset of outputs and corresponding queries from the plurality of queries, resulting in a plurality of embeddings; storing the plurality of embeddings in a cache separate from the at least one data storage device; receiving at least one query; comparing the at least one query to the plurality of embeddings to determine a matching embedding; and in response to determining the matching embedding, generating at least one output for the at least one query based on the matching embedding without querying the large language model or the at least one data storage device.

[0010] In non-limiting embodiments or aspects, the feedback data comprises at least one of the following: a selection of approval, a selection of disapproval, textual input, or any combination thereof. In non-limiting embodiments or aspects, the cache comprises temporary memory. In non-limiting embodiments or aspects, the method includes: receiving at least one second query; comparing the at least one second query to the plurality of embeddings to determine a second matching embedding; and in response to determining the second matching embedding, generating at least one output for the at least one query based on the second matching embedding by querying the large language model without querying the at least one data storage device. In non-limiting embodiments or aspects, the method includes: receiving at least one second query; comparing the at least one second query to the plurality of embeddings to determine that the at least one second query does not match a cached embedding; and in response to determining that the at least one second query does not match a cached embedding, generating at least one output for the at least one query based on querying the at least one data storage device and querying the large language model.66F1231. DOCX Page 3 of 28Attorney Docket No. 08223-2502249 (8485WO01)

[0011] In non-limiting embodiments or aspects, the method includes: receiving feedback data for the at least one output; and in response to determining that the feedback data for the at least one output comprises negative feedback, generating at least one new output for the at least one query based on identifying at least one document in the at least one data storage device and querying the large language model based on the at least one document. In non-limiting embodiments or aspects, the method includes: receiving feedback data for the at least one new output; and in response to determining that the feedback data for the at least one new output comprises negative feedback, generating at least one second new output for the at least one query based on identifying a subset of documents other than the at least one document in the at least one data storage device and querying the large language model based on the subset of documents.

[0012] According to non-limiting embodiments or aspects, provided is a computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, causes the at least one processor to: receive a plurality of queries; determine at least one document from a plurality of documents for each query of the plurality of queries, the plurality of documents stored in at least one data storage device; generate an output for each query of the plurality of queries by inputting the at least one document for each query into a large language model; receive feedback data for each output of each query of the plurality of queries; based on the feedback data, generate at least one embedding for each output and query pair of at least a subset of outputs and corresponding queries from the plurality of queries, resulting in a plurality of embeddings; store the plurality of embeddings in a cache separate from the at least one data storage device; receive at least one query; compare the at least one query to the plurality of embeddings to determine a matching embedding; and in response to determining the matching embedding, generate at least one output for the at least one query based on the matching embedding without querying the large language model or the at least one data storage device.

[0013] In non-limiting embodiments or aspects, the feedback data comprises at least one of the following: a selection of approval, a selection of disapproval, textual input, or any combination thereof. In non-limiting embodiments or aspects, the cache comprises temporary memory. In non-limiting embodiments or aspects, the program instructions further cause the at least one processor to: receive at least one second66F1231. DOCX Page 4 of 28Attorney Docket No. 08223-2502249 (8485WO01) query; compare the at least one second query to the plurality of embeddings to determine a second matching embedding; and in response to determining the second matching embedding, generate at least one output for the at least one query based on the second matching embedding by querying the large language model without querying the at least one data storage device. In non-limiting embodiments or aspects, the program instructions further cause the at least one processor to: receive at least one second query; compare the at least one second query to the plurality of embeddings to determine that the at least one second query does not match a cached embedding; and in response to determining that the at least one second query does not match a cached embedding, generate at least one output for the at least one query based on querying the at least one data storage device and querying the large language model. In non-limiting embodiments or aspects, the program instructions further cause the at least one processor to: receive feedback data for the at least one output; and in response to determining that the feedback data for the at least one output comprises negative feedback, generate at least one new output for the at least one query based on identifying at least one document in the at least one data storage device and querying the large language model based on the at least one document.

[0014] Other non-limiting embodiments or aspects will be set forth in the following numbered clauses:

[0015] Clause 1 : A system comprising: at least one processor configured to: receive a plurality of queries; determine at least one document from a plurality of documents for each query of the plurality of queries, the plurality of documents stored in at least one data storage device; generate an output for each query of the plurality of queries by inputting the at least one document for each query into a large language model; receive feedback data for each output of each query of the plurality of queries; based on the feedback data, generate at least one embedding for each output and query pair of at least a subset of outputs and corresponding queries from the plurality of queries, resulting in a plurality of embeddings; store the plurality of embeddings in a cache separate from the at least one data storage device; receive at least one query; compare the at least one query to the plurality of embeddings to determine a matching embedding; and in response to determining the matching embedding, generate at least one output for the at least one query based on the matching embedding without querying the large language model or the at least one data storage device.66F1231. DOCX Page 5 of 28Attorney Docket No. 08223-2502249 (8485WO01)

[0016] Clause 2: The system of clause 1 , wherein the feedback data comprises at least one of the following: a selection of approval, a selection of disapproval, textual input, or any combination thereof.

[0017] Clause 3: The system of clause 1 or 2, wherein the cache comprises temporary memory.

[0018] Clause 4: The system of any of clauses 1 -3, wherein the at least one processor is further configured to: receive at least one second query; compare the at least one second query to the plurality of embeddings to determine a second matching embedding; and in response to determining the second matching embedding, generate at least one output for the at least one query based on the second matching embedding by querying the large language model without querying the at least one data storage device.

[0019] Clause 5: The system of any of clauses 1 -4, wherein the at least one processor is further configured to: receive at least one second query; compare the at least one second query to the plurality of embeddings to determine that the at least one second query does not match a cached embedding; and in response to determining that the at least one second query does not match a cached embedding, generate at least one output for the at least one query based on querying the at least one data storage device and querying the large language model.

[0020] Clause 6: The system of any of clauses 1 -5, wherein the at least one processor is further configured to: receive feedback data for the at least one output; and in response to determining that the feedback data for the at least one output comprises negative feedback, generate at least one new output for the at least one query based on identifying at least one document in the at least one data storage device and querying the large language model based on the at least one document.

[0021] Clause 7: The system of any of clauses 1 -6, wherein the at least one processor is further configured to: receive feedback data for the at least one new output; and in response to determining that the feedback data for the at least one new output comprises negative feedback, generate at least one second new output for the at least one query based on identifying a subset of documents other than the at least one document in the at least one data storage device and querying the large language model based on the subset of documents.

[0022] Clause 8: A method comprising: receiving a plurality of queries; determining at least one document from a plurality of documents for each query of the plurality of66F1231. DOCX Page 6 of 28Attorney Docket No. 08223-2502249 (8485WO01) queries, the plurality of documents stored in at least one data storage device; generating an output for each query of the plurality of queries by inputting the at least one document for each query into a large language model; receiving feedback data for each output of each query of the plurality of queries; based on the feedback data, generating at least one embedding for each output and query pair of at least a subset of outputs and corresponding queries from the plurality of queries, resulting in a plurality of embeddings; storing the plurality of embeddings in a cache separate from the at least one data storage device; receiving at least one query; comparing the at least one query to the plurality of embeddings to determine a matching embedding; and in response to determining the matching embedding, generating at least one output for the at least one query based on the matching embedding without querying the large language model or the at least one data storage device.

[0023] Clause 9: The method of clause 8, wherein the feedback data comprises at least one of the following: a selection of approval, a selection of disapproval, textual input, or any combination thereof.

[0024] Clause 10: The method of clause 8 or 9, wherein the cache comprises temporary memory.

[0025] Clause 11 : The method of any of clauses 8-10, further comprising: receiving at least one second query; comparing the at least one second query to the plurality of embeddings to determine a second matching embedding; and in response to determining the second matching embedding, generating at least one output for the at least one query based on the second matching embedding by querying the large language model without querying the at least one data storage device.

[0026] Clause 12: The method of any of clauses 8-1 1 , further comprising: receiving at least one second query; comparing the at least one second query to the plurality of embeddings to determine that the at least one second query does not match a cached embedding; and in response to determining that the at least one second query does not match a cached embedding, generating at least one output for the at least one query based on querying the at least one data storage device and querying the large language model.

[0027] Clause 13: The method of any of clauses 8-12, further comprising: receiving feedback data for the at least one output; and in response to determining that the feedback data for the at least one output comprises negative feedback, generating at least one new output for the at least one query based on identifying at least one66F1231. DOCX Page 7 of 28Attorney Docket No. 08223-2502249 (8485WO01) document in the at least one data storage device and querying the large language model based on the at least one document.

[0028] Clause 14: The method of any of clauses 8-13, further comprising: receiving feedback data for the at least one new output; and in response to determining that the feedback data for the at least one new output comprises negative feedback, generating at least one second new output for the at least one query based on identifying a subset of documents other than the at least one document in the at least one data storage device and querying the large language model based on the subset of documents.

[0029] Clause 15: A computer program product comprising at least one non- transitory computer-readable medium including program instructions that, when executed by at least one processor, causes the at least one processor to: receive a plurality of queries; determine at least one document from a plurality of documents for each query of the plurality of queries, the plurality of documents stored in at least one data storage device; generate an output for each query of the plurality of queries by inputting the at least one document for each query into a large language model; receive feedback data for each output of each query of the plurality of queries; based on the feedback data, generate at least one embedding for each output and query pair of at least a subset of outputs and corresponding queries from the plurality of queries, resulting in a plurality of embeddings; store the plurality of embeddings in a cache separate from the at least one data storage device; receive at least one query; compare the at least one query to the plurality of embeddings to determine a matching embedding; and in response to determining the matching embedding, generate at least one output for the at least one query based on the matching embedding without querying the large language model or the at least one data storage device.

[0030] Clause 16: The computer program product of clause 15, wherein the feedback data comprises at least one of the following: a selection of approval, a selection of disapproval, textual input, or any combination thereof.

[0031] Clause 17: The computer program product of clause 15 or 16, wherein the cache comprises temporary memory.

[0032] Clause 18: The computer program product of any of clauses 15-17, wherein the program instructions further cause the at least one processor to: receive at least one second query; compare the at least one second query to the plurality of embeddings to determine a second matching embedding; and in response to66F1231. DOCX Page 8 of 28Attorney Docket No. 08223-2502249 (8485WO01) determining the second matching embedding, generate at least one output for the at least one query based on the second matching embedding by querying the large language model without querying the at least one data storage device.

[0033] Clause 19: The computer program product of any of clauses 15-18, wherein the program instructions further cause the at least one processor to: receive at least one second query; compare the at least one second query to the plurality of embeddings to determine that the at least one second query does not match a cached embedding; and in response to determining that the at least one second query does not match a cached embedding, generate at least one output for the at least one query based on querying the at least one data storage device and querying the large language model.

[0034] Clause 20: The computer program product of any of clauses 15-19, wherein the program instructions further cause the at least one processor to: receive feedback data for the at least one output; and in response to determining that the feedback data for the at least one output comprises negative feedback, generate at least one new output for the at least one query based on identifying at least one document in the at least one data storage device and querying the large language model based on the at least one document.

[0035] These and other features and characteristics of the present disclosure, as well as the methods of operation and functions of the related elements of structures and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the disclosed subject matter.BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Additional advantages and details are explained in greater detail below with reference to the non-limiting, exemplary embodiments that are illustrated in the accompanying schematic figures, in which:

[0037] FIG. 1 is a schematic diagram of a system for cache-based query generation according to some non-limiting embodiments or aspects;66F1231. DOCX Page 9 of 28Attorney Docket No. 08223-2502249 (8485WO01)

[0038] FIG. 2 is a flow diagram of a system and method for cache-based query generation according to some non-limiting embodiments or aspects;

[0039] FIG. 3 is another flow diagram of a system and method for cache-based query generation according to some non-limiting embodiments or aspects; and

[0040] FIG. 4 is a schematic diagram of example components of one or more devices according to some non-limiting embodiments or aspects.DETAILED DESCRIPTION

[0041] For purposes of the description hereinafter, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and derivatives thereof shall relate to the embodiments as they are oriented in the drawing figures. However, it is to be understood that the embodiments may assume various alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply exemplary embodiments or aspects of the disclosed subject matter. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein are not to be considered as limiting.

[0042] Some non-limiting embodiments or aspects are described herein in connection with thresholds. As used herein, satisfying a threshold may refer to a value being greater than the threshold, more than the threshold, higher than the threshold, greater than or equal to the threshold, less than the threshold, fewer than the threshold, lower than the threshold, less than or equal to the threshold, equal to the threshold, etc.

[0043] No aspect, component, element, structure, act, step, function, instruction, and / or the like used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more” and “at least one.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, and / or the like) and may be used interchangeably with “one or more” or “at least one.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise. In addition, reference to an66F1231. DOCX Page 10 of 28Attorney Docket No. 08223-2502249 (8485WO01) action being “based on” a condition may refer to the action being “in response to” the condition. For example, the phrases “based on” and “in response to” may, in some non-limiting embodiments or aspects, refer to a condition for automatically triggering an action (e.g., a specific operation of an electronic device, such as a computing device, a processor, and / or the like).

[0044] As used herein, the term “account identifier” may include one or more primary account numbers (PANs), tokens, or other identifiers associated with a customer account. The term “token” may refer to an identifier that is used as a substitute or replacement identifier for an original account identifier, such as a PAN. Account identifiers may be alphanumeric or any combination of characters and / or symbols. Tokens may be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases, and / or the like) such that they may be used to conduct a transaction without directly using the original account identifier. In some examples, an original account identifier, such as a PAN, may be associated with a plurality of tokens for different individuals or purposes.

[0045] An “application programming interface” (API) refers to computer code or other data sorted on a computer-readable medium that may be executed by a processor to facilitate the interaction between software components, such as a clientside front-end and / or server-side back-end for receiving data from the client. An “interface” refers to a generated display, such as one or more graphical user interfaces (GUIs) with which a user may interact, either directly or indirectly (e.g., through a keyboard, mouse, etc.).

[0046] As used herein, the term “communication” may refer to the reception, receipt, transmission, transfer, provision, and / or the like of data (e.g., information, signals, messages, instructions, commands, and / or the like). For one unit (e.g., a device, a system, a component of a device or system, combinations thereof, and / or the like) to be in communication with another unit means that the one unit is able to directly or indirectly receive information from and / or transmit information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, and / or the like) that is wired and / or wireless in nature. Additionally, two units may be in communication with each other even though the information transmitted may be modified, processed, relayed, and / or routed between the first and second unit. For example, a first unit may be in communication with a second unit even though the first unit passively receives66F1231. DOCX Page 1 1 of 28Attorney Docket No. 08223-2502249 (8485WO01) information and does not actively transmit information to the second unit. As another example, a first unit may be in communication with a second unit if at least one intermediary unit processes information received from the first unit and communicates the processed information to the second unit.

[0047] As used herein, the term “computing device” may refer to one or more electronic devices configured to process data. A computing device may, in some examples, include the necessary components to receive, process, and output data, such as a processor, a display, a memory, an input device, a network interface, and / or the like. A computing device may be a mobile device. As an example, a mobile device may include a cellular phone (e.g., a smartphone or standard cellular phone), a portable computer, a wearable device (e.g., watches, glasses, lenses, clothing, and / or the like), a personal digital assistant (PDA), and / or other like devices. A computing device may also be a desktop computer or other form of non-mobile computer.

[0048] As used herein, the term “issuer institution” may refer to one or more entities, such as a bank, that provide accounts to customers for conducting transactions (e.g., payment transactions), such as initiating credit and / or debit payments. For example, an issuer institution may provide an account identifier, such as a PAN, to a customer that uniquely identifies one or more accounts associated with that customer. The account identifier may be embodied on a portable financial device, such as a physical financial instrument, e.g., a payment card, and / or may be electronic and used for electronic payments. The term “issuer system” refers to one or more computer devices operated by or on behalf of an issuer institution, such as a server computer executing one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing a transaction.

[0049] As used herein, the term “merchant” may refer to an individual or entity that provides goods and / or services, or access to goods and / or services, to customers based on a transaction, such as a payment transaction. The term “merchant” or “merchant system” may also refer to one or more computer systems operated by or on behalf of a merchant, such as a server computer executing one or more software applications.

[0050] As used herein, the terms “client” and “client device” may refer to one or more client-side devices or systems (e.g., remote from a transaction service provider) used to initiate or facilitate a transaction (e.g., a payment transaction). As an example, a “client device” may refer to one or more point-of-sale (POS) devices used by a66F1231. DOCX Page 12 of 28Attorney Docket No. 08223-2502249 (8485WO01) merchant, one or more acquirer host computers used by an acquirer, one or more mobile devices used by a user, and / or the like. In some non-limiting embodiments or aspects, a client device may be an electronic device configured to communicate with one or more networks and initiate or facilitate transactions. For example, a client device may include one or more computers, portable computers, laptop computers, tablet computers, mobile devices, cellular phones, wearable devices (e.g., watches, glasses, lenses, clothing, and / or the like), PDAs, and / or the like. Moreover, a “client” may also refer to an entity (e.g., a merchant, an acquirer, and / or the like) that owns, utilizes, and / or operates a client device for initiating transactions (e.g., for initiating transactions with a transaction service provider).

[0051] As used herein, the term “server” may refer to or include one or more computing devices that are operated by or facilitate communication and processing for multiple parties in a network environment, such as the Internet, although it will be appreciated that communication may be facilitated over one or more public or private network environments and that various other arrangements are possible. Further, multiple computing devices (e.g., servers, point-of-sale (POS) devices, mobile devices, etc.) directly or indirectly communicating in the network environment may constitute a “system.”

[0052] As used herein, the term “system” may refer to one or more computing devices or combinations of computing devices and / or components of such (e.g., processors, servers, client devices, software applications, and / or the like). Reference to “a device,” “a server,” “a processor,” and / or the like, as used herein, may refer to a previously-recited device, server, or processor that is recited as performing a previous step or function, a different device, server, or processor, and / or a combination of devices, servers, and / or processors. For example, as used in the specification and the claims, a first device, a first server, or a first processor that is recited as performing a first step or a first function may refer to the same or different device, server, or processor recited as performing a second step or a second function.

[0053] Non-limiting embodiments may enhance the accuracy of retrieval- augmented generation (RAG) through the use of a feedback-supported cache. Nonlimiting embodiments also provide for reduced latency for RAG methods through the cache, reducing the number of API calls made to retrieve documents and / or reducing the amount of processing necessary by a large language model (LLM). Through the incorporation of feedback in connection with the cache, the results and output may be66F1231. DOCX Page 13 of 28Attorney Docket No. 08223-2502249 (8485WO01) continuously improved over time with increased usage and development of the cache. The feedback-caching mechanism provided by non-limiting embodiments provides for a closed-loop system that optimizes Al-based applications and allows for responses to be provided in real-time (e.g., before completion of another task, while a user is engaged with a chat bot, within an imperceptible amount of time / delay for a user, and / or the like). Non-limiting embodiments also provide for the utilization of high- quality enterprise data that may already be in possession of an operating entity, thereby increasing the accuracy of the outputs and avoiding the need to recreate documents or document repositories and maximizing the value derived from existing business data. By including automated processing of user feedback in the flow of operations, system learning, response generation, and system updates may all be improved with usage.

[0054] Referring to FIG. 1 , a system 1000 for cache-based query and generation is shown according to some non-limiting embodiments or aspects. A computing device 100 may serve as a platform that is accessed by a user device 106 (e.g., a client computing device) and other user devices (not shown in FIG. 1 ). The user device 106 may include a computing device operated by a user that is seeking information through a service such as, but not limited to, a chat bot, search query, and / or the like. The computing device 100 may include one or more server computers, as an example. The computing device 100 is configured to receive an initial query 1 12 from the user device 106. The initial query 1 12 may be a chat message, question, statement, request, and / or the like in response to which information can be provided, regardless of whether the query 1 12 explicitly asks for the return of information or an answer. The chat bot may be provided by a merchant system, issuer system, payment network, and / or any other service. The computing device 100 outputs a user response 1 14 after processing the initial query 1 12. The user response 1 14 may be in the form of a textual response to a chat message, an answer, a search result, and / or the like.

[0055] With continued reference to FIG. 1 , the computing device 100 may be in communication with a cache 108, a feedback database 109, and a knowledge database 110. The cache 108 may include memory, an allocation of a hard disk drive, and / or any data storage device physically and / or logically separate from a data storage device storing the knowledge database 1 10. The feedback database 109 may be stored on the same or different data storage device as the knowledge database 110. The knowledge database 1 10 may include a plurality of documents and / or data66F1231. DOCX Page 14 of 28Attorney Docket No. 08223-2502249 (8485WO01) structures with information. For example, the knowledge database 1 10 may include records and other documents associated with a user, an account, and / or the like.

[0056] Still referring to FIG. 1 , the computing device 100 may in communication with an embedding model 102. The embedding model 102 may be local or remote to the computing device 100. The embedding model 102 may be configured to generate one or more embeddings based on input data (e.g., such as input text, including queries and response pairs (question / answer pairs)).

[0057] In operation, the computing device 100 may receive an initial query 1 12 from a user device 106 and, based on the initial query 1 12, retrieve data (e.g., such as one or more documents or portions thereof) from the knowledge database 1 10. For example, if the initial query 1 12 refers to an account or a topic, the computing device 100 may retrieve documents from the knowledge database 1 10 corresponding to the account or topic (e.g., documents mentioning or labeled with the account or topic). The computing device 100 may then generate a prompt 1 16 for an LLM 104 based on the retrieved data. For example, the prompt 1 16 may include retrieved data and / or include an attachment with one or more documents. An LLM output 1 18 may be used to generate the user response 1 14. For example, the LLM output 1 18 may be formatted, translated, and / or combined with other data to form the user response 1 14. In some non-limiting embodiments, the LLM output 1 18 may be the same as the user response 1 14.

[0058] Still referring to FIG. 1 , in non-limiting embodiments, the computing device 100 may prompt an operator of the user device for feedback relating to the user response 1 14. For example, the operator may be prompted to provide binary feedback (e.g., approve or disapprove), metered feedback (e.g., a subjective score or option among multiple scores or options), and / or narrative feedback (e.g., a textual explanation or response that may not include an express approval or disapproval). The feedback may be stored in a feedback database 109 in association with the query 1 12, user, LLM query 1 16, and / or the like. For example, an indication of approval or disapproval may be stored in relation to an embedding based on the query 1 12 or LLM query 1 16.

[0059] In non-limiting embodiments, in response to positive feedback (e.g., an indication of approval, a score satisfying a threshold, a determination that subjective or narrative feedback is positive, and / or the like), the computing device 100 may store information in the cache 108 in association with the query 112. For example, at least66F1231. DOCX Page 15 of 28Attorney Docket No. 08223-2502249 (8485WO01) a portion of the LLM output and / or user response 114 may be stored in association with an embedding based on the query 112. The cache 108 may be used to store “truths” (e.g., ground truths, such as information that has been determined to be acceptable through feedback or otherwise). In non-limiting embodiments, the cache 108 may store information that is retrieved from the knowledge database1 10 to generate the LLM query 1 16 so that the knowledge database 1 10 does not have to be queried in future iterations.

[0060] In operation, when a query 1 12 is received, the computing device 100 may generate an embedding based on at least a portion of the query 1 12 or information sought by the query 1 12 with the embedding model 102. This embedding may be compared to embeddings in the cache 108 to determine if a match is found. A match may be found if the embeddings are the same and / or within a threshold distance (e.g., a Euclidean distance). If a matching embedding is found, the information associated with that embedding is retrieved and used to form the LLM query 1 16 without querying the knowledge database 1 10 or to generate the user response 1 14 without querying the knowledge database 1 10 or LLM 104.

[0061] Referring now to FIG. 2, shown is a flow diagram for a method for cachebased queries according to some non-limiting embodiments or aspects. The steps shown in FIG. 2 are for example purposes only. It will be appreciated that additional, fewer, different, and / or a different order of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, a step may be automatically performed in response to performance and / or completion of a prior step. A user query 212 is received and processed (e.g., parsed) at step 203 to isolate data elements (e.g., user identifier, account identifier, question, information being sought, and / or the like). At step 204, the data from the query 212 is compared with the cache by generating an embedding (e.g., representing the question or information being sought) with an embedding model 202 and comparing it to embeddings stored in a cache 208 (e.g., storing embeddings as vectors in association with information).

[0062] At step 204 of FIG. 2, if the similarity between the data from the query and an embedding in the cache 208 satisfies a threshold (e.g., “Th”), a response may be determined at step 205. The similarity comparison may be based on a Euclidean distance between embedding vectors, as an example. A first option at step 205 is to query an LLM 214 based on documents and / or information from a knowledge database 210 and the information from the cache 208 by generating a query at step66F1231. DOCX Page 16 of 28Attorney Docket No. 08223-2502249 (8485WO01)209. A second option at step 205 is to generate the user response 215 based on the information in the cache 208 without querying the knowledge database 210 and / or without querying the LLM 1 14. If the similarity between the data from the query and an embedding in the cache 208 does not satisfy a threshold (e.g., “Th”) at step 204, the process may continue to step 206 and the data from the query 212 may be compared with the knowledge database 210 by generating an embedding (e.g., representing the question or information being sought) with the embedding model 202 and comparing it to embeddings stored in the knowledge database 210 (e.g., stored embeddings as vectors in association with documents and information). The knowledge database may be a local and / or remote hard disk storage unit, as an example, that is physically and / or logically separate from the cache 108. At step 205, a selection between the first option (Option A) or the second option (Option B) may be determined based on a specification and / or parameters of the product (e.g., application). For example, if a faster response time and lower cost is desired for a product, then Option B may be selected. Nonlimiting embodiments may select Option A if output quality is desired, even with increased latency and cost. The determination at step 205 may be automated in nonlimiting embodiments with a software agent configured to automatically select between Option A and Option B for a particular query (e.g., based on predetermined rules, a decision model, and / or the like).

[0063] At step 206 of FIG. 2, the top n documents and / or data elements may be retrieved from the knowledge database 210. For example, the five (5), ten (10), or fifteen (15) closest matching documents and / or data elements may be retrieved based on comparing embeddings representing such documents and / or data elements in the knowledge database 210. In non-limiting embodiments, the value of n may be predetermined or may be dynamically determined during run-time. For example, the value of n may be determined automatically during operation based on the rate of change of the similarity score. In non-limiting embodiments, a larger downward change in similarity score (e.g., a drop in score satisfying a threshold or the like) may trigger a cutoff and all the documents (n documents) above that change may be used to determine the similarity score. At step 207, a query for the LLM 214 may be generated based on the top n documents and / or data elements from the knowledge database210. The number of documents or items of information may depend on a predetermined number of documents or on a threshold similarity (e.g., distance) without any predetermined minimum or maximum number of documents or items.66F1231. DOCX Page 17 of 28Attorney Docket No. 08223-2502249 (8485WO01)

[0064] Referring now to FIG. 3, shown is a flow diagram for a method for processing feedback for cache-based queries according to some non-limiting embodiments or aspects. The steps shown in FIG. 3 are for example purposes only. It will be appreciated that additional, fewer, different, and / or a different order of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, a step may be automatically performed in response to performance and / or completion of a prior step. At step 302, user feedback is received in response to a user response (e.g., user response 215 in FIG. 2). The feedback may be in different forms, such as binary feedback (e.g., approve or disapprove), metered feedback (e.g., a score or option among multiple scores or options), and / or subjective feedback (e.g., a textual explanation or response that may not include an express approval or disapproval).

[0065] At step 304 of FIG. 3 the type of feedback may be determined. If the feedback is determined to be binary and / or explicit in approval / disapproval, at step 306 the method may proceed to step 308. At step 308, it is determined if the feedback is positive (e.g., approval), and if so the method proceeds to step 310 to process the question / answer pair (e.g., the user query (input) and user response pair (output)) and store the pair in the cache at step 312. The pair may be stored as one or more embeddings, as an example, by inputting the text into an embedding model and storing the resulting embeddings. There may be an embedding for a pair or an embedding for each of the question (query) and answer (output or response). The generated embeddings for the question / answer pair may then be compared to the embeddings in the cache to determine if the question / answer pair is already stored in the cache and / or if a similar question / answer pair is already stored in the cache, thereby avoiding storing duplicate question / answer pairs in the cache and preserving cache storage space.

[0066] Still referring to FIG. 3, if the feedback is not positive at step 308, the method may proceed to step 314 to process the associated documents and / or portion of documents. At step 314, the user may be presented with a visual display of the topranked documents and / or information from documents (e.g., top five, top ten, top fifteen, and / or the like) to convey a reason for the output to the user that is unhappy with the answer. The system may generate a question / answer pair at step 316 and store it in the cache at step 312 to indicate non-relevant or less-relevant documents that should not be used for a future query. If the feedback is negative, a new LLM66F1231. DOCX Page 18 of 28Attorney Docket No. 08223-2502249 (8485WO01) query may be generated with different documents. For example, a more in-depth search of the knowledge database may be performed to identify different documents or information and / or additional documents or information relative to the initial prompt. In some non-limiting embodiments, a user may select the different and / or additional documents or information. In some non-limiting embodiments, different search queries and / or different models may be used to identify new documents or portions thereof.

[0067] With continued reference to FIG. 3, at step 306 if the feedback is not binary and / or explicit in approval / disapproval, such as subjective feedback with comments, a score, and / or any feedback that is non-explicit on approval / disapproval may be input into an LLM and / or other model for analysis at step 318. For example, the feedback may be input into an LLM with a prompt that asks the LLM to determine if the feedback is an approval or disapproval, and by what confidence level. At step 318, the LLM may be prompted to extract a reason for a disapproval or a correct answer, and at step 320 a new query may be generated for the LLM with one or more different documents and / or data elements (e.g., from a knowledge database). The correct answer, once determined, may be used to generate a question / answer pair at step 322 and may be stored in the cache at step 312.

[0068] Referring now to FIG. 4, shown is a diagram of example components of a device 400 according to non-limiting embodiments or aspects. Device 400 may correspond to at least one of the computing devices (e.g., computing device 100, 106) in FIG. 1 . In some non-limiting embodiments or aspects, such systems or devices may include at least one device 400 and / or at least one component of device 400. The number and arrangement of components shown in FIG. 4 are provided as an example. In some non-limiting embodiments or aspects, device 400 may include additional components, fewer components, different components, or differently arranged components than those shown in FIG. 4. Additionally, or alternatively, a set of components (e.g., one or more components) of device 400 may perform one or more functions described as being performed by another set of components of device 400.

[0069] As shown in FIG. 4, device 400 may include bus 402, processor 404, memory 406, storage component 408, input component 410, output component 412, and communication interface 414. Bus 402 may include a component that permits communication among the components of device 400. In some non-limiting embodiments or aspects, processor 404 may be implemented in hardware, firmware, or a combination of hardware and software. For example, processor 404 may include66F1231. DOCX Page 19 of 28Attorney Docket No. 08223-2502249 (8485WO01) a processor (e.g., a central processing unit (CPU), a GPU, an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component (e.g., a field-programmable gate array (FPGA), an applicationspecific integrated circuit (ASIC), etc.) that can be programmed to perform a function. Memory 406 may include random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 404.

[0070] With continued reference to FIG. 4, storage component 408 may store information and / or software related to the operation and use of device 400. For example, storage component 408 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, a solid state disk, etc.) and / or another type of computer-readable medium. Input component 410 may include a component that permits device 400 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). Additionally, or alternatively, input component 410 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). Output component 412 may include a component that provides output information from device 400 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). Communication interface 414 may include a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables device 400 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 414 may permit device 400 to receive information from another device and / or provide information to another device. For example, communication interface 414 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, and / or the like.

[0071] Device 400 may perform one or more processes described herein. Device 400 may perform these processes based on processor 404 executing software instructions stored by a computer-readable medium, such as memory 406 and / or storage component 408. A computer-readable medium may include any non-transitory memory device. A memory device includes memory space located inside of a single66F1231. DOCX Page 20 of 28Attorney Docket No. 08223-2502249 (8485WO01) physical storage device or memory space spread across multiple physical storage devices. Software instructions may be read into memory 406 and / or storage component 408 from another computer-readable medium or from another device via communication interface 414. When executed, software instructions stored in memory 406 and / or storage component 408 may cause processor 404 to perform one or more processes described herein. Additionally, or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, embodiments described herein are not limited to any specific combination of hardware circuitry and software. The term “configured to,” as used herein, may refer to an arrangement of software, device(s), and / or hardware for performing and / or enabling one or more functions (e.g., actions, processes, steps of a process, and / or the like). For example, “a processor configured to” may refer to a processor that executes software instructions (e.g., program code) that cause the processor to perform one or more functions.

[0072] Although embodiments have been described in detail for the purpose of illustration, it is to be understood that such detail is solely for that purpose and that the disclosure is not limited to the disclosed embodiments or aspects, but, on the contrary, is intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect can be combined with one or more features of any other embodiment or aspect. In fact, any of these features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.66F1231. DOCX Page 21 of 28

Claims

Attorney Docket No. 08223-2502249 (8485WO01)WHAT IS CLAIMED IS:1 . A system comprising: at least one processor configured to: receive a plurality of queries; determine at least one document from a plurality of documents for each query of the plurality of queries, the plurality of documents stored in at least one data storage device; generate an output for each query of the plurality of queries by inputting the at least one document for each query into a large language model; receive feedback data for each output of each query of the plurality of queries; based on the feedback data, generate at least one embedding for each output and query pair of at least a subset of outputs and corresponding queries from the plurality of queries, resulting in a plurality of embeddings; store the plurality of embeddings in a cache separate from the at least one data storage device; receive at least one query; compare the at least one query to the plurality of embeddings to determine a matching embedding; and in response to determining the matching embedding, generate at least one output for the at least one query based on the matching embedding without querying the large language model or the at least one data storage device.

2. The system of claim 1 , wherein the feedback data comprises at least one of the following: a selection of approval, a selection of disapproval, textual input, or any combination thereof.

3. The system of claim 1 , wherein the cache comprises temporary memory.

4. The system of claim 1 , wherein the at least one processor is further configured to:66F1231. DOCX Page 22 of 28Attorney Docket No. 08223-2502249 (8485WO01) receive at least one second query; compare the at least one second query to the plurality of embeddings to determine a second matching embedding; and in response to determining the second matching embedding, generate at least one output for the at least one query based on the second matching embedding by querying the large language model without querying the at least one data storage device.

5. The system of claim 1 , wherein the at least one processor is further configured to: receive at least one second query; compare the at least one second query to the plurality of embeddings to determine that the at least one second query does not match a cached embedding; and in response to determining that the at least one second query does not match a cached embedding, generate at least one output for the at least one query based on querying the at least one data storage device and querying the large language model.

6. The system of claim 1 , wherein the at least one processor is further configured to: receive feedback data for the at least one output; and in response to determining that the feedback data for the at least one output comprises negative feedback, generate at least one new output for the at least one query based on identifying at least one document in the at least one data storage device and querying the large language model based on the at least one document.

7. The system of claim 6, wherein the at least one processor is further configured to: receive feedback data for the at least one new output; and in response to determining that the feedback data for the at least one new output comprises negative feedback, generate at least one second new output for the at least one query based on identifying a subset of documents other than the66F1231. DOCX Page 23 of 28Attorney Docket No. 08223-2502249 (8485WO01) at least one document in the at least one data storage device and querying the large language model based on the subset of documents.

8. A method comprising: receiving a plurality of queries; determining at least one document from a plurality of documents for each query of the plurality of queries, the plurality of documents stored in at least one data storage device; generating an output for each query of the plurality of queries by inputting the at least one document for each query into a large language model; receiving feedback data for each output of each query of the plurality of queries; based on the feedback data, generating at least one embedding for each output and query pair of at least a subset of outputs and corresponding queries from the plurality of queries, resulting in a plurality of embeddings; storing the plurality of embeddings in a cache separate from the at least one data storage device; receiving at least one query; comparing the at least one query to the plurality of embeddings to determine a matching embedding; and in response to determining the matching embedding, generating at least one output for the at least one query based on the matching embedding without querying the large language model or the at least one data storage device.

9. The method of claim 8, wherein the feedback data comprises at least one of the following: a selection of approval, a selection of disapproval, textual input, or any combination thereof.

10. The method of claim 8, wherein the cache comprises temporary memory.1 1 . The method of claim 8, further comprising: receiving at least one second query;66F1231. DOCX Page 24 of 28Attorney Docket No. 08223-2502249 (8485WO01) comparing the at least one second query to the plurality of embeddings to determine a second matching embedding; and in response to determining the second matching embedding, generating at least one output for the at least one query based on the second matching embedding by querying the large language model without querying the at least one data storage device.

12. The method of claim 8, further comprising: receiving at least one second query; comparing the at least one second query to the plurality of embeddings to determine that the at least one second query does not match a cached embedding; and in response to determining that the at least one second query does not match a cached embedding, generating at least one output for the at least one query based on querying the at least one data storage device and querying the large language model.

13. The method of claim 8, further comprising: receiving feedback data for the at least one output; and in response to determining that the feedback data for the at least one output comprises negative feedback, generating at least one new output for the at least one query based on identifying at least one document in the at least one data storage device and querying the large language model based on the at least one document.

14. The method of claim 13, further comprising: receiving feedback data for the at least one new output; and in response to determining that the feedback data for the at least one new output comprises negative feedback, generating at least one second new output for the at least one query based on identifying a subset of documents other than the at least one document in the at least one data storage device and querying the large language model based on the subset of documents.66F1231. DOCX Page 25 of 28Attorney Docket No. 08223-2502249 (8485WO01)15. A computer program product comprising at least one non- transitory computer-readable medium including program instructions that, when executed by at least one processor, causes the at least one processor to: receive a plurality of queries; determine at least one document from a plurality of documents for each query of the plurality of queries, the plurality of documents stored in at least one data storage device; generate an output for each query of the plurality of queries by inputting the at least one document for each query into a large language model; receive feedback data for each output of each query of the plurality of queries; based on the feedback data, generate at least one embedding for each output and query pair of at least a subset of outputs and corresponding queries from the plurality of queries, resulting in a plurality of embeddings; store the plurality of embeddings in a cache separate from the at least one data storage device; receive at least one query; compare the at least one query to the plurality of embeddings to determine a matching embedding; and in response to determining the matching embedding, generate at least one output for the at least one query based on the matching embedding without querying the large language model or the at least one data storage device.

16. The computer program product of claim 15, wherein the feedback data comprises at least one of the following: a selection of approval, a selection of disapproval, textual input, or any combination thereof.

17. The computer program product of claim 15, wherein the cache comprises temporary memory.

18. The computer program product of claim 15, wherein the program instructions further cause the at least one processor to: receive at least one second query;66F1231. DOCX Page 26 of 28Attorney Docket No. 08223-2502249 (8485WO01) compare the at least one second query to the plurality of embeddings to determine a second matching embedding; and in response to determining the second matching embedding, generate at least one output for the at least one query based on the second matching embedding by querying the large language model without querying the at least one data storage device.

19. The computer program product of claim 15, wherein the program instructions further cause the at least one processor to: receive at least one second query; compare the at least one second query to the plurality of embeddings to determine that the at least one second query does not match a cached embedding; and in response to determining that the at least one second query does not match a cached embedding, generate at least one output for the at least one query based on querying the at least one data storage device and querying the large language model.

20. The computer program product of claim 15, wherein the program instructions further cause the at least one processor to: receive feedback data for the at least one output; and in response to determining that the feedback data for the at least one output comprises negative feedback, generate at least one new output for the at least one query based on identifying at least one document in the at least one data storage device and querying the large language model based on the at least one document.66F1231. DOCX Page 27 of 28

Images