Large language model reasoning via querying database

Abstract

A method performed using a computer comprises an encoder, a database return encoder, a feature fusion model, and a decoder. The method includes the encoder receiving a question. The encoder can output a plaintext database query, a data processing instruction, and a question embedding for the question. The computer can obtain a set of data elements from a database using the plaintext database query. The database return encoder can receive the set of data elements and the data processing instruction. The database return encoder can output, to the feature fusion model, an intermediate embedding based on the set of data elements and the data processing instruction. The feature fusion model can output, to the decoder, an output embedding using the intermediate embedding and the question embedding. The decoder can determine a text output based on the output embedding.

Description

LARGE LANGUAGE MODEL REASONING VIA QUERYING DATABASECROSS-REFERENCES TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 520,045, filed August 16, 2023, which is herein incorporated by reference in its entirety for all purposes.BACKGROUND

[0002] Large language models can process input text to determine an output answer. Some large language models attempt to include reasoning into the text processing method. Reasoning is included into the large language model by manually writing prompts to guide the large language model in the reasoning processing. For example, a large language model can be provided with the following question (Q), reason (R), and answer (A) during training: Q) Smith had 5 apples, and bought two dozen apples; how many does he get then?; R) Smith starts with 5 apples. Two dozen is 2*12=24 apples. 5+24=29; A) The answer is 29. The large language model is then trained with many different custom reasoning examples.

[0003] Such reasoning requires the assumption that the factual knowledge is common sense and is already understood by the large language model. However, typically, factual knowledge is not common sense and is based in facts or data. The large language model does not have access to current facts or data. Large language models are rarely trained on historical databases of data and are typically only trained on text corpuses.

[0004] Current large language models can also “understand” data upon which they were trained. Therefore any factual data that occurs after the training of the large language model is not within the large language model’s knowledge base. If a question were asked of the large language model regarding current data (e.g., what is the stock price of XYZ), the large language model would not be able to answer the question with true data and would rather be hard coded to not answer or would givemade up numbers based on past text patterns that the large language model was trained on.

[0005] Further, such custom reasoning prompts are time consuming to create and the mass creation of them is not scalable. Additionally, the prompts are non- transferable, meaning that they cannot be used across different tasks for the large learning model.

[0006] Embodiments of the disclosure address this problem and other problems individually and collectively.SUMMARY

[0007] One embodiment is related to a method performed using a computer comprising an encoder, a database return encoder, a feature fusion model, and a decoder, the method comprising: receiving, by the encoder, a question; outputting, by the encoder, a plaintext database query, a data processing instruction, and a question embedding for the question; obtaining a set of data elements from a database using the plaintext database query; receiving, by the database return encoder, the set of data elements and the data processing instruction; determining, by the database return encoder, an intermediate embedding based on the set of data elements and the data processing instruction; determining, by the feature fusion model, an output embedding using the intermediate embedding and the question embedding; and determining, by the decoder, a text output based on the output embedding.

[0008] Another embodiment is related to a computer comprising: a processor; and a computer-readable medium coupled to the processor, the computer-readable medium comprising code executable by the processor for implementing a method comprising: receiving a question; outputting, by an encoder in the computer, a plaintext database query, a data processing instruction, and a question embedding for the question; obtaining a set of data elements from a database using the plaintext database query; receiving, by a database return encoder in the computer, the set of data elements and the data processing instruction; determining, by the database return encoder, an intermediate embedding based on the set of data elements and the data processing instruction; determining, by a feature fusion model in the computer, an output embedding using the intermediate embedding and the question embedding;and determining, by a decoder in the computer, a text output based on the output embedding.

[0009] Another embodiment is related to a system comprising: a database storing a plurality of data elements; and a computer in operative communication with the database, the computer comprising: a processor; and a computer-readable medium coupled to the processor, the computer-readable medium comprising code executable by the processor for implementing a method comprising: receiving a question; outputting, by an encoder in the computer, a plaintext database query, a data processing instruction, and a question embedding for the question; obtaining a set of data elements from the database using the plaintext database query; receiving, by a database return encoder in the computer, the set of data elements and the data processing instruction; determining, by the database return encoder, an intermediate embedding based on the set of data elements and the data processing instruction; determining, by a feature fusion model in the computer, an output embedding using the intermediate embedding and the question embedding; and determining, by a decoder in the computer, a text output based on the output embedding.

[0010] Further details regarding embodiments of the disclosure can be found in the Detailed Description and the Figures.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG. 1 shows a block diagram of a large language model reasoning system according to embodiments.

[0012] FIG. 2 shows a block diagram of components of an processing computer according to embodiments.

[0013] FIG. 3 shows a hybrid flow diagram illustrating a large language model reasoning and database querying method according to embodiments.

[0014] FIG. 4 shows a hybrid flow diagram illustrating details of a large language model encoder method according to embodiments.DETAILED DESCRIPTION

[0015] Prior to discussing embodiments of the disclosure, some terms can be described in further detail.

[0016] A “large language model” (LLM) can include a probabilistic model of a natural language. A large language model can generate probabilities of a series of words, based on text corpora in one or multiple languages it was trained on. Large language models can include a combination of feedforward neural networks and transformers. A large language model can include artificial neural networks that contain from tens of millions and up to billions of weights and can be (pre-)trained using self-supervised learning and semi-supervised learning. A large language model accept an input text and can repeatedly predict a next token or word as an output.

[0017] A “transformer” can include a deep learning architecture that utilizes attention mechanisms. A transformers can include primary components of 1 ) tokenizers, which convert text into tokens, 2) embedding layers, which convert tokens into semantically meaningful representations, and 3) transformer layers, which carry out the reasoning capabilities, and can consist of Attention and MLP layers. Transformer layers can be one of two types, encoder and decoder. A transformer can include an encoder, a decoder, or both an encoder and a decoder. Bidirectional encoder representations from transformers (BERT) are an example of encoder-only models, while generative pre-trained transformers (GPT) are an example of decoder- only models.

[0018] An “encoder” can include a type of neural network that converts data from a raw format into another form. An encoder can include a self-attention mechanism and a feed-forward neural network. The self-attention mechanism accepts input encodings from the previous encoder and weights the encodings’ relevance to each other to generate output encodings. The feed-forward neural network further processes each output encoding individually. These output encodings are then passed to the next encoder as its input in a looping manner to encode a full input text. The encoder can be bidirectional in terms of attention, for example, attention can be placed on tokens before and after the current token.

[0019] A “decoder” can include a type of neural network that generates output from encoded representations. A decoder can include attention mechanisms (e.g., a self-attention mechanism and an attention mechanism over the encodings) and a feedforward neural network. A decoder can function in a similar fashion to an encoder, but an additional attention mechanism is inserted, which instead draws relevant information from the encodings generated by the encoders. This mechanism can also be called the encoder-decoder attention.

[0020] An “embedding” can include vector representation of data. An embedding can be a low-dimensional learned continuous vector representation of discrete variables. An embedding can be generated using a neural network that is trained to convert input data into output vector representations of the input data.

[0021] A “question” can include a sentence worded or expressed so as to elicit information. A question can include text strings in a plaintext format. A question can be an input message into a large language model. A question can be a text sentence created by a user.

[0022] A “plaintext database query” can include a request for data. A plaintext database query can include plaintext (e.g., not encrypted or in a vector space). A plaintext database query can include a query for a database to access, manipulate, delete, or retrieve data from the database. The plaintext database query can be written using a specific syntax.

[0023] A “data element” can include a unit of data. In some embodiments, the data element can have a precise meaning or precise semantics. A data element can include, for example, zip code, merchant identifier, user identifier, amount, IP address, date, time, etc. A data element can be associated with a value, for example, 94016, merchant_1234, user_1234, $19.99, 123.123.12.123, 01 / 01 / 2015, 11 :00 AM PT.

[0024] A “processor” may include a device that processes something. In some embodiments, a processor can include any suitable data computation device or devices. A processor may comprise one or more microprocessors working together to accomplish a desired function. The processor may include a CPU comprising at least one high-speed data processor adequate to execute program components for executing user and / or system -generated requests. The CPU may be amicroprocessor such as AMD's Athlon, Duron and / or Opteron; IBM and / or Motorola's PowerPC; IBM's and Sony's Cell processor; Intel's Celeron, Itanium, Pentium, Xeon, and / or XScale; and / or the like processor(s).

[0025] A “memory” may be any suitable device or devices that can store electronic data. A suitable memory may comprise a non-transitory computer readable medium that stores instructions that can be executed by a processor to implement a desired method. Examples of memories may comprise one or more memory chips, disk drives, etc. Such memories may operate using any suitable electrical, optical, and / or magnetic mode of operation.

[0026] A “server computer” may include a powerful computer or cluster of computers. For example, the server computer can be a large mainframe, a minicomputer cluster, or a group of servers functioning as a unit. In one example, the server computer may be a database server coupled to a Web server. The server computer may comprise one or more computational apparatuses and may use any of a variety of computing structures, arrangements, and compilations for servicing the requests from one or more client computers.

[0027] Embodiments provide for large language model reasoning via querying a database and without manually written prompts. A reasoning process can be performed to determine an answer to a prompt using reasoning. A computer can encode questions and generate queries given the questions using a large language model (LLM) encoder. The queries can be database queries that are based on the questions. The computer can query a database using the queries. The computer can then encode the information returned from the database queries. Upon receiving the return data (e.g., a set of data elements) from the database, the computer can fuse the large language model feature(s) and encoding database feature(s) into fused features (e.g., output embeddings). The computer can then decode the fused features to an output answer using an LLM decoder.

[0028] As an example, a computer can include an LLM encoder, a database return encoder, a feature fusion model, and an LLM decoder. The LLM encoder of the computer can receive a question (e.g., from a client device). The question can be text provided by a user. The LLM encoder can output a plaintext database query, a data processing instruction, and a question embedding for the question. The computer canthen obtain a set of data elements from a database using the plaintext database query. The database return encoder of the computer can receive the set of data elements and the data processing instruction. The database return encoder can generate an intermediate embedding based on the set of data elements and the data processing instruction. The database return encoder can output the intermediate embedding to the feature fusion model. The feature fusion model can generate an output embedding using the intermediate embedding and the question embedding. The feature fusion model can output the output embedding to the LLM decoder. The LLM decoder can determine a text output based on the output embedding, where the text output is a response to the question. In some embodiments, the computer can include the set of data elements from the database into the text output.

[0029] FIG. 1 shows a system 100 according to embodiments of the disclosure. The system 100 comprises a client device 102, an processing computer 104, and a database 106. The processing computer 104 can be in operative communication with the client device 102 and the database 106.

[0030] For simplicity of illustration, a certain number of components are shown in FIG. 1. It is understood, however, that embodiments of the invention may include more than one of each component. In addition, some embodiments of the invention may include fewer than or greater than all of the components shown in FIG. 1 .

[0031] Messages between the devices in the system 100 in FIG. 1 can be transmitted using a secure communications protocols such as, but not limited to, Secure Hypertext Transfer Protocol (HTTPS), SSL, ISO (e.g., ISO 8583) and / or the like. The communications network may include any one and / or the combination of the following: a direct interconnection; the Internet; a Local Area Network (LAN); a Metropolitan Area Network (MAN); an Operating Missions as Nodes on the Internet (OMNI); a secured custom connection; a Wide Area Network (WAN); a wireless network (e.g., employing protocols such as, but not limited to a Wireless Application Protocol (WAP), l-mode, and / or the like); and / or the like. The communications network can use any suitable communications protocol to generate one or more secure communication channels. A communications channel may, in some instances, comprise a secure communication channel, which may be established in any knownmanner, such as through the use of mutual authentication and a session key, and establishment of a Secure Socket Layer (SSL) session.

[0032] The client device 102 can include a device that accesses services made available by a server (e.g., the processing computer 104). The client device 102 can act as a client part of the client-server model of computer networks. The client device 102 can be operated by a user. The client device 102 can provide questions, which can be created by the user, to the processing computer 104. In some embodiments, one or more client devices can be present and can provide questions to the processing computer 104. The client device 102 can be a user device such as a laptop computer, a desktop computer, a smartphone, tablet, a smartwatch, and / or a device that can be operated by a user and can connect to the processing computer 104 remotely.

[0033] The processing computer 104 can include a computer or a server computer that processes data. The processing computer 104 can process received questions. The processing computer 104 can include a large language model. The processing computer 104 can utilize the large language model as well as the database 106 to generate responses to received questions.

[0034] The database 106 can include any suitable database. The database may be a conventional, fault tolerant, relational, scalable, secure database such as those commercially available from Oracle™ or Sybase™. The database 106 can store data elements (e.g., transaction data, weather data, image data, network data, etc.). The database 106 can receive plaintext database queries from the processing computer 104 and can return a set of data elements to the processing computer 104 based on the plaintext database queries. The database 106 can include one or more data tables. Each data table can be identified by a data table identifier.

[0035] FIG. 2 shows a block diagram of the processing computer 104 according to embodiments. The exemplary processing computer 104 may comprise a processor 204. The processor 204 may be coupled to a memory 202, a network interface 206, and a computer readable medium 208. The computer readable medium 208 can comprise one or more modules. The computer readable medium 208 can include a large language model encoder 210, a database return encoder 212, a feature fusion model 214, and a large language model decoder 216.

[0036] The memory 202 can be used to store data and code. For example, the memory 202 can store data, questions, plaintext database query, embeddings, etc. The memory 202 may be coupled to the processor 204 internally or externally (e.g., cloud based data storage), and may comprise any combination of volatile and / or nonvolatile memory, such as RAM, DRAM, ROM, flash, or any other suitable memory device.

[0037] The computer readable medium 208 may comprise code, executable by the processor 204, for performing a method comprising: receiving a question; outputting a plaintext database query, a data processing instruction, and a question embedding for the question; obtaining a set of data elements from a database using the plaintext database query; receiving the set of data elements and the data processing instruction; determining an intermediate embedding based on the set of data elements and the data processing instruction; determining an output embedding using the intermediate embedding and the question embedding; and determining a text output based on the output embedding.

[0038] The large language model encoder 210 can include an encoder. The large language model encoder 210 can receive a question from a client device. The large language model encoder 210 can process the question to generate a plaintext database query, a data processing instruction, and a question embedding for the question.

[0039] The large language model encoder 210 can understand and extract relevant information from an input text (e.g., a question). The large language model encoder 210 can output a continuous representation (e.g., an embedding) of the input text. As such, the large language model encoder 210 can generate a question embedding (which can be a vector of values) for the question.

[0040] The large language model encoder 210 can also generate a plaintext database query. The large language model encoder 210 can generate the plaintext database query based on the question and / or the question embedding. The plaintext database query can be a query for data from the database 106. The large language model encoder 210 can determine a data table from a plurality of data tables in the database 106. The large language model encoder 210 can then determine one or more filters for the query from a plurality of filters. The large language model encoder210 can also select a mode that indicates an output sequence for the result of the query. The mode can indicate how the output sequence is formatted.

[0041] The large language model encoder 210 can also generate a data processing instruction. The data processing instruction can be a value that indicates how the database return encoder 212 is to encode a set of data obtained from the database 106. The data processing instruction can be an integer, a flag, a structure of data, or other data value. For example, the data processing instruction can indicate a length of embedding to create, a number of embeddings to create, a method of creating an embedding, and / or other parameters that can indicate how an encoder is to encode data into an embedding.

[0042] Further details regarding the large language model encoder 210 are described in reference to FIG. 4.

[0043] The database return encoder 212 can include an encoder. The database return encoder 212 can receive a set of data elements from the database 106 and the data processing instruction from the large language model encoder 210. The database return encoder 212 can generate an intermediate embedding using the set of data elements based on the data processing instruction. The database return encoder 212 can encode inputs (e.g., a set of data elements from a database) in a similar manner to the large language model encoder 210. The database return encoder 212 can encode the set of data elements based on the data processing instruction.

[0044] For example, the database return encoder 212 can receive a set of data elements that include data elements for five transactions. The database return encoder 212 can also receive a data processing instruction that indicates to create a single intermediate embedding based on the data elements for the five transactions. The database return encoder 212 can then generate a single intermediate embedding based on the data elements for the five transactions. For example, the five transactions can be concatenated into a single vector of data. The vector of data can be input into the database return encoder 212 to generate the intermediate embedding.

[0045] The feature fusion model 214 can combine embeddings. The feature fusion model 214 can generate an output embedding using the intermediate embedding and the question embedding. The feature fusion model 214 can be a machine learning model that is trained to combine two embeddings into one embedding.

[0046] For example, the feature fusion model 214 can train weights that are multiplied by the received embeddings. The weights can modify how much influence each received embedding has on the output embedding. As an illustrative example, the feature fusion model 214 can include a first weight w1 and a second weight w2. The first weight w1 can be multiplied with the question embedding, while the second weight w2 can be multiplied with the intermediate embedding. The result of each multiplication can be added together to form the output embedding. It is understood that the feature fusion model 214 can combine the received embeddings in other mathematical manners.

[0047] The large language model decoder 216 can include a decoder. In some embodiments the large language model decoder 216 can include a self-attention mechanism, an attention mechanism over the encodings, and a feed-forward neural network. The large language model decoder 216 can determine a text output based on the output embedding. The large language model decoder 216 can iteratively be fed the numerical elements in the embedding vectors to determine a next word output for the text output.

[0048] The network interface 206 may include an interface that can allow the processing computer 104 to communicate with external computers. The network interface 206 may enable the processing computer 104 to communicate data to and from another device (e.g., the client device 102, the database 106, etc.). Some examples of the network interface 206 may include a modem, a physical network interface (such as an Ethernet card or other Network Interface Card (NIC)), a virtual network interface, a communications port, a Personal Computer Memory Card International Association (PCMCIA) slot and card, or the like. The wireless protocols enabled by the network interface 206 may include Wi-Fi™. Data transferred via the network interface 206 may be in the form of signals which may be electrical, electromagnetic, optical, or any other signal capable of being received by the externalcommunications interface (collectively referred to as “electronic signals” or “electronic messages”). These electronic messages that may comprise data or instructions may be provided between the network interface 206 and other devices via a communications path or channel. As noted above, any suitable communication path or channel may be used such as, for instance, a wire or cable, fiber optics, a telephone line, a cellular link, a radio frequency (RF) link, a WAN or LAN network, the Internet, or any other suitable medium.

[0049] FIG. 3 shows a hybrid flow diagram illustrating a large language model reasoning and database querying method according to embodiments. The method illustrated in FIG. 3 can be performed by a computer such as the processing computer 104. The processing computer 104 can include an LLM encoder, a database return encoder, a feature fusion model, and an LLM decoder. In some embodiments, the processing computer 104 can include the database 106. In other embodiments, the database 106 can be a database outside of the processing computer 104.

[0050] At step 302, the processing computer 104 can receive a question from the client device 102. The question can include plaintext string data. The question can be a sentence expressed so as to elicit information.

[0051] As an illustrative example, a user of the client device 102 can input a question into the client device 102. The question, for example, can be “how is AirlineXYZ performing in the first quarter of 2023?”

[0052] At step 304, after receiving the question from the client device 102, the processing computer 104 can provide the question to the LLM encoder. The LLM encoder can generate and output a plaintext database query, a data processing instruction, and a question embedding for the question. Generating the plaintext database query, the data processing instruction, and the question embedding for the question are described in further detail herein in reference to FIG. 4.

[0053] The plaintext database query can be a query that is formatted to obtain data from the database 106 based on the question. The plaintext database query can request data from the database 106 that is related to the question. For example, the plaintext database query can query the database 106 for data related to “AirlineXYZ.” The plaintext database query can include a filter that filters data based on the airlinename. The plaintext database query can also include a filter that filters data based on date such that data in the “first quarter of 2023” is obtained (e.g., 1-1 -2023 to 3-31 - 2023).

[0054] The data processing instruction can be a value that indicates how the database return encoder is to encode a set of data obtained from the database 106. The data processing instruction can indicate to the database return encoder on how to generate an embedding using a set of data elements returned from the database 106. The data processing instruction can indicate how many embeddings to create from the data (e.g., create 1 embedding, 2 embeddings, 5 embeddings, etc.). The data processing instruction can also indicate an encoding method (e.g., indicate how to process the data in order to create the embeddings). For example, the data processing instruction can indicate that the data is to be concatenated into a single sentence, from which an embedding is generated. As another example, the data processing instruction can indicate that numerical or textual data from the database 106 is to be used to generate an embedding. As another example, the data processing instruction can indicate for the database return encoder to generate one embedding per received data element and then combine, based on a mathematical operator (e.g., concatenate, multiply, average, etc.) the generated embeddings into an intermediate embedding.

[0055] The question embedding can be an embedding generated based on the question. For example, the LLM encoder can convert the question, which is a text value, into a vector. The LLM encoder can encode the vector into the question embedding. The LLM encoder can be previously trained to generate embeddings. For example, during training, models are exposed to vast amounts of text, allowing the models to learn complex patterns of language usage and meaning. The learning process involves adjusting the parameters of the neural networks (e.g., in the encoder) to minimize the difference between the model’s output and the expected output, a process known as backpropagation.

[0056] As an illustrative example, the LLM encoder can determine a plurality of tokens based on the question to vectorize the text question into a numerical format. For example, the question of “how is AirlineXYZ performing in the first quarter of 2023” can be vectorized into “<3, 53, 63, 100, 453, 52, 90, 32, 403, 234>.” Each word in thequestion can correspond to a token value, which can be determined from a database of tokens that can be created during training of the LLM encoder. It is noted that the some values in this application, such as the vector values above may not be actual values, but are illustrative of the concepts described herein. Actual values may differ from the values discussed.

[0057] The LLM encoder can generate a question embedding based on the vector of values that represent the question. For example, the LLM encoder can generate a question embedding of “<50, 122, 412, 53, 304>.”

[0058] At step 306, after generating the plaintext database query for the question, the processing computer 104 can provide the plaintext database query to the database 106. The processing computer 104 can provide the plaintext database query to the database 106 in any suitable manner.

[0059] At step 308, after receiving the plaintext database query, the database 106 can search for data stored in the database 106 based on the received plaintext database query. The database 106 can identify data elements in the database to provide to the processing computer 104.

[0060] For example, the plaintext database query can be a query for data related to “AirlineXYZ.” The database 106 can filter the data in the database based on the airline name of “AirlineXYZ.” The database 106 can obtain a set of data elements that are related to “AirlineXYZ.” For example, the database 106 can store transaction data that can include transactions involving the “AirlineXYZ.” The set of data elements can include a plurality of transaction data. The database 106 can also filter the data based on the filter indicating the date range that is included in the plaintext database query (e.g., 1 -1 -2023 to 3-31-2023).

[0061] As an illustrative example, the database 106 can obtain 58 data elements that satisfy the plaintext database query and the filters therein. The database 106 can return the data elements comprising a plurality of values illustrated in Table 1 , below. Each row is a data element that contains a plurality of values.Table 1 : example data elements

[0062] At step 310, after identifying and obtaining the set of data elements based on the plaintext database query, the database 106 can return the set of data elements to the processing computer 104.

[0063] The processing computer 104 can provide the received set of data elements to the database return encoder in the processing computer 104.

[0064] At step 312, after generating the data processing instruction and receiving the set of data elements, the processing computer 104 can process the set of data elements using the data processing instruction in the database return encoder. The database return encoder can encode the data elements into embedding(s) that represent the data elements of the set of data elements. The embeddings output by the database return encoder can be referred to as intermediate embeddings.

[0065] The database return encoder can generate the intermediate embeddings based on the data processing instruction. For example, the data processing instruction can indicate for the database return encoder to generate one intermediate embedding for each data element. As another example, the data processing instruction can indicate for the database return encoder to generate fewer intermediate embeddings than the number of data elements. As another example, the data processing instruction can indicate for the database return encoder to generate one intermediate embedding that represents all of the data elements.

[0066] For example, the database return encoder can evaluate the data processing instruction to determine how to generate the intermediate embedding(s). The database return encoder can determine that the data processing instruction indicates that a single intermediate embedding is to be generated based on a singleinput vector. As such, the database return encoder can combine and encode the received set of data elements into a single vector. For example, the database return encoder can concatenate each data element of the set of data elements to form a single sentence. The database return encoder can then encode the single sentence into vector format. The database return encoder can then process the vector to generate an intermediate embedding.

[0067] As an illustrative example, the database return encoder can vectorize each data element of the set of data elements to obtain a plurality of vectors. For example, entry 1 in the Table 1 , above, can be vectorized as “<1 , 1120231341 , 5555, 4123456789012345, 512>.” The database return encoder can then concatenate each vector of the plurality of vectors into a single input vector. The database return encoder can generate an intermediate embedding using the input vector. The output intermediate embedding can be, for example, “<132, 158, 139, 9, 78>.”

[0068] At step 314, after generating the question embedding and the intermediate embedding(s), the processing computer 104 can process the question embedding and the intermediate embedding(s) using the feature fusion model.

[0069] The feature fusion model can combine the question embedding and the intermediate embedding into an output embedding. The output embedding can be a vector that represents both the question embedding vector and the intermediate embedding vector.

[0070] For example, the feature fusion model can be a machine learning model that is trained to combine a plurality of embeddings into a single embedding. The feature fusion model can include a number of weights, which can be applied (e.g., multiplied) to each input embedding. For example, the feature fusion model can include a first weight and a second weight. The feature fusion model can multiply the first weight and the question embedding. The feature fusion model can multiply the second weight and the intermediate embedding. The feature fusion model can then combine the weighted question embedding and the weighted intermediate embedding. For example, the feature fusion model can add, multiply, average, etc. the weighted question embedding and the weighted intermediate embedding.

[0071] As an illustrative example, the feature fusion model can receive the question embedding of “<50, 122, 412, 53, 304>” and the intermediate embedding of “<132, 158, 139, 9, 78>.” The feature fusion model can multiply each input embedding by a weight value or, in some embodiments, a weight vector or a weight matrix. The feature fusion model can multiply the question embedding by a first weight W1 that is equal to 1 .2. The feature fusion model can multiply the intermediate embedding by a second weight W2 that is equal to 0.9. The feature fusion model can compute:< 50, 122, 412, 53, 304 > * 1.2 = < 60, 146.4, 494.4, 54.2, 364.8 >< 132, 158, 139, 9, 78 > * 0.9 = < 118.8, 142.2, 125.1, 8.1, 70.2 >

[0072] After weighting the question embedding and the intermediate embedding the feature fusion model can mathematically combine the two embeddings into a single embedding. As an example, the feature fusion model can average the question embedding and the intermediate embedding to obtain an output embedding of “<89.4, 144.3, 309.75, 31.15, 217.5>.” However, it is understood that other mathematical operations can be performed on the question embedding and the intermediate embedding to combine the two embeddings into a single embedding. In some embodiments, the combination of the two embeddings can involve additional weight values in the machine learning model that are pretrained.

[0073] At step 316, after generating the output embedding, the processing computer 104 can provide the output embedding to the LLM decoder. The LLM decoder can decode the output embedding from a vector into a text output. The LLM decoder can include a type of neural network that generates output from encoded representations. The LLM decoder can include attention mechanisms (e.g., a selfattention mechanism and an attention mechanism over the encodings) and a feedforward neural network. The LLM decoder can be trained to determine a next word or token in an output sentence based on the input embedding. As such, the LLM decoder can convert the output embedding into a text.

[0074] For example, the LLM decoder can determine the text output from the output embedding of “<89.4, 144.3, 309.75, 31.15, 217.5>.” The text output can be, for example, “AirlineXYZ had good income in the first quarter of 2023.”

[0075] At step 318, the processing computer 104 can determine a summary output based on the text output from the LLM decoder and based on the set of data elements received from the database 106. The summary output can include one or more sections. Each section can include information related to the question. For example, the summary output can include a first section 318A and a second section 318B.

[0076] The first section 318A can include data from the set of data elements obtained from the database 106. The first section 318A can include one or more data elements and / or derived values from the one or more data elements (e.g., sum, average, median, min, max, etc.). For example, the first section 318A can include a derived value that is the sum total amount of all transactions that were performed that involved the AirlineXYZ in the first quarter of 2023.

[0077] The second section 318B can include the text output from the LLM decoder that includes a sentence that answers the question of “how is AirlineXYZ performing in the first quarter of 2023?”

[0078] The processing computer 104 can provide the summary output to the client device 102 in response to the question.

[0079] FIG. 4 shows a hybrid flow diagram illustrating details of a large language model encoder method according to embodiments. The method illustrated in FIG. 4 can be performed by a computer such as the processing computer 104. FIG. 4 illustrates further details regarding the generation of a plaintext database query, a data processing instruction, and a question embedding for the question.

[0080] At step 402, the processing computer 104 can input the question into the LLM encoder. The question can be in a plaintext format and can be converted into a plurality of tokens that represent the words of the question (e.g., be vectorized) prior to being input into the LLM encoder, or can be converted by the LLM encoder.

[0081] For example, the question can be “how does CarCompanyABC compare to CarCompanyXYZ in 2024?” The question can be vectorized such that each word in the sentence is converted into a numerical value (e.g., token). The vector can then be processed by the LLM encoder.

[0082] At step 404, after obtaining the vector corresponding to the question, the LLM encoder can initiate generation of the plaintext database query, the data processing instruction, and the question embedding for the question. The LLM encoder can process the question during steps 406-416.

[0083] At step 406, the LLM encoder can generate the plaintext database query, which the processing computer 104 can provide to the database. The plaintext database query can have a syntax or a format that allows the plaintext database query to be properly read by the database. The plaintext database query can include a name of a database that is to be searched for data elements. For example, the name of the database can indicate a database that stores company data. The plaintext database query can also include a table name, one or more filters, and, in some embodiments, a mode that indicates what the output data sequence should be.

[0084] The one or more filters can be in a format of [filter name] [condition] [filter content]. The filter name can be a name of the filter that is being used and can identify the filter to the database. For example, a filter name can be datetime, resource provider identifier, user identifier, etc. The condition can be a constraint that is used with the filter name. The condition can be a resource provider identifier or name (e.g., CarCompanyXYZ), a date range (e.g., 2024), etc. The filter content can be an indication of how to sort the return of the query. The filter content can be, for example, an indication to sort the return data by date, in alphabetical order, etc. In some embodiments, the one or more filters can be formatted as an SQL query. For example, the one or more filters can include a filter of “SELECT column from table where data>'2021 -01-01 ' order by column.”

[0085] The LLM encoder can generate the plaintext database query by determining a database name and / or a table name that is to be searched for data elements related to the question. For example, the LLM encoder can identify terms in the question that relate to data elements that can be obtained. The LLM encoder can identify the words of “CarCompanyABC,” “CarCompanyXYZ,” and “2024” from the question. The LLM encoder can perform a multi-class classification process to determine what database and / or table of a plurality of databases and / or a plurality of tables would include data related to the words. The multi-class classification process can determine one or more predefined database(s) and / or table name(s). As anexample, the LLM encoder may have access to a database that stores transaction data associated with merchants. The LLM encoder can identify the words “CarCompanyABC” and “CarCompanyXYZ” as being company names and can thus be related to the transaction data database. The LLM encoder can include the database name and / or table name into the beginning of the plaintext database query.

[0086] The LLM encoder can then generate one or more filters that can be applied to filter the data elements of the database based on the question. The LLM encoder can generate the one or more filters based on the input question using a machine learning process that is trained to identify related filters. For example, the LLM encoder can generate filters that relate to “CarCompanyABC,” “CarCompanyXYZ,” and “2024” due to these words being included in the question. For example, a first filter can be utilized to filter the database by data related to CarCompanyABC. A second separate filter can be to filter the database by data related to CarCompanyXYZ. A third filter can be applied in conjunction with the two aforementioned filter to a date range of 2024. As such, the plaintext query can be: [[Database: merchant transactions] [Merchant name: CarCompanyABC OR CarCompanyXYZ] [Time from: 01 / 01 / 2024; Time to: 12 / 31 / 2024]].

[0087] At step 408, the LLM encoder can provide the plaintext database query to the database.

[0088] At step 410, the LLM encoder can generate the data processing instruction, which the processing computer 104 can provide to the database return encoder. The data processing instruction can include a identifier or instructions that indicate how the database return encoder is to encode a set of data elements returned from the database. For example, the data processing instruction can indicate for the database return encoder to encode each data element of a set of data elements into a different intermediate embedding. As another example, the data processing instruction can indicate for the database return encoder to encode each data element of a set of data elements into a single intermediate embedding. As another example, the data processing instruction can indicate for the database return encoder to encode each data element of a set of data elements into a predetermined number of intermediate embeddings.

[0089] The LLM encoder can be trained during a machine learning process, which adjusts decision weights in a machine learning model, to select between a plurality of different data processing instructions. For example, the LLM encoder can select between instructing the database return encoder to generate one embedding per data element, to generate a predetermined number of embeddings for the set of data elements, or to generate one embedding that represents all data elements.

[0090] At step 412, the LLM encoder can provide the data processing instruction to the database return encoder.

[0091] At step 414, the LLM encoder can generate the question embedding, which the processing computer 104 can provide to the feature fusion model. The LLM encoder can generate the question embedding by converting the input question, which is plaintext, into a vector. The LLM encoder can generate the question embedding using an attention mechanism and a feedforward neural network. The LLM encoder can be trained to convert input text into output vectors that represent the input text. During the conversion from input text into output vectors, the LLM encoder can perform an attention process that places different weights, which are ultimately represented in the output vectors, between the relationships between different words in the input text.

[0092] At step 416, the LLM encoder can provide the question embedding to the feature fusion model.

[0093] In some embodiments, the LLM encoder can perform steps 406, 410, and 414 in any order.

[0094] Embodiments of the disclosure have a number of advantages. For example, the large language model can query a database and can utilize a database return encoder so that reasoning and facts can be included into large language model answers.

[0095] Current large language models only understand data upon which they were trained. Therefore any factual data that occurs after the training of the large language model is not within the large language model’s knowledge base. Thus, current large language models cannot answer questions regarding factual data since they have no access to any factual data. Embodiments solve the technical problem of using recent factual data in a large language model. Embodiments provide for thetechnical solution of incorporating factual data that was obtained or otherwise created after training of the large language model into the responses generated by the large language model.

[0096] For example, provided a question of “what is a current stock price of company ABC and how has it performed in the past week,” previous large language models would need to create fake data for the current stock price and other performance data since it does not have access to this data. As an example, current large language models can provide a response of “the stock price of company ABC is $291 and the company ABC is doing well.” However, the correct stock price of company ABC is $563.44. In contrast to previous large language models, embodiments provide for systems that can create a correct response to the aforementioned question using factual data from a database. For example, embodiments can provide for a response of “the stock price of company ABC is $563.44 according to [source_database] and the company’s stock price has decreased by 2.4% since last week.”

[0097] Embodiments also eliminate the need for a manual process to generate fine-tuning reasoning prompt data. For example, during the training process for the large language model, thousands of reasoning prompts do not need to be custom created and then input into the large language model for training so that the large language model is trained to reason.

[0098] Although the steps in the flowcharts and process flows described above are illustrated or described in a specific order, it is understood that embodiments of the invention may include methods that have the steps in different orders. In addition, steps may be omitted or added and may still be within embodiments of the invention.

[0099] Any of the software components or functions described in this application may be implemented as software code to be executed by a processor using any suitable computer language such as, for example, Java, C, C++, C#, Objective-C, Swift, or scripting language such as Perl or Python using, for example, conventional or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer readable medium for storage and / or transmission, suitable media include random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an opticalmedium such as a compact disk (CD) or DVD (digital versatile disk), flash memory, and the like. The computer readable medium may be any combination of such storage or transmission devices.

[0100] Such programs may also be encoded and transmitted using carrier signals adapted for transmission via wired, optical, and / or wireless networks conforming to a variety of protocols, including the Internet. As such, a computer readable medium according to an embodiment of the present invention may be created using a data signal encoded with such programs. Computer readable media encoded with the program code may be packaged with a compatible device or provided separately from other devices (e.g., via Internet download). Any such computer readable medium may reside on or within a single computer product (e.g. a hard drive, a CD, or an entire computer system), and may be present on or within different computer products within a system or network. A computer system may include a monitor, printer, or other suitable display for providing any of the results mentioned herein to a user.

[0101] The above description is illustrative and is not restrictive. Many variations of the invention will become apparent to those skilled in the art upon review of the disclosure. The scope of the invention should, therefore, be determined not with reference to the above description, but instead should be determined with reference to the pending claims along with their full scope or equivalents.

[0102] One or more features from any embodiment may be combined with one or more features of any other embodiment without departing from the scope of the invention.

[0103] As used herein, the use of "a," "an," or "the" is intended to mean "at least one," unless specifically indicated to the contrary.

Claims

WHAT IS CLAIMED IS:1 . A method performed using a computer comprising an encoder, a database return encoder, a feature fusion model, and a decoder, the method comprising: receiving, by the encoder, a question; outputting, by the encoder, a plaintext database query, a data processing instruction, and a question embedding for the question; obtaining a set of data elements from a database using the plaintext database query; receiving, by the database return encoder, the set of data elements and the data processing instruction; determining, by the database return encoder, an intermediate embedding based on the set of data elements and the data processing instruction; determining, by the feature fusion model, an output embedding using the intermediate embedding and the question embedding; and determining, by the decoder, a text output based on the output embedding.

2. The method of claim 1 , wherein the set of data elements includes data elements representing transaction data, weather data, image data, or network data.

3. The method of claim 1 , wherein the computer is a processing computer, wherein receiving the question comprises: receiving the question from a client device.

4. The method of claim 3 further comprising: providing the text output to the client device.

5. The method of claim 1 , wherein the text output is an answer to the question.

6. The method of claim 1 , wherein the encoder is a large language model encoder and the decoder is a large language model decoder.

7. The method of claim 1 further comprising: determining one or more filters, wherein the plaintext database query includes the one or more filters.

8. The method of claim 1 , wherein the data processing instruction indicates an encoding method.

9. The method of claim 1 further comprising: generating, by the computer, a first section of a summary output using the set of data elements; and generating, by the computer, a second section of the summary output using the text output.

10. The method of claim 1 , wherein the feature fusion model is a machine learning model that is trained to combine a plurality of embeddings including the intermediate embedding and the question embedding into one embedding.11 . The method of claim 1 , wherein the question is a text sentence created by a user.

12. A computer comprising: a processor; and a computer-readable medium coupled to the processor, the computer- readable medium comprising code executable by the processor for implementing a method comprising: receiving a question; outputting, by an encoder in the computer, a plaintext database query, a data processing instruction, and a question embedding for the question; obtaining a set of data elements from a database using the plaintext database query; receiving, by a database return encoder in the computer, the set of data elements and the data processing instruction;determining, by the database return encoder, an intermediate embedding based on the set of data elements and the data processing instruction; determining, by a feature fusion model in the computer, an output embedding using the intermediate embedding and the question embedding; and determining, by a decoder in the computer, a text output based on the output embedding.

13. The computer of claim 12 further comprising: determining one or more data tables in the database using a machine learning model, wherein the plaintext database query includes the one or more data tables.

14. The computer of claim 12 further comprising: generating a first section of a summary output using the set of data elements; generating a second section of the summary output using the text output; and providing the summary output to a client device in response to receiving the question from the client device.

15. The computer of claim 14, wherein generating the first section comprises: generating one or more derived values from the set of data elements, wherein the first section includes the one or more derived values.

16. The computer of claim 14, wherein the first section comprises one or more data elements of the set of data elements.

17. The computer of claim 12, wherein the computer is a processing computer, wherein the text output is an answer to the question, wherein the data processing instruction indicates an encoding method, wherein receiving the question comprises:receiving the question from a client device, wherein the client device is a user device operated by a user, wherein the question is a text sentence created by the user.

18. A system comprising: a database storing a plurality of data elements; and a computer in operative communication with the database, the computer comprising: a processor; and a computer-readable medium coupled to the processor, the computer-readable medium comprising code executable by the processor for implementing a method comprising: receiving a question; outputting, by an encoder in the computer, a plaintext database query, a data processing instruction, and a question embedding for the question; obtaining a set of data elements from the database using the plaintext database query; receiving, by a database return encoder in the computer, the set of data elements and the data processing instruction; determining, by the database return encoder, an intermediate embedding based on the set of data elements and the data processing instruction; determining, by a feature fusion model in the computer, an output embedding using the intermediate embedding and the question embedding; and determining, by a decoder in the computer, a text output based on the output embedding.

19. The system of claim 18, wherein the method further comprises: generating one or more derived values from the set of data elements; generating a first section of a summary output using the one or more derived values;generating a second section of the summary output using the text output; and providing the summary output to a client device in response to receiving the question from the client device.

20. The system of claim 18, wherein the method further comprises: determining a plurality of tokens based on the question, wherein the encoder determines the question embedding based on the plurality of tokens.

Images