Computer-implemented method, computer program, and computing system (serverless functional routing for large language model inference service)

Abstract

To provide a computer-implemented method for serving a large language model (LLM) application via a serverless function router communicative with multiple endpoints that each have a set of subject matter expert models stored thereon.SOLUTION: A computer-implemented method includes: receiving a prompt; querying a database comprising multiple datasets for an indication as to which one of the multiple datasets has a highest level of similarity with the prompt; recognizing one of the multiple endpoints as having the set of expert models stored thereon which have a closest match with the one of the multiple datasets; and routing the prompt to the one of the multiple endpoints having the set of the expert models stored thereon which have the closest match with the one of the multiple datasets.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates generally to large-scale language models in computing systems, and more particularly to cost-effective and quality-guaranteed serverless functional routing for large-scale language model (LLM) inference services. Summary of the Invention [Problem to be solved by the invention]

[0002] LLMs are language models known for their ability to achieve general-purpose language generation. They achieve this ability by learning statistical relationships from text documents during a computationally intensive self-supervised and semi-supervised training process. LLMs are generally artificial neural networks that can be built with transformer-based architectures, although recent implementations are based on alternative architectures such as variants of recurrent neural networks.

[0003] As an example, LLMs can be used for text generation and other forms of generative artificial intelligence (AI). In these and other cases, an LLM receives input text and repeatedly predicts the next token or word. Until recently, the only way a given LLM could be adapted to perform a specific task was through fine-tuning. However, modern large-scale LLMs have proven capable of being promptly engineered to produce positive results by acquiring knowledge of the syntax, semantics, and ontologies inherent in human languages. [Means for solving the problem]

[0004] An aspect of the present disclosure provides a computer-implemented method for servicing large language model (LLM) applications via a serverless function router capable of communicating with a plurality of endpoints, each endpoint storing a set of subject matter expert models. The computer-implemented method includes receiving a prompt, querying a database including a plurality of datasets for an indication of which one of the datasets most closely resembles the prompt, identifying one of the plurality of endpoints storing the set of expert models that has the highest percentage match with the one of the plurality of datasets, and routing the prompt to the one of the plurality of endpoints storing the set of expert models that has the highest percentage match with the one of the plurality of datasets. In additional or alternative embodiments, the computer-implemented method provides a rapid response to the prompt without wasting valuable computing resources.

[0005] An aspect of the present disclosure provides a computer program product for servicing large language model (LLM) applications via a serverless function router capable of communicating with a plurality of endpoints, each endpoint storing a set of subject matter expert models. The computer program product includes one or more computer-readable storage media having computer-readable program code collectively stored thereon. The computer-readable program code is executable by a processor of a computer system to cause the computer system to perform a method. The method includes receiving a prompt; querying a database including a plurality of datasets for an indication of which one of the datasets most closely resembles the prompt; identifying one of the plurality of endpoints as storing the set of expert models that has the highest degree of agreement with the one of the plurality of datasets; and routing the prompt to the one of the plurality of endpoints storing the set of expert models that has the highest degree of agreement with the one of the plurality of datasets. In additional or alternative embodiments, the method provides a rapid response to the prompt without wasting valuable computing resources.

[0006] Certain aspects of the present disclosure provide a computing system including a processor, a memory coupled to the processor, and one or more computer-readable storage media coupled to the processor. The one or more computer-readable storage media collectively include instructions executed by the processor via the memory to implement a method for servicing large language model (LLM) applications via a serverless function router capable of communicating with a plurality of endpoints, each endpoint having a set of subject matter expert models stored thereon. In additional or alternative embodiments, the method for servicing LLM applications via a serverless function router capable of communicating with a plurality of endpoints, each endpoint having a set of subject matter expert models stored thereon, provides rapid response to prompts without wasting valuable computing resources.

[0007] Further technical features and advantages are realized through the techniques of the present invention. Embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, reference is made to the detailed description and drawings. [Brief explanation of the drawings]

[0008] The exclusive details set forth herein are particularly pointed out and distinctly claimed in the claims at the conclusion of this specification. The above-discussed and other features and advantages of embodiments of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0009] [Figure 1] 1 is a schematic diagram of a computing environment for performing a computer-implemented method for operating a chip processing assembly, according to one or more embodiments;

[0010] [Figure 2] FIG. 1 is a block diagram of components of a machine learning training and inference system according to one or more embodiments;

[0011] [Figure 3] FIG. 1 is a flow diagram illustrating a computer-implemented method for serving large language model (LLM) applications via a serverless function router capable of communicating with multiple endpoints, each of which stores a set of subject matter expert models, according to one or more embodiments;

[0012] [Figure 4] 4 is a schematic diagram of an architecture for performing the computer-implemented method of FIG. 3 , according to one or more embodiments;

[0013] [Figure 5] 1 is a table illustrating a model, hypothetical prompts for the model, and data embedding, according to one or more embodiments;

[0014] [Figure 6] is a formula showing the Euclidean distance between a prompt and a model, according to one or more embodiments; and

[0015] [Figure 7] 1 is a table illustrating models and distance data between prompts and the models, respectively, according to one or more embodiments.

[0016] The figures shown herein are exemplary. There may be many variations to the figures or the operations described therein without departing from the spirit of the invention. For example, actions may be performed in a different order, or actions may be added, deleted, or modified. Additionally, the term "coupled" and variations thereof describe the existence of a communication path between two elements, and do not imply a direct connection between the elements without an intervening element / connection between them. All such variations are considered part of this specification.

[0017] In the following detailed description of the illustrated embodiments and in the accompanying figures, various elements shown in the figures are labeled with two- or three-digit reference numbers. With few exceptions, the left-most digit(s) of each reference number corresponds to the figure in which the element is first shown. DETAILED DESCRIPTION OF THE INVENTION

[0018] An aspect of the present disclosure provides a computer-implemented method for servicing large language model (LLM) applications via a serverless function router capable of communicating with a plurality of endpoints, each of which stores a set of subject matter expert models. The computer-implemented method includes receiving a prompt, querying a database including the plurality of datasets for an indication of which one of the plurality of datasets is most similar to the prompt, identifying one of the plurality of endpoints as storing a set of expert models that have a highest percentage match with one of the plurality of datasets, and routing the prompt to one of the plurality of endpoints that stores the set of expert models that have a highest percentage match with one of the plurality of datasets. In additional or alternative embodiments, the computer-implemented method provides a rapid response to the prompt without wasting valuable computing resources.

[0019] Receiving the prompt, querying the database, recognizing one of multiple endpoints, and routing the prompt are performed by a serverless function router, so that all of these operations are contained within a single operational and computational entity.

[0020] The database includes a vector database configured to facilitate or enable directing queries to the particular endpoints with the highest match rate.

[0021] Each of the plurality of datasets has a highest match with the subject matter expert model stored in one of the plurality of endpoints. In this manner, the computer-implemented method avoids sending queries to endpoints with lower matches.

[0022] The subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to medical subjects, the subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to financial subjects, and the subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to technical subjects. These particular subject matter experts are merely examples and cover some of the most valuable subjects.

[0023] A subject matter expert model includes one or more foundation models and one or more fine-tuned models, so that each subject matter expert can approach a query in a different way.

[0024] The serverless function router is a prompt-aware router, and the routing includes prompt-aware routing that responds to user input.

[0025] An aspect of the present disclosure provides a computer program product for servicing large language model (LLM) applications via a serverless function router capable of communicating with a plurality of endpoints, each endpoint storing a set of subject matter expert models. The computer program product includes one or more computer-readable storage media having computer-readable program code collectively stored thereon. The computer-readable program code is executable by a processor of the computer system to cause the computer system to perform a method. The method includes receiving a prompt; querying a database including the plurality of datasets for an indication as to which one of the plurality of datasets is most similar to the prompt; identifying one of the plurality of endpoints as storing a set of expert models that have a highest degree of agreement with one of the plurality of datasets; and routing the prompt to one of the plurality of endpoints storing the set of expert models that have a highest degree of agreement with one of the plurality of datasets. In additional or alternative embodiments, the method provides a rapid response to the prompt without wasting valuable computing resources.

[0026] Receiving the prompt, querying the database, recognizing one of multiple endpoints, and routing the prompt are performed by a serverless function router, so that all of these operations are contained in a single operational and computational entity.

[0027] The database includes a vector database configured to facilitate or enable directing queries to the particular endpoints with the highest match rate.

[0028] Each of the multiple datasets has a highest match with a subject matter expert model stored on one of the multiple endpoints. In this manner, the method avoids sending queries to endpoints with lower matches.

[0029] The subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to medical subjects, the subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to financial subjects, and the subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to technical subjects. These particular subject matter experts are merely examples and cover some of the most valuable subjects.

[0030] A subject matter expert model includes one or more foundation models and one or more fine-tuned models, so that each subject matter expert can approach a query in a different way.

[0031] The serverless function router is a prompt-aware router, and the routing includes prompt-aware routing that responds to user input.

[0032] Certain aspects of the present disclosure provide a computing system including a processor, a memory coupled to the processor, and one or more computer-readable storage media coupled to the processor. The one or more computer-readable storage media collectively include instructions executed by the processor via the memory to implement a method for servicing large language model (LLM) applications via a serverless function router capable of communicating with a plurality of endpoints, each endpoint having a set of subject matter expert models stored thereon. In additional or alternative embodiments, the method for servicing LLM applications via a serverless function router capable of communicating with a plurality of endpoints, each endpoint having a set of subject matter expert models stored thereon, provides rapid response to prompts without wasting valuable computing resources.

[0033] Receiving the prompt, querying the database, recognizing one of multiple endpoints, and routing the prompt are performed by a serverless function router, so that all of these operations are contained within a single operational and computational entity.

[0034] The database includes a vector database configured to facilitate or enable directing queries to the particular endpoints with the highest match rate.

[0035] Each of the plurality of data sets has a highest match with a subject matter expert model stored on one of the plurality of endpoints. In this manner, a method for serving an LLM application via a serverless function router capable of communicating with a plurality of endpoints, each of which stores a set of subject matter expert models, avoids sending queries to endpoints with a lower match.

[0036] The subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to medical subjects, the subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to financial subjects, and the subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to technical subjects. These particular subject matter experts are merely examples and cover some of the most valuable subjects.

[0037] A subject matter expert model includes one or more foundation models and one or more fine-tuned models, so that each subject matter expert can approach a query in a different way.

[0038] The serverless function router is a prompt-aware router, and the routing includes prompt-aware routing that responds to user input.

[0039] Various aspects of the present disclosure are described by text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in computer program product (CPP) embodiments. For any flowchart, depending on the technology involved, operations may be performed in a different order than shown in a given flowchart. For example, again depending on the technology involved, two operations shown in successive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in at least partially overlapping time.

[0040] A computer program product embodiment ("CPP embodiment" or "CPP") is a term used in this disclosure to describe any set of one or more storage media (also referred to as "media"), collectively contained in one or more storage devices, that collectively contain machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A "storage device" is any tangible device that can hold and store instructions for use by a computer processor. The computer-readable storage medium may be, but is not limited to, an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded device (such as pits / lands formed on a major surface of a punch card or disk), or any suitable combination of the foregoing. Computer-readable storage media, as the term is used in this disclosure, is not to be construed as storage of transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, light pulses passing through fiber optic cables, electrical signals communicated over wires, and / or other transmission media. As will be appreciated by those skilled in the art, data typically moves at some infrequent time during the normal operation of a storage device, such as during access, defragmentation, or garbage collection, but this does not make the storage device transient, as data is not transient while stored.

[0041] Referring to Figure 1, a computer or computing device 100 implements a computer-implemented method for servicing large language model (LLM) applications through a serverless function router capable of communicating with a plurality of endpoints, each endpoint having a set of subject matter expert models stored thereon. The computer or computing device 100 of Figure 1 includes an example environment for executing at least a portion of the computer code involved in performing a method of the present invention, such as block 1001 of the computer-implemented method for servicing large language model (LLM) applications through a serverless function router capable of communicating with a plurality of endpoints, each endpoint having a set of subject matter expert models stored thereon.

[0042] In addition to the computer-implemented method of block 1001 for serving a large language model (LLM) application via a serverless function router capable of communicating with multiple endpoints, each of which stores a set of subject matter expert models, a computer or computing device 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end-user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a set of processors 110 (including processing circuitry 120 and cache 121), a communications fabric 111, volatile memory 112, persistent storage 113 (including an operating system 122 and the computer-implemented method of block 1001 as shown above), a set of peripheral devices 114 (including a set of user interface (UI) devices 123, storage 124, and a set of Internet of Things (IoT) sensors 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud orchestration module 141, a set of host physical machines 142, a set of virtual machines 143, and a set of containers 144.

[0043] Computer 101 may take the form of a desktop computer, a laptop computer, a tablet computer, a smartphone, a smartwatch or other wearable computer, a mainframe computer, a quantum computer, or any other form of computer or mobile device now known or later developed that is capable of executing programs, accessing a network, or querying a database, such as remote database 130. As is well understood in the art of computer technology, depending on the technology, execution of a computer-implemented method may be distributed among multiple computers and / or multiple locations. While this presentation of the computer-implemented method focuses the detailed discussion on a single computer, specifically computer 101, to keep the presentation as simple as possible. Computer 101 may be located within a cloud, even if not depicted within the cloud in FIG. 1 . However, computer 101 need not be within a cloud except to any extent that may be categorically indicated.

[0044] Processor set 110 includes one or more computer processors of any type now known or later developed. Processing circuitry 120 may be distributed across multiple packages, e.g., multiple coordinated integrated circuit chips. Processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. Cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for fast access by threads or cores executing on processor set 110. Cache memory is typically configured with multiple levels depending on relative proximity to the processing circuitry. Alternatively, some or all of the caches for a processor set may be located “off-chip.” In some computing environments, processor set 110 may be designed to operate with qubits and perform quantum computing.

[0045] Computer-readable program instructions are typically loaded into computer 101 and a series of operational steps are executed by processor set 110 of computer 101, thereby performing a computer-implemented method, such that the instructions so executed instantiate the methods specified in the computer-implemented method flowcharts and / or descriptions contained in this document (collectively referred to as the "methods of the present invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media discussed below. The program instructions and associated data are accessed by processor set 110 to control and direct the execution of the methods of the present invention. In the computer-implemented method, at least some of the instructions for performing the methods of the present invention may be stored in persistent storage 113 in block 1001 of the computer-implemented method.

[0046] Communications fabric 111 is the signal-conducting pathway that allows various components of computer 101 to communicate with one another. Typically, this fabric is made up of switches and conductive pathways, such as those that make up buses, bridges, physical input / output ports, and the like. Other types of signal communication pathways, such as fiber optic and / or wireless communication pathways, may also be used.

[0047] Volatile memory 112 may be any type of volatile memory now known or later developed. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory 112 is characterized by random access, although this is not required unless expressly indicated. In computer 101, volatile memory 112 is located in a single package and is internal to computer 101, although alternatively or additionally, volatile memory may be distributed across multiple packages and / or located external to computer 101.

[0048] Persistent storage 113 is any form of non-volatile computer storage, now known or later developed. The non-volatility of this storage means that stored data remains regardless of whether power is supplied to computer 101 and / or power is supplied directly to persistent storage 113. Persistent storage 113 may be read-only memory (ROM), but typically at least a portion of persistent storage allows data to be written, data to be deleted, and data to be rewritten. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. Operating system 122 may take several forms, such as an open-source Portable Operating System Interface-type operating system employing a kernel or various known proprietary operating systems. The code included in block 1001 of the computer-implemented method typically includes at least a portion of the computer code involved in performing the method of the present invention.

[0049] Peripheral device set 114 includes the set of peripheral devices of computer 101. Data communication connections between peripheral devices and other components of computer 101 may be implemented in various ways, such as Bluetooth® connections, near field communication (NFC) connections, connections made by cable (such as a universal serial bus (USB)-type cable), insertable connections (e.g., a Secure Digital (SD) card), connections made through a local area communication network, and even connections made through a wide area network such as the Internet. In various embodiments, UI device set 123 may include components such as a display screen, speakers, microphones, wearable devices (such as goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. Storage 124 is external storage, such as an external hard drive, or insertable storage, such as an SD card. Storage 124 may be persistent and / or volatile. In some embodiments, storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 101 needs to have a large amount of storage (e.g., computer 101 stores and manages a large database locally), this storage may be provided by a peripheral storage device designed to store extremely large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. IoT sensor set 125 consists of sensors that may be used in Internet of Things applications. For example, one sensor may be a thermometer and another sensor may be a motion detector.

[0050] The network module 115 is a collection of computer software, hardware, and firmware that enables the computer 701 to communicate with other computers over the WAN 102. The network module 115 may include hardware such as a modem or Wi-Fi® signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for communicating data over the Internet. In some embodiments, the network control and network forwarding functions of the network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing Software-Defined Networking (SDN)), the control and forwarding functions of the network module 115 are performed on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for carrying out the methods of the present invention may be downloaded to the computer 101 from an external computer or external storage device, typically through a network adapter card or network interface included in the network module 115.

[0051] WAN 102 is any wide area network (e.g., the Internet) capable of communicating computer data over non-local distances by any now known or later developed technology for communicating computer data. In some embodiments, WAN 102 may be replaced and / or supplemented by a local area network (LAN) designed to communicate data between devices located in a local area, such as a Wi-Fi® network. WANs and / or LANs typically include copper transmission cables, optical transmission fiber, wireless transmission, and computer hardware such as routers, firewalls, switches, gateway computers, and edge servers.

[0052] End-user device (EUD) 103 is any computer system used and controlled by an end user (e.g., a customer of the enterprise that operates computer 101) and may take any of the forms discussed above in connection with computer 101. EUD 103 typically receives useful and useful data from the operation of computer 101. For example, in the hypothetical case where computer 101 is designed to provide recommendations to the end user, the recommendations would typically be communicated from network module 115 of computer 101 over WAN 102 to EUD 103. In this manner, EUD 103 can display or otherwise present the recommendations to the end user. In some embodiments, EUD 103 may be a client device such as a thin client, a heavy client, a mainframe computer, a desktop computer, and the like.

[0053] Remote server 104 is any computer system that services at least some data and / or functionality to computer 101. Remote server 104 may be controlled and used by the same entity that operates computer 101. Remote server 104 represents a machine that collects and stores useful and useful data for use by other computers, such as computer 101. For example, in the hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 101 from remote database 130 of remote server 104.

[0054] A public cloud 105 is any computer system available for use by multiple entities that provides on-demand availability of computer system resources and / or other computer capabilities, particularly data storage (cloud storage) and computing power, without direct, active management by users. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct, active management of the computing resources of the public cloud 105 is performed by the computer hardware and / or software of a cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that comprise a host physical machine set 142, which is the universe of physical computers within and / or available to the public cloud 105. A virtual computing environment (VCE) typically takes the form of a virtual machine from a virtual machine set 143 and / or a container from a container set 144. It is understood that these VCEs may be stored as images and may be transferred between various physical machine hosts as images or after instantiation of the VCE. The cloud orchestration module 141 manages the transfer and storage of images, deploys new instantiations of VCE, and manages active instantiations of VCE deployments. The gateway 140 is a collection of computer software, hardware, and firmware that enables the public cloud 105 to communicate over the WAN 102.

[0055] We now provide some further explanation of virtualized computing environments (VCEs). A VCE can be stored as an "image." A new, active instance of a VCE can be instantiated from the image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to a feature of an operating system in which the kernel allows the existence of multiple isolated user space instances called containers. These isolated user space instances typically function as actual computers from the perspective of programs running within them. A computer program running on a typical operating system can utilize all of the computer's resources, such as attached devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running inside a container can only use the contents of the container and of the devices allocated to the container; this feature is known as containerization.

[0056] A private cloud 106 is similar to a public cloud 105, except that its computing resources are available only for use by a single enterprise. While the private cloud 106 is shown in communication with the WAN 102, in other embodiments, the private cloud may be completely disconnected from the Internet and accessible only through a local / private network. A hybrid cloud is a composite of multiple clouds of different types (e.g., private, community, or public cloud types), often implemented by multiple different vendors. While each of the multiple clouds remains a separate, discrete entity, the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the constituent clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0057] Turning now to an overview of technologies more specifically related to embodiments of the present invention, LLMs are becoming increasingly common in AI applications. However, many issues remain regarding the proper operation and functionality of LLMs. These issues include the fact that training costs for LLMs can be prohibitively high, and model sizes can be prohibitively large (i.e., several GB in size), resulting in prohibitively long load, reload, and startup times for LLMs in serverless function architectures. These issues further include challenges with routing. While LLM inference services have prompts for multiple different domain contexts, such as medical, financial, and technical, which may improve the effectiveness of routing, such that a given query is routed to an appropriate serverless function with an appropriate model, it has been shown that routing end-user prompts to appropriate LLM serverless functions differs from traditional L3 / L5 / L7 routing.

[0058] Turning now to an overview of aspects of the present invention, one or more embodiments of the present invention address the above-mentioned shortcomings of the prior art by providing a model-of-model (MoM) system in which queries are routed to "expert" models. LLM serverless functions are built with a foundation model (FM) and a fine-tuned model, such as a model trained via low-rank adaptation (LoRA) or other training process. The fine-tuned model complements the FM but is extremely lightweight to load and invoke in a serverless function. Each serverless function service endpoint is annotated with dataset or fine-tuned model information. Prompt-aware routing is used instead of L3 / L5 / L7 routing to route prompts to the appropriate serverless function with the best FM and fine-tuned model.

[0059] Among other things, a method of prompt-to-model routing is provided in which a router queries a vector in which a subset of a fine-tuning training dataset is located. The vector returns a similarity result indicating how close the query is to a particular dataset. The router uses the similarity result to route the prompt to an appropriate serverless function service endpoint. The computer-implemented method includes receiving a prompt, querying a database including multiple datasets for an indication of which one of the multiple datasets is most similar to the prompt, identifying one of the multiple endpoints as storing a set of expert models that have the highest percentage match with one of the multiple datasets, and routing the prompt to one of the multiple endpoints storing the set of expert models that have the highest percentage match with one of the multiple datasets.

[0060] The above aspects of the present invention address the shortcomings of the prior art by providing a computer-implemented method for serving large language model (LLM) applications via a serverless function router capable of communicating with multiple endpoints, each of which stores a set of subject matter expert models.

[0061] Turning now to a more detailed description of aspects of the present invention, FIG. 2 illustrates a block diagram of components of a machine learning training and inference system 200. The machine learning training and inference system 200, according to one or more embodiments of the present invention, can perform tasks utilizing machine learning techniques, such as a computer-implemented method for serving large language model (LLM) applications via a serverless function router capable of communicating with multiple endpoints, each of which stores a set of subject matter expert models. Embodiments of the present invention utilize AI, including various so-called machine learning techniques. The phrase “machine learning” broadly refers to the ability of electronic systems to learn from data. A machine learning system, engine, or module can include trainable machine learning algorithms that can be trained, such as in an external cloud environment, to learn functional relationships between inputs and outputs, and the resulting model (sometimes referred to as a “trained neural network,” “trained model,” and / or “trained machine learning model”) can be used, for example, to manage information during a web conference. In one or more embodiments of the present invention, machine learning functionality can be implemented using an artificial neural network (ANN) capable of being trained to perform functions. In machine learning and cognitive science, ANNs are a family of statistical learning models inspired by biological neural networks, particularly the brains of animals. ANNs can be used to estimate or approximate systems and functions that depend on a large number of inputs. Convolutional neural networks (CNNs) are a class of deep feed-forward ANNs that are particularly useful for tasks such as, but not limited to, natural language processing (NLP) and visual image analysis. Recurrent neural networks (RNNs) are another class of deep feed-forward ANNs that are particularly useful for tasks such as, but not limited to, unsegmented connected handwriting recognition and speech recognition. Other types of neural networks are also known and may be used in accordance with one or more embodiments of the present invention described herein.

[0062] An ANN can be embodied as a so-called "neuromorphic" system of interconnected processor elements that act as simulated "neurons" and exchange "messages" with each other in the form of electronic signals. Similar to the so-called "plasticity" of synaptic neurotransmitter connections that transmit messages between biological neurons, connections in an ANN that transmit electronic messages between simulated neurons are provided with numerical weights that correspond to the strength or weakness of a given connection. These weights can be adjusted and refined based on experience, allowing the ANN to adapt to inputs and learn. For example, an ANN for handwritten character recognition is defined by a set of input neurons that can be activated by pixels in an input image. The activation of these input neurons is weighted and transformed by a function determined by the network designer, and then passed on to other downstream neurons (often referred to as "hidden" neurons). This process is repeated until an output neuron is activated. The activated output neuron determines which character has been input. It should be understood that these same techniques can be applied to locating a target object referenced by a synthetic representation from a set of images with similar visual elements, as described herein.

[0063] The machine learning training and inference system 200 performs training 202 and inference 204. During training 202, a training engine 216 trains a model (e.g., a trained model 218) to perform a task. Inference 204 is the process of implementing the trained model 218 to perform a task in the context of a larger system (e.g., a system 226).

[0064] Training 202 begins with training data 212, which may be structured or unstructured data. A training engine 216 receives the training data 212 and a model format 214. The model format 214 represents an untrained base model. The model format 214 may have preset weights and biases that may be adjusted during training. It should be understood that the model format 214 may be selected from many different model formats depending on the task to be performed. For example, if training 202 is to train a model to perform image classification, the model format 214 may be a convolutional neural network (CNN) model format. Training 202 may involve supervised learning, semi-supervised learning, unsupervised learning, reinforcement learning, and / or the like, including combinations and / or multiples thereof. For example, supervised learning may be used to train a machine learning model to classify objects of interest in images. To do this, the training data 212 includes labeled images, including images of objects of interest with associated labels (ground truth), and other images that do not include the objects of interest with associated labels. In this example, the training engine 216 takes training images from the training data 212 as input, makes predictions to classify the images, and compares the predictions with known labels. The training engine 216 then adjusts the weights and / or biases of the model based on the results of the comparison, such as by using backpropagation. The training 202 may be performed multiple times (referred to as "epochs") until a suitable model (e.g., the trained model 218) is trained.

[0065] Once trained, the trained model 218 can be used to perform inferences 204 to execute tasks. The inference engine 220 applies the trained model 218 to new data 222 (e.g., real, non-training data). For example, if the trained model 218 is trained to classify images of a particular object, such as a chair, the new data 222 can be images of chairs that were not part of the training data 212. In this manner, the new data 222 represents data that the model 218 has not seen. The inference engine 220 makes a prediction 224 (e.g., classification of the object in the image of the new data 222) and passes the prediction 224 to the system 226. The system 226 can take an action, perform a behavior, perform an analysis, and / or the like, including a combination and / or multiple thereof, based on the prediction 224. In some embodiments of the present invention, the system 226 can add and / or modify the new data 222 based on the prediction 224.

[0066] According to one or more embodiments of the present invention, the predictions 224 generated by the inference engine 220 are periodically monitored and validated to ensure that the inference engine 220 is operating as expected. Based on that validation, further training 202 can be performed using the trained model 218 as a starting point. The further training 202 can include all or a subset of the original training data 212 and / or new training data 212. According to one or more embodiments of the present invention, the training 202 includes updating the trained model 218 to account for changes in the expected input data.

[0067] Referring to FIG. 3 , a computer-implemented method 300 is provided for serving large language model (LLM) applications via a serverless function router capable of communicating with multiple endpoints, each of which stores a set of subject matter expert models.

[0068] 3, the computer-implemented method 300 includes initially building and training subject matter expert models to process prompts of a particular type from a user (block 301) and storing each subject matter expert model of the particular type at one of a plurality of endpoints (block 302). The computer-implemented method 300 also includes storing a plurality of datasets in a vector database (block 303). Each of the plurality of datasets stored in the vector database corresponds to and has the highest percentage match with each subject matter expert model of the particular type stored at one of the plurality of endpoints.

[0069] Thus, in an exemplary case, one of the multiple endpoints may have a subject matter expert model associated with a medical prompt (i.e., a foundation model and a fine-tuned model), and a corresponding one of the multiple datasets stored in the vector database may include data and information that corresponds to and has the highest percentage agreement with the subject matter expert model associated with the medical prompt. In another exemplary case, one of the multiple endpoints may have a subject matter expert model associated with a financial prompt (i.e., a foundation model and a fine-tuned model), and a corresponding one of the multiple datasets stored in the vector database may include data and information that corresponds to and has the highest percentage agreement with the subject matter expert model associated with the financial prompt. In yet another exemplary case, one of the multiple endpoints may have a subject matter expert model associated with a technical prompt (i.e., a foundation model and a fine-tuned model), and a corresponding one of the multiple datasets stored in the vector database may include data and information that corresponds to and has the highest percentage agreement with the subject matter expert model associated with the technical prompt.

[0070] Of course, it should be understood that other types of subject matter expert models may be provided, and that the types listed above are a non-exhaustive list.

[0071] The computer-implemented method 300 further includes receiving a prompt (block 304), querying a vector database including the plurality of datasets for an indication of which one of the plurality of datasets is most similar to the prompt (block 305), identifying one of the plurality of endpoints as storing a set of expert models that have the highest percentage match with one of the plurality of datasets (block 306), and routing the prompt to one of the plurality of endpoints storing the set of expert models that have the highest percentage match with one of the plurality of datasets (block 307). According to an embodiment, the receiving of the prompt in block 304, the querying of the database in block 305, the identifying one of the plurality of endpoints in block 306, and the routing of the prompt in block 307 are each performed by a serverless function router. According to a further embodiment, the serverless function router may include or be provided as a prompt-aware router, and the routing in block 307 may include prompt-aware routing (block 3071).

[0072] Referring to FIG. 4, an architecture 400 for executing the computer-implemented method 300 of FIG. 3 is provided. The architecture 400 includes a prompt-aware serverless function router 410, a vector database 420, and endpoints 430, 440, and 450. Each of the endpoints 430, 440, and 450 may include or be provided as a server. The endpoint 430 stores a foundation model and a fine-tuned model 431 previously built and trained to process a particular type of prompt (i.e., a medical prompt). The endpoint 440 stores a foundation model and a fine-tuned model 441 previously built and trained to process a particular type of prompt (i.e., a financial prompt). The endpoint 450 stores a foundation model and a fine-tuned model 451 previously built and trained to process a particular type of prompt (i.e., a technical prompt). Datasets 421, 422, and 423 are stored in the vector database 420. Each of datasets 421, 422, 423 includes data or information that corresponds to or has the highest agreement with the foundation model and fine-tuned model of one of endpoints 430, 440, 450. For example, dataset 421 may correspond to and have a high agreement with the foundation model and fine-tuned model 431 of endpoint 430, dataset 422 may correspond to and have a high agreement with the foundation model and fine-tuned model 441 of endpoint 440, and dataset 423 may correspond to and have a high agreement with the foundation model and fine-tuned model 451 of endpoint 450.

[0073] When a prompt is received by the prompt-aware serverless function router 410, the prompt-aware serverless function router 410 queries the vector database 420 for an indication as to which of the datasets 421, 422, 423 have foundation and fine-tuned models that most closely resemble the prompt. When the prompt-aware serverless function router 410 receives an indication from the vector database 420, it determines, based on the indication, to which of the endpoints 430, 440, 450 to send the prompt, and then routes the prompt to that endpoint.

[0074] It should be appreciated that because only the subject matter expert model is stored on each of the endpoints 430, 440, and 450, and this subject matter expert model is typically much smaller than the entire LLM, resource consumption at each of the endpoints 430, 440, and 450 is limited (less than 10 GB for the subject matter expert model compared to hundreds or tens of GB for the LLM). In this manner, subject matter expert models can be updated and retrained as needed in a relatively short amount of time and consuming relatively few computing resources.

[0075] With continued reference to Figure 4 and further reference to Figures 5-7, the operation of the prompt-aware serverless function router 410 of Figure 4 is based, at least in part, on the similarity of incoming prompts and the foundation and fine-tuned models of endpoints 430, 440, and 450. In the examples of Figures 5-7, Model A is specialized for medicine, Model B is specialized for programming, and Model C is specialized for finance, with the hypothetical prompts used for Models A, B, and C representing the nature of each domain, as shown in the table of Figure 5.

[0076] If the incoming prompt is "I'm not feeling well," the embedding may be [0.5, 0.3, 0.4, 0.4, 0.4]. Then, the Euclidean distance of the similarity distance may be calculated using the formula shown in FIG. 6. This distance is the distance between the incoming prompt and the three hypothetical prompts, and is shown in the table in FIG. 7. From FIG. 7, it can be seen that the incoming prompt is closer to "Is fever a symptom of the flu?", which represents model A. Therefore, in this example, the prompt-aware serverless function router 10 will select a server of model A and route this prompt there.

[0077] Various embodiments of the present invention are described herein with reference to the associated drawings. Alternative embodiments may be devised without departing from the scope of the present invention. While the following description and drawings describe various connections and relationships between elements (e.g., above, below, adjacent, etc.), those skilled in the art will recognize that many of the relationships described herein are independent of orientation, provided that the described functionality is maintained even when the orientation is changed. These connections and / or relationships may be direct or indirect, unless otherwise specified, and the present invention is not intended to be limited in this respect. Thus, a connection between entities may refer to a direct or indirect connection, and a relationship between entities may be a direct or indirect relationship. As an example of an indirect relationship, reference in this description to forming layer "A" on layer "B" includes a situation in which one or more intermediate layers (e.g., layer "C") are between layers "A" and "B," as long as the relative properties and functionality of layers "A" and "B" are not substantially altered by the intermediate layer or layers.

[0078] The following definitions and abbreviations should be used in interpreting the claims and the specification. As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," "contains," or "containing," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a complex, mixture, process, method, article, or device that includes a list of elements is not necessarily limited to only those elements but can include other elements not expressly listed or that are inherent to such complex, mixture, process, method, article, or device.

[0079] Additionally, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. The terms "at least one" and "one or more" are understood to include any integer greater than or equal to one, i.e., one, two, three, four, etc. The term "a plurality" is understood to include any integer greater than or equal to two, i.e., two, three, four, five, etc. The term "connection" can include indirect and direct "connections."

[0080] References herein to "one embodiment," "an embodiment," "an example embodiment," or the like indicate that the described embodiment may include a particular feature, structure, or characteristic, but all embodiments may or may not include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with one embodiment, it is believed to be within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments, whether or not explicitly described.

[0081] For purposes of the following description, the terms “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” and their derivatives refer to the structures and methods being described as oriented in the drawings. The terms “overlying,” “atop,” “on top,” “positioned on,” or “positioned atop” mean that a first element, such as a first structure, is above a second element, such as a second structure, where there may be an intervening element, such as an interfacial structure, between the first and second elements. The term “direct contact” means that a first element, such as a first structure, and a second element, such as a second structure, are connected without an intermediate conductive, insulating, or semiconducting layer at the interface between the two elements.

[0082] Spatially relative terms such as, for example, "beneath," "below," "lower," "above," and "upper" may be used herein for ease of description to describe the relationship of one element or feature to another, as shown in the figures. It will be understood that the spatially relative terms are intended to encompass multiple different orientations of the device during use or operation in addition to the orientation shown in the figures. For example, if a device in the figures were turned over, elements described as "below" or "below" other elements or features would be oriented "above" the other elements or features. Thus, the term "below" can encompass both an upper and lower orientation. A device may also be oriented differently (rotated 90 degrees or to other orientations), and the spatially relative descriptors used herein would be interpreted accordingly.

[0083] For example, the phrase "selective to," such as "a first element selective to a second element," means that the first element can be etched and the second element can act as an etch stop.

[0084] The terms "about," "substantially," "approximately," and variations thereof are intended to include the degree of error associated with measurement of the particular quantity based on the equipment available at the time of filing. For example, "about" can include a range of ±8%, or 5%, or 2% of a given value.

[0085] The term "conformal" (e.g., conformal layer) means that the thickness of the layer is substantially the same on all surfaces or that the thickness varies by less than 15% of the nominal thickness of the layer.

[0086] The terms "epitaxial growth and / or deposition" and "epitaxially formed and / or grown" refer to the growth of one semiconductor material (crystalline substance) on the deposition surface of another semiconductor material (crystalline substance), where the growing semiconductor material (crystalline overlayer) has substantially the same crystallinity as the semiconductor material of the deposition surface (seed material). In an epitaxial deposition process, chemical reactants provided by source gases can be controlled, and system parameters can be set so that the depositing atoms arrive at the deposition surface of a semiconductor substrate with sufficient energy to travel across the surface, resulting in the depositing atoms orienting themselves to the crystalline arrangement of the atoms of the deposition surface. An epitaxially grown semiconductor material can have substantially the same crystallinity as the deposition surface on which it is formed. For example, epitaxially grown semiconductor material deposited on a {100}-oriented crystalline surface can have a {100} orientation. In some embodiments of the present invention, the epitaxial growth and / or deposition process may be selective to formation on semiconductor surfaces and not allow material to be deposited on exposed surfaces, such as silicon dioxide or silicon nitride surfaces.

[0087] As noted herein above, for the sake of brevity, this specification may or may not describe in detail conventional techniques associated with the manufacture of semiconductor devices and integrated circuits (ICs). However, by way of background, a more general description of semiconductor device manufacturing processes that may be utilized in implementing one or more embodiments of the present invention is now provided. While the specific manufacturing operations used in implementing one or more embodiments of the present invention may be individually known, the combination of operations described and / or resulting structures of the present invention are unique. Thus, the unique combination of operations described in connection with the manufacture of semiconductor devices in accordance with the present invention utilizes a variety of individually known physical and chemical processes performed on a semiconductor (e.g., silicon) substrate, some of which are described in the immediately following paragraphs.

[0088] Generally, the various processes used to form microchips that will be packaged into ICs fall into four general categories: film deposition, removal / etching, semiconductor doping, and patterning / lithography. Deposition is any process that grows, coats, or otherwise transfers material onto a wafer. Available techniques include physical vapor deposition (PVD), chemical vapor deposition (CVD), electrochemical deposition (ECD), molecular beam epitaxy (MBE), and more recently, atomic layer deposition (ALD), among others. Removal / etching is any process that removes material from the wafer. Examples include etching processes (wet or dry) and chemical-mechanical planarization (CMP), and the like. Semiconductor doping is the modification of electrical properties by doping the source and drain of transistors, for example, typically by diffusion and / or ion implantation. These doping steps are followed by furnace annealing or rapid thermal annealing (RTA). The annealing serves to activate the implanted dopants. Films of both conductors (e.g., polysilicon, aluminum, copper, etc.) and insulators (e.g., various forms of silicon dioxide, silicon nitride, etc.) are used to connect and separate transistors and their components. Selective doping of various regions of a semiconductor substrate allows the conductivity of the substrate to be changed by applying a voltage. By creating structures of these various components, millions of transistors can be constructed and wired together to form the complex circuits of modern microelectronic devices. Semiconductor lithography is the formation of a three-dimensional relief image or pattern on a semiconductor substrate and the subsequent transfer of the pattern into the substrate. In semiconductor lithography, the pattern is formed with a photosensitive polymer called a photoresist. The lithography and etching pattern transfer steps are repeated multiple times to build the intricate structures that make up a transistor and the many wires that connect the millions of transistors in a circuit.Each pattern printed on the wafer is aligned to a previously formed pattern, and conductors, insulators, and selectively doped regions are slowly built up to form the final device.

[0089] The flowcharts and block diagrams in the figures illustrate possible implementations of methods of manufacture and / or operation according to various embodiments of the present invention. In the flow diagrams, various functions / acts of the method are represented by blocks. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved.

[0090] The description of various embodiments of the present invention is presented for purposes of illustration and is not intended to be comprehensive or limited to the described embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best explain the principles of the embodiments, practical applications of or technical improvements to technologies found in the market, or to enable others skilled in the art to understand the embodiments described herein.

Claims

1. 1. A computer-implemented method for serving large language model (LLM) applications via a serverless function router capable of communicating with a plurality of endpoints, each endpoint storing a set of subject matter expert models, the method comprising: receiving a prompt; querying a database including the plurality of data sets for an indication as to which one of the plurality of data sets most closely resembles the prompt; identifying one of the plurality of endpoints as having stored thereon the set of subject matter expert models that have the highest percent match with the one of the plurality of datasets; and routing the prompt to the one of the plurality of endpoints on which the set of subject matter expert models having the highest match with the one of the plurality of data sets is stored.

1. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the receiving of the prompt, the querying of the database, the identifying of the one of the plurality of endpoints, and the routing of the prompt are performed by the serverless function router.

3. The computer-implemented method of claim 1 , wherein the database comprises a vector database.

4. The computer-implemented method of claim 1 , wherein each of the plurality of datasets has a highest percent agreement with the subject matter expert model stored on one of the plurality of endpoints.

5. the subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to a medical subject; the subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to a financial subject; and the subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to a technical subject matter. The computer-implemented method of claim 4.

6. The computer-implemented method of claim 4 , wherein the subject matter expert models include one or more foundation models and one or more fine-tuned models.

7. The computer-implemented method of claim 1 , wherein the serverless function router is a prompt-aware router and the routing comprises prompt-aware routing.

8. 1. A computer program for serving large language model (LLM) applications via a serverless function router capable of communicating with a plurality of endpoints, each endpoint storing a set of subject matter expert models, the computer program comprising computer readable program code that, when executed by a processor of a computer system, causes the computer system to: receiving a prompt; querying a database including the plurality of data sets for an indication as to which one of the plurality of data sets most closely resembles the prompt; identifying one of the plurality of endpoints as having stored thereon the set of subject matter expert models that have the highest percent match with the one of the plurality of datasets; and routing the prompt to the one of the plurality of endpoints on which the set of subject matter expert models having the highest match with the one of the plurality of data sets is stored. performing a method including Computer program.

9. 9. The computer program product of claim 8, wherein the receiving of the prompt, the querying of the database, the identifying of the one of the plurality of endpoints, and the routing of the prompt are performed by the serverless function router.

10. The computer program product of claim 8 , wherein the database comprises a vector database.

11. 11. The computer program of claim 8, wherein each of the plurality of datasets has a highest percent agreement with the subject matter expert model stored on one of the plurality of endpoints.

12. the subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to a medical subject; the subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to a financial subject; and the subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to a technical subject matter.

12. A computer program according to claim 11.

13. The computer program product of claim 11 , wherein the subject matter expert models include one or more foundation models and one or more fine-tuned models.

14. 11. The computer program product of claim 8, wherein the serverless function router is a prompt-aware router and the routing comprises prompt-aware routing.

15. 1. A computing system comprising: processor; a memory coupled to the processor; and one or more computer-readable storage media coupled to the processor, the one or more computer-readable storage media collectively comprising instructions that are executed by the processor via the memory to implement a method for servicing large language model (LLM) applications via a serverless function router capable of communicating with a plurality of endpoints, each endpoint having a set of subject matter expert models stored thereon, the method comprising: receiving a prompt; querying a database including the plurality of data sets for an indication as to which one of the plurality of data sets most closely resembles the prompt; identifying one of the plurality of endpoints as having stored thereon the set of subject matter expert models that have the highest percent match with the one of the plurality of datasets; and routing the prompt to the one of the plurality of endpoints on which the set of subject matter expert models having the highest match with the one of the plurality of data sets is stored. Including, A computing system comprising:

16. 16. The computing system of claim 15, wherein the receiving of the prompt, the querying of the database, the identifying of the one of the plurality of endpoints, and the routing of the prompt are performed by the serverless function router.

17. the database includes a vector database; each of the plurality of datasets has a highest percent match with the subject matter expert model stored on one of the plurality of endpoints; 17. A computing system according to claim 15 or 16.

18. the subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to a medical subject; the subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to a financial subject; and the subject matter expert model of a first one of the plurality of endpoints is configured to process prompts related to a technical subject matter.

20. The computing system of claim 17.

19. The computing system of claim 17 , wherein the subject matter expert models include one or more foundation models and one or more fine-tuned models.

20. 17. The computing system of claim 15 or 16, wherein the serverless function router is a prompt-aware router and the routing comprises prompt-aware routing.

Images