Persistent cache for on-device large language model

Abstract

A method includes providing a prompt to a large language model. The prompt includes information for a plurality of different functions that are related to one or more applications installed on a device. The method also includes generating, using the large language model, key and value (KV) pairs for each function of the plurality of different functions in response to receiving the prompt. The method includes storing the KV pairs at a persistent KV cache that is accessible to the large language model. The method includes providing a user query for performing a function to the large language model. The method includes retrieving the KV pairs stored at the persistent KV cache in response to receiving the user query. The method includes generating, using the large language model, a large language model output based on the KV pairs.

Description

[0001] Attorney Docket No. 24-0970-WO

[0002] PERSISTENT CACHE FOR ON-DEVICE LARGE LANGUAGE MODEL

[0003] BACKGROUND

[0004] [1] Devices may include one or more large language models to process user queries. To illustrate, a device may include a large action model that enables the device to autonomously perform tasks, make decisions, and / or provide personalized experiences. However, each time the large action model is invoked, the large action model has to process a prompt that includes functions dynamically made available to the large action model (e.g., functions installed on the device). Repeatedly processing the same prompt each time the large action model is invoked may be an energy and time intensive process.

[0005] SUMMARY

[0006] [2] A device may include a large language model (e.g., a large action model) that is configured to process user queries. Processing the user queries may result in generating executable commands to perform functions associated with the user queries. When the device is connected to a power source, a prompt may be provided to the large language model. The prompt includes functions that are dynamically available to the large language model. In particular, the prompt includes functions for each application (or a subset of applications) installed on the device. The large language model may process the prompt to generate key and value (KV) pairs that are indicative of the underlying function. After the KV pairs are generated, the KV pairs may be cached at a persistent KV cache that is accessible to the large language model.

[0007] [3] After the KV values are cached, the large language model may process user queries and leverage functionality associated with applications installed on a user device without having to regenerate the KV values. For example, in response to receiving a user query, the stored KV values may be retrieved, as opposed to regenerated, and the large language model may process the user query based on the stored KV values. Using cached KV values, as opposed to regenerating the KV values, may reduce processing time for processing the user query.

[0008] [4] In a first example embodiment, a method includes providing a prompt to a large language model. The prompt includes information for a plurality of different functions that are related to one or more applications installed on a device. The method also includes generating, Attorney Docket No. 24-0970-WO using the large language model, key and value (KV) pairs for each function of the plurality of different functions in response to receiving the prompt. The method also includes storing the KV pairs at a persistent KV cache that is accessible to the large language model. The method also includes providing a user query for performing a function to the large language model. The method also includes retrieving the KV pairs stored at the persistent KV cache in response to receiving the user query. The method also includes generating, using the large language model, a large language model output based on the KV pairs stored at the persistent KV cache. The large language model output is usable to generate a command executable by the processor to cause an application of the one or more applications to perform the function.

[0009] [5] In a second example embodiment, a device includes a memory and a processor coupled to the memory. The processor is configured to provide a prompt to a large language model. The prompt includes information for a plurality of different functions that are related to one or more applications installed on the device. The processor is also configured to generate, using the large language model, key and value (KV) pairs for each function of the plurality of different functions in response to receiving the prompt. The processor is also configured to store the KV pairs at a persistent KV cache that is accessible to the large language mode. The processor is also configured to provide a user query for performing a function to the large language model. The processor is also configured to retrieve the KV pairs stored at the persistent KV cache in response to receiving the user query. The processor is also configured to generate, using the large language model, a large language model output based on the KV pairs stored at the persistent KV cache. The large language model output is usable to generate a command executable by the processor to cause an application of the one or more applications to perform the function.

[0010] [6] In a third example embodiment, a non-transitory computer-readable medium includes instructions that, when executed by a processor of a device, cause the processor to perform operations. The operations include providing a prompt to a large language model. The prompt includes information for a plurality of different functions that are related to one or more applications installed on the device. The operations also include generating, using the large language model, key and value (KV) pairs for each function of the plurality of different functions in response to receiving the prompt. The operations also include storing the KV pairs at a persistent KV cache that is accessible to the large language model. The operations also include providing a user query for performing a function to the large language model. The Attorney Docket No. 24-0970-WO operations also include retrieving the KV pairs stored at the persistent KV cache in response to receiving the user query. The operations also include generating, using the large language model, a large language model output based on the KV pairs stored at the persistent KV cache. The large language model output is usable to generate a command executable by the processor to cause an application of the one or more applications to perform the function.

[0011] [7] In a fourth example embodiment, a computer program product includes computer-executable program code. The computer-executable program code, when executed by a computer, causes the computer to provide a prompt to a large language model. The prompt includes information for a plurality of different functions that are related to one or more applications installed on the device. The computer-executable program code, when executed by the computer, causes the computer to generate, using the large language model, key and value (KV) pairs for each function of the plurality of different functions in response to receiving the prompt. The computer-executable program code, when executed by the computer, causes the computer to store the KV pairs at a persistent KV cache that is accessible to the large language mode. The computer-executable program code, when executed by the computer, causes the computer to provide a user query for performing a function to the large language model. The computer-executable program code, when executed by the computer, causes the computer to retrieve the KV pairs stored at the persistent KV cache in response to receiving the user query. The computer-executable program code, when executed by the computer, causes the computer to generate, using the large language model, a large language model output based on the KV pairs stored at the persistent KV cache. The large language model output is usable to generate a command executable by the computer to cause an application of the one or more applications to perform the function.

[0012] [8] In a fifth example embodiment, a system may include various means for carrying out each of the operations of the first example embodiment.

[0013] [9] These, as well as other embodiments, aspects, advantages, and alternatives, will become apparent to those of ordinary skill in the art by reading the following detailed description, with reference where appropriate to the accompanying drawings. Further, this summary and other descriptions and figures provided herein are intended to illustrate embodiments by way of example only and, as such, that numerous variations are possible. For instance, structural elements and process steps can be rearranged, combined, distributed, Attorney Docket No. 24-0970-WO eliminated, or otherwise changed, while remaining within the scope of the embodiments as claimed.

[0014] Attorney Docket No. 24-0970-WO

[0015] BRIEF DESCRIPTION OF THE DRAWINGS

[0016]

[0010] Figure 1 illustrates a device, in accordance with examples described herein.

[0017]

[0011] Figure 2 illustrates the device, in accordance with examples described herein.

[0018]

[0012] Figure 3 illustrates the device, in accordance with examples described herein.

[0019]

[0013] Figure 4 illustrates multiple examples of caching function sets, in accordance with examples described herein.

[0020]

[0014] Figure 5 is a diagram of a process, in accordance with examples described herein.

[0021]

[0015] Figure 6 illustrates a flow chart, in accordance with examples described herein.

[0022] Attorney Docket No. 24-0970-WO

[0023] DETAILED DESCRIPTION

[0024]

[0016] Example methods, devices, and systems are described herein. It should be understood that the words “example” and “exemplary” are used herein to mean “serving as an example, instance, or illustration.” Any embodiment or feature described herein as being an “example,” “exemplary,” and / or “illustrative” is not necessarily to be construed as preferred or advantageous over other embodiments or features unless stated as such. Thus, other embodiments can be utilized and other changes can be made without departing from the scope of the subject matter presented herein.

[0025]

[0017] Accordingly, the example embodiments described herein are not meant to be limiting. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0026]

[0018] Further, unless context suggests otherwise, the features illustrated in each of the figures may be used in combination with one another. Thus, the figures should be generally viewed as component aspects of one or more overall embodiments, with the understanding that not all illustrated features are necessary for each embodiment.

[0027]

[0019] Particular embodiments are described herein with reference to the drawings. In the description, common features are designated by common reference numbers throughout the drawings. In some figures, multiple instances of a particular type of feature are used. Although these features are physically and / or logically distinct, the same reference number is used for each, and the different instances are distinguished by addition of a letter to the reference number. When the features as a group or a type are referred to herein (e.g., when no particular one of the features is being referenced), the reference number is used without a distinguishing letter. However, when one particular feature of multiple features of the same type is referred to herein, the reference number is used with the distinguishing letter. For example, referring to Figure 1, key and value (KV) pairs are illustrated and associated with reference numbers 150A, 150B, and 150C. When referring to a particular KV pair, such as the KV pair 150A, the distinguishing letter “A” is used. However, when referring to any arbitrary KV pair or to the KV pairs as a group, the reference number 150 is used without a distinguishing letter.

[0028]

[0020] Additionally, any enumeration of elements, blocks, or steps in this specification or the claims is for purposes of clarity. Thus, such enumeration should not be interpreted to Attorney Docket No. 24-0970-WO require or imply that these elements, blocks, or steps adhere to a particular arrangement or are carried out in a particular order. Unless otherwise noted, figures are not drawn to scale.

[0029] I. Overview

[0030]

[0021] The techniques described herein improve performance of a large language model, such as a large action model, running on a device by introducing a persistent KV cache mechanism. In particular, the techniques described herein reduce processing time and energy consumption associated with a large language model running on a performance-constrained device and / or an energy-constrained device, such as a smartphone, a tablet, etc.

[0031]

[0022] A large action model is a type of large language model that enables a device to autonomously perform tasks, make decisions, and / or provide personalized experiences. The large action model may be used to generate commands that, when executed by a processor of the device, enable the device to interact with external services. Non-limiting examples of the commands may include (i) a command to remote start a car using a vehicle application installed on the device, (ii) a command to set an alarm on an external clock using a clock application installed on the device, (iii) a command to control a television using a television application installed on the device, etc.

[0032]

[0023] To generate the commands, the processor prompts (e.g., provides an input prompt to) the large action model. For example, the processor may be configured to (i) enumerate a plurality of functions made available by installed applications on the device and (ii) construct an appropriate prompt (e.g., the input prompt) for the large action model that covers each function of the plurality of functions. In particular, the input prompt includes text identifying the functions that are dynamically made available to the large action model, such as functions from the different applications installed on the device.

[0033]

[0024] The large action model processes the input prompt and generates KV pairs based on the text. For example, the input prompt may be run through the large action model to generate the KV pairs for each head / layer combination in a transformer-based structure of the large action model. The KV pairs correspond to internal representations of corresponding tokens, which in turn, are used to generate a large language model output, such as an executable-command to perform a particular function. However, each time the large action model is invoked to generate a command, the input prompt may need to be provided to, and processed by, the large action model to regenerate the KV pairs. This may result in repeated Attorney Docket No. 24-0970-WO processing of the same input prompt, which may consume a substantial amount of time and energy.

[0034]

[0025] As a non-limiting example, if one-hundred fifty (150) functions are available to the large action model from applications installed on the device and each function utilizes forty (40) tokens, the large action model may generate KV pairs for six thousand (6000) tokens each time the input prompt is provided to the large action model. Furthermore, in some scenarios, extra tokens (e.g., approximately five-hundred (500) tokens) may be added for a description of the function format, totaling approximately sixty-five hundred (6500) tokens. Thus, each time the large action model is invoked to generate a command, the large action model may process the input prompt to regenerate the KV pairs for sixty -five hundred (6500) tokens, which may result in excessive energy consumption and unnecessary processing time.

[0035]

[0026] To reduce the processing time and energy associated with processing the same input prompt each time the large action model is invoked to generate a command, the first time the input prompt is processed by the large action model, resulting KV pairs generated by the large action model may be stored at a persistent KV cache that is accessible to the large action model. As a result of storing the KV pairs at the persistent KV cache, when the large action model is invoked to generate a command in the future, the KV pairs may be retrieved from the persistent KV cache to bypass the time-consuming and energy-intensive process of regenerating the KV pairs (e.g., processing the input prompt to regenerate the same tokens).

[0036]

[0027] For example, instead of the large action model regenerating KV pairs for sixty- five hundred (6500) tokens each time the large action model is invoked to generate a command, relevant KV pairs are simply accessed by the large language model from the persistent KV cache. If the rate at which the tokens are stored in memory is approximately six hundred (600) tokens per second, storing the KV pairs in the persistent KV cache may save over ten (10) seconds of latency and battery drain if the persistent KV cache is already in random access memory (RAM). If the persistent KV cache is not in RAM, reading the KV pairs from other storage may be performed in under half a second.

[0037]

[0028] The persistent KV cache may be configured (e.g., optimized) to facilitate quick access for retrieving the KV pairs and may have a small storage footprint. In some scenarios, compression algorithms may be used to reduce the size of the persistent KV cache. In some scenarios, cache eviction policies may be based on usage patterns and storage constraints. For example, KV pairs indicative of functions that are frequently used may be more readily Attorney Docket No. 24-0970-WO accessible in the persistent cache than KV pairs indicative of rarely used functions. In some scenarios, a background service may monitor (i) application installation at the device or (ii) model updates to update the persistent KV cache, accordingly.

[0038]

[0029] The above-described techniques provide time and energy savings because, when generating an output, the large action model (e.g., a transformer-based large language model) may reference previous tokens (e.g., the input prompt context) via internal representations, which are structured in KV pairs for each head within each layer of the attention mechanism. The structure of the large action model enables the large action model to dynamically weight the importance of different parts of the input prompt, as the input prompt includes context for generating each token in the output. Once KV pairs are generated for a specific token, assuming that all previous tokens are unchanged, the KV pairs remain static. In the context of transformer KV caching, when the KV pairs are generated and stored, processing of the resulting tokens is no longer required if the large action model is invoked. Thus, the techniques described herein enables storage of the contents of the transformer KV cache for a large action model prompt that includes a listing of the available on-device functions. As a result, the repeated KV generation against the tokens may be circumvented, as the listing of the available on-device functions does not vary from request to request.

[0039]

[0030] It should be appreciated that by caching the KV pairs generated in response to processing the input prompt, the large action model may bypass the time-consuming KV pair regeneration process in the future, resulting in faster response times for users. As indicated above, the latency may be improved by at least ten (10) seconds compared to regenerating the KV pairs each time the large action model is invoked. Additionally, by eliminating redundant computations by regenerating the KV pairs each time the large action model is invoked, energy consumption may be reduced, which in turn, extends battery life. Thus, by caching the KV pairs, the techniques described herein may result in faster and more energy-efficient large language model performance, which translates into a smoother and more enjoyable user experience.

[0040]

[0031] In some embodiments, to further conserve energy (e.g., battery life), the input prompt can be processed by (e.g., run through) the large action model when the device is connected to a power source. Thus, pre-generating the KV pairs while the device is connected to the power source and charging enables KV pairs to be generated and stored when energy availability is less of a limiting factor. Similarly, prior to updating the persistent KV cache Attorney Docket No. 24-0970-WO with updated KV pairs based on an updated input prompt, the processor may ensure that the device is connected to the power source to conserve energy. Thus, if there are updates to the available functions since the persistent KV cache was last generated (e.g., if a user installed, removed, or upgraded applications), generation of the updated KV pairs may be performed when the device is connected to the power source.

[0041]

[0032] In some embodiments, the KV pairs that are generated and cached may be based on user-specific data. For example, the KV pairs that are generated and cached may be based on a user history indicative of applications that a user commonly accesses (e.g., uses). By generating and caching KV pairs based on the user history, additional efficiency may be achieved by bypassing generation of KV pairs for applications that are rarely used.

[0042]

[0033] In some embodiments, additional efficiency may be achieved where multiple cached function sets (e.g., function sets for different applications) exist. To illustrate, the prompt provided to the large action model may include a first list of functions for a first application installed on the device and a second list of functions for a second application installed on the device. The first list of functions and the second list of functions may include generic information that indicates how each function in the first and second list of functions are declared. In this scenario, because the generic information that indicates how each function is declared is the same for the first and second list of functions, the large action model can generate KV pairs for the generic information and store the KV pairs in the persistent KV cache as a prefix to both the first and second list of functions. As a result, storage overhead may be reduced.

[0043]

[0034] With respect to embodiments that include interactions by the device with cloudbased servers or otherwise involve sharing data with other devices, a user may be provided with controls allowing the user to make an election as to both if and when systems, programs, or features described herein may enable collection of user information (e.g., information about a user’s social network, social actions, or activities, profession, a user’s preferences, or a user’s current location), and if the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user’s identity may be treated so that no personally identifiable information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may Attorney Docket No. 24-0970-WO have control over what information is collected about the user, how that information is used, and what information is provided to the user.

[0044] II. Example Device

[0045]

[0035] Figure 1 illustrates a diagram of a device 100, in accordance with examples described herein. In some implementations, the device 100 may be a mobile device, such as a mobile phone, a tablet, a laptop computer, etc. As described below, with regards to processing a user query with a large language model, the device 100 may be configured to increase processing speed and reduce energy consumption.

[0046]

[0036] The device 100 includes a processor 102, a memory 104 coupled to the processor 102, and a user interface 106 coupled to the processor 102. The memory 104 can be a non-transitory computer-readable medium that stores instructions 105 that are executable by the processor 102 to perform the operations described herein. Specifically, the instructions 105 can be executable to cause processor 102 to store KV pairs 150, generated by a large language model 112 (e.g., a large action model), at persistent KV cache 114 accessible to the large language model to increase the processing speed and reduce energy consumption associated with processing a user query 160.

[0047]

[0037] Executable instructions to run one or more applications 108 may also be stored at the memory 104. To illustrate, in Figure 1, the memory 104 stores executable instructions to run an application 108 A, an application 108B, and an application 108C. As used herein, the executable instructions to run any particular application 108 are referred to by the application 108. Although three applications 108 are stored in the memory 104, in other embodiments, additional (or fewer) applications 108 may be stored in the memory 104. As a non-limiting example, in some embodiments, the memory 104 may store fifty applications 108. As another non-limiting example, in some embodiments, the memory 104 may store a single application 108.

[0048]

[0038] The applications 108 stored at the memory 104 may correspond to external applications that are responsive to commands generated by the device 100 (e.g., external applications controllable by the device 100). As non-limiting examples, the application 108 A may be a vehicle application installed on the device 100 that enables the device 100 to send remote commands to a vehicle, the application 108B may be a clock application installed on the device 100 that enables the device 100 to send remote commands to an external clock, and the application 108C may be a television application installed on the device 100 that enables Attorney Docket No. 24-0970-WO the device 100 to send remote commands to a television. It should be understood the above examples of the applications 108 are merely for illustrative purposes and should not be construed as limiting. In other embodiments, one or more of the applications 108 may correspond to different applications.

[0049]

[0039] The user interface 106 may be configured to receive a user query 160 from a user of the device 100. In some embodiments, the user interface 106 may be an audio interface (e.g., a microphone) such that an audio query from the user can be detected. In these embodiments, the user query 160 may be an audio command to perform a function 170. Nonlimiting examples of the function 170 may include starting a vehicle, setting an alarm on a clock, turning the volume down on a television, etc. In other embodiments, the user interface 106 may be a visual interface (e.g., a camera) such that a visual query from the user can be detected. In yet other embodiments, the user interface 106 may receive textual queries from the user of the device 100. As described below, the processor 102 may use the large language model 112 to quickly process the user query 160 using previously generated KV pairs 150 stored at the persistent KV cache 114.

[0050]

[0040] The processor 102 may be configured to run an operating system 110. The operating system 110 may be implemented by executing the instructions 105 stored at the memory 104. Additionally, the processor 102 includes the large language model 112 and the persistent KV cache 114. Although the large language model 112 is external to the operating system 110 in Figure 1, in some embodiments, the large language model 112 may be integrated into the operating system 110 such that the operating system 110 manages operation of the large language model 112. Similarly, although the persistent KV cache 114 is external to the operating system 110 in Figure 1, in some embodiments, the persistent KV cache 114 may be managed by the operating system 110 to provide the large language model 112 access to the persistent KV cache 114.

[0051]

[0041] As described herein, the large language model 112 may employ a machine learning inference process to make predictions and / or output results. For example, the large language model 112 may be trained using a training dataset and a deep learning framework. Based on a pre-trained machine learning algorithm stemming from the training dataset and the deep learning framework, the large language model 112 may make predictions and / or output results. In some embodiments, techniques such as retrieval-augmented generation (RAG) may be utilized to enhance the accuracy and reliability of the large language model 112. In some Attorney Docket No. 24-0970-WO embodiments, techniques such as low-rank adaptation (LoRA) may be used to reduce the number of trainable parameters.

[0052]

[0042] According to some implementations, one or more components of the processor 102 can be implemented using dedicated circuitry. As non-limiting examples, one or more components of the processor 102 can be implemented using application-specific integrated circuits (ASICs) or field-programmable gate array (FPGA) devices. According to some implementations, one or more components of the processor 102 can be implemented using software. As a non-limiting example, the processor 102 can execute the instructions 105 stored in the memory 104 to perform the operations of one or more components of the processor 102.

[0053]

[0043] The operating system 110 includes a power source monitor 120, a prompt generation unit 122, and a function execution unit 124. The power source monitor 120 may be configured to determine whether the device 100 is connected to a power source. For example, the power source monitor 120 may determine whether the device 100 is connected to an alternating current (AC) power source or to a non-battery direct current (DC) power source.

[0054]

[0044] The prompt generation unit 122 may be configured to provide a prompt 130 to the large language model 112 in response to a determination, by the power source monitor 120, that the device 100 is connected to a power source. The prompt 130 may include application information 132 for each application 108 installed on the device 100. For example, the prompt 130 may include application information 132A for the application 108A, application information 132B for the application 108B, and application information 132C for the application 108C. The application information 132A may include a set of functions 140A related to the application 108 A, the application information 132B may include a set of functions 140B related to the application 108B, and the application information 132C may include a set of functions 140C related to the application 108C.

[0055]

[0045] To illustrate, the set of functions 140A for the application 108A may indicate an example function for remote starting a vehicle, an example function for adjusting the temperature in the vehicle, an example function for flashing the headlights of the vehicle, etc. The set of functions 140B for the application 108B may indicate an example function for setting an alarm on a clock, an example function for setting a timer, etc. The set of functions 140C for the application 108C may indicate an example function for turning on a television, an example function for changing a channel, an example function for recording a program, an example function for adjusting the television volume, etc. It should be understood that the Attorney Docket No. 24-0970-WO above functions 140 are merely for illustrative purposes and should not be construed as limiting. In other embodiments, the functions 140 in the prompt 130 may differ.

[0056]

[0046] The large language model 112 may be configured to generate KV pairs for each function 140 in response to receiving the prompt 130. To illustrate, the large language model 112 may generate one or more KV pairs 150A for the set of functions 140A associated with the application 108A, one or more KV pairs 150B for the set of functions 140B associated with the application 108B, and one or more KV pairs 150C for the set of functions 140C associated with the application 108C. After the large language model 112 generates the KV pairs 150, the processor 102 may store the KV pairs 150 at the persistent KV cache 114.

[0057]

[0047] As described below, the persistent KV cache 114 is accessible to the large language model 112, which enables retrieval of the stored KV pairs 150 when the large language model 112 is prompted with a user query 160. As a result, the large language model 112 may bypass the time-consuming KV pair regeneration process in the future, resulting in faster response times for users. For example, by eliminating redundant computations by regenerating the KV pairs 150 each time the large language model 112 is invoked, energy consumption may be reduced, which in turn, extends battery life. Thus, by caching the KV pairs 150, the techniques described herein may result in faster and more energy-efficient large language model performance, which translates into a smoother and more enjoyable user experience.

[0058]

[0048] To illustrate, a user of the device 100 may use the user interface 106 to provide the user query 160. For illustrative purposes, let’s assume that the user audibly says “Turn on my car”. In this example, the function 170 to be performed by the device 100 includes sending a remote command to a vehicle to start the vehicle. To perform this function, the large language model 112 generates the command (e.g., a large language model output 180) based on the set of functions 140A provided in the prompt 130. To generate the large language model output 180, the large language model 180 may use the KV pairs 150A from the set of functions 140A related to the relevant application 108 A (e.g., the vehicle application). However, because the KV pairs 150A are stored in the persistent KV cache 114, in response to receiving the user query 160, the large language model 112 does not have to undergo the time and energy intensive process of receiving the prompt 130 and regenerating the KV pairs 150.

[0059]

[0049] For example, in response to receiving the user query 160, the large language model 112 may be configured to retrieve the KV pairs 150 stored at the persistent KV cache Attorney Docket No. 24-0970-WO

[0060] 114. Based on the relevant KV pairs 150A, the large language model 112 may be configured to generate the large language model output 180. In some scenarios, the large language model output 180 is a command executable by the processor 102 to perform the function 170 (e.g., an executable command to start the vehicle).

[0061]

[0050] The function execution unit 124 may be configured to execute the command (e.g., the large language model output 180) to facilitate performance of the function 170. Thus, the large language model 112 can generate the command associated with the user query 160 based on KV pairs 150 cached at the persistent KV cache 114, and the function execution unit 124 may execute the command such that the device 100 performs the desired function 170 indicated by the user query 160.

[0062]

[0051] The techniques described with respect to Figure 1 reduce latency associated with processing the user query 160. For example, by caching the KV pairs 150 generated in response to processing the input prompt 130, the large language model 112 may bypass the time-consuming KV pair regeneration process in the future, resulting in faster response times for users. In particular, the latency may be improved by at least ten (10) seconds compared to regenerating the KV pairs 150 each time the large language model 112 is invoked. Additionally, by eliminating redundant computations by regenerating the KV pairs 150 each time the large language model 112 is invoked, energy consumption may be reduced, which in turn, extends battery life. Thus, by caching the KV pairs 150 in the persistent KV cache 114, the techniques described with respect to Figure 1 may result in faster and more energy-efficient large language model performance, which translates into a smoother and more enjoyable user experience.

[0063]

[0052] Figure 2 illustrates another diagram of the device 100, in accordance with examples described herein. However, in Figure 2, an updated prompt 230 is provided to the large language model 112.

[0064]

[0053] As depicted in Figure 2, the application 108 A has been updated to an application 108AA, the application 108C has been deleted, and a new application 108D has been installed on the device 100. The updated prompt 230 may be generated in response to (i) a determination that a new application 108D is installed on the device 100, (ii) a determination that the application 108 A has been updated (to application 108AA), or (iii) a determination that the application 108C has been deleted. Attorney Docket No. 24-0970-WO

[0065]

[0054] To generate and provide the updated prompt 230 to the large language model 112, the power source monitor 120 may determine whether the device 100 is connected to the power source. The prompt generation unit 122 may be configured to provide the updated prompt 230 to the large language model 112 in response to a determination, by the power source monitor 120, that the device 100 is connected to a power source. The updated prompt 230 may include application information 132 for each application 108 installed on the device 100. For example, the updated prompt 230 may include application information 132AA for the updated application 108AA, the application information 132B for the application 108B, and application information 132D for the new application 108D. The application information 132AA may include a set of functions 140AA related to the updated application 108AA, the application information 132B may include the set of functions 140B related to the application 108B, and the application information 132D may include a set of functions 140D related to the new application 108D.

[0066]

[0055] The large language model 112 may be configured to generate KV pairs for each function 140 in response to receiving the updated prompt 130. To illustrate, the large language model 112 may generate one or more KV pairs 150AA for the set of functions 140AA associated with the updated application 108AA, the one or more KV pairs 150B for the set of functions 140B associated with the application 108B, and one or more KV pairs 150D for the set of functions 140D associated with the new application 108D. After the large language model 112 generates the KV pairs 150, the processor 102 may store the KV pairs 150 at the persistent KV cache 114.

[0067]

[0056] Thus, according to the techniques described with respect to Figure 2, prior to updating the persistent KV cache 114 with updated KV pairs 150 based on the updated prompt 250, the processor 102 may ensure that the device 100 is connected to the power source to conserve energy. Thus, if there are updates to the available functions 140 since the persistent KV cache 114 was last generated (e.g., if a user installed, removed, or upgraded applications), generation of the updated KV pairs 150 may be performed when the device 100 is connected to the power source.

[0068]

[0057] Figure 3 is another diagram of the device 100, in accordance with examples described herein. In Figure 3, a user query 360 is prepended with changes to the functions 140 related to the applications 108 installed on the device 100. In particular, the changes to the Attorney Docket No. 24-0970-WO functions 140 are indicated by, or included within, the updated prompt functions 340 prepended to the user query 360.

[0069]

[0058] Similar to Figure 2, as depicted in Figure 3, the application 108A has been updated to the application 108AA, the application 108C has been deleted, and a new application 108D has been installed on the device 100. The updated prompt functions 340 are based on the updated application 108AA, the application 108B, and the new application 108D. The updated prompt functions 340 may be prepended to the user query 360 before the user query 360 is provided to the large language model 112.

[0070]

[0059] Based on the updated prompt functions 340, the prompt generation unit 122 may be configured to generate an updated prompt 330. Similar to Figure 2, in Figure 3, the updated prompt 330 may be generated in response to (i) a determination that a new application 108D is installed on the device 100, (ii) a determination that the application 108 A has been updated (to application 108AA), or (iii) a determination that the application 108C has been deleted.

[0071]

[0060] The updated prompt 330 may include application information 132 for each application 108 installed on the device 100. For example, the updated prompt 330 may include application information 132AA for the updated application 108AA, the application information 132B for the application 108B, and application information 132D for the new application 108D. The application information 132AA may include the set of functions 140AA related to the updated application 108AA, the application information 132B may include the set of functions 140B related to the application 108B, and the application information 132D may include the set of functions 140D related to the new application 108D.

[0072]

[0061] Thus, in Figure 3, the user query 360 is prepended with changes to the plurality of different functions 140 since the KV pairs 150 were stored at the persistent cache 114 before providing the user query 360 to the large language model 112.

[0073] III. Example Processes of Caching Functions

[0074]

[0062] Figure 4 illustrates multiple examples of caching function sets, in accordance with examples described herein. For example, Figure 4 depicts a first example 400 of caching functions and a second example 410 of caching functions.

[0075]

[0063] According to the first example 400, the set of functions 140A (e.g., the vehicle functions) and the set of functions 140B (e.g., clock functions) are provided to the large language model 112 as part of the prompt 130. In the first example 400, the vehicle functions Attorney Docket No. 24-0970-WO in the set of functions 140A have a different prompt prefix than clock functions in the set of functions MOB.

[0076]

[0064] In the first example 400, the large language model 112 may be configured to generate KV pairs 450A for the prompt prefix of the vehicle functions 140A and KV pairs 450C for the vehicle functions 140A. For example, the KV pairs 450A may indicate how the vehicle functions 140A are declared, and the KV pairs 450C may indicate the actual vehicle functions 140A. In some embodiments, the KV pairs 450A and the KV pairs 450C may correspond to the KV pairs 150A.

[0077]

[0065] Similarly, in the first example 400, the large language model 112 may be configured to generate KV pairs 450B for the prompt prefix of the clock functions MOB and KV pairs 450D for the clock functions MOB. For example, the KV pairs 450B may indicate how the clock functions MOB are declared, and the KV pairs 450D may indicate the actual clock functions MOB. In some embodiments, the KV pairs 450B and the KV pairs 450D may correspond to the KV pairs 150B.

[0078]

[0066] According to the second example 410, the set of functions MOB (e.g., the vehicle functions) and the set of functions MOB (e.g., clock functions) are provided to the large language model 112 as part of the prompt 130. In the second example 410, the vehicle functions in the set of functions 140 A have the same prompt prefix as the clock functions in the set of functions MOB.

[0079]

[0067] To reduce storage associated with caching the functions 140, in the second example 410, the large language model 112 may be configured to (i) determine that the set of functions 140 A and the set of functions MOB share a generic prompt prefix and (ii) generate one or more KV pairs 450E for the generic prompt prefix. The generic prompt prefix includes information that indicates how the set of functions 140 A and the set of functions MOB are declared. Additionally, the large language model 112 may (i) generate one or more KV pairs 450F for the remaining portions of the set of vehicle functions 140A and (ii) generate one or more KV pairs 450G for the remaining portions of the set of clock functions MOB. In some embodiments, the KV pairs 450E and the KV pairs 450F may correspond to the KV pairs 150A, and the KV pairs 450E and the KV pairs 450G may correspond to the KV pairs 150B.

[0080]

[0068] Thus, according to the second example 410, additional efficiency may be achieved where multiple cached function sets (e.g., function sets for different applications) exist. To illustrate, the prompt 130 provided to the large language model 112 may include the Attorney Docket No. 24-0970-WO list of functions 140A for the application 108A installed on the device 100 and the list of functions 140B for the application 108B installed on the device 100. The list of functions 140A and the list of functions 140B may include generic information that indicates how each function in the list of functions 140A, 140B are declared. In this scenario, because the generic information that indicates how each function is declared is the same for the each list of functions 140A, 140B, the large language model 112 can generate KV pairs 450E for the generic information and store the KV pairs 450E in the persistent KV cache 114 as a prefix to both the lists of functions. As a result, storage overhead may be reduced

[0081] IV. Example Process

[0082]

[0069] Figure 5 illustrates an example of a process 500, in accordance with examples described herein. The process 500 may be performed by the device 100 of Figure 1.

[0083]

[0070] According to the process 500, at step 502, the processor 102 may provide a prompt to a large language model based on installed applications. For example, referring to Figure 1, the processor 102 may provide the prompt 130 to the large language model 112. The prompt 130 includes information (e.g., functions 140) based on the installed applications 108. After step 502, the process 500 proceeds to step 504.

[0084]

[0071] At step 504, the large language model may generate KV pairs based on the prompt. For example, referring to Figure 1, the large language model 112 may generate the KV pairs 150 based on the prompt 130. After step 504, the process 500 proceeds to step 506.

[0085]

[0072] At step 506, the KV pairs may be stored at a persistent KV cache. For example, referring to Figure 1, the KV pairs 150 may be stored at the persistent KV cache 114. After step 506, the process 500 proceeds to step 508.

[0086]

[0073] At step 508, the processor may provide a user query to the large language model. For example, referring to Figure 1, the processor 102 may provide the user query 160 to the large language model 112. After step 508, the process 500 proceeds to step 510.

[0087]

[0074] At step 510, the KV pairs may be retrieved from the persistent KV cache. For example, referring to Figure 1 , the KV pairs 150 may be retrieved from the persistent KV cache 114 in response to the large language model 112 receiving the user query 160. After step 510, the process 500 proceeds to step 512.

[0088]

[0075] At step 512, the large language model may generate an executable command based on the query using the stored KV pairs. For example, referring to Figure 1, the large language model 112 may generate the large language model output 180 (e.g., an executable Attorney Docket No. 24-0970-WO command to perform the function 170) based on the user query 160 using the stored KV pairs 150. After step 512, the process 500 proceeds to decision step 514.

[0089]

[0076] At decision step 514, the process 500 determines whether a change has occurred regarding the installed applications. For example, referring to Figure 1, the processor 102 may determine whether a change has occurred regarding the installed applications 108. The change may correspond to an update to an installed application 108, removal (e.g., deletion of an installed application 108), or installation of a new application 108. If a change has not occurred at decision step 514, the process 500 may proceed back to step 508. However, if a change has occurred at decision step 516, the process processed to decision step 518.

[0090]

[0077] At decision step 518, the process 500 determines whether the device is connected to a power source. For example, referring to Figure 1, the power source monitor 120 determines whether the device 100 is connected to a power source. If the device is not connected to a power source at decision step 516, the process 500 proceeds to step 518 where device 100 waits for a predetermined time period before checking whether the power source is connected to the device 100. However, if the device is connected to a power source at decision step 516, the process 500 proceeds to step 520.

[0091]

[0078] At step 520, a new prompt is provided to the large language model based on the change to the installed applications. For example, referring to Figure 2, the updated prompt 230 is provided to the large language model 112. After step 520, the process 500 returns to step 504.

[0092]

[0079] The process 500 of Figure 5 reduces latency associated with processing the user query 160. For example, by caching the KV pairs 150 generated in response to processing the input prompt 130, the large language model 112 may bypass the time-consuming KV pair regeneration process in the future, resulting in faster response times for users. In particular, the latency may be improved by at least ten (10) seconds compared to regenerating the KV pairs 150 each time the large language model 112 is invoked. Additionally, by eliminating redundant computations by regenerating the KV pairs 150 each time the large language model 112 is invoked, energy consumption may be reduced, which in turn, extends battery life. Thus, by caching the KV pairs 150 in the persistent KV cache 114, the process 500 may result in faster and more energy-efficient large language model performance, which translates into a smoother and more enjoyable user experience. Attorney Docket No. 24-0970-WO

[0093] V. Additional Example Operations

[0094]

[0080] Figure 6 illustrates a flow chart of a method 600 related to a new technology. The method 600 may be carried out by the device 100 among other possibilities. The embodiments of Figure 6 may be simplified by the removal of any one or more of the features shown therein. Further, these embodiments may be combined with features, aspects, and / or implementations of any of the previous figures or otherwise described herein.

[0095]

[0081] The method 600 includes providing a prompt to a large language model, at block 602. The prompt includes information for a plurality of different functions that are related to one or more applications installed on the device. For example, referring to Figure 1, the processor 102 may provide the prompt 130 to the large language model 112. The prompt 130 includes information 132 from a plurality of different functions 140 that are related to applications 108 installed on the device 100.

[0096]

[0082] The method 600 includes generating, using the large language model, KV pairs for each function of the plurality of different functions in response to receiving the prompt, at block 604. For example, referring to Figure 1, the large language model 112 generates the KV pairs 150 for each function 140 of the plurality of different functions 140 in response to receiving the prompt 130.

[0097]

[0083] The method 600 includes storing the KV pairs at a persistent KV cache that is accessible to the large language model, at block 606. For example, referring to Figure 1, the KV pairs 150 are stored at the persistent KV cache 114 that is accessible to the large language model 112.

[0098]

[0084] The method 600 includes providing a user query for performing a function to the large language model, at block 608. For example, referring to Figure 1, the processor 102 may provide the query 160 for performing the function 170 to the large language model 112.

[0099]

[0085] The method 600 includes retrieving the KV pairs stored at the persistent KV cache in response to receiving the user query, at block 610. For example, referring to Figure 1, the KV pairs 150 stored at the persistent KV cache 114 may be retrieved in response to the large language model 112 receiving the user query 160.

[0100]

[0086] The method 600 includes generating, using the large language model, a large language model output based on the KV pairs stored at the persistent KV cache, at block 612. The large language model output is usable to generate a command executable by the processor to cause an application of the one or more applications to perform the function. For example, Attorney Docket No. 24-0970-WO referring to Figure 1, the large language model 112 may generate the large language model output 180 based on the KV pairs 150 stored at the persistent KV cache 114. The large langue model output 180 is usable to generate a command executable by the processor 102 to perform the function 170.

[0101]

[0087] According to one implementation of the method 600, the one or more applications comprise a first application and a second application. The plurality of different functions comprise a first set of functions related to the first application and a second set of functions related to the second application. Generating the KV pairs for each function may include generating one or more first KV pairs for the first set of functions and generating one or more second KV pairs for the second set of functions.

[0102]

[0088] According to one implementation of the method 600, the one or more applications comprise a first application and a second application. The plurality of different functions comprise a first set of functions related to the first application and a second set of functions related to the second application. Generating the KV pairs for each function may include (i) determining that the first set of functions and the second set of functions share a generic prompt prefix, (ii) generating one or more KV pairs for the generic prompt prefix, (iii) generating one or more first KV pairs for remaining portions of the first set of functions from the generic prompt prefix, and (iv) generating one or more second KV pairs for remaining portions of the second set of functions from the generic prompt prefix. According to one implementation of the method 600, the generic prompt prefix comprises information that indicates how the first set of functions and the second set of functions are declared.

[0103]

[0089] According to one implementation, the method 600 may include determining, by the processor, that the device is connected to a power source. The KV pairs for each function of the plurality of different functions are generated in response to a determination that the device is connected to the power source.

[0104]

[0090] According to one implementation, the method 600 may include determining, by the processor, that the device is connected to a power source. In response to a determination that the device is connected to the power source, the method 600 may include providing, by the processor, an updated prompt to the large language model. The updated prompt may include updated information for the plurality of different functions that are related to the one or more applications. In response to a determination that the device is connected to the power source, the method 600 may also include generating, using the large language model, updated Attorney Docket No. 24-0970-WO

[0105] KV pairs for each function of the plurality of different functions in response to receiving the updated prompt. In response to a determination that the device is connected to the power source, the method 600 may also include storing the updated KV pairs at the persistent KV cache.

[0106]

[0091] According to one implementation of the method 600, the updated prompt is generated in response to a determination that the one or more applications comprise a new application installed on the device. According to one implementation of the method 600, the updated prompt is generated in response to a determination that at least one application of the one or more applications has been updated. According to one implementation of the method 600, the updated prompt is generated in response to a determination that at least one application of the one or more applications has been deleted.

[0107]

[0092] According to one implementation of the method 600, the large language model stores the KV pairs at the persistent KV cache. According to one implementation of the method 600, the large language model provides the processor access to the KV pairs, and the processor stores the KV pairs at the persistent KV cache.

[0108]

[0093] According to one implementation of the method 600, the large language model is a transformer-based large language model. According to one implementation of the method 600, the large language model comprises a large action model.

[0109]

[0094] According to one implementation of the method 600, the user query is prepended with changes to the plurality of different functions since the KV pairs were stored at the persistent cache before providing the user query to the large language model.

[0110]

[0095] The method 600 of Figure 6 reduces latency associated with processing the user query 160. For example, by caching the KV pairs 150 generated in response to processing the input prompt 130, the large language model 112 may bypass the time-consuming KV pair regeneration process in the future, resulting in faster response times for users. In particular, the latency may be improved by at least ten (10) seconds compared to regenerating the KV pairs 150 each time the large language model 112 is invoked. Additionally, by eliminating redundant computations by regenerating the KV pairs 150 each time the large language model 112 is invoked, energy consumption may be reduced, which in turn, extends battery life. Thus, by caching the KV pairs 150 in the persistent KV cache 114, the method 600 may result in faster and more energy-efficient large language model performance, which translates into a smoother and more enjoyable user experience. Attorney Docket No. 24-0970-WO

[0111]

[0096] A non-transitory computer-readable medium includes instructions that, when executed by a processor 102 of a device 100, cause the processor 102 to perform operations. The operations include providing a prompt 130 to a large language model 112 hosted on the device 100. The prompt 130 includes information for a plurality of different functions 140 that are related to one or more applications 108 installed on the device 100. The operations also include generating, using the large language model 112, KV pairs 150 for each function 140 of the plurality of different functions 140 in response to receiving the prompt 130. The operations also include storing the KV pairs 150 at a persistent KV cache 114 that is accessible to the large language model 112. The operations also include providing a user query 160 for performing a function 170 to the large language model 112. The operations also include retrieving, using the large language model 112, the KV pairs 150 stored at the persistent KV cache 114 in response to receiving the user query 160. The operations also include generating, using the large language model 112, a large language model output 180 based on the KV pairs 150 stored at the persistent KV cache 114. The large language model output 180 is usable to generate a command executable by the processor 102 to perform the function 170.

[0112]

[0097] A computer program product includes computer-executable program code. The computer-executable program code, when executed by a computer, causes the computer to provide a prompt to a large language model hosted on the device. The prompt includes information for a plurality of different functions that are related to one or more applications installed on the device. The computer-executable program code, when executed by the computer, causes the computer to generate, using the large language model, key and value (KV) pairs for each function of the plurality of different functions in response to receiving the prompt. The computer-executable program code, when executed by the computer, causes the computer to store the KV pairs at a persistent KV cache that is accessible to the large language mode. The computer-executable program code, when executed by the computer, causes the computer to provide a user query for performing a function to the large language model. The computer-executable program code, when executed by the computer, causes the computer to retrieve the KV pairs stored at the persistent KV cache in response to receiving the user query. The computer-executable program code, when executed by the computer, causes the computer to generate, using the large language model, a large language model output based on the KV pairs stored at the persistent KV cache. The large language model output is usable to generate a command executable by the computer to perform the function. Attorney Docket No. 24-0970-WO

[0113] VI. Conclusion

[0114]

[0098] The present disclosure is not to be limited in terms of the particular embodiments described in this application, which are intended as illustrations of various aspects. Many modifications and variations can be made without departing from its scope, as will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses within the scope of the disclosure, in addition to those described herein, will be apparent to those skilled in the art from the foregoing descriptions. Such modifications and variations are intended to fall within the scope of the appended claims.

[0115]

[0099] The above detailed description describes various features and operations of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The example embodiments described herein and in the figures are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0116]

[0100] With respect to any or all of the message flow diagrams, scenarios, and flow charts in the figures and as discussed herein, each step, block, and / or communication can represent a processing of information and / or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages can be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved. Further, more or fewer blocks and / or operations can be used with any of the message flow diagrams, scenarios, and flow charts discussed herein, and these message flow diagrams, scenarios, and flow charts can be combined with one another, in part or in whole.

[0117]

[0101] A step or block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including Attorney Docket No. 24-0970-WO related data). The program code may include one or more instructions executable by a processor for implementing specific logical operations or actions in the method or technique. The program code and / or related data may be stored on any type of computer readable medium such as a storage device including random access memory (RAM), a disk drive, a solid state drive, or another storage medium.

[0118]

[0102] The computer readable medium may also include non-transitory computer readable media such as computer readable media that store data for short periods of time like register memory, processor cache, and RAM. The computer readable media may also include non-transitory computer readable media that store program code and / or data for longer periods of time. Thus, the computer readable media may include secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, solid state drives, compactdisc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.

[0119]

[0103] Moreover, a step or block that represents one or more information transmissions may correspond to information transmissions between software and / or hardware modules in the same physical device. However, other information transmissions may be between software modules and / or hardware modules in different physical devices.

[0120]

[0104] The particular arrangements shown in the figures should not be viewed as limiting. It should be understood that other embodiments can include more or less of each element shown in a given figure. Further, some of the illustrated elements can be combined or omitted. Yet further, an example embodiment can include elements that are not illustrated in the figures.

[0121]

[0105] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for the purpose of illustration and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

Attorney Docket No. 24-0970-WOCLAIMSWhat is claimed is:

1. A method comprising: providing a prompt to a large language model, the prompt including information for a plurality of different functions that are related to one or more applications installed on a device; generating, using the large language model, key and value (KV) pairs for each function of the plurality of different functions in response to receiving the prompt; storing the KV pairs at a persistent KV cache that is accessible to the large language model; providing a user query for performing a function to the large language model; retrieving the KV pairs stored at the persistent KV cache in response to receiving the user query; and generating, using the large language model, a large language model output based on the KV pairs stored at the persistent KV cache, wherein the large language model output is usable to generate a command executable by a processor to cause an application of the one or more applications to perform the function.

2. The method of claim 1, wherein the one or more applications comprise a first application and a second application, wherein the plurality of different functions comprise a first set of functions related to the first application and a second set of functions related to the second application, and wherein generating the KV pairs for each function comprises: generating one or more first KV pairs for the first set of functions; and generating one or more second KV pairs for the second set of functions.

3. The method of claim 1, wherein the one or more applications comprise a first application and a second application, wherein the plurality of different functions comprise a first set of functions related to the first application and a second set of functions related to the second application, and wherein generating the KV pairs for each function comprises: determining that the first set of functions and the second set of functions share a generic prompt prefix;Attorney Docket No. 24-0970-WO generating one or more KV pairs for the generic prompt prefix; generating one or more first KV pairs for remaining portions of the first set of functions from the generic prompt prefix; and generating one or more second KV pairs for remaining portions of the second set of functions from the generic prompt prefix.

4. The method of claim 3, wherein the generic prompt prefix comprises information that indicates how the first set of functions and the second set of functions are declared.

5. The method of claim 1, further comprising determining, by the processor, that the device is connected to a power source, wherein the KV pairs for each function of the plurality of different functions are generated in response to a determination that the device is connected to the power source.

6. The method of claim 1, further comprising: determining, by the processor, that the device is connected to a power source; and in response to a determination that the device is connected to the power source: providing, by the processor, an updated prompt to the large language model, the updated prompt including updated information for the plurality of different functions that are related to the one or more applications; generating, using the large language model, updated KV pairs for each function of the of the plurality of different functions in response to receiving the updated prompt; and storing the updated KV pairs at the persistent KV cache.

7. The method of claim 6, wherein the updated prompt is generated in response to a determination that the one or more applications comprise a new application installed on the device.

8. The method of claim 6, wherein the updated prompt is generated in response to a determination that at least one application of the one or more applications has been updated.Attorney Docket No. 24-0970-WO9. The method of claim 6, wherein the updated prompt is generated in response to a determination that at least one application of the one or more applications has been deleted.

10. The method of claim 1, wherein the large language model stores the KV pairs at the persistent KV cache.

11. The method of claim 1, wherein the large language model provides the processor access to the KV pairs, and wherein the processor stores the KV pairs at the persistent KV cache.

12. The method of claim 1, wherein the large language model comprises a large action model.

13. The method of claim 1, wherein the user query is prepended with changes to the plurality of different functions since the KV pairs were stored at the persistent cache before providing the user query to the large language model.

14. A device comprising: a memory; and a processor coupled to the memory, the processor configured to: provide a prompt to a large language model, the prompt including information for a plurality of different functions that are related to one or more applications installed on the device; generate, using the large language model, key and value (KV) pairs for each function of the plurality of different functions in response to receiving the prompt; store the KV pairs at a persistent KV cache that is accessible to the large language model; provide a user query for performing a function to the large language model; retrieve the KV pairs stored at the persistent KV cache in response to receiving the user query; and generate, using the large language model, a large language model output based on the KV pairs stored at the persistent KV cache, wherein the largeAttorney Docket No. 24-0970-WO language model output is usable to generate a command executable by the processor to cause an application of the one or more applications to perform the function.

15. The device of claim 14, wherein the one or more applications comprise a first application and a second application, wherein the plurality of different functions comprise a first set of functions related to the first application and a second set of functions related to the second application, and wherein to generate the KV pairs for each function, the processor is configured to: generate one or more first KV pairs for the first set of functions; and generate one or more second KV pairs for the second set of functions.

16. The device of claim 14, wherein the one or more applications comprise a first application and a second application, wherein the plurality of different functions comprise a first set of functions related to the first application and a second set of functions related to the second application, and wherein to generate the KV pairs for each function, the processor is configured to: determine that the first set of functions and the second set of functions share a generic prompt prefix; generate one or more KV pairs for the generic prompt prefix; generate one or more first KV pairs for remaining portions of the first set of functions from the generic prompt prefix; and generate one or more second KV pairs for remaining portions of the second set of functions from the generic prompt prefix.

17. The device of claim 16, wherein the generic prompt prefix comprises information that indicates how the first set of functions and the second set of functions are declared.

18. The device of claim 14, wherein the processor is configured to determine that the device is connected to a power source, and wherein the KV pairs for each function of the plurality of different functions are generated in response to a determination that the device is connected to the power source.Attorney Docket No. 24-0970-WO19. The device of claim 14, wherein the processor is configured to: determine that the device is connected to a power source; and in response to a determination that the device is connected to the power source: provide an updated prompt to the large language model, the updated prompt including updated information for the plurality of different functions that are related to the one or more applications; generate, using the large language model, updated KV pairs for each function of the of the plurality of different functions in response to receiving the updated prompt; and store the updated KV pairs at the persistent KV cache.

20. The device of claim 19, wherein the updated prompt is generated in response to a determination that the one or more applications comprises a new application installed on the device.

21. The device of claim 19, wherein the updated prompt is generated in response to a determination that at least one application of the one or more applications has been updated.

22. The device of claim 19, wherein the updated prompt is generated in response to a determination that at least one application of the one or more applications has been deleted.

23. The device of claim 14, wherein the large language model stores the KV pairs at the persistent KV cache.

24. The device of claim 14, wherein the large language model provides the processor access to the KV pairs, and wherein the processor stores the KV pairs at the persistent KV cache.

25. The device of claim 14, wherein the large language model comprises a large action model.Attorney Docket No. 24-0970-WO26. The device of claim 14, wherein the user query is prepended with changes to the plurality of different functions since the KV pairs were stored at the persistent cache before providing the user query to the large language model.

27. A non-transitory computer-readable medium comprising instructions that, when executed by a processor of a device, cause the processor to perform operations comprising: providing a prompt to a large language model, the prompt including information for a plurality of different functions that are related to one or more applications installed on the device; generating, using the large language model, key and value (KV) pairs for each function of the plurality of different functions in response to receiving the prompt; storing the KV pairs at a persistent KV cache that is accessible to the large language model; providing a user query for performing a function to the large language model; retrieving the KV pairs stored at the persistent KV cache in response to receiving the user query; and generating, using the large language model, a large language model output based on the KV pairs stored at the persistent KV cache, wherein the large language model output is usable to generate a command executable by the processor to cause an application of the one or more applications to perform the function.

28. A computer program product comprising computer-executable program code, the computer-executable program code, when executed by a computer, causes the computer to: provide a prompt to a large language model, the prompt including information for a plurality of different functions that are related to one or more applications installed on a device; generate, using the large language model, key and value (KV) pairs for each function of the plurality of different functions in response to receiving the prompt; store the KV pairs at a persistent KV cache that is accessible to the large language model; provide a user query for performing a function to the large language model;Attorney Docket No. 24-0970-WO retrieve the KV pairs stored at the persistent KV cache in response to receiving the user query; and generate, using the large language model, a large language model output based on the KV pairs stored at the persistent KV cache, wherein the large language model output is usable to generate a command executable by the computer to cause an application of the one or more applications to perform the function.

Images