Hybrid Generative Artificial Intelligence Model

Abstract

Certain aspects of this disclosure provide techniques and apparatus for generating responses to input queries using a generative artificial intelligence model. Exemplary methods generally include receiving input for processing. A prompt representing the received input is generated based on the received input, contextual information associated with the received prompt, and a prompt-generating artificial intelligence model. The generated prompt is output to the generative artificial intelligence model for processing. A response to the generated prompt is received from the generative artificial intelligence model and output as a response to the received input.

Description

Technical Field

[0001] (Cross - Reference to Related Applications)

[0001] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 462,198, filed on April 26, 2023, entitled "Hybrid Generative Artificial Intelligence Models", and claims the priority of U.S. Patent Application No. 18 / 543,533, filed on December 18, 2023, entitled "Hybrid Generative Artificial Intelligence Models". Both of those applications have been assigned to the assignee of this application and are hereby incorporated by reference in their entirety.

Background Art

[0002]

[0002] Aspects of the present disclosure relate to generative artificial intelligence models, and more particularly, to hybrid generative artificial intelligence models that execute on edge devices and within cloud environments.

[0003]

[0003] Generative artificial intelligence models can be used in a variety of environments to generate responses to input queries. For example, a generative artificial intelligence model can be used in a chatbot application where large language models are used to generate answers or at least responses to input queries. Other examples where a generative artificial intelligence model can be used include Stable Diffusion where the model generates an image from an input text description of the desired image's content, and Decision Transformer where future actions are predicted based on a sequence of previous actions within a given environment.

[0004]

[0004] In general, generating responses to queries using generative artificial intelligence models can be computationally expensive. For example, in a chatbot deployment where a large language model is used to generate responses to queries formatted as text queries, the response to the query may be generated using a path through the large language model for each token (e.g., a word or part of a word) generated as part of the response. The output of each path may be a probability distribution for a set of tokens (e.g., a word(s) or part of a word) from which the next token may be selected by sampling or on the basis of maximum likelihood. Since the path through the large language model is used to generate each word (or token(s)) in the response to the query, the computational cost can be modeled as the product of the number of words in the response and the computational resource cost (e.g., in terms of processing power, memory bandwidth, or other computational resources used) of implementing the path through the large language model, which generally increases as the number of parameters in the large language model increases. [Overview of the project]

[0005]

[0005] Some aspects of the present disclosure provide a method for generating a response to an input query using a generative artificial intelligence model. The method generally includes receiving an input for processing. A prompt representing the received input is generated based on the received input, contextual information associated with the received input, and a prompt-generating artificial intelligence model. The generated prompt is output to the generative artificial intelligence model for processing. A response to the generated prompt is received from the generative artificial intelligence model and output as a response to the received input.

[0006]

[0006] Some aspects of the present disclosure provide a method for generating a response to an input query using a generative artificial intelligence model. The method generally includes receiving an input prompt for processing. Based on user information associated with the received prompt, context information associated with the received prompt is requested from and received from the personal knowledge repository. A query is generated based on the input prompt and the context information associated with the input prompt. The generated query is output to the generative artificial intelligence model for processing. A response to the generated query is received from the generative artificial intelligence model, and the response to the generated query is output as a response to the input prompt.

[0007]

[0007] Another embodiment provides a processing system configured to carry out the above-described method and the method described herein; a non-temporary computer-readable medium containing instructions that, when executed by one or more processors of the processing system, cause the processing system to carry out the above-described method and the method described herein; a computer program product embodied on a computer-readable storage medium containing code for carrying out the above-described method and the method described herein; and means for carrying out the above-described method and the method described herein.

[0008]

[0008] The following description and related drawings describe in detail certain exemplary features of one or more embodiments. [Brief explanation of the drawing]

[0009]

[0009] The attached figures illustrate only some aspects of the present disclosure and should therefore not be considered as limiting the scope of the present disclosure. [Figure 1]

[0010] This figure shows an exemplary hybrid computing environment in which queries are dispatched to be processed on different devices within a hybrid computing environment, according to aspects of this disclosure. [Figure 2]

[0011] This figure shows an exemplary hybrid computing environment in which a generative model on an edge device functions as a proxy for a generative model running in a cloud environment, according to an aspect of this disclosure. [Figure 3]

[0012] This figure shows an exemplary hybrid computing environment in which queries are dynamically offloaded for processing from one edge device to another within a hybrid environment, according to aspects of the present disclosure. [Figure 4]

[0013] This figure shows an exemplary hybrid computing environment in which queries are processed by a generative artificial intelligence model based on speculative decoding, according to an aspect of this disclosure. [Figure 5]

[0014] This figure shows an exemplary operation for processing queries by a generative artificial intelligence model on an edge device that acts as a proxy for a generative artificial intelligence model running in a cloud environment, according to an aspect of this disclosure. [Figure 6]

[0015] This figure shows an exemplary hybrid computing environment in which external resources are used to extend query processing using generative artificial intelligence models on one or more devices within the hybrid computing environment, according to aspects of the present disclosure. [Figure 7]

[0016] This figure shows an exemplary architecture for orchestrating query processing using a generative artificial intelligence model in a hybrid computing environment, according to aspects of the present disclosure. [Figure 8]

[0017] This figure shows an exemplary operation for generating responses to input queries using a generative artificial intelligence model and external knowledge resources, according to an aspect of this disclosure. [Figure 9]

[0018] This figure shows an exemplary processing system configured to implement various aspects of this disclosure.

[0010]

[0019] For ease of understanding, the same reference numeral is used to designate identical elements common to multiple drawings, where possible. The intention is that elements and features of one embodiment can be usefully incorporated into other embodiments without further detail. [Modes for carrying out the invention]

[0011]

[0020] Aspects of this disclosure provide apparatus, methods, processing systems, and computer-readable media for generating responses to input queries using generative artificial intelligence models in a hybrid computing environment. The term “generative artificial intelligence model” is used interchangeably with the term “generative model” throughout this disclosure. The term “query” may also be used interchangeably with the term “prompt” throughout this disclosure.

[0012]

[0021] Generally, generative artificial intelligence models generate responses to queries input into the model. For example, a large-scale language model deployed within a chatbot may generate responses to queries using multiple paths through the large-scale language model, each successive path based on the query and tokens (or words) generated using previous paths through the large-scale language model. Generally, these large-scale language models may contain a large number of weights or parameters (e.g., billions or trillions). Due to the size of these models, and the actions performed on each token to predict what the next token should be based on the query and previously generated tokens, deploying large-scale language models on various devices, such as those with limited memory, storage, and / or processing power compared to cloud compute instances on which large-scale language models typically operate, may be impractical or even impossible. Furthermore, the memory bandwidth involved in generating responses to queries provided as input to the model may prevent computing resources from being used for other tasks.

[0013]

[0022] To enable generative artificial intelligence models to be used across different devices, generative AI models of different sizes can be trained and deployed for different devices according to the computing power of those devices. For example, a compact model (e.g., trained with 7 billion to 20 billion tokens) can be deployed on edge devices (or a first set or type of device) such as laptop computers, tablet computers, and smartphones, while a larger model (e.g., a model trained with 20 billion to 70 billion tokens, or a model trained with 20 billion to 200 billion tokens) can be deployed on other devices (or a second set or type of device) such as server computers, cloud computing instances, or other computing devices with broader computing power. By using models of different sizes, generative models can be used to generate responses to queries on various devices. However, generally speaking, the size of a model may relate to the model's ability to generate accurate responses to input queries. For example, a more compact model may be able to generate accurate responses to a smaller range of queries than a larger model, but as mentioned above, it may be deployed on devices where it may not be possible to run these larger models due to a lack of available computing resources.

[0014]

[0023] Aspects of this disclosure provide techniques for orchestrating query processing by generative artificial intelligence models in a hybrid computing environment. When orchestrating or otherwise coordinating query processing across different devices in a hybrid computing environment, including edge devices and cloud computing environments, queries may be executed on specific devices based on the properties of the query. In some aspects, information available on these devices may be used to extend the response generated by the generative artificial intelligence model in the hybrid computing environment. Thus, queries may be routed for execution on one or more devices in the hybrid computing environment, thereby enabling the generation of accurate responses while allowing computing resources on other devices to remain available to process other queries using the generative artificial intelligence model.

[0015]

[0024] Aspects of the present disclosure provide techniques for processing queries using a generative artificial intelligence model based on context information retrieved from one or more external knowledge repositories (such as a knowledge graph in which relationships between different items of data are modeled as a graph where connections between different items represent the relationships between these different items). Generally, these external knowledge repositories can store various data. The data can be, for example, specific to a user of an edge device, specific to a related group of users (such as users within the same family, users within the same organization, etc.), or general across various users. The context information is grounded in the context information and can thus be used to generate a response to an input query that is related to a specific user associated with the input query. By doing so, aspects of the present disclosure can improve the accuracy of responses generated using a generative artificial intelligence system. The improved accuracy of these responses minimizes or at least reduces the amount of resources consumed when regenerating the response to account for previous or otherwise unknown contexts, and then enables computing resources to remain available for processing other queries using the generative artificial intelligence model.

[0016] Exemplary hybrid computing environment for processing queries using a generative artificial intelligence model

[0025] FIG. 1 shows an exemplary hybrid computing environment 100 in which a query is dispatched for processing on different devices within a hybrid computing environment, according to an aspect of the present disclosure.

[0017]

[0026] As shown, the hybrid computing environment 100 includes an edge inference system 110, a local inference system 120, and a cloud inference system 130. The edge inference system 110 may correspond to a personal computing device in which a generative artificial intelligence model, such as a large language model (LLM) trained to generate text responses to text queries, is deployed. The local inference system 120 may correspond to a computing system that exposes the generative artificial intelligence model to a defined group of users, such as family members, members of a computing network, or some other defined group of users. Finally, the cloud inference system 130 may correspond to a publicly available computing system hosted within a cloud computing environment or another computing environment that exposes the generative artificial intelligence model to the general public. Figure 1 shows a hybrid computing environment including an edge inference system 110, a local inference system 120, and a cloud inference system 130, but it should be recognized that the hybrid computing environment 100 may include any number of them in any combination (for example, it may include the edge inference system 110 and the cloud inference system 130, the edge inference system 110 and the local inference system 120, the local inference system 120 and the cloud inference system 130, etc.).

[0018]

[0027] Generally, edge inference system 110 may correspond to one or more devices (or a first set or type of devices) having the most limited computing capabilities (e.g., processing speed, memory capacity, memory bandwidth, etc.) within hybrid computing environment 100. Local inference system 120 may generally correspond to one or more devices (or a third set or type of devices) having broader computing capabilities than edge inference system 110. Cloud inference system 130 may generally correspond to one or more devices (or a second set or type of devices) having broader computing capabilities than local inference system 120.

[0019]

[0028] Edge inference system 110 generally includes one or more peripheral devices 112, one or more generative models 114, an orchestrator 116, and a personal knowledge repository 118 (also referred to as a personal knowledge graph). One or more peripheral devices 112 generally enable edge inference system 110 to capture queries and context information related to the captured queries. Peripheral devices 112 included in or connected to (e.g., communicatively coupled to) edge inference system 110 may include, for example, audio-visual capture devices that may capture audio-visual data, sensors that may capture motion data, and other devices that may provide context information related to the use of the edge inference system.

[0020]

[0029] The orchestrator 116 generally identifies which system in the hybrid computing environment 100 should process the ingested query (or part thereof) and routes the ingested query to the identified system for processing. In some embodiments, to identify which system should process the ingested query, the orchestrator 116 may examine information about the query's topic and estimate the query's complexity (e.g., a complexity metric) based on the query's topic. For topics with a complexity metric below a defined threshold, or topics included in a defined set of topics that can be addressed using the generative model 114 in the edge inference system 110, the orchestrator 116 may dispatch the ingested query (and, in some embodiments, contextual information derived from peripheral devices 112 and / or knowledge from the personal knowledge repository 118) to the generative model 114 for processing. In some embodiments, the information based on the generative model 114 generating a response to a ingested query may be further supplemented by data extracted by the orchestrator 116 from one or more external resources (e.g., an external tool 136 hosted in the cloud inference system 130) and / or one or more internal resources.

[0021]

[0030] In some embodiments, the orchestrator 116 may determine that the ingested query is of a sufficiently high level of complexity, or may suggest information to be hosted in either the local inference system 120 or the cloud inference system 130. In such cases, the orchestrator 116 may offload the ingested query to the local inference system 120 (for example, to process it using a generative model 124 hosted in the local inference system 120) and / or to the cloud inference system 130 (for example, to process it using a generative model 134 hosted in the cloud inference system 130). Subsequently, the orchestrator 116 receives a response from the system to which the ingested query is offloaded, and may output the received response to the user of the edge inference system 110 (for example, by rendering the response on a display that is communicably coupled to or integrated with the edge inference system 110, by sending one or more electronic messages containing the response to the user of the edge inference system 110, or by outputting the received response as audio output (for example, as speech output generated by a text-to-speech system) to the user of the edge inference system 120).

[0022]

[0031] In some embodiments, the orchestrator 116 may use contextual information and / or information retrieved from the personal knowledge repository 118 related to the ingested query to identify devices in the hybrid computing environment 100 to which the ingested query (or part thereof) should be dispatched for processing. For example, as will be described in more detail below, peripheral devices 112 may capture data in various data modalities, such as visual data (e.g., image data, video data, etc.), audio data, gesture data, etc., which provide context for the ingested query. In some embodiments, this contextual data may be captured simultaneously with the ingested query. In other embodiments, this contextual data may include historical data captured before the time the ingested query was received. If the orchestrator 116 determines that the ingested query includes multimodal data that can be used to supplement the input query (or, if the input query is captured as audio data (e.g., speech or lyrics) and converted to a text string, a text representation of the input query), the orchestrator 116 may dispatch the ingested query and context information to the local inference system 120 and / or the cloud inference system 130 for processing, as the local inference system 120 and / or the cloud inference system 130 may have sufficient computing power or otherwise be better suited to generating a response to the multimodal data.

[0023]

[0032] In some embodiments, the edge inference system 110 (e.g., generative model 114, orchestrator 116, and / or other models (not shown in Figure 1)) can transform the ingested query, contextual information associated with the ingested query, and / or information retrieved from the personal knowledge repository 118 into a query that can be input to another generative artificial intelligence model (e.g., generative model 124 in the local inference system 120 and / or generative model 134 in the cloud inference system 130). As will be described in more detail below, the generated query may be a text query incorporating one or more of the ingested query and contextual information associated with the ingested query, an embedded representation of the ingested query and contextual information, or a set of features derived from the ingested query and contextual information. Based on various metrics, such as the complexity metric associated with the ingested query and whether the query suggests the use of data in a particular knowledge graph (or other knowledge repository), the orchestrator 116 may dispatch the generated query to the appropriate system in the hybrid computing environment 100 for processing.

[0024]

[0033] The local inference system 120 generally includes an orchestrator 122 that receives requests from the edge inference system 110 to process input queries, one or more generative models 124, and a private knowledge repository 126 that can be used (as described in further detail herein) to extend the responses generated by the generative models 124. The private knowledge repository 126 may be, for example, a knowledge graph or other knowledge repository containing information related to a user or a specific group of a user. The information in the private knowledge repository 126 may be access-controlled so that the knowledge contained therein is accessible and usable by a user or a member of a group (e.g., a family member or a defined and restricted group of users). The generative model(s) 124 deployed on the local inference system 120 may be a larger model than the generative model 114 deployed on the edge inference system 110, and therefore may be used to provide answers to queries that are more complex than those of the model running on the edge inference system 110. In some embodiments, the orchestrator 122 in the local inference system 120 may interact with an external tool 136 (for example, hosted within a cloud computing environment such as a compute instance on which the cloud inference system is hosted) to extend the response generated by the generative artificial intelligence model 124.

[0025]

[0034] The cloud inference system 130 includes, as shown, at least the edge inference system 110, one or more generative models 134, and an orchestrator 132 that can be used to extend the responses generated by the generative models (one or more) 134 on the cloud inference system 130 (as described in more detail herein), and / or the responses generated by other generative artificial intelligence models in the hybrid computing environment 100 (e.g., generative model 114 hosted in the edge inference system 110 and / or generative model 124 hosted in the local inference system 120), to process input queries from one or more external tools 136 (e.g., plug-ins, knowledge graphs (public and / or proprietary), etc.).

[0026]

[0035] In general, the generative model(s) 134 deployed on the cloud inference system 130 may be larger than the generative models 114 and 124 deployed on the edge inference system 110 and the local inference system 120, respectively. In some embodiments, the generative model(s) 134 deployed on the cloud inference system 130 may be used to validate the responses generated by the generative models 114 and 124 deployed on the edge inference system 110 and the local inference system 120, respectively. In further or alternative embodiments, the generative model(s) 134 deployed on the cloud inference system 130 may be used to generate responses to queries offloaded from the edge inference system 110.

[0027]

[0036] In some embodiments, one or more external tools 136 may enable access-controlled grounding of responses generated by the generative model 134, where data within these external tools 136 is restricted to specific users, and as a result, responses generated by the generative model 134 may be modified, or contingent upon, by user-specific information that may differ for multiple different users of the cloud inference system 130.

[0028]

[0037] Figure 2 shows an exemplary hybrid computing environment 200 in which a generative model on an edge device functions as a proxy for a generative model running in a cloud environment, according to an aspect of this disclosure.

[0029]

[0038] As shown, the edge inference system 110 includes a plurality of prompt generation models 210 associated with the peripheral device 112. In order to enable the prompt generation models 210 (and / or one or more generation models 114) to act as proxies for generation models 134 running on the cloud inference system 130, the prompt generation models 210 may function as one or more prompt generation models to preprocess user input to the edge inference system 110 and generate prompts for processing based on the received input and contextual information associated with the input.

[0030]

[0039] As shown, the prompt generation model 210 may include a first model 212 (e.g., a low-power model) and a second model 214 (e.g., a high-performance model). In other embodiments, more prompt generation models 210 may have more than two models. In some embodiments, each of the prompt generation models 210 may have different performance levels, sizes, parameter counts, etc. (e.g., the first model is a low-power model, the second model is a medium-power model, and the third model is a high-power model).

[0031]

[0040] To begin processing a query, the edge inference system 110 receives input through one or more peripheral devices 112. In some embodiments, a low-power model 212, which may run continuously (e.g., as a background process, daemon, service, etc.), may capture signals and other data generated by the peripheral devices 112 to determine when captured data is associated with a query. For example, the low-power model 212 may run continuously to identify the presence of specific keywords indicating a user presenting a query for processing from speech captured by a microphone or other audio capture device connected to or integrated with the edge inference system 110. These specific keywords may be identified simultaneously with or before the query is captured. The above is just one example of a low-power model for detecting that a user is entering a query for processing, and it should be recognized that the low-power model 212 may also employ other techniques to detect that captured data is associated with a query.

[0032]

[0041] When the low-power model 212 determines that a user has begun inputting a query into the edge inference system 110, the low-power model 212 may activate the high-performance model 214 to generate text data and feature outputs from data captured by the peripheral device 112, which may be provided as input to a generative artificial intelligence model (e.g., a generative model 114 deployed on the edge inference system 110, a generative model 134 deployed on the cloud inference system 130, or other generative models not shown in Figure 2, deployed on other devices within the hybrid computing environment 200). The high-performance model 214 may, for example, enable the integration of multiple data modalities into a multimodal query output to the generative model for further processing. In doing so, text, audio, video, sensors, and other data modalities may be combined to enable the inclusion of broader contextual information in the query, thereby enabling the generative model to generate a response to the query that leverages additional information captured by various peripheral devices as context for the textual representation of the query.

[0033]

[0042] In some embodiments, the low-power model 212 may be omitted, and the high-performance model 214 may be invoked on demand to process input from peripherals integrated into or connected to the edge inference system 110 and to generate data (e.g., text data) and / or feature outputs that can be provided as input to a generative artificial intelligence model.

[0034]

[0043] The orchestrator 116 in the edge inference system 110 may use data and / or feature outputs generated by the prompt generation model 210 to generate prompts and send them to the cloud inference system (or the local inference system 120 (not shown in Figure 2)) for processing. Accordingly, the generative model 134 in the cloud inference system 130 generates a response based on the received prompt and sends the generated response to the orchestrator 116 in the edge inference system 110. The orchestrator 116 may then output the generated response to the user of the edge inference system 110 (e.g., in text, visually, audibly, etc.).

[0035]

[0044] In some embodiments, the generative model 114 deployed on the edge inference system may be deactivated or otherwise not used when generating responses to input queries. The low-power model 212 and the high-performance model 214 may be used as proxies for the generative model 134 deployed on the cloud inference system. As described above, when functioning as a proxy for the generative model 134, the prompt generative model 210 may perform various preprocessing tasks on the input captured from the peripheral device 112 in the edge inference system 110 before sending the query to the cloud inference system 130 to offload the task from the generative model 134 in order to generate and process the query. By doing so, embodiments of the disclosure may improve the responsiveness, or at least perceived responsiveness, of the generative model 134 to input queries received in the edge inference system 110 and dispatched to the cloud inference system 130 for processing.

[0036]

[0045] Figure 3 shows an exemplary hybrid computing environment 300 in which queries are dynamically offloaded for processing from an edge inference system (e.g., 110) to another system (e.g., 120 or 130) within the hybrid computing environment 300, according to an aspect of the present disclosure.

[0037]

[0046] In the hybrid computing environment 300, the edge inference system 110 may determine whether to use one or more generative models 114 deployed on the edge inference system 110 to generate a response to an incoming query, or whether to offload the processing of the incoming query to another system (e.g., a local inference system 120 or a cloud inference system 130) (e.g., by offloading a message 302 or 304 instructing the receiving system to generate a response to the incoming query). The orchestrator 116 in the edge inference system 110 may determine which system should be used to generate the response. The determination may be based on, for example (but not limited to), a predefined assignment of a particular task to a particular system within the hybrid computing environment 300, an evaluation of the response generated by the generative models 114 deployed on the edge inference system 110, or an evaluation of the complexity of the task related to the incoming query.

[0038]

[0047] In general, the decision to offload queries to the local inference system 120 or to the cloud inference system 130 may be made in an attempt to maintain the accuracy of responses generated within the hybrid computing environment 300 for a wide range of queries. In other embodiments, the decision to offload queries to the local inference system 120 or to the cloud inference system 130 may be made based on one or more other criteria (e.g., resource utilization on one or more of the systems, the cost of processing on one or more of the systems, the latency or amount of time due to processing on one or more of the systems).

[0039]

[0048] In some embodiments, the orchestrator 116 in the edge inference system 110 may first route the received query to a generative model 114 deployed on the edge inference system 110 in order to generate an initial response. In some embodiments, the orchestrator 116 may determine whether the initial response is correct or incorrect. For example, the orchestrator 116 may use various tools, such as a local compiler and a local knowledge repository (e.g., a personal knowledge repository 118 in the edge inference system 110), to determine whether the initial response is correct or incorrect. For example, in the case of code generation where the query relates to generating source code (e.g., in Python, C++, or some other programming language) to perform a particular task, the orchestrator 116 may use a compiler / interpreter and a unit test framework to check the generated source code. If the compiler / interpreter fails to successfully execute the generated code, or if one or more tests performed on the generated source code fail, the orchestrator 116 may determine that the query should be executed on a different device and may offload the query for processing on the local inference system 120 or the cloud inference system 130. In some embodiments, the orchestrator 116 may predict (for example, based on the confidence level or accuracy score associated with the initial response) whether the initial response is likely to be correct or incorrect, and based on that prediction, may determine whether to offload the query for processing on the local inference system 120 or the cloud inference system 130. Thus, embodiments of the disclosure may enable the response to be output quickly when the initial response is determined to be correct, and when the initial response is determined to be incorrect, the correct response may be generated by a system with more processing power (e.g., the local inference system 120 or the cloud inference system 130).

[0040]

[0049] If the orchestrator 116 in the edge inference system 110 determines that an incoming query should be offloaded from the edge inference system 110 for processing, the orchestrator may provide the query to either or both the local inference system 120 or the cloud inference system 130 for processing. The determination of whether to offload an incoming query to the local inference system 120 or the cloud inference system 130 may be made, for example, based on a complexity metric associated with the incoming query (which may, in some embodiments, be related to a metric such as a confidence score or precision score, where a lower confidence or precision score of the initial response corresponds to a more complex query that should be offloaded to another system for processing, and a higher confidence or precision score of the initial response corresponds to a less complex query whose execution can remain in the edge inference system 110). The most complex queries (e.g., queries associated with a defined complexity level above a defined threshold) may be offloaded to the cloud inference system 130, while other less complex queries may be offloaded to the local inference system 120 for processing.

[0041]

[0050] In some embodiments, the edge inference system 110, and either or both of the local inference system 120 or the cloud inference system 130, may generate responses to incoming queries received by the edge inference system 110. In such cases, the user of the edge inference system 110 may receive an initial response from the edge inference system 110, and then receive responses from other systems to which the received query is offloaded or routed for processing (which may be more accurate due to the increasing size of generative models 124 and 134 relative to the size of generative model 114 on the edge inference system 110, as described above) (for example, responses may be received sequentially from the edge inference system 110, the local inference system 120, and the cloud inference system 130). In some embodiments, responses generated by the edge inference system 110, and either or both of the local inference system 120 or the cloud inference system 130 may be presented simultaneously so that the user of the edge inference system 110 can select the response that the user considers to be the most accurate response to the received query.

[0042]

[0051] Figure 4 shows an exemplary hybrid computing environment 400 in which queries are processed by a generative artificial intelligence model based on speculative decoding, according to an aspect of the present disclosure. It should be understood that the aspects shown in Figure 4 and described herein may be implemented using or in the hybrid computing environments 100, 200, and 300 described above.

[0043]

[0052] In the hybrid computing environment 400, generative models 410 and 420 deployed on the edge inference system 110 and the cloud inference system 130, respectively, can work together to generate responses to received queries. In some embodiments, the generative model 410 deployed on the edge inference system 110 may generate a partial response or multiple candidate partial responses and provide the partial response to the generative model 420 deployed on the cloud inference system 130 for verification. The generative model 420 deployed on the cloud inference system 130 may identify the correct response from the generated partial response or candidate partial responses and return the identified correct response to the edge inference system 110 for use in generating further parts of the response to the query until the response is complete. The generative model 410 deployed on the edge inference system 110 may hereafter be referred to as the draft model, and the generative model 420 deployed on the cloud inference system 130 may hereafter be referred to as the target model.

[0044]

[0053] In a speculative decryption pipeline, the draft model 410 can speculatively generate n tokens autoregressively according to the following equation:

[0045]

number

[0046] During the ceremony,

[0047]

number

[0048] , token x0~x t Probability distribution for the t-th token based on conditional probabilities assuming the selection of t

[0049]

number

[0050] This corresponds to the t+1th token generated from [the source].

[0051]

[0054] The target model 420 generates a probability distribution p for each of the n tokens by employing the generated n tokens according to the following formula and processing the n tokens in parallel.

[0052]

number

[0053]

[0055] Next, the target model 420 can verify the tokens generated by the draft model 410 by comparing the distributions from the draft model 410 and the target model 420 to determine whether the tokens are accepted or rejected.

[0054]

number

[0055] In this case, given token

[0056]

number

[0057] A token may be accepted, and for a function f and a threshold α, the threshold α is chosen such that the probability of an accepted token being a valid token is high in order to include it in the response to the input query. Otherwise, the token may be rejected. The final token is then rejected at the first rejection position or the last position n.

[0058]

number

[0059] It can be generated as a function of (for example, function

[0060]

number

[0061] (Represented by...)

[0062]

[0056] In some embodiments, the draft model 410 may speculatively generate tokens on a group basis. In doing so, groups of tokens may be selected together as candidate responses to an input query, and these candidate responses are represented as a tree data structure having the input query as the root node of the tree. The target model 420 may generate an output distribution for each subpath using a single path through the target model by including all tree nodes in the generated tree as token inputs and performing masked self-attention and position coding for each subpath in the tree.

[0063]

[0057] In some embodiments, speculative decoding may be performed recursively. In this case, the target model 420 recursively performs rejection sampling on the tokens generated by the draft model 410 and contained in the generated tree, and on the probability distribution q provided as input to the target model 420. Rejection sampling may be performed recursively at each node in the generated tree. When performing rejection sampling recursively, the target model 420 may adjust the probability distribution used to accept or reject tokens and to validate subsequent tokens in the generated tree. If a token is rejected, an updated probability distribution q'=(qp) may be generated for use when evaluating subsequent tokens in the tree, where p represents the probability associated with the rejected token from the original probability distribution q. The updated probability distribution q' may then be used to evaluate the next token in the tree.

[0064]

[0058] In some embodiments, speculative decoding can be achieved using a single generative model that combines the functionality of the draft model and target model described above. In doing so, draft token generation, target token generation, and verification can be parallelized in a single generative artificial intelligence model. By using a single generative model, for example, the computational costs involved in generating both the target model 420 and the draft model 410 can be reduced, the performance of the generative task can be improved by performing token verification and speculative generation in a single pass through a single generative model, and the amount of memory used to store the model used for speculative decoding in the generative task can be reduced.

[0065]

[0059] In some embodiments, the draft model 410 and the target model 420 may operate in parallel or substantially in parallel, so that the draft model 410 (deployed on an edge inference system) speculatively generates tokens in response to queries, while the target model 420 (deployed on a cloud inference system) validates a set of tokens previously generated by the draft model 410. In doing so, in order to maximize or at least increase throughput (e.g., the number of tokens generated per second), the draft model 410 running on an edge device may successively generate batches (or sets of tokens) of candidate tokens and output these tokens to the target model 420 running on a server (e.g., in a cloud computing environment). Once the target model 420 returns the accepted set of tokens, the draft model 410 prunes a sample tree to fit the accepted set of tokens. In some embodiments, when the set of accepted tokens is a null set (for example, when the target model 420 does not accept any tokens), the draft model 410 may backtrack to the last token accepted by the target model 420 and resume speculative token generation from that last token. For example, to backtrack, the draft model 410 may prune the generated tree of sample tokens to the last accepted token and resume speculative generation based on the pruned tree.

[0066]

[0060] Examples of speculative generation and decoding in generative artificial intelligence models are described in more detail in U.S. Provisional Patent Application No. 63 / 454,605, filed on 24 March 2023, and U.S. Provisional Patent Application No. 63 / 460,850, filed on 20 April 2023, both of which are incorporated herein by reference in their entirety.

[0067]

[0061] Figure 4 shows the use of the edge inference system 110 and the cloud inference system 130 to perform speculative decryption using a draft model 410 hosted on the edge inference system 110 and a target model 420 hosted on the cloud inference system 130, but it should be understood that a single system can host both the draft model 410 and the target model 420. In such a case, the draft model 410 can directly provide batches of draft tokens to the target model 420 for processing without sending these tokens to different computing systems, as described above.

[0068]

[0062] Figure 5 shows an exemplary operation 500 for processing queries by a generative artificial intelligence model on an edge device that acts as a proxy for a generative artificial intelligence model running in a cloud environment, according to an aspect of the present disclosure. In some aspects, operation 500 may be performed by an edge inference system 110 having, for example, a prompt-generating artificial intelligence model (e.g., a prompt-generating model 210 as shown in Figure 2). In some aspects, operation 500 may perform or include any one or more features of a hybrid computing environment 100-400.

[0069]

[0063] As shown, operation 500 begins in block 510 by receiving an input to process.

[0070]

[0064] In block 520, operation 500 proceeds to generate a prompt representing the received input. In some embodiments, the prompt may be generated based on the received input, contextual information associated with the received input, and a prompt generation artificial intelligence model.

[0071]

[0065] In some embodiments, the prompt-generating artificial intelligence model includes a model that generates prompts based on multimodal context data associated with one or more sensor inputs captured in connection with receiving inputs for processing (e.g., in the meantime or before, but providing the context for that).

[0072]

[0066] In some embodiments, generating a prompt representing a received input includes generating text output based on multimodal context data input to a prompt generation artificial intelligence model.

[0073]

[0067] In some embodiments, generating a prompt involves generating a set of multimodal features based on multimodal context data input to a prompt generation artificial intelligence model.

[0074]

[0068] In some embodiments, multimodal context data includes one or more audio data, image data, or motion data captured while receiving input for processing. Audio data may be captured via one or more audio capture peripherals (e.g., microphones) that are communicatively coupled to or integrated with the edge device. Image data may be captured via one or more imaging device peripherals (e.g., still cameras and / or video cameras) that are communicatively coupled to or integrated with the edge device. Motion data may be captured via one or more imaging device peripherals (e.g., still cameras and / or video cameras), motion detection sensors (e.g., photodiodes, Hall effect sensors, etc.) that are communicatively coupled to or integrated with the edge device.

[0075]

[0069] In block 530, operation 500 proceeds to output the generated prompt to the generative artificial intelligence model for processing. In some embodiments, the generative artificial intelligence model may be a different model from the prompt-generating artificial intelligence model. In some embodiments, the prompt-generating artificial intelligence model may be the same model as the generative artificial intelligence model.

[0076]

[0070] In some embodiments, outputting a generated prompt to a generative artificial intelligence model involves identifying a model from among several generative models deployed in a distributed computing environment to process the generated prompt. In some embodiments, the identification of a model to process the generated prompt may be based at least in part on the task identified within the generated prompt. The generated prompt may be output to a model identified from among several generative models.

[0077]

[0071] In some embodiments, outputting a generated prompt to a generative artificial intelligence model may include generating an initial response based on a local generative artificial intelligence model. The quality of the initial response may be evaluated, and based on the determination that the quality of the initial response does not meet a threshold quality metric, the generated prompt may be output to a remote generative artificial intelligence model for processing. In some embodiments, outputting a generated prompt to a remote generative artificial intelligence model may include estimating the complexity of the task associated with the generated prompt. The model to which the generated prompt should be output may be selected from a plurality of generative models based on the estimated complexity. The generated prompt may be output to the model selected from the plurality of generative models.

[0078]

[0072] In block 540, operation 500 proceeds to receive a response from the generative artificial intelligence model to the generated prompt.

[0079]

[0073] In block 550, operation 500 proceeds to output the received response as a response to the received input.

[0080]

[0074] In some embodiments, receiving input for processing includes detecting query input from a user of the computing device (e.g., via a query detection artificial intelligence model). Based on detecting query input from a user of the computing device, a prompt generation artificial intelligence model may be activated. The prompt generation artificial intelligence model may then be deactivated based on outputting a received response as a response to the received input.

[0081] Exemplary knowledge resource-based grounding of responses generated by generative artificial intelligence models in a hybrid computing environment

[0075] Figure 6 shows an exemplary hybrid computing environment 600 in which external resources are used to augment query processing using generative artificial intelligence models on one or more devices within the hybrid computing environment 600, according to aspects of the present disclosure. The use of external knowledge resources to augment query processing using generative artificial intelligence models may be carried out in some aspects using any of the various aspects discussed herein (for example, in the hybrid computing environments 100, 200, 300, and 400 in Figures 1 to 4 above (among others not shown).

[0082]

[0076] In the hybrid computing environment 600, the orchestrator 116 in the edge inference system 110 may determine (i) whether to use external resources (e.g., among others, the personal knowledge repository 118, the private knowledge repository 126, and / or external tools 136) to extend the processing of incoming queries, and / or (ii) how to use these external resources. These external tools may include, for example, plugins deployed on the edge inference system 110, the local inference system 120, and / or the cloud inference system 130, knowledge repositories 118, 126, and external tools 136 (e.g., knowledge graphs) deployed on the edge inference system 110, the local inference system 120, and / or the cloud inference system 130, respectively.

[0083]

[0077] In some embodiments, the orchestrator 116 in the edge inference system 110 may augment an incoming query using information retrieved from external resources before dispatching the augmented query to a generative artificial intelligence model (e.g., one or more of the generative model 114 in the edge inference system 110, the generative model 124 in the local inference system 120, and / or the generative model 134 in the cloud inference system 130) for processing. In another example, the orchestrator 116 in the edge inference system 110 may retrieve information from the local inference system 120 and / or the cloud inference system 130 (e.g., via grounding 610 or 620 shown in Figure 6) for use in augmenting an incoming query before dispatching it to a generative artificial intelligence model for processing.

[0084]

[0078] The orchestrators 122 and 132 in the local inference system 120 or the cloud inference system 130 may use access control to identify which external resources can be accessed, and other information related to identifying users of the edge inference system, in order to extract relevant information for use when augmenting the received queries and provide it to the edge inference system. Generally, these access controls may prevent users in one workgroup from accessing private knowledge repositories associated with different workgroups.

[0085]

[0079] In some embodiments, the orchestrator 116 in the edge inference system 110 may use external resources to perform various checks on the response generated by the generative artificial intelligence model in the hybrid computing environment 600. If the orchestrator 116 in the edge inference system 110 determines that the response generated by the generative artificial intelligence model in the hybrid computing environment 600 is incorrect or does not match information associated with a particular user of the edge inference system, the orchestrator 116 in the edge inference system 110 may determine that a different generative artificial intelligence model should be used to generate the response, and / or modify the response based on knowledge associated with that user (for example, using a personal knowledge repository 118 stored in the edge inference system 110, and / or a private knowledge repository located in the local inference system 120 or the cloud inference system 130).

[0086]

[0080] In some embodiments, external resources may be used to minimize or at least reduce memory usage during operation using the generative artificial intelligence model. For example, these external resources may be used to improve response quality by maintaining a repository in which current queries and candidate responses can be stored (e.g., so that such candidate responses do not need to be regenerated), a conversation history may be maintained, and as a result, answers to queries previously presented within the hybrid computing environment 600 may be provided as responses without calling response generation operations using the generative artificial intelligence model.

[0087]

[0081] Figure 7 shows an exemplary hybrid computing environment architecture 700 for orchestrating query processing using a generative artificial intelligence model in a hybrid computing environment, according to aspects of the present disclosure. The query processing orchestration shown in Figure 7 may, in some aspects, be carried out using any of the various aspects discussed herein (for example, in the hybrid computing environments 100, 200, 300, 400, and 600 of Figures 1-4 and 6 (among others not shown) described above).

[0088]

[0082] In order to enable various applications to operate within the hybrid computing environment, as described above, the hybrid computing environment architecture 700 may expose the orchestrators 116, 132 and generative models 114, 134 of the edge inference system 110 and the cloud inference system 130 as services (e.g., background processes, daemons, etc.) located on devices within the hybrid computing environment.

[0089]

[0083] As shown, on an edge inference system 110 (e.g., a laptop computer, tablet computer, smartphone, etc.), the orchestrator 116 and generative models (one or more) 114 may be deployed within the edge stack 710 as a hybrid artificial intelligence service 714, positioned between the operating system 712 installed on the edge inference system 110 and the application 718 that uses the hybrid artificial intelligence service 714. The hybrid artificial intelligence service 714 may enable the orchestrator 116 to be model-independent and application-independent, thereby enabling many different applications to take advantage of the performance benefits provided by offloading queries to other devices in the hybrid computing environment for processing. Similarly, as shown, various applications 718 may run on top of the hybrid artificial intelligence service 714 located on a cloud inference system. In some embodiments, the edge stack 710 may further include an artificial intelligence software stack 716, which may include one or more artificial intelligence models that generate queries or at least a characterized version of the input to the edge inference system 110 based on inputs captured by peripheral devices 112 in the edge inference system 110.

[0090]

[0084] The hybrid artificial intelligence service 714 included in the edge stack 710 associated with the edge inference system 110, and the hybrid artificial intelligence service 722 included in the cloud stack 720 associated with the cloud inference system 130, can manage orchestration, communication, and data security. Furthermore, the hybrid artificial intelligence services 714 and 722 can perform various data type conversions (e.g., integer to floating-point conversion, quantization, etc.) to provide data consistency across models used in the hybrid computing environment. Finally, since orchestration and communication are performed by the service rather than directly by the applications 718 and 724 using the generative artificial intelligence model, a common interface can be used across different applications, thereby minimizing or at least reducing the risk of interface fragmentation across different applications.

[0091] Exemplary behavior for generating responses to input queries using generative artificial intelligence models and knowledge resource grounding.

[0085] Figure 8 shows an exemplary operation 800 for generating a response to an input query using a generative artificial intelligence model and external knowledge resources, according to an aspect of the present disclosure. It should be understood that operation 800 can be carried out in conjunction with the various aspects described above with respect to Figures 1 to 7.

[0092]

[0086] As shown, operation 800 begins in block 810 by receiving an input prompt to process.

[0093]

[0087] In block 820, operation 800 proceeds to request context information associated with the received prompt from the knowledge repository. Generally, the request may be based on user information associated with the received prompt. User information may include, for example, information identifying the user account logged into the device or service from which the input prompt was received.

[0094]

[0088] In some embodiments, requesting contextual information from a personal knowledge repository involves identifying the knowledge repository from a population of repositories to which the user associated with the user information has access permissions. For example, a user associated with user information may have access to personal data repositories located on edge devices, a subset of personal data repositories located on local servers (e.g., the local inference system 120 shown in Figure 7), and knowledge repositories located on cloud inference systems.

[0095]

[0089] In some embodiments, requesting context information from a personal knowledge repository may include requesting context information from multiple knowledge repositories. Context information may be received from one or more of the multiple knowledge repositories. Generally, one or more knowledge repositories from which context information is received include knowledge repositories that the user associated with the user information has permission to access. Knowledge repositories that the user does not have permission to access may, for example, return a null dataset or an error indicating that the user does not have permission to access these repositories.

[0096]

[0090] In some embodiments, the knowledge repository includes a knowledge repository located in the same location as the generative artificial intelligence model. For example, the personal knowledge repository and the generative artificial intelligence model may be located in the same location on the edge device that receives the input prompt.

[0097]

[0091] In some embodiments, the knowledge repository includes a knowledge repository that is hosted on a local network and accessible by a group of users, including users associated with user information.

[0098]

[0092] In some embodiments, the knowledge repository includes a public knowledge repository located on a remote computing system.

[0099]

[0093] In block 830, operation 800 proceeds to retrieve context information associated with the received prompt from the knowledge repository.

[0100]

[0094] In block 840, operation 800 proceeds to generate a query based on the input prompt and the context information associated with the input prompt.

[0101]

[0095] In block 850, operation 800 proceeds to output the generated query to the generating artificial intelligence model for processing.

[0102]

[0096] In block 860, operation 800 proceeds to receive a response to the generated query from the generative artificial intelligence model.

[0103]

[0097] In block 870, operation 800 proceeds to output the received response as a response to the input prompt.

[0104]

[0098] In some embodiments, operation 800 further includes retrieving query-related information from an external resource. The received response may be updated based on the information retrieved from the external resource, and the modified response may be output as a response to an input prompt.

[0105] Exemplary processing system for handling queries using generative artificial intelligence models in a hybrid computing environment

[0099] Figure 9 shows an exemplary processing system 900 for generating responses to query inputs to artificial intelligence models generated within a hybrid environment (e.g., hybrid computing environments 100, 200, 300, 400, and 600, among others) as described herein with respect to Figures 5 and 8, for example.

[0106]

[0100] The processing system 900 includes a central processing unit (CPU) 902, which in some embodiments may be a multi-core CPU. Instructions executed in the CPU 902 may be loaded, for example, from program memory associated with the CPU 902, or from a memory partition (for example, memory 924).

[0107]

[0101] The processing system 900 also includes additional processing components tuned to specific functions, such as a graphics processing unit (GPU) 904, a digital signal processor (DSP) 906, a neural processing unit (NPU) 908, and a connectivity component 912.

[0108]

[0102] NPUs such as the NPU908 are generally dedicated circuits configured to perform control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. An NPU may also be referred to as a neural signal processor (NSP), tensor processing unit (TPU), neural network processor (NNP), intelligence processing unit (IPU), vision processing unit (VPU), or graph processing unit.

[0109]

[0103] NPUs such as the NPU908 are configured to accelerate the performance of common machine learning tasks such as image classification, machine translation, object detection, and various other predictive models. In some embodiments, multiple NPUs may be instantiated on a single chip, such as a system on a chip (SoC), while in other embodiments, such NPUs may be part of a dedicated neural network accelerator.

[0110]

[0104] The NPU can be optimized for training or inference, or in some cases, it can be configured to balance performance between both. With respect to an NPU capable of performing both training and inference, the two tasks can still generally be performed independently.

[0111]

[0105] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is an extremely computationally intensive operation that involves inputting an existing dataset (often labeled or tagged), iterating through that dataset, and then adjusting model parameters such as weights and biases to improve model performance. Generally, optimization based on incorrect predictions involves backpropagating through layers of the model to determine gradients to reduce prediction errors.

[0112]

[0106] NPUs designed to accelerate inference are generally configured to operate on a complete model. Therefore, such an NPU may be configured to take new data as input and process this new data quickly through a model that has already been trained to produce model outputs (e.g., inferences).

[0113]

[0107] In some implementations, the NPU908 is part of one or more of the CPU902, GPU904, and / or DSP906. These may be located on the user equipment (UE) of the wireless communication system or on another computing device.

[0114]

[0108] In some embodiments, the connection component 912 may include, for example, subcomponents for third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., Long-Term Evolution, LTE), fifth-generation (5G) connectivity (e.g., New Radio, NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The connection component 912 may be further coupled to one or more antennas 914.

[0115]

[0109] The processing system 900 may also include one or more sensor processing units 916 associated with any type of sensor, one or more image signal processors (ISPs) 918 associated with any type of image sensor, and / or a navigation processor 920 which may include satellite-based positioning system components (e.g., GPS or GLONASS), and inertial positioning system components.

[0116]

[0110] The processing system 900 may also include one or more input and / or output devices 922, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, a microphone, etc.

[0117]

[0111] In some embodiments, one or more of the processors of the processing system 900 may be based on an ARM instruction set or a RISC-V instruction set.

[0118]

[0112] The processing system 900 also includes memory 924 which represents one or more static memories and / or dynamic memories, such as dynamic random access memory and flash-based static memory. In this embodiment, memory 924 includes computer executable components which may be executed by one or more of the aforementioned processors of the processing system 900.

[0119]

[0113] In particular, in this embodiment, the memory 924 includes a query receiving component 924A, a device identification component 924B, a request sending component 924C, a response receiving component 924D, a response output component 924E, a generation model 924F, and a personal knowledge repository 924G. The components shown, and other components not shown, may be configured to carry out various aspects of the methods described herein.

[0120]

[0114] Generally, the processing system 900 and / or its components may be configured to carry out the methods described herein.

[0121] Exemplary clause

[0115] Details of various implementations of this disclosure are described in the following numbered clauses.

[0122]

[0116] Clause 1: A method implemented by a processor, comprising: receiving an input for processing; generating a prompt representing the received input based on the received input, contextual information associated with the received input, and a prompt-generating artificial intelligence model; outputting the generated prompt to the generative artificial intelligence model for processing; receiving a response from the generative artificial intelligence model to the generated prompt; and outputting the received response as a response to the received input.

[0123]

[0117] Clause 2: The method according to Clause 1, wherein receiving input for processing includes detecting query input from a user of a computing device, and the method further includes activating a prompt-generating artificial intelligence model based on detecting the query input.

[0124]

[0118] Clause 3: The method according to Clause 2, further comprising deactivating a prompt-generating artificial intelligence model based on outputting a received response as a response to a received input.

[0125]

[0119] Clause 4: The method according to any one of Clauses 1 to 3, comprising a prompt generation artificial intelligence model that generates prompts based on multimodal context data associated with one or more sensor inputs captured in connection with receiving input for processing.

[0126]

[0120] Clause 5: The method according to Clause 4, wherein generating a prompt representing a received input includes generating text output based on multimodal context data input to a prompt-generating artificial intelligence model.

[0127]

[0121] Clause 6: The method according to Clause 4 or 5, wherein generating a prompt includes generating a set of multimodal features based on multimodal context data input to a prompt-generating artificial intelligence model.

[0128]

[0122] Clause 7: The method according to any one of Clauses 4 to 6, wherein the multimodal context data includes one or more audio data, image data, or motion data captured while receiving input for processing.

[0129]

[0123] Clause 8: The method of any one of Clauses 1 to 7, wherein outputting a generated prompt to a generating artificial intelligence model includes identifying a model from a group of generating models deployed in a distributed computing environment to process the generated prompt, and outputting the generated prompt to the model identified from the group of generating models.

[0130]

[0124] Clause 9: The method of any one of Clauses 1 to 8, wherein outputting a generated prompt to a generating artificial intelligence model is, based on generating an initial response based on a first generating artificial intelligence model, evaluating the quality of the initial response, and determining that the quality of the initial response does not meet a threshold quality metric, the generated prompt is output to a second generating artificial intelligence model for processing, wherein the second generating artificial intelligence model is remote from the first generating artificial intelligence model.

[0131]

[0125] Clause 10: The method according to Clause 9, wherein the first generating artificial intelligence model is deployed on the same device on which the input is received.

[0132]

[0126] Clause 11: The method according to Clause 9 or 10, wherein the second generative artificial intelligence model includes a model that is deployed on a device remote from the device on which the first generative artificial intelligence model is deployed.

[0133]

[0127] Clause 12: The method according to any one of Clauses 9 to 11, wherein outputting a generated prompt to a remote generative artificial intelligence model includes estimating the complexity of the task associated with the generated prompt, selecting from a plurality of generative models to which the generated prompt should be output based on the estimated complexity, and outputting the generated prompt to the selected model from the plurality of generative models.

[0134]

[0128] Clause 13: A method implemented by a processor, comprising: receiving an input prompt for processing; requesting context information associated with the received prompt from a knowledge repository based on user information associated with the received prompt; retrieving context information associated with the input prompt from the knowledge repository; generating a query based on the input prompt and the context information associated with the input prompt; outputting the generated query to a generative artificial intelligence model for processing; receiving a response from the generative artificial intelligence model to the generated query; and outputting the received response as a response to the input prompt.

[0135]

[0129] Clause 14: The method of Clause 13, wherein requesting contextual information from a knowledge repository includes identifying the knowledge repository from a population of repositories to which the user associated with the user information has access permissions.

[0136]

[0130] Clause 15: The method according to Clause 13 or 14, wherein requesting context information from a knowledge repository includes requesting context information from multiple knowledge repositories, and retrieving context information includes receiving context information from one or more of the multiple knowledge repositories, wherein one or more of the knowledge repositories include knowledge repositories that the user associated with the user information has permission to access.

[0137]

[0131] Clause 16: The method according to any of Clauses 13-15, wherein the knowledge repository includes a knowledge repository located in the same location as the generated artificial intelligence model.

[0138]

[0132] Clause 17: The method according to Clause 16, wherein the knowledge repository and the generative artificial intelligence model are located in the same location on the edge device that receives the input prompt.

[0139]

[0133] The method of the

[0140]

[0134] Clause 19: The method of any of Clauses 13 to 18, which includes a knowledge repository, the personal knowledge repository being hosted on a local network and accessible by a group of users, including the user associated with the user information.

[0141]

[0135] Clause 20: The method described in any of Clauses 13 to 19, wherein the personal knowledge repository includes a public knowledge repository located on a remote computing system.

[0142]

[0136] Clause 21: A processing system comprising: at least one memory storing executable instructions; and one or more processors configured to execute executable instructions in order to cause the processing system to carry out the method described in any of Clauses 1 to 20.

[0143]

[0137] Clause 22: A processing system comprising means for carrying out the method described in any of Clauses 1 to 20.

[0144]

[0138] Clause 23: A non-temporary computer-readable medium wherein instructions, when executed by one or more processors, perform the operations described in any of Clauses 1 to 20.

[0145] Additional considerations

[0139] The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. The embodiments discussed herein do not limit the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to a person skilled in the art, and the general principles defined herein may also be applied to other embodiments. For example, changes may be made to the function and configuration of the elements discussed without departing from the scope of the disclosure. Various embodiments may omit, substitute, or add various procedures or components as needed. For example, the methods described may be carried out in an order different from the order described, and various steps may be added, omitted, or combined. Also, features described in some embodiments may be combined in some other embodiments. For example, an apparatus may be implemented or a method may be practiced using any number of embodiments described herein. Furthermore, the scope of the disclosure is intended to encompass apparatus or methods that are practiced using other structures, functionalities, or structures and functions in addition to, or other than, the various embodiments of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the claims.

[0146]

[0140] As used herein, the term “exemplary” means “to serve as an example, case, or illustration.” No embodiment described herein as “exemplary” should be construed as necessarily preferable or advantageous to any other embodiment.

[0147]

[0141] As used herein, the phrase “at least one of” the list of items refers to any combination of those items, including a single member. For example, “at least one of a, b, or c” is intended to include a, b, c, ab, ac, bc, and abc, as well as any combination having multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other sequence of a, b, and c).

[0148]

[0142] As used herein, the term “determining” encompasses a wide variety of actions. For example, “determining” may include calculating, calculating, processing, deriving, investigating, searching (e.g., searching a table, database, or other data structure), and confirming. It may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and resolving, selecting, choosing, and establishing.

[0149]

[0143] The methods disclosed herein include one or more steps or actions to achieve those methods. The steps and / or actions of those methods can be replaced with one another without departing from the claims. In other words, unless a particular order of steps or actions is specified, the order of any particular steps and / or actions, and / or the use of those steps and / or actions, can be modified without departing from the claims. Furthermore, the various operations of the methods described above can be carried out by any preferred means capable of performing the corresponding functions. These means may include, but are not limited to, a variety of hardware components and / or software components, including circuits, application-specific integrated circuits (ASICs), or processors, and / or a variety of hardware modules and / or software modules. Generally, where operations are shown in the figures, those operations may have corresponding equivalent means-plus-function components, similarly numbered.

[0150]

[0144] The following claims are not intended to be limited to the embodiments shown herein, but the full scope consistent with the language of the claims should be recognized. In the claims, a singular reference to an element is intended to mean "one or more" rather than "one and only one" unless otherwise explicitly stated. Unless otherwise explicitly stated, the term "several" refers to one or more. No element of a claim should be construed under Section 112(f) of the U.S. Patent Act unless it is explicitly enumerated using the phrase "means for..." or, in the case of a method claim, enumerated using the phrase "steps for...". All structural and functional equivalents of elements of various embodiments described throughout this disclosure, which are known to those skilled in the art or will become known later, are expressly incorporated by reference herein and are intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be made public, regardless of whether such disclosure is explicitly enumerated in the claims.

Claims

1. A processing system, At least one memory location storing executable instructions, One or more processors, wherein the processing system includes: It receives input for processing, Based on the received input, the context information associated with the received input, and the prompt generation artificial intelligence model, a prompt representing the received input is generated. The generated prompt is output to the generating artificial intelligence model for processing. The artificial intelligence model generates a response to the generated prompt, As a response to the received input, the received response is output. One or more processors configured to execute the aforementioned executable instructions, A processing system equipped with the following features.

2. In order to receive the input for the processing described above, one or more processors are configured to cause the processing system to detect query input from a user of a computing device. The one or more processors are further configured to cause the processing system to activate the prompt generation artificial intelligence model based on detecting the input of the query. The processing system according to claim 1.

3. The processing system according to claim 1, wherein the prompt generation artificial intelligence model further includes a model that generates the prompt based on multimodal context data associated with one or more sensor inputs captured in connection with receiving the input for processing.

4. The processing system according to claim 3, wherein, in order to generate the prompt representing the received input, one or more processors are configured to cause the processing system to generate text output based on the multimodal context data input to the prompt generation artificial intelligence model.

5. The processing system according to claim 3, wherein, in order to generate the prompt representing the received input, one or more processors are configured to cause the processing system to generate a set of multimodal features based on the multimodal context data input to the prompt generation artificial intelligence model.

6. The processing system according to claim 3, wherein the multimodal context data includes one or more audio data, image data, or motion data captured while receiving the input for processing.

7. In order to output the generated prompt to the generating artificial intelligence model, one or more processors in the processing system To process the generated prompt, the system identifies a model from multiple generative models deployed in a distributed computing environment. The generated prompt is output to the model identified from the plurality of generation models. The processing system according to claim 1, configured as described above.

8. In order to output the generated prompt to the generating artificial intelligence model, one or more processors, An initial response is generated based on the first generative artificial intelligence model. Evaluate the quality of the initial response, Based on the determination that the quality of the initial response does not meet the threshold quality metric, the generated prompt is output to a second generative artificial intelligence model for processing, and the second generative artificial intelligence model is remote from the first generative artificial intelligence model. The processing system according to claim 1, configured as described above.

9. The processing system according to claim 8, wherein the first generative artificial intelligence model includes a model deployed on the same device as the device that receives the input.

10. The processing system according to claim 9, wherein the second generative artificial intelligence model includes a model deployed on a device remote from the device on which the first generative artificial intelligence model is deployed.

11. In order to output the generated prompt to the remotely generated artificial intelligence model, one or more processors in the processing system To estimate the complexity of the task associated with the generated prompt, Based on the estimated complexity, select the model from among multiple generative models that should output the generated prompt. The generated prompt is output to the selected model from the plurality of generation models. The processing system according to claim 8, configured as described above.

12. A method implemented by a processor, Receiving input for processing, Based on the received input, the context information associated with the received input, and the prompt generation artificial intelligence model, a prompt representing the received input is generated. The generated prompt is output to the generative artificial intelligence model for processing, The artificial intelligence model generates responses to the generated prompts, As a response to the received input, the received response is output, Methods that include...

13. Receiving input for the aforementioned processing includes detecting query input from a user of the computing device, The method further includes activating the prompt generation artificial intelligence model based on detecting the input of the query, The method according to claim 12.

14. The method according to claim 12, wherein the prompt generation artificial intelligence model includes a model that generates the prompt based on multimodal context data associated with one or more sensor inputs captured in connection with receiving the input for processing.

15. The method according to claim 14, wherein generating the prompt representing the received input includes generating text output based on the multimodal context data input to the prompt generation artificial intelligence model.

16. The method according to claim 14, wherein generating the prompt includes generating a set of multimodal features based on the multimodal context data input to the prompt generation artificial intelligence model.

17. The method according to claim 14, wherein the multimodal context data includes one or more audio data, image data, or motion data captured while receiving the input for processing.

18. Outputting the generated prompt to the generating artificial intelligence model is To process the generated prompt, identify a model from multiple generative models deployed in a distributed computing environment, The generated prompt is output to the identified model from the plurality of generation models, The method according to claim 12, including the method described in claim 12.

19. Outputting the generated prompt to the generating artificial intelligence model is To generate an initial response based on a first generative artificial intelligence model, To evaluate the quality of the initial response, Based on the determination that the quality of the initial response does not meet the threshold quality metric, the generated prompt is output to a second generative artificial intelligence model for processing, wherein the second generative artificial intelligence model is remote from the first generative artificial intelligence model. The method according to claim 12, including the method described in claim 12.

20. The method according to claim 19, wherein the first generative artificial intelligence model includes a model deployed on the same device as the device on which the input is received.

21. The method according to claim 20, wherein the second generative artificial intelligence model includes a model that is deployed on a device remote from the device on which the first generative artificial intelligence model is deployed.

22. Outputting the generated prompt to the remote artificial intelligence model is To estimate the complexity of the task associated with the generated prompt, Based on the estimated complexity, select from multiple generative models the model that should output the generated prompt. The generated prompt is output to a model selected from the multiple generation models, The method according to claim 19, including the method described in claim 19.

23. A processing system, Means for receiving input for processing, Means for generating a prompt representing the received input based on the received input, context information associated with the received input, and a prompt generation artificial intelligence model, Means for outputting the generated prompt to a generating artificial intelligence model for processing, Means for receiving a response to the generated prompt from the aforementioned artificial intelligence model, As a response to the received input, means for outputting the received response, A processing system equipped with the following features.

24. The means for receiving input for processing includes means for detecting query input from a user of a computing device, The processing system further comprises means for activating the prompt generation artificial intelligence model based on detecting the input of the query. The processing system according to claim 23.

25. The processing system according to claim 23, wherein the prompt generation artificial intelligence model includes a model that generates the prompt based on multimodal context data associated with one or more sensor inputs captured in connection with receiving the input for processing.

26. The means for generating the prompt representing the received input is Means for generating text output based on the multimodal context data input to the prompt generation artificial intelligence model, or Means for generating a set of multimodal features based on the multimodal context data input to the prompt generation artificial intelligence model, The processing system according to claim 25, comprising at least one or more of the above.

27. The means for outputting the generated prompt to the generating artificial intelligence model is Means for identifying a model from multiple generative models deployed in a distributed computing environment to process the generated prompt, Means for outputting the generated prompt to the identified model from the plurality of generation models, The processing system according to claim 23, comprising:

28. Outputting the generated prompt to the generating artificial intelligence model is Means for generating an initial response based on a first generative artificial intelligence model, Means for evaluating the quality of the initial response, Means for outputting the generated prompt to a second generative artificial intelligence model for processing, based on the determination that the quality of the initial response does not meet a threshold quality metric, wherein the second generative artificial intelligence model is remote from the first generative artificial intelligence model. The processing system according to claim 23, comprising:

29. The means for outputting the generated prompt to the remotely generated artificial intelligence model is A means for estimating the complexity of the task associated with the generated prompt, A means for selecting from a plurality of generative models, based on the estimated complexity, the model from which the generated prompt should be output. Means for outputting the generated prompt to the selected model from the plurality of generation models, The processing system according to claim 19, comprising:

30. A non-temporary computer-readable medium storing executable instructions, wherein when the instructions are executed by one or more processors, Receiving input for processing, Based on the received input, the context information associated with the received input, and the prompt generation artificial intelligence model, a prompt representing the received input is generated. The generated prompt is output to the generative artificial intelligence model for processing, The artificial intelligence model generates responses to the generated prompts, As a response to the received input, the received response is output, A non-temporary computer-readable medium that performs operations including [specific actions].

Images