Self-adjusting assistant LLM to enable robust interaction with traffic LLM

Abstract

A method includes receiving a natural language query specifying an action for an assistant interface to perform and selecting one or more business large language models (LLMs) for the assistant interface to interact with to perform the performance of the action. For each business LLM, the method further includes accessing an adapter module to construct the natural language query as a respective prompt specifically formulated for the corresponding business LLM, issuing the respective prompt for input to the corresponding business LLM, and transmitting the natural language query to the corresponding business LLM. And receiving, from the corresponding traffic LLM, corresponding response content conveying details regarding execution of the corresponding portion of the action. The method further includes presenting presentation content for output from the user device based on the corresponding response content received from each corresponding traffic LLM.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to self-tuning assistant large language models (LLMs) that enable robust interactions with business LLMs. BACKGROUND

[0002] Large language models are increasingly being used to provide conversational experiences between users and digital assistant interfaces executing on user devices. Generally, a user provides a query / prompt in natural language to an LLM requesting information, and the LLM generates a response conveying the requested information based on the query / prompt. As LLMs are currently opening a wide range of applications due to their powerful understanding and generation capabilities that they can operate on textual, image, and / or audio inputs, LLMs are being customized to operate and provide specific services for users. SUMMARY

[0003] One aspect of the present disclosure provides a computer-implemented method executing on data processing hardware that causes the data processing hardware to perform operations for implementing a self-tuning assistant large language model (LLM) that enables robust interactions with business LLMs. The operations include receiving, at an assistant interface, a natural language query input by a user to a user device, where the natural language query specifies an action for the assistant interface to perform on behalf of the user. The operations further include selecting, by the assistant interface, one or more business large language models (LLMs) for the assistant interface to interact with to fulfill performance of the action, where each of the one or more business LLMs is selected to fulfill performance of a corresponding portion of the action. For each corresponding business LLM among the one or more business LLMs selected by the assistant interface, the operations include accessing, by the assistant interface, an adaptor module to structure the natural language query as a respective prompt specifically formulated for the corresponding business LLM to fulfill performance of the corresponding portion of the action, issuing, by the assistant interface, for input to the corresponding business LLM, where the respective prompt is specifically formulated for the corresponding business LLM to fulfill performance of the corresponding portion of the action, and receiving, at the assistant interface, from the corresponding business LLM, corresponding response content conveying details regarding performance of the corresponding portion of the action. The operations also include providing presentation content for output from the user device based on the corresponding response content received from each corresponding business LLM of the selected one or more business LLMs.

[0004] Implementations of the disclosure can include one or more of the following optional features. In some implementations, the operations further include identifying, by the assistant interface, an intermediate list of candidate business LLMs each capable of performing at least a portion of the action. Here, the intermediate list of candidate business LLMs includes a first business LLM and a different second business LLM both capable of performing the same respective portion of the action. In these implementations, the operations further include: prompting, by the assistant interface, the user to select which of the first business LLM or the second business LLM the user prefers for the assistant interface to interact with for performance of the respective portion of the action; and receiving, at the assistant interface, a selection input by the user indicating, the selection input indicating indicating a selection of the first business LLM for the assistant interface to interact with for performance of the respective portion of the action. Here, the one or more business LLMs selected by the assistant interface for the assistant interface to interact with for performance of the action include the first business LLM and do not include the second business LLM.

[0005] In some examples, the operations further include presenting selection information associated with the one or more business LLMs selected by the assistant interface for output from the user device. Here, for each respective business LLM of the one or more business LLMs, the selection information indicates: a name of the respective business LLM; and a description of a respective portion of the action that the respective business LLM will perform on behalf of the user. Additionally, one of the one or more business LLMs selected by the assistant interface for the assistant interface to interact with for performance of the action can be selected to perform performance of at least two different respective portions of the action.

[0006] In some examples, the operations further include receiving, at the assistant interface, one or more interoperability configuration inputs, where each interoperability configuration input specifies one or more candidate business LLMs to add to a preferred set of business LLMs for the assistant interface to interact with to perform the action on behalf of the user. Here, at least one of the one or more business LLMs selected by the assistant interface for the assistant interface to interact with for performance of the action is selected by the assistant interface from the preferred set of business LLMs. In these implementations, at least one of the one or more interoperability configuration inputs received from the user can be provided by the user as unstructured natural language input specifying one or more candidate business LLMs to add to the preferred set of business LLMs.

[0007] In some implementations, accessing the adapter module to construct the natural language query as a respective prompt for a corresponding business LLM among the one or more business LLMs includes converting the natural language query into a respective natural language prompt that allows the assistant interface to communicate with the corresponding business LLM via natural language. Here, the respective natural language prompt includes a respective prompt that is specifically formulated for the corresponding business LLM to perform execution of a corresponding portion of the action.

[0008] In some examples, accessing the adapter module to construct the natural language query as a respective prompt for a corresponding business LLM among the one or more business LLMs includes converting the natural language query into a respective soft prompt that is specifically formulated to include a prompt structure that is published by the corresponding business LLM. In some implementations, accessing the adapter module to construct the natural language query as a respective prompt for a corresponding business LLM among the one or more business LLMs includes: accessing the adapter module to view previous prompts that were issued to the corresponding business LLM, where the previous prompts were constructed from previous natural language queries; and constructing the natural language query as the respective prompt based on a structure of any of the previous prompts that were issued to the corresponding business LLM that were deemed successful.

[0009] In some examples, for a corresponding business LLM among the one or more corresponding business LLMs that is selected by the assistant interface, accessing the adapter module includes processing the natural language query using a respective adaptation model associated with the corresponding business LLM to generate a respective prompt that is specifically formatted for interacting with the corresponding business LLM. Here, the respective adaptation model is trained to construct prompts for interacting with the corresponding business LLM from natural language input. In these examples, the assistant interface can include a personal LLM having an encoder network and a decoder network, and the respective adaptation model associated with the corresponding business LLM can include a prefix to the decoder of the assistant LLM. Additionally, the operation can further include activating the respective adaptation model associated with the corresponding business LLM in response to selecting the at least one corresponding business LLM. For another corresponding business LLM among the one or more corresponding business LLMs that is selected by the assistant interface, accessing the adapter module can include processing the natural language query using another respective adaptation model associated with the other corresponding business LLM to generate a respective prompt that is specifically formatted for interacting with the other corresponding business LLM. Here, the respective adaptation model associated with the other corresponding business LLM is trained to construct prompts for interacting with the corresponding business LLM from natural language input.

[0010] In some implementations, one or more corresponding business LLMs selected by the assistant interface include a first business LLM and a second business LLM, wherein the first business LLM is operated by a first cloud provider, and the second business LLM is operated by a second cloud provider different from the first cloud provider; and the corresponding prompts constructed based on natural language queries for the first business LLM are formatted differently from the corresponding prompts constructed based on natural language queries for the second business LLM. In some examples, the operation further includes: after providing presentation content for output from a user device, receiving user feedback indicating user dissatisfaction with the corresponding response content that conveys details about the execution of a corresponding part of an action performed by a corresponding business LLM among the one or more corresponding business LLMs; determining a loss based on the corresponding response content and the user feedback indicating user dissatisfaction with the corresponding response content; and accessing an adapter module to fine-tune the corresponding prompts specifically formulated for a corresponding business LLM among the one or more corresponding business LLMs by updating the gradient of the corresponding prompts based on the loss while keeping the parameters of the corresponding business LLMs fixed.

[0011] Another aspect of this disclosure provides a system comprising: data processing hardware; and memory hardware communicating with the data processing hardware and storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving, at an assistant interface, a natural language query input by a user into a user device, the natural language query specifying an action to be performed by the assistant interface on behalf of the user. The operations further include selecting, by the assistant interface, one or more business large language models (LLMs) for interaction with the assistant interface to perform the execution of the action, wherein each of the one or more business LLMs is selected to perform the execution of a corresponding portion of the action. For each corresponding business LLM selected by the assistant interface, the operations include: accessing an adapter module by the assistant interface to construct the natural language query as a corresponding prompt specifically tailored for the corresponding business LLM to perform the execution of the corresponding portion of the action; publishing by the assistant interface for input to the corresponding business LLM, wherein the corresponding prompt is specifically tailored for the corresponding business LLM to perform the execution of the corresponding portion of the action; and receiving, at the assistant interface, a corresponding response content from the corresponding business LLM conveying details about the execution of the corresponding portion of the action. The operation also includes providing presentation content based on the corresponding response content received from each of the selected service LLMs for output from the user device.

[0012] This aspect may include one or more of the following optional features. In some implementations, the operation further includes: an intermediate list of candidate service LLMs, each capable of performing at least a portion of an action, identified by the assistant interface. Here, the intermediate list of candidate service LLMs includes a first service LLM capable of performing the same corresponding portion of the action and different second service LLMs. In these implementations, the operation further includes: prompting the user through the assistant interface to select either the first service LLM or the second service LLM preferred by the user for interaction with the assistant interface to perform the corresponding portion of the action; and receiving a selection input indication at the assistant interface, the selection input indication indicating the selection of the first service LLM for interaction with the assistant interface to perform the corresponding portion of the action. Here, the one or more service LLMs selected by the assistant interface for interaction with the assistant interface to perform the action include the first service LLM and do not include the second service LLM.

[0013] In some examples, the operation further includes presenting selection information associated with one or more service LLMs selected by the assistant interface for output from the user device. Here, for each corresponding service LLM among the one or more service LLMs, the selection information indicates: the name of the corresponding service LLM; and a description of the corresponding portion of the action that the corresponding service LLM will represent the user in performing. Additionally, one of the one or more service LLMs selected by the assistant interface for the assistant interface to interact with in order to perform the execution of an action can be selected to perform the execution of at least two different corresponding portions of the action.

[0014] In some examples, the operation further includes receiving one or more interoperability configuration inputs at the assistant interface, wherein each interoperability configuration input specifies one or more candidate business LLMs to be added to a preferred group of business LLMs for the assistant interface to interact with in order to perform actions on behalf of the user. Here, at least one business LLM among the one or more business LLMs selected by the assistant interface for the assistant interface to interact with in order to perform the execution of actions is selected by the assistant interface from the preferred group of business LLMs. In these examples, at least one of the one or more interoperability configuration inputs received from the user may be provided by the user as unstructured natural language input specifying one or more candidate business LLMs to be added to the preferred group of business LLMs.

[0015] In some implementations, accessing the adapter module to construct a corresponding prompt for a natural language query for one of one or more business LLMs includes converting the natural language query into a corresponding natural language prompt that allows the assistant interface to communicate with the corresponding business LLM via natural language. Here, the corresponding natural language prompt includes a prompt specifically tailored for the corresponding business LLM to perform the execution of the corresponding part of an action.

[0016] In some examples, accessing the adapter module to construct a natural language query as a corresponding prompt for one of one or more business LLMs includes converting the natural language query into a corresponding soft prompt that is specifically designed to include a prompt structure published by the corresponding business LLM. In some implementations, accessing the adapter module to construct a natural language query as a corresponding prompt for one of one or more business LLMs includes: accessing the adapter module to view previous prompts published to the corresponding business LLM, wherein the previous prompts are constructed based on previous natural language queries; and constructing the natural language query as a corresponding prompt based on the structure of any of the previous prompts that are considered to have been successfully published to the corresponding business LLM.

[0017] In some examples, for one of one or more corresponding business LLMs selected by the assistant interface, the access adaptor module includes processing a natural language query using a corresponding adaptation model associated with the corresponding business LLM to generate a response prompt specifically formatted for interacting with the corresponding business LLM. Here, the corresponding adaptation model is trained to construct a prompt for interacting with the corresponding business LLM based on natural language input. In these examples, the assistant interface may include a personal LLM with an encoder network and a decoder network, and the corresponding adaptation model associated with the corresponding business LLM may include a prefix of the decoder of the assistant LLM. In these examples, the operation may further include activating the corresponding adaptation model associated with the corresponding business LLM in response to selecting at least one corresponding business LLM. In these examples, for another corresponding business LLM among one or more corresponding business LLMs selected by the assistant interface, the access adaptor module may include processing a natural language query using another corresponding adaptation model associated with the other corresponding business LLM to generate a response prompt specifically formatted for interacting with the other corresponding business LLM. Here, a corresponding adaptation model associated with another business LLM is trained to construct prompts for interacting with that corresponding business LLM based on natural language input.

[0018] In some implementations, one or more corresponding business LLMs selected by the assistant interface include a first business LLM and a second business LLM, wherein the first business LLM is operated by a first cloud provider, and the second business LLM is operated by a second cloud provider different from the first cloud provider; and the corresponding prompts constructed based on natural language queries for the first business LLM are formatted differently from the corresponding prompts constructed based on natural language queries for the second business LLM. In some examples, after providing presentation content for output from a user device, the operation further includes: receiving user feedback indicating user dissatisfaction with the corresponding response content, which conveys details about the execution of a corresponding part of an action performed by a corresponding business LLM among the one or more corresponding business LLMs; determining a loss based on the corresponding response content and the user feedback indicating user dissatisfaction with the corresponding response content; and accessing an adapter module to fine-tune the corresponding prompts specifically formulated for a corresponding business LLM among the one or more corresponding business LLMs by updating the gradient of the corresponding prompts based on the loss while keeping the parameters of the corresponding business LLMs fixed.

[0019] Details of one or more implementations of this disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the specification, drawings, and claims. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of an example system used to adapt the assistant interface to interact with the business large language model (LLM) to perform actions on behalf of the user.

[0021] Figure 2 This is a schematic diagram of an example process for configuring the assistant interface to interoperate with the business LLM.

[0022] Figure 3 This is a schematic diagram of an example screen that displays selection information 310 for one or more business LLMs for the corresponding part selected to perform an action.

[0023] Figure 4 This is a flowchart of an example layout for adapting the assistant interface to interact with the business LLM to perform actions on behalf of the user.

[0024] Figure 5 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0025] In the various figures, the same reference numerals indicate the same elements. Detailed Implementation

[0026] Humans can engage in human-computer dialogue using interactive software applications known as "chatbots," "voice bots," "automated assistants," "interactive personal assistants," "intelligent personal assistants," and "conversational agents" via various computing devices. As an example, these chatbots can correspond to machine learning models or combinations of different machine learning models and can be used to perform various tasks on behalf of users.

[0027] Chatbots employing large language models (LLMs) are currently opening up a wide range of applications due to their powerful understanding and generation capabilities on text, image, and / or audio input. These models are also extended to have actuation capabilities through integration mechanisms with various service providers.

[0028] As LLMs become increasingly prevalent, it's clear that not only will users have their own personalized assistant LLMs, but business entities will also develop LLMs as a crucial mechanism for delivering their services to end users in business-to-consumer (B2C) settings. In a B2C setting, companies will have their own versions of LLMs supported by different cloud providers. Here, business entities may create and operate their own business LLMs to provide services on their behalf. Thus, assistant LLMs will interact with the business LLMs on behalf of their users to accomplish tasks.

[0029] However, due to the varying levels of capabilities offered by different business LLMs, the assistant LLM cannot operate all business LLMs in the same way. The implementation described herein relates to an assistant interface capable of handling natural language queries input by a user specifying a particular action the user wants the assistant interface to perform. This assistant interface interacts with one or more business LLMs to perform the specific action by constructing appropriate prompts based on the natural language query for input into one or more business LLMs. More specifically, upon receiving a natural language query input by the user, the implementation describes an assistant interface selecting one or more business LLMs for which it interacts to perform the action. For each of the selected business LLMs, an adapter module is accessed to construct the natural language query as an appropriate prompt specifically tailored for that corresponding business LLM to perform the corresponding part of the action. The assistant interface then publishes the appropriate prompts, constructed based on the natural language query and specifically tailored for the corresponding business LLM to perform the corresponding part of the action, for input into each corresponding business LLM. The implementation of this disclosure also includes an assistant interface receiving corresponding response content from each corresponding service LLM, conveying details about the execution of the corresponding part of the action, and providing presentation content for output from the user device based on the corresponding response content received from each of the selected one or more service LLMs. In more detail below, the selected service LLMs can be chosen from a preferred group of service LLMs pre-configured by the user based on interoperability configuration input.

[0030] Figure 1An example system 100 is shown, comprising an LLM interoperability system 105 that allows an assistant interface 150 to interact with different service LLMs 160 to perform actions on behalf of a user 10 associated with the assistant interface 150. Generally, the user 10 inputs a natural language query 116 via a user device 110 to the assistant interface 150 specifying a particular action that the assistant interface 150 wants to perform on behalf of the user 10, and the assistant interface 150 selects one or more service LLMs 160, 160 an for interaction with it to perform the action. Here, the assistant interface 150 processes the natural language query 116 by performing query interpretation to determine the specific action to be performed. The performance of a specific action may require the execution of multiple parts or sub-actions / tasks that commonly define that specific action. Thus, the assistant interface 150 can select each service LLM 160 to perform the execution of the corresponding part of the action specified by the natural language query 116 input to the assistant interface 150. For each corresponding service LLM 160 selected by the assistant interface 150, the assistant interface 150 issues a corresponding prompt 152 specifically tailored for the corresponding service LLM 160 to perform the corresponding part of an action for input to the corresponding service LLM 160, and receives from the corresponding service LLM 160 a corresponding response content 162 conveying details about the performance of the corresponding part of the action performed by the corresponding service LLM 160. The assistant interface 150 may facilitate multiple interactions with the corresponding service LLM 160, with or without input from the user 10, until the corresponding part of the action has been performed. Based on the corresponding response content 162 received from each corresponding service LLM 160, the assistant interface 150 is configured to provide presentation content 180 for output from the user device 110. The user device 110 may audibly output the presentation content 180 as synthesized speech from an audio output device (e.g., an acoustic speaker) 117. Additionally or alternatively, user device 110 may display graphics, text, and / or other visual information on screen 112 in communication with user device 110 that convey details of the presented content 160.

[0031] System 100 includes a user device 110, a telecomputing system 120, and a network 130. User device 110 includes data processing hardware 113 and memory hardware 114. User device 110 may include, or communicate with, an audio capture device 115 (e.g., an array of one or more microphones) for converting spoken natural language query 116 into corresponding audio data 102 (e.g., electrical signals or digital data) spoken by user 10. Instead of verbal input, user 10 may input a text representation of natural language query 116 via a user interface 150 executed on user device 110. In the scenario where the user speaks natural language query 116 captured by microphone 115 of user device 110, an automatic speech recognition (ASR) system 140 executed on user device 110 or telecomputing system 120 may process the corresponding audio data 102 to generate a transcription of query 116. Here, the transcription conveys natural language query 116 as a text representation for input to assistant interface 150. The ASR system 140 can implement any number and / or type of past, present, or future speech recognition systems, models, and / or methods, including but not limited to end-to-end speech recognition models such as streaming speech recognition models with recurrent neural network-transducer (RNN-T) model architectures, hidden Markov models, acoustic models, articulation models, language models, and / or Naive Bayes classifiers.

[0032] User device 110 can be any computing device capable of communicating with remote computing system 120 via network 130. User device 110 includes, but is not limited to, desktop computing devices and mobile computing devices, such as laptops, tablets, smartphones, smart speakers / displays, digital assistant devices, smart appliances, Internet of Things (IoT) devices, infotainment systems, vehicle infotainment systems, and wearable computing devices (e.g., headphones, smart glasses, and / or watches).

[0033] Remote computing system 120 may be a distributed system with scalable and elastic resources (e.g., a cloud computing environment). Resources include computing resources 123 (e.g., data processing hardware) and / or storage resources 124 (e.g., memory hardware). Additionally or alternatively, remote computing system 120 may be a centralized system. Network 130 may be wired, wireless, or a combination thereof, and may include private networks and / or public networks, such as the Internet.

[0034] Continue to refer to Figure 1The LLM interoperability system 105 includes an ASR system 140, an assistant LLM 150, multiple business LLMs 160, 160an, and a user interface 170. The ASR system 140 may be optional or utilized only if the user 10 prefers verbal input for the natural language query 116 rather than typed input. In some implementations, the LLM interoperability system 105 executes on both the data processing hardware 113 of the user device 110 and the data processing hardware 123 of the remote computing system 120. For example, one or more components of the interoperability system 105 may execute on the data processing hardware 113 of the user device 110, while one or more other components of the interoperability system 105 may execute on the remote computing system 120. Although not shown, the business LLM 160 may execute on different remote computing systems depending on the service provider operating the business LLM 160. Thus, the assistant interface 150 can interact with the interoperability system 105 across a diverse set of remote computing systems operated by different providers, performing different business LLMs 160.

[0035] A specific business may develop and offer its own version of a business LLM160, supported by a specific cloud service provider. Here, a first business LLM provided by a first business entity can be contracted through a first cloud service provider, and a second business LLM provided by a second business entity can be contracted through a second cloud service provider. In this example, the first business LLM may include a first pre-trained LLM (e.g., Google Cloud LLM) customized for the first business entity, which has a significantly larger number of LLM parameters (e.g., 540 billion parameters) than a second business LLM (e.g., 11 billion parameters) that includes a second pre-trained LLM customized for the second business entity (e.g., Ascenty LLM). Here, the first business entity may provide training samples including training cues paired with corresponding ground truth responses to create the first business LLM as a customized version of the first pre-trained LLM. Similarly, the second business entity may provide its own training samples including training cues paired with corresponding ground truth responses to create the second LLM as a customized version of the second pre-trained LLM.

[0036] The training, or more specifically the customization process used to create a business LLM, can result in each business entity having different LLM capabilities. For example, the customization process can include various levels of customization for the resulting business LLM with disparate capabilities. While the number of LLM parameters provided by each particular cloud service provider, the available plugins, and / or application programming interfaces (APIs) can constrain the LLM capabilities of the resulting business LLM, various training techniques such as fine-tuning, cue tuning, and / or reinforcement learning (RL) fine-tuning can provide additional levels of customization for the LLM capabilities provided by the business LLM. For example, a business entity can use few-shot learning to create a customized version of an existing pre-trained LLM provided by a cloud service provider. On the other hand, cue tuning can be implemented to learn how to create soft cues that guide an existing pre-trained LLM provided by a cloud service provider to provide responses customized for the business entity, while the parameters of the pre-trained LLM remain fixed. In other words, a business entity can fine-tune (e.g., few-shot examples, soft cues tuned via prompts, and / or individual adapter weights) inputs from an existing pre-trained LLM that is already capable of being used for more generalized dialogue and / or for fine-tuning cues input into that existing pre-trained LLM, without needing to fine-tune the pre-trained LLM itself. This disclosure is not limited to how business LLMs are created and customized. This disclosure relates to techniques for implementing interoperability between an assistant interface 150 and multiple different business LLMs 160 spanning a diverse set of LLM capabilities.

[0037] In some implementations, the assistant interface 150 is a personalized assistant LLM 150 for user 10. The assistant LLM 150 can act as a personal chatbot capable of conversing with user 10 in natural language and performing tasks / actions on behalf of the user. In some examples, the assistant LLM 150 includes instances of Bard, LaMDA, BERT, Meena, ChatGPT, or any other previously trained LLM. These previously trained LLMs have been previously trained on a large and diverse amount of data and are able to engage in corresponding conversations with the user in a natural and intuitive way. However, these LLMs have multiple machine learning (ML) layers and hundreds of millions to trillions of ML parameters. Therefore, in implementations where the assistant LLM 150 is an instance of a previously trained LLM that is locally fine-tuned at user device 110, the previously trained LLM obtained and fine-tuned to provide a personalized assistant LLM 150 for user 10 can be a sparse version of the previously trained LLM. Conversely, in an implementation where the assistant LLM is an instance of a previously trained LLM that is remotely fine-tuned from a client device, the previously trained LLM obtained and fine-tuned to provide assistant LLM 150 can be a denser version of the previously trained LLM. Compared to the virtually unlimited resources of the remote system, due to various hardware and / or software constraints at user device 110, the sparser version of the previously trained LLM can have fewer ML layers, fewer ML parameters, masked weights, and / or other sparsity aspects to reduce the size of the previously trained LLM.

[0038] The assistant LLM allows unstructured, free-form natural language input that conveys details of the action / task to be performed, but does not define any corresponding dialogue state diagram (e.g., no dialogue state or dialogue state transitions are defined). In response to receiving a query 116 as unstructured, free-form natural language input, the assistant LLM 150 interacts with the business LLM 160, which is associated with the business entity and capable of performing the action / task specified by query 116, by constructing a prompt for input to the business LLM 160 that causes the business LLM 160 to perform the action / task on behalf of the user 10. The business LLM 160 may return response content 162 to the assistant LLM 150 conveying details of the action / task performed, and the assistant LLM 150 may provide presentation content 180 for output from the user device 10, which acts as a response to query 116 by conveying information associated with the response content 162 returned from one or more business LLMs 160. Assistant LLM 150 can determine presentation content 180 based on the response content 162 returned by each business LLM 160 that performs the corresponding part of the action on behalf of user 10. Further, presentation content 180 may include, for example, the corresponding result of one or more tasks performed by the business LLM, a corresponding summary of the corresponding task, and / or other content. Continuing the example above, Assistant LLM 150 can also perform actions or parts of actions on behalf of user 10 without needing to interact with any business LLM 160.

[0039] In other implementations, assistant interface 150 includes a conventional virtual digital assistant that does not utilize LLM functionality but can use heuristics / rules to interoperate with a business LLM for performing actions on behalf of user 10. For simplicity, this disclosure refers to assistant interface 150 as assistant LLM 150; however, assistant interface 150 of this disclosure may include a conventional virtual digital assistant.

[0040] A business LLM 160, which can be used by the assistant LLM 150 to interact with and perform actions on behalf of user 10, can be pre-configured based on interoperability configuration inputs 202 received by the assistant LLM 150. Each interoperability configuration input 202 can specify one or more candidate business LLMs to be added to a preferred group of business LLMs for the assistant LLM 150 to interact with and perform actions on behalf of user 10. Here, the interoperability configuration input 202 can cause the assistant LLM 150 to send an interoperability request 250 to the business LLM 160 requesting that the business LLM 160 interact with the assistant LLM 150. The business LLM 160 or its associated entity can return an adaptation package 260 to the assistant LLM 150, which provides details for the assistant LLM 150 to optimally adapt when interoperating with the business LLM 160, thereby most effectively realizing the intent of user 10. The assistant LLM 150 may include or communicate with an adapter module 210, which receives an adaptation packet 260 for configuring the assistant LLM 150 to be adapted for interoperability with each service LLM 160.

[0041] Figure 2 A schematic diagram 200 illustrates an example configuration process performed by Assistant LLM 150 to configure Assistant LLM 150 for interoperability with Example Business LLM 160. Assistant LLM 150 can perform the configuration process for each Business LLM 160 with which Assistant LLM 150 wants to interoperate. In some examples, the interoperability configuration input 202 received by Assistant LLM 150 includes a natural language configuration request entered by User 10, explicitly specifying one or more candidate Business LLM 160s to be added to a preferred group of Business LLMs. For example, the natural language configuration request might state, “I’d like to order from eat.ch most of my dishes, except Indian ones for which I’d ​​like to use smood.ch.” Here, the assistant LLM 150 can be configured to convert a natural language configuration request into a configuration by sending a configuration request 250 to the corresponding service LLM 160 provided by eat.ch and smood.ch, thereby allowing the service LLM 160 to return a corresponding adaptation package 260.

[0042] In some additional examples, the interoperability configuration input 202 received by the assistant LLM 150 includes user preferences that may indicate services preferred by user 10, services used by the user based on user history, user feedback, and / or applications installed on the user's device. For example, the assistant LLM 150 may know that user 10 always books flights with Delta Airlines and collects Delta Airlines reward points via a dedicated credit card. Furthermore, the interoperability configuration input 202 may instruct a discovery search from user 10 that requests the assistant LLM 150 to search for a business LLM with the capabilities of the services specified in the discovery search. Here, the business LLM 160 may have memory enhancements utilizing the external data storage of service 165, which user 10 can query or search by feeding discovery prompts to the assistant LLM 150.

[0043] In some additional examples, interoperability configuration input 202 indicates a canonical service LLM 160 associated with a popular service LLM 160 used to perform common tasks among a group of users. If the canonical service LLM 160 is associated with a service entity that has already been authorized by the user, the canonical service LLM 160 can be automatically entered into the preferred group of candidate service LLMs 160 for the assistant LLM 150. If the user 10 has not yet authorized a service entity associated with the canonical service LLM 160 specified in configuration input 202, the assistant LLM 150 can suggest including the canonical service LLM 160 in the preferred group of candidate service LLMs 160, whereby the user 10 can explicitly select the canonical service LLM 160 to be included in the preferred group via a checkbox displayed by the user interface 170. By the same logic, user 10 can remove any business LLM 150 from the preferred group of candidate business LLMs 160 at any time, for example, by deselecting the associated checkbox for the business LLM that user 10 wants to remove, as displayed by user interface 170. The exemplary business LLMs 160 deemed available may depend on the geographic region where user 10 is located. For example, a business LLM provided by a food delivery service operating only in the United States would not be available to user 10 residing in the United Kingdom.

[0044] Continue to refer to Figure 2The adaptation package 260 returned from the business LLM 160 can demonstrate the LLM capabilities of the business LLM 160 to inform the assistant LLM 150 how best to adapt when interoperating with the business LLM 150. The adaptation package 260 may include an adaptation model 212, prompt examples, natural language constraints, the size of the business LLM (e.g., the number of parameters), and / or the capabilities of the LLM. The adaptation model 212 may be published by the business LLM 160 and specific to the business LLM 150 so that the assistant LLM 150 can use it to generate prompts that are specifically formatted for interacting with the corresponding business LLM. Here, the entity associated with the business LLM 150 can train the corresponding adaptation model 212 to construct prompts for interacting with the business LLM 160 based on natural language. In some examples, the adaptor module 210 stores multiple adaptation models 212, 212 an, each associated with a corresponding business LLM 160 that the assistant LLM 150 is configured to interoperate with. In these examples, and described in more detail below, the assistant LLM 150 can activate a corresponding adaptation model 212 associated with each business LLM 160 with which the assistant LLM 150 has chosen to interoperate to perform actions on behalf of the user 10. The adaptation model 212, previously trained by the business entity, can be fine-tuned by the assistant LLM 150 based on positive / negative interactions from the user regarding the response content returned from the business LLM according to the previous prompts constructed from the adaptation model 212. In some implementations, the assistant LLM 150 includes an encoder network 151 and a decoder network 153. The encoder network 151 is configured to encode the natural language query 116 into an encoded representation, and the decoder network 153 is configured to decode the encoded representation into a resulting prompt that is specifically formatted for the business LLM to perform the corresponding portion of the action specified by the natural language query 116. In these implementations, the adaptor module 210 can activate the corresponding adaptor model 212 associated with the corresponding service LLM, such that the activated adaptor model 212 includes a prefix to the decoder network 153 of the assistant LLM 150. The adaptor model 212 can act as a sub-model of the assistant LLM 150, thereby biasing how to construct hints for interaction with the corresponding service LLM 160.

[0045] In some cases, adapter module 210 uses prompt examples included in adaptation package 260 that convey the prompt structure published by the business LLM. Here, adapter module 210 can use the prompt examples to adapt assistant LLM 150 to convert natural language queries 116 input to assistant LLM into corresponding soft prompts specifically designed to include the prompt structure conveyed by the prompt examples. Soft prompts may include digital representations (e.g., vectors) that can be provided as input to the business LLM, rather than natural language prompts. The prompt examples included in adaptation package 260 may include few-sample examples operable to fine-tune business LLM 160 to perform a specific task or provide response content 162 within a specific domain.

[0046] The adapter module 210 may additionally or alternatively use natural language constraints included in the adaptation package 260 to rephrase the natural language query 116 into a format suitable for prompting the business LLM 160. Here, the natural language constraints provide constraints regarding how the assistant LLM 150 and the business LLM 160 communicate via natural language. Thus, the natural language constraints allow the adapter module 210 to convert the natural language query into a corresponding natural language prompt that allows the assistant LLM 150 to communicate with the corresponding business LLM 160 via natural language. For example, the business LLM 160 may require the natural language prompt to include terms spelled in a certain way, or must reduce the content included in the original natural language query 116. In some examples, the assistant LLM 150 and / or the adapter module 210 use natural language constraints to generate templates for converting natural language queries entered by the user into the assistant LLM 150 into natural language prompts specifically formatted for the business LLM 160.

[0047] The adapter module 210 can receive an adaptation package 260 for configuring the assistant LLM 150 to interact with the business LLM 160. Notably, the adapter module 210 configures the assistant LLM 160 to translate natural language queries input to the assistant LLM 160 into corresponding prompts that are specially formatted for the business LLM 150 to execute the corresponding part of the action specified by the natural language query. Based on the principle that the business LLM 160 includes a large and diverse set of LLM capabilities and is provided by multiple different cloud service providers, the assistant LLM 150 must access the adapter module 210 to determine how to interoperate with each business LLM 160 on a case-by-case basis. For example, for two different business LLMs 150 that can each book flights, a prompt generated by the assistant LLM 150 for calling one of the business LLMs to book a flight may not be suitable for calling the other business LLM to book the same flight. In other words, the Assistant LLM 150 access adapter module 210 is used to adapt the Assistant LLM 150 to construct prompts specific to the business LLM with which the Assistant LLM 150 is interoperating in a given instance.

[0048] exist Figure 1In the example, Assistant LLM 150 receives a natural language query 116 input by a user (e.g., via voice captured by the microphone of user device 10 or via text input via user interface 170 executed on user device 110) stating “I'm traveling to Detroit from July 24–31. Book a roundtrip flight and show me hotels in downtown”. Here, query 116 specifies the action of booking a round-trip flight and returning available hotels in downtown Detroit. Assistant LLM 150 processes natural language query 116 to identify an intermediate list of candidate business LLMs 160C, each capable of performing at least a portion of the action. Assistant LLM 150 may further resolve ambiguous details required to perform the action that are not explicitly stated in natural language query 116. In the example, Assistant LLM 150 may determine, based on the knowledge that user 10 lives near Dallas, that the departure flight should depart from Dallas and the return flight should arrive in Dallas. In the example shown, the intermediate list of candidate business LLM 160C includes a first business LLM 160a associated with an airline booking entity and capable of booking / reserving flights instead of hotels, a second business LLM 160b associated with a travel agency entity and capable of booking / reserving both flights and hotels, and a third business LLM 160c associated with an accommodation entity and capable of booking / reserving hotels without interacting with the flight provider. It is noteworthy that in this example, both the first business LLM 160a and the different second business LLM 160b are capable of performing the same corresponding part of the action of booking / reserving flights. Here, the assistant LLM 150 may issue a user prompt 52 to the user 10, prompting the user 10 to choose which of the first business LLM 160a or the second business LLM 160b the user prefers for interacting with the assistant LLM 150 to perform the corresponding part of the action (i.e., booking a round-trip flight to Detroit). The assistant interface 170 can audibly or visually present user prompts 52 from user device 10, and the user can provide selection input instructions 54 via the user interface 170 to select the first service LLM 160a for interaction with the assistant LLM. The user 10 can provide selection input instructions 54 to the assistant interface 170 via voice, touch, gesture, or other input methods. Instead of prompting the user 10 via user prompts 52, the assistant LLM 150 can refer to past interactions where the user 10 has provided feedback 56 indicating that the user 10 prefers to use the first service LLM 160a to book flights relative to the second service LLM 150b.Here, the confidence level of the first business LLM 150a can be increased, and the assistant LLM 150 can select the first business LLM 160a without additional input from user 10.

[0049] When identifying which business LLMs are capable of performing at least a portion of the actions specified by the natural language query 116, the assistant LLM 150 may consider the richness and detail of the natural language query 116. For example, a smaller business LLM (e.g., approximately 11 billion parameters) may only be able to handle the constructed query, while a larger business LLM may be able to handle queries with a richer language or knowledge base.

[0050] Continuing the example, Assistant LLM 150 selects one or more Business LLMs 160 to interact with in order to perform an action, wherein each of the one or more Business LLMs 160 is selected to perform the corresponding part of the action. Here, First Business LLM 160a and Third Business LLM 160c are... Figure 1 The image is shaded to depict the selection of the first business LLM 160a and the second business LLM 160c. The first business LLM 160a can be selected based on user input instruction 54, causing the second business LLM 160b to be excluded from the selected one or more business LLMs 160. Notably, when the assistant LLM 150 is confident in selecting the third business LLM 160c, it can automatically select the third business LLM 160c to perform the corresponding part of the action to show available hotels. For example, the assistant LLM 150 can assign a confidence score for selecting business LLM 160c based on user preferences, past user interactions / feedback, the richness / detail of the underlying natural language query, and / or the LLM capabilities of the business LLM. In this example, the interoperability configuration input 202 may have already instructed the user to only want to stay at hotels belonging to a hotel group. Alternatively, Assistant LLM 150 may still provide notification to user 10 via user interface 170 that the third service LLM 160c has been selected.

[0051] refer to Figure 1 and Figure 3In some implementations, Assistant LLM 150 presents selection information associated with one or more business LLMs for output from the user device. For each corresponding business LLM, the selection information may indicate the name of the corresponding business LLM and a description of the corresponding part of the action that the corresponding business LLM will perform on behalf of the user 10. For example, continuing with the example of selecting a first business LLM 160a to book a round-trip flight and selecting a third business LLM 160c to show available hotels in downtown Detroit, the selection information 310 may include the name 310a of the first business LLM 160a (i.e., Flight LLM affiliated with ABC Airlines) and a description 310b of the part of the action that the first business LLM 160a will perform (i.e., Booking a roundtrip flight from Dallas to Detroit). Similarly, selection information 310 may include the name 310a of the third business LLM 160c (i.e., Hotel LLM affiliated with 123 Hotel Group) and a description 310b of the portion of the action that the third business LLM 160c will perform (i.e., Showing available hotels in downtown Detroit from July 24–31). User interface 170 may allow the user to confirm the selected assistant LLM 150 or reject the selection of any business LLM. Although not shown, a business LLM 160 may be selected to perform at least two different corresponding portions of the action.

[0052] Return to reference Figure 1 For each corresponding business LLM 160 among one or more business LLMs 160 selected by the assistant LLM 150, the assistant LLM 150 can access the adapter module 210 to construct the natural language query 116 into a corresponding prompt 152 specifically tailored for the corresponding business LLM 160 to perform the corresponding part of the action. In one example, when configuring the first business LLM 160a for interoperability with the assistant LLM 150a, the adapter module 210 has an adaptation package 260 based on the first business LLM 160a. Figure 2The natural language constraints and / or prompt examples included in the adaptation package 260 are used to feed natural language prompts to the first service LLM 160. Therefore, the assistant LLM 150 can access the adapter module 210 to convert the natural language query 116 into a corresponding natural language prompt 152, thereby enabling the first service LLM 160a to perform the execution of the corresponding part of the action. In this example, the adapter module 210 has the knowledge to feed soft prompts to the third service LLM 160c based on prompt examples included in the adaptation package 260 provided by the third service LLM 160c. Therefore, the assistant LLM 150 can access the adapter module 210 to convert the natural language query 116 into a corresponding soft prompt 152 that is specifically designed to include a prompt structure published by the third service LLM 160c. The soft prompt 152 may include a numerical representation (e.g., a vector) to be provided as input to the third service LLM 160c.

[0053] In some implementations, for at least one corresponding business LLM 160 among one or more corresponding business LLMs selected by the assistant LLM 150, the adaptor module 210 activates a corresponding adaptation model 212 associated with the corresponding business LLM to generate a corresponding prompt 152 specifically designed for interacting with the corresponding business LLM 160. The assistant LLM 150 may activate the corresponding adaptation model 212 in response to selecting the corresponding business LLM 160. As previously described, the corresponding adaptation model 212 may be trained to construct prompts for interacting with the corresponding business LLM 160 based on natural language input. In these implementations, when the assistant LLM 150 includes an encoder network 151 and a decoder network 153, the corresponding adaptation model 212 associated with the corresponding business LLM 160 may include a prefix of the decoder network 153. For another corresponding business LLM among the selected one or more corresponding business LLMs, the adaptor module 210 can further activate another corresponding adaptation model associated with the other corresponding business LLM to generate a corresponding prompt that is specifically formatted for interacting with the other corresponding business LLM. Here, the corresponding adaptation model 212 associated with the other corresponding business LLM 160 can be trained to construct prompts for interacting with the corresponding business LLM based on natural language input.

[0054] In some additional implementations, the assistant LLM 150 accesses the adapter module 210 to view previous prompts published to the corresponding business LLM 160. Here, the previous prompts are constructed based on previous natural language queries 116. Based on the previous prompts, the assistant LLM 150 can construct the natural language query 116 into a corresponding prompt based on the structure of any previous prompts that were considered successful and published to the corresponding business LLM.

[0055] Continue to refer to Figure 1 For each corresponding business LLM 160 among one or more business LLMs 160 selected by Assistant LLM 150, Assistant LLM 150 can issue a corresponding prompt 152 specifically designed for the corresponding business LLM 160 to perform the corresponding part of the action for input into the corresponding business LLM. Continuing the example, Assistant LLM 150 can issue a first prompt specifically designed for a first business LLM 160a to book a round-trip flight to Detroit, while Assistant LLM 150 can issue a second prompt 152 specifically designed for a third business LLM 160c to provide information on available hotels in downtown Detroit. The first prompt 152 issued to the first business LLM 160a can be formatted / constructed differently from the second prompt 152 issued to the third business LLM 160c.

[0056] After issuing a prompt 152 to each of one or more corresponding business LLMs 160 selected by the assistant LLM 150, the assistant LLM 150 receives a corresponding response content 162 from each corresponding business LLM 160, conveying details about the execution of the corresponding part of the action. Continuing the example, the assistant LLM 150 issues a prompt 152 to a first business LLM 160a for executing the corresponding part of an action including booking a round-trip flight to Detroit, and issues a prompt to a third business LLM 160c for executing the corresponding part of an action including retrieving available hotels in downtown Detroit. Details may indicate whether the corresponding part of the action has been performed or not. In some examples, the assistant LLM 150 and at least one of the corresponding business LLMs 160 engage in multiple interactions between them, issuing prompts 152 and returning corresponding response content 162, until the corresponding part of the action has been performed.

[0057] Based on the corresponding response content 162 received from each of the selected service LLMs, assistant LLM 150 uses user interface 170 to provide presentation content 180 for user 10 as a response to the natural language query 116 initially input to assistant LLM 150 by user 10, for output from user device 110. Assistant LLM 150 may generate presentation content 180 based on all received response content 162. In some scenarios, assistant LLM 150 refines or filters response content 162 to provide personalized presentation content 180 for user 10. In these scenarios, assistant LLM 150 may have knowledge of user preferences or past interactions between user 10 and assistant LLM 150. For example, assistant LLM 150 may filter response content 162 received from a third service LLM 160c so that only hotels offering free European breakfasts are presented to user 10 in the presentation content. Furthermore, Assistant LLM 150 can prompt User 10 at any time to disambiguate between two or more options related to the action to be performed. For example, when User 10 clearly indicates in Natural Language Query 116 that User 10 wants Assistant LLM 150 to book a round-trip flight to Detroit for travel from July 24th to 31st, Assistant LLM 150 can issue a user prompt to determine whether User 10 is willing to depart for Detroit on the evening of July 23rd to ensure that User 10 is in Detroit to fulfill any obligations User 10 may have on July 24th. Assuming User 10 responds affirmatively to Prompt 52, Assistant LLM 150 can create and issue a corresponding prompt 152 to First Business LLM 160c to enable First Business LLM 160c to book / reserve a flight from Dallas to Detroit on the evening of July 23rd. Furthermore, the positive response from user 10, indicating that user 10 wants to arrive in Detroit on July 23 instead of July 24, causes assistant LLM 150 to refine the corresponding prompt 152 constructed for third service LLM 160c, such that the corresponding prompt 152 instructs third service LLM 160c to search for hotels available in downtown Detroit from July 23 (instead of July 24) to July 31.

[0058] User interface 170 can audibly output presentation content 180 as a synthesized speech representation conveying details of actions performed in response to natural language query 116. Here, user interface 170 can access a text-to-speech (TTS) system (not shown) that converts the text representation of presentation content 180 output by assistant LLM 150 into a synthesized speech representation. The TTS system is not limited and may include a TTS model and a vocoder. Continuing the example, user interface 170 can provide the synthesized speech representation of presentation content 180 for audible output from acoustic speaker 117 of user device 10, including “Here are the details for the flight I booked to Detroit. The following hotels in downtown are available during your stay.” Additionally or alternatively, Assistant LLM 150 may provide a visual or graphical representation of Presentation Content 180 for output from User Device 10 by displaying text and / or graphics on Screen 112 of User Device 112. In some examples, providing a visual or graphical representation of Presentation Content 180 for output complements the synthesized speech representation of Presentation Content 180. For example, User Interface 170 may graphically display flight details for each of the departure and return flights booked by First Business LLM 160, whereby User 10 can select graphics to determine more detailed information about the departure and return flights. Additionally, User Interface 170 may graphically display a list of all available hotels in the city center, whereby User 10 can interact with the graphical representation of each available hotel 10 to view the accommodations offered, location, price, etc. User 10 can then definitively select (e.g., by selecting a graphical element) which hotel in the list of available hotels User 10 wants Assistant LLM 150 to reserve on behalf of User 10. Assistant LLM 150 can continue by creating a new prompt to issue to Third Business LLM 160c, which instructs Third Business LLM 160c to book / reserve a hotel selected by User 10.

[0059] After providing presentation content 180, assistant LLM 150 can determine whether the action was successfully performed based on user feedback 56. In some examples, assistant LLM 150 receives user feedback 56 instructing user 10 to perform an action unrelated to the previously entered natural language query 116. Here, assistant LLM 150 can infer that user 10 is satisfied with presentation content 180 and mark the interaction between assistant LLM 150 and each of one or more corresponding business LLMs selected to perform the action as successful. In some examples, assistant LLM 150 stores each successful interaction instance as a positive example, which includes any combination of the natural language query 116 input to assistant LLM 150, the business LLM selected to perform the action, the corresponding prompt 152 created and published to the business LLM, the corresponding response content 162, and presentation content 180.

[0060] In other examples, Assistant LLM 150 receives user feedback 56 that instructs corrections or follow-up queries on presented content 180 to perform additional actions related to the original action specified in the initial natural language query 116. For example, User 10 could follow up with another natural language query stating “Book dinner for the first night I arrive there,” which would cause Assistant LLM 150 to interact with additional business LLMs to reserve the dinner and also store the initial interaction as a successful interaction instance. On the other hand, user feedback 56 instructing corrections to presented content 180 could be used to negate a previous action and cause Assistant LLM 150 to mark previous interactions between Assistant LLM 150 and each of one or more corresponding business LLMs selected to perform the action as unsuccessful. As an example, User 10 could provide feedback 56 stating “This is not suiting my plans, please redo this booking with new provider Y.” In some examples, the assistant LLM 150 stores each successful interaction instance as a negative example, which includes any combination of the natural language query 116 input to the assistant LLM 150, the corresponding part of the business LLM selected to perform the action, the corresponding prompt 152 created and published to the business LLM, the corresponding response content 162, and the presentation content 180.

[0061] In some implementations, the assistant LLM 150 uses stored positive examples associated with successful interactions between the assistant LLM 150 and the corresponding business LLM 160 to further refine future interactions between the assistant LLM 150 and the corresponding business LLM 160, in order to perform the same or similar actions more efficiently in the future. In these implementations, the business LLM 160 involved in one of the interactions with the assistant LLM 150 can provide a training pattern in which the corresponding adaptation model 212 associated with the business LLM 160 can be tuned, or the corresponding prompt 152 generated by the corresponding adaptation model 212 for input to the business LLM 160 can be prompted to reduce or at least better customize the interaction between the assistant LLM 150 and the business LLM 160.

[0062] In a more insightful approach, the assistant LLM 150 can identify and associate stored positive examples with natural language queries 116 similar to the current natural language query 116 input to the assistant LLM 150. These stored positive examples can assist the assistant LLM 150 in selecting previously successful business LLMs and allow the assistant LLM 150 to construct prompts 152 similar to those used in the identified stored positive examples.

[0063] In some additional implementations, the assistant LLM 150 uses stored negative examples associated with unsuccessful interactions between the assistant LLM 150 and the corresponding business LLM 160 to refine the selection process for how the assistant LLM 150 selects the business LLM 160 and / or how the assistant LLM 150 creates and issues prompts 152 to the business LLM 160. For example, a negative example determined based on user feedback 56 can indicate that one of the business LLMs selected to perform the corresponding part of the action is unsuitable. Further, a negative example can indicate that the corresponding prompt 152 constructed based on the natural language query 116 is incorrect and causes one of the business LLMs 160 to fail to perform the execution of the corresponding part of the action.

[0064] It is worth noting that the assistant LLM 150 can use stored positive and negative examples to fine-tune the prompts published to the business LLM 160 using reinforcement learning (RLHF) based on human feedback. Here, the interaction between the assistant LLM 150 and the business LLM 160 can be improved via reinforcement learning (RL) optimization techniques. For the interaction between the assistant LLM 150 and a specific business LLM 160, RLHF can optimize how the assistant LLM 150 interacts with the specific business LLM 160 in the future, where the parameters of the assistant LLM 150 are tuned, while the parameters of the business LLM 160 remain fixed. In the same concept, the parameters of the corresponding adaptation model 212 associated with the specific business LLM 160 can be tuned / updated to guide the adaptation model 212 to learn and construct prompts published to the specific business LLM 160 that result in a more suitable response content 162 for the user 10.

[0065] After providing presentation content 180, assistant LLM 150 can receive user feedback 56 indicating user dissatisfaction with the corresponding response content 162 returned by one of the corresponding business LLMs 160. Here, interactions with assistant LLM 150 and the corresponding business LLM 160 can be stored as negative examples, allowing assistant LLM 160 to determine a loss based on the corresponding response content 162 and the user feedback 56 indicating user dissatisfaction with the corresponding response content 162. In some examples, assistant LLM 150 applies RLHF by accessing adaptor module 210 to fine-tune the corresponding cue 152 (and / or corresponding adaptation model 212, if available) specifically designed for the corresponding business LLM 160 by updating the gradient of the corresponding cue based on the loss while keeping the parameters of the corresponding business LLM fixed.

[0066] Figure 4 This is a flowchart illustrating an example layout of a method 400 for adapting an assistant LLM 150 to interact with a business LLM 160 to perform actions on behalf of a user 10 associated with the assistant LLM 150. Method 400 may be based on data stored in memory hardware 520. Figure 5 Instructions on the data processing hardware 510 ( Figure 5The data processing hardware 510 includes data processing hardware 113 of user device 110, and memory hardware 520 includes memory hardware 114 of user device 110. In other examples, data processing hardware 510 includes data processing hardware 123 of remote computing system 120, and memory hardware 520 includes data processing hardware 124 of remote computing system 120. In operation 402, method 400 includes receiving a natural language query input by a user to the user device at an assistant interface. The natural language query specifies an action to be performed by the assistant interface on behalf of the user.

[0067] In operation 404, method 400 includes selecting one or more business large language models (LLMs) by the assistant interface for interaction with the assistant interface to perform the execution of an action. Here, each of the one or more business LLMs is selected to perform the execution of the corresponding part of the action.

[0068] For each corresponding business LLM among one or more business LLMs selected by the assistant interface, operations 406, 408, and 410 are performed. In operation 406, method 400 includes accessing an adapter module via the assistant interface to construct a natural language query as a response specifically tailored for the corresponding business LLM to perform the execution of the corresponding part of an action. In operation 408, method 400 includes issuing a response specifically tailored for the corresponding business LLM to perform the execution of the corresponding part of an action via the assistant interface for input into the corresponding business LLM. In operation 410, method 400 includes receiving corresponding response content at the assistant interface from the corresponding business LLM conveying details about the execution of the corresponding part of the action. In operation 412, method 400 includes providing presentation content for output from the user device based on the corresponding response content received from each corresponding business LLM among the selected one or more business LLMs.

[0069] A software application (i.e., a software resource) can refer to computer software that instructs a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0070] Non-transitory memory can be a physical device used for temporary or permanent storage of programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0071] Figure 5 This is a schematic diagram of an example computing device 500 that can be used to implement the systems and methods described in this document. The computing device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementations of the invention described and / or claimed in this document.

[0072] Computing device 500 includes a processor 510, memory 520, storage device 530, a high-speed interface / controller 540 connected to memory 520 and high-speed expansion port 550, and a low-speed interface / controller 560 connected to low-speed bus 570 and storage device 530. Each of components 510, 520, 530, 540, 550, and 560 is interconnected using various buses and can be mounted on a common motherboard or otherwise. Processor 510 can process instructions for execution within computing device 500, including instructions stored in memory 520 or storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 580 coupled to high-speed interface 540. In other implementations, multiple processors and / or multiple buses, as well as multiple memories and various types of memory, can be used as appropriate. Moreover, multiple computing devices 500 can be connected, with each device providing a portion of the necessary operation (e.g., as a server group, blade server cluster, or multiprocessor system).

[0073] Memory 520 stores information non-temporarily within computing device 500. Memory 520 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-temporary memory 520 may be a physical means for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0074] Storage device 530 provides mass storage for computing device 500. In some implementations, storage device 530 is a computer-readable medium. In various implementations, storage device 530 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or device array (including devices arranged in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.

[0075] High-speed controller 540 manages bandwidth-intensive operations of computing device 500, while low-speed controller 560 manages lower bandwidth-intensive operations. This assignment of responsibilities is merely exemplary. In some implementations, high-speed controller 540 is coupled to memory 520, display 580 (e.g., via a graphics processor or accelerator), and high-speed expansion port 550 which can accept various expansion cards (not shown). In some implementations, low-speed controller 560 is coupled to storage device 530 and low-speed expansion port 590. Low-speed expansion port 590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled, for example, via a network adapter to one or more input / output devices such as a keyboard, pointing device, scanner, or a networking device such as a switch or router.

[0076] The computing device 500 can be implemented in a variety of different forms, as shown in the figure. For example, the computing device can be implemented as a standard server 500a or multiple times in a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0077] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system, which includes at least one programmable processor, which may be dedicated or general-purpose and is coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transfer data and instructions to the storage system, at least one input device, and at least one output device.

[0078] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0079] The processes and logic flows described in this specification can be executed by one or more programmable processors, also known as data processing hardware, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to said mass storage device, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry systems.

[0080] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen) and possibly a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0081] Various implementations have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other implementations are within the scope of the following claims.

Claims

1. A computer-implemented method (400) executed on data processing hardware (510), the computer-implemented method causing the data processing hardware (510) to perform operations, the operations including: The assistant interface (150) receives a natural language query (116) input by the user into the user device, the natural language query (116) specifying an action to be performed by the assistant interface (150) on behalf of the user; One or more business large language models (160) are selected by the assistant interface (150) for the assistant interface (150) to interact with it to perform the execution of the action, and each of the one or more business LLMs (160) is selected to perform the execution of the corresponding part of the action; For each of the one or more service LLMs (160) selected by the assistant interface (150): The adapter module (210) is accessed via the assistant interface (150) to construct the natural language query (116) as a corresponding prompt (52) specifically designated for the execution of the corresponding part of the corresponding business LLM (160) to perform the action; The assistant interface (150) issues the corresponding prompt (52) specifically designed for the corresponding business LLM (160) to perform the corresponding part of the action for input into the corresponding business LLM (160); as well as At the assistant interface (150), a corresponding response content (162) is received from the corresponding business LLM (160) conveying details about the execution of the corresponding part of the action; and Presentation content (116) is provided based on the corresponding response content (162) received from each of the selected one or more service LLMs (160) for output from the user device.

2. The computer-implemented method (400) as described in claim 1, wherein, The operation further includes: The assistant interface (150) identifies an intermediate list of candidate service LLMs (160) that are each capable of performing at least a portion of the action, wherein the intermediate list of candidate service LLMs (160) includes a first service LLM (160) that is capable of performing the same corresponding portion of the action and a different second service LLM (160). The assistant interface (150) prompts the user to select either the first service LLM (160) or the second service LLM (160) for the execution of the corresponding portion of the action performed by the assistant interface (150) in accordance with the user's preference; and The assistant interface (150) receives a selection input indication (54) from the user, the selection input indication indicating a selection of the first service LLM (160) for the execution of the corresponding part of the interaction between the assistant interface (150) and the user to perform the action. The one or more service LLMs (160) selected by the assistant interface (150) for the assistant interface (150) to interact with it to perform the action include the first service LLM (160) and do not include the second service LLM (160).

3. The computer-implemented method (400) as described in claim 1 or 2, wherein, The operation further includes presenting selection information (310) associated with the one or more service LLMs (160) selected by the assistant interface (150) for output from the user device, wherein for each corresponding service LLM (160) among the one or more service LLMs (160), the selection information (310) indicates: The name (310a) of the corresponding business LLM (160); and The corresponding business LLM (160) will represent the description (310b) of the corresponding part of the action performed by the user.

4. The computer-implemented method (400) as described in any one of claims 1 to 3, wherein, Select one of the one or more business LLMs (160) selected by the assistant interface (150) for the assistant interface (150) to interact with in order to perform the execution of the action to perform the execution of at least two different corresponding parts of the action.

5. The computer-implemented method (400) as described in any one of claims 1 to 4, wherein the operation further comprises: The assistant interface (150) receives one or more interoperability configuration inputs (202), each interoperability configuration input (202) specifying one or more candidate service LLMs (160) to be added to a preferred group of service LLMs (160) used by the assistant interface (150) to perform actions on behalf of the user. At least one of the one or more business LLMs (160) selected by the assistant interface (150) for the assistant interface (150) to interact with in order to perform the action is selected by the assistant interface (150) from the preferred group of business LLMs (160).

6. The computer-implemented method (400) as described in claim 5, wherein, At least one of the one or more interoperability configuration inputs (202) received from the user is provided by the user as unstructured natural language input of the one or more candidate service LLMs (160) specifying the preferred group to be added to the service LLM (160).

7. The computer-implemented method (400) as described in any one of claims 1 to 6, wherein, Accessing the adapter module (210) to construct the natural language query (116) for a corresponding prompt (52) for one of the one or more business LLMs (160) includes converting the natural language query (116) into a corresponding natural language prompt (52) that allows the assistant interface (150) to communicate with the corresponding business LLM (160) via natural language, the corresponding natural language prompt (52) including the prompt (52) specifically formulated for the corresponding business LLM (160) to perform the corresponding part of the action.

8. The computer-implemented method (400) as described in any one of claims 1 to 7, wherein, Accessing the adapter module (210) to construct the natural language query (116) into the corresponding prompt (52) for one of the one or more business LLMs (160) includes converting the natural language query (116) into a corresponding soft prompt (52) specifically designed to include a prompt structure published by the corresponding business LLM (160).

9. The computer-implemented method (400) as described in any one of claims 1 to 8, wherein, Accessing the adapter module (210) to construct the natural language query (116) into a corresponding prompt (52) for one of the one or more business LLMs (160) includes: Access the adapter module (210) to view the previous prompt (152) published to the corresponding business LLM (160), the previous prompt (152) being constructed based on a previous natural language query (116); and Based on the structure of any of the previous prompts (152) that are considered to have been successfully published to the corresponding business LLM (160), the natural language query (116) is constructed as the corresponding prompt (52).

10. The computer-implemented method (400) as described in any one of claims 1 to 9, wherein, For one of the one or more corresponding business LLMs (160) selected by the assistant interface (150), accessing the adapter module (210) includes processing the natural language query (116) using a corresponding adaptation model (212) associated with the corresponding business LLM (160) to generate the corresponding prompt (52) specifically formatted for interacting with the corresponding business LLM (160), the corresponding adaptation model (212) being trained to construct the prompt (152) for interacting with the corresponding business LLM (160) based on the natural language input.

11. The computer-implemented method (400) as described in claim 10, wherein: The assistant interface (150) includes a personal LLM with an encoder network (151) and a decoder network (153); and The corresponding adaptation model (212) associated with the corresponding service LLM (160) includes a prefix to the decoder of the assistant LLM (150).

12. The computer-implemented method (400) as described in claim 10 or 11, wherein, The operation further includes activating the corresponding adaptation model (212) associated with the corresponding service LLM (160) in response to selecting the at least one corresponding service LLM (160).

13. The computer-implemented method (400) as described in any one of claims 10 to 12, wherein, For another corresponding business LLM (160) among the one or more corresponding business LLMs (160) selected by the assistant interface (150), accessing the adapter module (210) includes processing the natural language query (116) using another corresponding adaptation model (212) associated with the other corresponding business LLM (160) to generate the corresponding prompt (52) specifically formatted for interacting with the other corresponding business LLM (160), wherein the corresponding adaptation model (212) associated with the other corresponding business LLM (160) is trained to construct the prompt (52) for interacting with the corresponding business LLM (160) based on the natural language input.

14. The computer-implemented method (400) as described in any one of claims 1 to 13, wherein: The one or more corresponding service LLMs (160) selected by the assistant interface (150) include a first service LLM (160) and a second service LLM (160), wherein the first service LLM (160) is operated by a first cloud provider, and the second service LLM (160) is operated by a second cloud provider different from the first cloud provider; and The corresponding prompts for the first business LLM (160) constructed based on the natural language query (116) are formatted differently from the corresponding prompts for the second business LLM (160) constructed based on the natural language query (116).

15. The computer-implemented method (400) as described in any one of claims 1 to 14, wherein, The operation further includes: after providing the presentation content (116) for output from the user device: Receive user feedback (56) indicating user dissatisfaction with the corresponding response content (162) that conveys details of the execution of the corresponding part of the action performed by the corresponding one of the one or more corresponding service LLMs (160). The loss is determined based on the corresponding response content (162) and the user feedback (56) indicating user dissatisfaction with the corresponding response content (162); and Access the tuner module (210) to fine-tune the corresponding prompt (52) specifically formulated for one of the one or more corresponding service LLMs (160) by updating the gradient of the corresponding prompt (52) based on the loss while the parameters of the corresponding service LLM (160) remain fixed.

16. A system (100) comprising: Data processing hardware; as well as A memory hardware that communicates with and stores instructions on the data processing hardware, the instructions causing the data processing hardware to perform operations when executed on the data processing hardware, the operations including: The assistant interface (150) receives a natural language query (116) input by the user into the user device, the natural language query (116) specifying an action to be performed by the assistant interface (150) on behalf of the user; One or more business large language models (160) are selected by the assistant interface (150) for the assistant interface (150) to interact with it to perform the execution of the action, and each of the one or more business LLMs (160) is selected to perform the execution of the corresponding part of the action; For each of the one or more service LLMs (160) selected by the assistant interface (150): The adapter module (210) is accessed via the assistant interface (150) to construct the natural language query (116) as a corresponding prompt (52) specifically designated for the execution of the corresponding part of the corresponding business LLM (160) to perform the action; The assistant interface (150) issues a corresponding prompt (52) specifically designed for the corresponding business LLM (160) to perform the corresponding part of the action for input into the corresponding business LLM (160); and At the assistant interface (150), a corresponding response content (162) is received from the corresponding business LLM (160) conveying details about the execution of the corresponding part of the action; and Presentation content (116) is provided based on the corresponding response content (162) received from each of the selected one or more service LLMs (160) for output from the user device.

17. The system (100) as claimed in claim 16, wherein, The operation further includes: The assistant interface (150) identifies an intermediate list of candidate service LLMs (160) that are each capable of performing at least a portion of the action, wherein the intermediate list of candidate service LLMs (160) includes a first service LLM (160) that is capable of performing the same corresponding portion of the action and a different second service LLM (160). The assistant interface (150) prompts the user to select either the first service LLM (160) or the second service LLM (160) for the execution of the corresponding portion of the action performed by the assistant interface (150) in accordance with the user's preference; and The assistant interface (150) receives a selection input indication (54) from the user, the selection input indication indicating a selection of the first service LLM (160) for the execution of the corresponding part of the interaction between the assistant interface (150) and the user to perform the action. The one or more service LLMs (160) selected by the assistant interface (150) for the assistant interface (150) to interact with it to perform the action include the first service LLM (160) and do not include the second service LLM (160).

18. The system (100) as claimed in claim 16 or 17, wherein, The operation further includes presenting selection information (310) associated with the one or more service LLMs (160) selected by the assistant interface (150) for output from the user device, wherein for each corresponding service LLM (160) among the one or more service LLMs (160), the selection information (310) indicates: The name (310a) of the corresponding business LLM (160); and The corresponding business LLM (160) will represent the description (310b) of the corresponding part of the action performed by the user.

19. The system (100) as claimed in any one of claims 16 to 18, wherein, Select one of the one or more business LLMs (160) selected by the assistant interface (150) for the assistant interface (150) to interact with in order to perform the execution of the action to perform the execution of at least two different corresponding parts of the action.

20. The system (100) of any one of claims 16 to 19, wherein the operation further comprises: The assistant interface (150) receives one or more interoperability configuration inputs (202), each interoperability configuration input (202) specifying one or more candidate service LLMs (160) to be added to a preferred group of service LLMs (160) used by the assistant interface (150) to perform actions on behalf of the user. At least one of the one or more business LLMs (160) selected by the assistant interface (150) for the assistant interface (150) to interact with in order to perform the action is selected by the assistant interface (150) from the preferred group of business LLMs (160).

21. The system (100) as claimed in claim 20, wherein, At least one of the one or more interoperability configuration inputs (202) received from the user is provided by the user as unstructured natural language input of the one or more candidate service LLMs (160) specifying the preferred group to be added to the service LLM (160).

22. The system (100) as claimed in any one of claims 16 to 21, wherein, Accessing the adapter module (210) to construct the natural language query (116) for a corresponding prompt (52) for one of the one or more business LLMs (160) includes converting the natural language query (116) into a corresponding natural language prompt (52) that allows the assistant interface (150) to communicate with the corresponding business LLM (160) via natural language, the corresponding natural language prompt (52) including the prompt (52) specifically formulated for the corresponding business LLM (160) to perform the corresponding part of the action.

23. The system (100) as claimed in any one of claims 16 to 22, wherein, Accessing the adapter module (210) to construct the natural language query (116) into the corresponding prompt (52) for one of the one or more business LLMs (160) includes converting the natural language query (116) into a corresponding soft prompt (52) specifically designed to include a prompt structure published by the corresponding business LLM (160).

24. The system (100) as claimed in any one of claims 16 to 23, wherein, Accessing the adapter module (210) to construct the natural language query (116) into a corresponding prompt (52) for one of the one or more business LLMs (160) includes: Access the adapter module (210) to view the previous prompt (152) published to the corresponding business LLM (160), the previous prompt (152) being constructed based on a previous natural language query (116); and Based on the structure of any of the previous prompts (152) that are considered to have been successfully published to the corresponding business LLM (160), the natural language query (116) is constructed as the corresponding prompt (52).

25. The system (100) as claimed in any one of claims 16 to 24, wherein, For one of the one or more corresponding business LLMs (160) selected by the assistant interface (150), accessing the adapter module (210) includes processing the natural language query (116) using a corresponding adaptation model (212) associated with the corresponding business LLM (160) to generate the corresponding prompt (52) specifically formatted for interacting with the corresponding business LLM (160), the corresponding adaptation model (212) being trained to construct the prompt (152) for interacting with the corresponding business LLM (160) based on the natural language input.

26. The system (100) as claimed in claim 25, wherein: The assistant interface (150) includes a personal LLM with an encoder network (151) and a decoder network (153); and The corresponding adaptation model (212) associated with the corresponding service LLM (160) includes a prefix to the decoder of the assistant LLM (150).

27. The system (100) as claimed in claim 25 or 26, wherein, The operation further includes activating the corresponding adaptation model (212) associated with the corresponding service LLM (160) in response to selecting the at least one corresponding service LLM (160).

28. The system (100) as claimed in any one of claims 25 to 27, wherein, For another corresponding business LLM (160) among the one or more corresponding business LLMs (160) selected by the assistant interface (150), accessing the adapter module (210) includes processing the natural language query (116) using another corresponding adaptation model (212) associated with the other corresponding business LLM (160) to generate the corresponding prompt (52) specifically formatted for interacting with the other corresponding business LLM (160), wherein the corresponding adaptation model (212) associated with the other corresponding business LLM (160) is trained to construct the prompt (52) for interacting with the corresponding business LLM (160) based on the natural language input.

29. The system (100) as claimed in any one of claims 16 to 28, wherein: The one or more corresponding service LLMs (160) selected by the assistant interface (150) include a first service LLM (160) and a second service LLM (160), wherein the first service LLM (160) is operated by a first cloud provider, and the second service LLM (160) is operated by a second cloud provider different from the first cloud provider; and The corresponding prompts for the first business LLM (160) constructed based on the natural language query (116) are formatted differently from the corresponding prompts for the second business LLM (160) constructed based on the natural language query (116).

30. The system (100) as claimed in any one of claims 16 to 29, wherein, The operation further includes: after providing the presentation content (116) for output from the user device: Receive user feedback (56) indicating user dissatisfaction with the corresponding response content (162) that conveys details of the execution of the corresponding part of the action performed by the corresponding one of the one or more corresponding service LLMs (160). The loss is determined based on the corresponding response content (162) and the user feedback (56) indicating user dissatisfaction with the corresponding response content (162); and Access the tuner module (210) to fine-tune the corresponding prompt (52) specifically formulated for one of the one or more corresponding service LLMs (160) by updating the gradient of the corresponding prompt (52) based on the loss while the parameters of the corresponding service LLM (160) remain fixed.

Images