Automotive assistant with hierarchy having a backbone and domain-specific delegees

Abstract

An automotive assistant in an infotainment system of a vehicle includes a hierarchy that receives a top-level query via a speech interface and that provides a top-level response to the query. The hierarchy includes a top-level agent and a first and second domain with a related domain specific query. Both domains are queried using prompts comprising natural language. The top level response is formulated based on first and second domain specific activities.

Description

AUTOMOTIVE ASSISTANT WITH HIERARCHY HAVING A BACKBONE AND DOMAIN-SPECIFIC DELEGEESCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. provisional application Serial No. 63 / 685,291 filed August 21, 2024, the disclosure of which is hereby incorporated in its entirety by reference herein.TECHNICAL FIELD

[0002] Disclosed herein are systems and methods for an automotive assistant with hierarchy having a backbone and domain-specific delegees.BACKGROUND]0003[ An automobile includes various software-controlled features. Control of these features is carried out by providing structured text, such as API calls.

[0004] In response to a prompt, a large language model outputs text. With appropriate training, nothing prevents this text from being an API call. A suitably trained model is thus able to make an API call. Information received as a result of this API call can then be made available for further use, either as a final answer or as a basis for further reasoning steps.

[0005] The set of all API calls used in a modem vehicle is quite large. Although in principle it is possible to train a model to accommodate a large set of API calls, the model’s reliability appears to decrease as the number of API calls increases.SUMMARY

[0006] Disclosed herein is a system to address the challenge of scaling to thousands of functions and API calls. It does so by redistributing the training load among plural large language models instead of training one model for all API calls. This is implemented by providing a top-level agentand a hierarchy of domain-specific delegees, each of which handles function calls in a particular domain. The top-level agent receives prompts and delegates particular tasks to selected ones of the domain-specific delegees.

[0007] Each domain is a semantic grouping of similar requests. Examples include a navigation domain, a music domain, a general -knowledge domain, and a car-control domain. Each domainspecific delegee makes use of a large language model that has been specifically trained for making API calls in its domain.

[0008] The foregoing architecture offers the advantage of avoiding the diminishing return that arises when one attempts to train a single model to generate too many different kinds of API calls. In this architecture, the top-level agent is an orchestrator or backbone that is trained to recognize which domain-specific delegees to use for delegating the task of generating an API call and how to ask that domain-specific delegee to generate that API call.

[0009] Among the foregoing components, it is only the top-level agent that enjoys direct communication with the occupant. Upon receiving a request from the occupant, the top-level agent determines a domain set. This domain set comprises one or more domains that are pertinent to the request. The top-level agent then communicates with the relevant domain-specific delegees. It does so one after the other until it has completed a suitable response to the original request.

[0010] A significant feature of the architecture is that the top-level agent includes its own large language model. As a result, decisions concerning which domain-specific delegee to use are made within the top-level agent.

[0011] Yet another feature of the architecture is the unidirectional communication between the top-level agent and the domain-specific delegees. While the top-level agent can prompt the domain-specific delegee, the converse is not true. In addition, domain-specific delegees do not communicate with each other.

[0012] Another feature of the architecture is that inputs to different domain-specific delegees differ from each other. Thus, the architecture does not include the possibility of different domain-specific delegees receiving the same inputs and then somehow combining the results from those domain-specific models.(0013] Finally, within the disclosed architecture, the models of the domain-specific delegees remain separate. There is no merger between them.|0014] As used herein, a “domain” refers to a semantic grouping of similar requests. Examples of a “domain” include a navigation domain, a music domain, a vehicle-control domain, and a general -knowledge domain. A domain-specific model for a particular domain is tuned to handle requests within that domain. The domain-specific model is a model that is typically smaller than that of the top-level agent and that has been configured specifically for interaction with a particular application program interface.|0015| It should be noted that the relationship between a domain-specific delegee and a domain is not necessarily a one-to-one relationship. It is possible to have plural domain-specific delegees that are specific to the same domain.

[0016] The foregoing architecture is highly amenable to recursion. In particular, it is possible for a domain-specific delegee to itself have second-level delegees that specialize in subdomains of the domain of that domain-specific delegee of which it is a second-level delegee. In such cases, the domain-specific delegee behaves as a top-level agent with respect to its subdomain-specific delegees. It should be noted that a particular entity can in principle behave as either a domainspecific delegee or as a top-level agent for much the same reason that a particular human being can play the role of either child or parent. The foregoing recursive architecture has no clear bound and can in principle proceed ad infinitum.

[0017] In one aspect, the invention features an automotive assistant in an infotainment system of a vehicle. The infotainment system, which is in electromagnetic communication with loudspeaker and a microphone, also includes a hierarchy that receives a top-level query via a speech interface and that provides a top-level response to the query. The hierarchy itself includes a top-level agent and plural delegees. The top-level agent comprises a top-level model and a prompt builder, the latter being configured to generate a top-level prompt in response to the top-level query and to-0013 -W0-0RG provide the top-level prompt to the top-level model. The top-level model responds to the top-level prompt by providing a first output for generating a domain-specific query. The top-level agent provides the first output to a delegee from the plurality of delegees, receives a response from the delegee, and uses information in the response to provide the top-level response to the user. Each of the delegees in the plurality of delegees is a domain-specific delegee that has been trained to provide a response within a corresponding domain. Each of the responses is thus a domain-specific response.BRIEF DESCRIPTION OF THE DRAWINGS|0018| The embodiments of the present disclosure are pointed out with particularity in the appended claims. However, other features of the various embodiments will become more apparent and will be best understood by referring to the following detailed description in conjunction with the accompany drawings in which:(0019] FIG. 1 shows a vehicle that includes an automotive assistant;

[0020] FIG. 2 shows a hierarchy within the automotive assistant of FIG. 1;

[0021] FIG. 3 shows an output of the top-level model shown in FIG. 2 after having interacted with a first domain-specific delegee; and10022] FIG. 4 shows an output of the top-level model shown in FIG. 2 after having interacted with a second domain-specific delegee.DETAILED DESCRIPTION

[0023] As required, detailed embodiments of the present invention are disclosed herein; however, it is to be understood that the disclosed embodiments are merely exemplary of the invention that may be embodied in various and alternative forms. The figures are not necessarily to scale; some features may be exaggerated or minimized to show details of particular components. Therefore, specific structural and functional details disclosed herein are not to be interpreted aslimiting, but merely as a representative basis forteaching one skilled in the art to variously employ the present invention.(0024] FIG. 1 shows a vehicle 10 having an infotainment system 12 that executes an automotive assistant 14. The infotainment system 12 communicates with a remote server 16 via a network interface 18.100251 During the course of occupying the vehicle 10, an occupant 20 occasionally interacts with the automotive assistant 14 through a speech interface 22. The speech interface 22 permits the occupant 20 and the automotive assistant 14 to exchange information using a microphone 24 and a loudspeaker 26 within the vehicle 10.

[0026] In some cases, the automotive assistant 14 also receives context information 28. This context information 28 comes from an internal context-source 30 or an external context source 32. Examples of such context information 28 include: occupant preferences, stored dialog history, location, and state of the vehicle 10, weather, and traffic. The context information 28 may also be provided by sensors or other vehicle components including GPS, etc. The external context- source 32 may provided context information that includes real-time information such as average vehicle speed of other vehicles. This information may be used to identify a specific domain that is pertinent to the top-level query. In another example, meteorological data may be received from the external context source and the meteorological data may be used to identify a specific domain that is pertinent to the top-level query

[0027] During the course of its interaction with the occupant 20, the automotive assistant 14 interacts with an external application 34a, 34b from an external-application set 36. For example, if the occupant 20 asks whether a particular flight is arriving on time, the automotive assistant 14 will have no way of knowing this on its own. Instead, it identifies an external application 34a, 34b that maintains real-time information on flight arrivals, interacts with that external application 34a, 34b, and relays the result to the occupant 20.

[0028] A typical external application 34a, 34b interacts with other entities through its application-program interface 38a, 38b Therefore, in order to interact with an external application34a, 34b, the automotive assistant 14 uses that external application’s application-program interface 38a, 38b. Each external application 34a, 34b in the external-application set 36 will have its own application-program interface 38a, 38b. This means that the automotive assistant 14 must somehow know how to interact with many different application-program interfaces 38a, 38b.

[0029] In those cases where the external-application set 36 has only a few external applications 34a, 34b, it is practical to construct an automotive assistant 14 that knows the relevant applicationprogram interfaces 38a, 38b. However, as the number of external applications 34a, 34b increases, so too does the task of ensuring that the automotive assistant 14 will be able to use them all effectively. Moreover, the application-program interfaces 38a, 38b are not necessarily static. They are apt to change over time as developers of the external application 38a, 38b add or delete functions or as they make changes to existing functions. A technical problem to be solved is therefore that of easing the burden of ensuring that the automotive assistant 14 will be able to interact with an ever-changing panoply of application-program interfaces 38a, 38b.

[0030] The vehicle 10 may be configured to include various types of components, processors, and memory, and may communicate with a communication network. The communication network may be referred to as a “cloud” and may involve data transfer via wide area and / or local area networks, such as the Internet, Global Positioning System (GPS), cellular networks, Wi-Fi, Bluetooth, etc. The communication network may provide for communication between the vehicle and an external or remote server and / or database, as well as other external applications, systems, vehicles, etc. This communication network may provide navigation, music or other audio, program content, marketing content, internet access, speech recognition, cognitive computing, artificial intelligence, to the vehicle.

[0031] The remote server 16 and any databases may include one or more computer hardware processors coupled to one or more computer storage devices for performing steps of one or more methods as described herein and may enable the vehicle to communicate and exchange information and data with systems and subsystems external to the vehicle and local to or onboard the vehicle. The vehicle 10 and / or the system 12 may include one or more processors configured to perform certain instructions, commands and other routines as described herein. Internal vehicle networksmay also be included, such as a vehicle controller area network (CAN), an Ethernet network, and a media oriented system transfer (MOST), etc. The internal vehicle networks may allow the processor to communicate with other vehicle systems, such as a vehicle modem, a GPS module and / or Global System for Mobile Communication (GSM) module configured to provide current vehicle location and heading information, and various vehicle electronic control units (ECUs) configured to corporate with the processor.

[0032] The processor(s) may execute instructions for certain vehicle applications, including navigation, infotainment, climate control, etc. Specifically the processor may execute instructions for the processes discussed herein, including those with respect to at least FIGs. 4„ 5 and 7. Instructions for the respective vehicle systems may be maintained in a non-volatile manner using a variety of types of computer-readable storage medium. The computer-readable storage medium (also referred to herein as memory, or storage) includes any non-transitory medium (e.g., a tangible medium) that participates in providing instructions or other data that may be read by the processor. Computer-executable instructions may be compiled or interpreted from computer programs created using a variety of programming languages and / or technologies, including, without limitation, and either alone or in combination, Java, C, C++, C#, Objective C, Fortran, Pascal, Java Script, Python, Perl, and PL / structured query language (SQL).|0033] FIG. 2 shows a hierarchy 40 upon which the automotive assistant 14 relies to interact with application-program interfaces 38a, 38b. The hierarchy 40 includes a first level 41a and a second level 41b. The first level comprises a top-level agent 42, which can be referred to as the “backbone” or “orchestrator” of the hierarchy 40. The second level 41b comprises domain-specific del egees 44a, 44b which can be referred to as “domain-specific actions or activities” or “domain-specific tools.”

[0034] In the illustrated embodiment, the first level 41a happens to be the top level of the hierarchy 40. However, the architecture shown in FIG. 2 is inherently recursive. Nothing prevents each domain-specific del egee 44a, 44b from playing the role of a first level 41a and having subdomain-specific delegees under it.

[0035] Each domain-specific delegee 44a, 44b is configured to respond to prompts that are pertinent to a particular domain. Examples of a domain include: a navigation domain, a music domain, a general-knowledge domain, and a car-control domain. In general, a unit of text may have more than one meaning. To ascertain which of several meanings of text is to be applied, additional information is required. This additional information identifies a “domain.” As an example, the term “lay down a track,” when accompanied by information that a “music” domain is to be applied, would be interpreted to mean the act of recording music. In contrast, when accompanied by information that a “railroad” domain is to be applied, the same text would be interpreted to mean the act of laying down pairs of rails for use by a train. It is often useful to divide requests into domains so that a given set of words can be assigned to an appropriate meaning.

[0036] The top-level agent 42 comprises a top-level prompt-builder 46 and a top-level model 48.

[0037] In a preferred embodiment, the top-level model 48 is a large language model, hereafter referred to as a “model.” A model, in this context, receives a textual input and provides an output consistent with its training. In some embodiments, the output of the top-level model 48 is natural language text. However, this need not be the case. For instance, in some embodiments, the output of the top-level model 48 comprises structured text, such as that which forms an API command. This output can then be used to carry out functions in response to a textual input.

[0038] The top-level prompt builder 46 receives a top-level query 50. This top-level query 50 comes from the occupant 20. The top-level prompt builder 46 uses this top-level query 50, the context information 28, and domain information 47c to construct a top-level prompt 52, which it then provides to the top-level model 48.

[0039] The top-level prompt 52 does more than merely echo the top-level query 50. The toplevel prompt 52 urges the top-level model 48 to respond with: information that identifies plural domains that are pertinent to processing the top-level query 50, reasoning steps relied upon to identify those domains, and a plan of action for responding to the top-level query 50.

[0040] FIG. 3 shows an example of a first output 54 that resulted from the top-level prompt 52 that the top-level prompt-builder 46 built from the occupant’s top-level query 50. As is apparent, in a single step, the top-level model 48 has responded by: providing a thought section 56 that analyzes the top-level query 50 into separate and distinct first and second domain-specific tasks; providing an action section 58 that identifies a first domain-specific delegee 44a to carry out the first domain-specific task; and providing an action-input section 60 that discloses how to instigate interaction with that first domain-specific delegee 44a to promote the return of useful information from that first domain-specific delegee 44a.|0041| The action-input section 60 includes an input that serves as a basis for a second query 62 that is to be provided to the domain-specific delegee 44a. Examples of an input include an executable command and a question to be responded to. If the input depends on the context and contains anaphoric relationships, anaphoric pronouns, implicit contextual reference, non-verbal responses, and / or ellipsis, the input is paraphrased based on conversational history to resolve contextual dependencies before using it as input for the domain-specific delegee 44a. Input for the domain-specific delegee 44a should always be understandable as a self-contained question or command.

[0042] It has been found that by forcing the top-level model 48 to provide a thought section 56 that includes reasoning steps to arrive at the action section 58, the top-level model 48 becomes constrained to more reliably produce an effective second prompt 52.|0043| FIG. 4 shows a second output 62 that results in the top-level agent 42 having interacted with the first domain-specific delegee 44a in the manner indicated in the action-input section 60. This second output 62 is structurally similar to that shown in FIG.3.

[0044] Like the first output 54, the second output 62 includes: an observation section 64 that discloses the information acquired from the first domain-specific delegee 44a; another action section 58 identifying a second domain-specific delegee 44b, and an action-input section 60 to instigate the interaction with the second domain-specific delegee 44b, and a thought section 56 that provides the reasoning steps used to arrive at the proposal in the action section 58.

[0045] The top-level agent 42 also includes a top-level query builder 68 and a top-level receiver 70.

[0046] The top-level query builder 68 uses the first output 54 to build domain-specific queries 72a, 72b and to provide them to corresponding domain-specific delegees 44a, 44b identified in the action section 58 of the first output 54. Each domain-specific query 72a, 72b uses information derived from the action-input section 60 of the same first output 54.

[0047] For each domain-specific delegee 44a, 44b the domain-specific query 72a, 72b is a query not unlike the top-level query 50 that came from the occupant 20. This means that the top-level query builder 68 need not be privy to details about interacting with the external applications 34a, 34b. This knowledge is contained in the relevant domain-specific delegee 44a, 44b. The domainspecific query 72a, 72b need only be specific enough to trigger the application of the knowledge contained in the domain-specific delegee 44a, 44b in the correct way.

[0048] As a result of the foregoing architecture, with its incorporated hierarchy 40, it becomes possible to bypass training the top-level model 48 to output all kinds of API calls. Instead, the toplevel model 48 only needs to be trained to identify which domain-specific agent 44a, 44b will know how to generate a particular kind of API call.|0049| Each domain-specific delegee 44a, 44b provides a corresponding domain-specific response 76a, 76b back to the top-level agent 42, and in particular, to the top-level receiver 70. The top-level receiver 70 weaves the domain-specific response 76a, 76b from the different domain-specific delegees 44a, 44b into a coherent top-level response 80, which it then provides back to the occupant 20 in response to the top-level query 50.|0050] The top-level agent 42 thus carries out a form of triage, in which it receives a top-level query 50, identifies various specialists, i.e., domain-specific delegees 44a, 44b that will be needed to process the top-level query 50, and then provides those domain-specific delegees 44a, 44b with corresponding domain-specific queries 72a, 72b.

[0051] Having discussed the structure and operation of the top-level agent 42, it is now useful to consider the structure and operation of a representative domain-specific delegee 44a, 44b. Thestructure and operation of the representative domain-specific delegee 44a, 44b is representative of all the domain-specific del egees 44a, 44b.(0052] As shown in FIG. 2, a domain-specific delegee 44a, 44b comprises a structure very much like that of the top-level agent 42. The domain-specific delegee 44a, 44b features a domain-specific prompt builder 82a, 82b and a domain-specific model 84a, 84b. Like the top-level model 48, the domain-specific model 84a, 84b is a large language model.

[0053] A domain-specific delegee 44a, 44b has been trained to use application-program interfaces 38a, 38b associated with selected external applications 34a, 34b that are pertinent to its domain. Accordingly, the domain-specific delegee 44a, 44b has been trained to accommodate some number of application-program interfaces 38a, 38b. However, because the domain-specific delegee 44a, 44b is specific to a single domain, that number is not very high. And most importantly, it is not high enough to degrade the agent’s overall performance.

[0054] The domain-specific prompt builder 82a, 82b uses the domain-specific query 72a, 72b and domain information 47a to construct a domain-specific prompt 86a, 86b, which it then provides to the domain-specific model 84a. The domain-specific model 84a provides an application-specific output 88a, 88b to an API builder 90a, 90b, which then constructs an application-specific API call 92a, 92b and provides it to the relevant application 34a, 34b.

[0055] In response to the API call 92a, 92b, the application 34a, 34b provides an applicationspecific response 94a, 94b to the domain-specific delegee’ s application-specific receiver 96a, 96b. The application-specific receiver 96a, 96b then transforms the application-specific response 94a, 94b into the domain-specific response 76a, 76b that ultimately goes to the top-level receiver 70 as already discussed.10056] The agent hierarchy 40 relieves the top-level agent 42 from having to know the details of the various application-program interfaces 38a, 38b. Instead, the backbone 42 only needs to be able to recognize what domains are pertinent to the top-level query 50 and to articulate what is needed from each of the domain-specific delegees 44a, 44b.

[0057] The descriptions of the various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0058] Aspects of the present embodiments may be embodied as a system, method or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “module” or “system.” Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

[0059] Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (erasable programmable read-only memory (EPROM) or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.[0060 Aspects of the present disclosure are described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchartillustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / acts specified in the flowchart and / or block diagram block or blocks. Such processors may be, without limitation, general purpose processors, special-purpose processors, application-specific processors, or field- programmable.

Claims

WHAT IS CLAIMED IS:

1. A method for using a hierarchy in an automotive assistant to provide a top-level response to a top-level query from an occupant of a vehicle, the hierarchy comprising a top-level agent that comprises a top-level model, a first domain-specific delegee that comprises a first domain-specific model, and a second domain-specific delegee that comprises a second domain-specific model, wherein the method comprises: receiving a top-level query, building a top-level response; identifying a first domain-specific activity that is required to complete the top-level response; formulating a first domain-specific query; providing the first domain-specific query to the first domain-specific model, thereby causing the first domain-specific model to execute the first domain-specific activity, identifying a second domain-specific activity that is required to complete the toplevel response; formulating a second domain-specific query that incorporates information resulting from execution of the first domain-specific activity; providing the second domain-specific query to the second domain-specific delegee, thereby causing the second domain-specific delegee to execute the second domain-specific activity; completing formulation of the top-level response, the top-level response being based on having completed at least the first and second domain-specific activities; and providing the completed top-level response to the occupant; wherein the top-level model, the first domain-specific model, and the second domain-specific model comprise at least one large language model; wherein the first and second domain-specific delegees are specific to first and second domains corresponding to the first and second domain-specific activities, respectively; andwherein the first and second domain-specific queries comprise prompts that comprise natural language.

2. The method of claim 1, wherein carrying out at least one of the first and second domain-specific activities comprises causing a non-verbal response.

3. The method of claim 1, wherein carrying out at least one of the first and second domain-specific activities comprises the hierarchy receiving at least one of first and second domain-specific information to be incorporated into the top-level response.

4. The method of claim 1, further comprising the hierarchy receiving context information from a context source and using the context information to identify a specific domain that is pertinent to the top-level query.

5. The method of claim 1, further comprising the hierarchy receiving occupant preferences from an internal context-source that stores occupant-preferences and using the stored occupant-preferences to identify a specific domain that is pertinent to the top-level query.

6. The method of claim 1, further comprising the hierarchy receiving stored dialog history from an internal context-source and using the saved dialog history to identify a specific domain that is pertinent to the top-level query.

7. The method of claim 1, further comprising using context information that represents a current location of the vehicle to identify a specific domain that is pertinent to the toplevel query.

8. The method of claim 1, further comprising receiving vehicle-state information from one or more sensors of the vehicle and using the vehicle-state information to identify a specific domain that is pertinent to the top-level query.

9. The method of claim 1, further comprising receiving real time meteorological data from an external context source and using the meteorological data to identify a specific domain that is pertinent to the top-level query.

10. The method of claim 1, further comprising establishing communication with an external context source that provides context information that comprises real-time information on average vehicle speed of other vehicles and using the real-time information on the average vehicle speed to identify a specific domain that is pertinent to the top-level query.

11. The method of claim 1, wherein the automotive assistant is configured to interact with each of a plurality of external applications, wherein each of the external applications receives structured text, and wherein each of the domain-specific delegees has been trained to provide the structured text to at least one of the external applications.

12. The method of claim 1, further comprising causing the first domain-specific delegee to generate an API call in response to the first domain-specific query and to provide the API call to an external application.

13. The method of claim 1, further comprising generating a top-level prompt to urge the top-level model to respond with an output that comprises: an identification of one or more domains that are pertinent to processing the top-level query, reasoning steps relied upon to identify the domains, and a plan of action for responding to the top-level query.

14. The method of claim 1, further comprising selecting the top-level model to be one that has been trained to respond to a top-level prompt by providing an output that comprises: a thought section, an action section, and an action-input section, wherein the thought section analyzes the top-level query into separate and distinct first and second domain-specific tasks; wherein the action section identifies the first domain-specific delegee to execute the first domainspecific task; and wherein the action-input section discloses how to instigate interaction with the first domain-specific delegee to promote return of useful information from the first domainspecific delegee.

15. The method of claim 1, further comprising selecting the domains from the group consisting of: a navigation domain, a music domain, a general -knowledge domain, and a car-control domain.

16. The method of claim 1, further comprising selecting the top-level model to be a large language model that receives a textual input and provides an output that comprises natural language.

17. The method of claim 1, further comprising selecting the top-level model to be a large language model that receives a textual input and wherein the first output comprises structured text.

18. The method of claim 1 further comprising selecting the top-level model to be a large language model that receives a textual input and wherein the first output comprises an API call.

19. The method of claim 1, further comprising the top-level agent executing a triage function that comprises identifying one or more of the top-level delegees that will be required to process the top-level query.

20. The method of claim 1 , wherein each of the domain-specific delegees comprises: an API builder; a domain-specific-prompt builder that uses the domain-specific query and domain information to construct a domain-specific prompt; a domain-specific model, the domain-specific model being a large language model that receives the domain-specific prompt and generates an application-specific output that is provided to the API builder; wherein, in response to receiving the application-specific output, the API builder constructs an application-specific API call and provides the application-specific API call to an external application.

21. An apparatus comprising an automotive assistant in an infotainment system of a vehicle, the infotainment system being in electromagnetic communication with loudspeaker and a microphone, wherein the automotive assistant operates a hierarchy that receives a top-levelquery via a speech interface from an occupant of the vehicle and that provides the occupant of the vehicle with a top-level response to the top-level query, the hierarchy including at least a first level and a second level, wherein the first level of the hierarchy comprises a top-level agent that interacts with the second level of the hierarchy during generation of the top-level response; wherein the second level of the hierarchy comprises a plurality of domain-specific delegees, each of which has been configured to provide domain-specific responses for a corresponding domain, each of the domain-specific delegees comprising a corresponding domainspecific model; wherein the top-level agent comprises a top-level model and a top-level prompt builder; wherein the top-level prompt builder is configured to generate a top-level prompt in response to the top-level query and to provide the top-level prompt to the top-level model; wherein the top-level model is configured to respond to the top-level prompt by providing a first output that is to be used for generating domain-specific queries for the domainspecific delegees, the domain-specific queries comprising prompts that comprise natural language; wherein the top-level agent is configured: to provide one of the domain-specific queries to a corresponding domainspecific delegee from the plurality of domain-specific delegees, to receive a domain-specific response to the domain-specific query from the domain-specific delegee, and to use information in the domain-specific response to build the top-level response, which is to be provided to the occupant of the vehicle.

Images