Techniques for determining a conversational intent

Abstract

The present disclosure relates to systems and methods for inferring intent of spoken language by employing large language models (LLMs) to enhance interactions between users and automated agents, such as digital assistants. The invention involves continuously monitoring ambient audio, converting speech to text, and utilizing LLMs to determine whether the spoken language is intended for an automated agent. Structured prompts, including converted text and specific instructions, are sent to the LLM, which is fine-tuned to handle domain-specific prompts. The LLM provides structured output in a standardized format indicating the user's intent. The system can involve multiple prompts to perform separate tasks, such as identifying intent and generating additional context-specific data. This approach facilitates a more natural and intuitive user experience by eliminating the need for wake words and enabling seamless conversational interactions with virtual assistants across various platforms and devices.

Description

Priority requirements

[0001] This patent application claims the benefit of priority to U.S. Application Serial No. 18 / 612,539, filed March 21, 2024, which is incorporated herein by reference in its entirety. Technical Field

[0002] This application relates to the technical field of artificial intelligence (AI), and more specifically, to automated agents or assistants commonly referred to as digital agents, digital assistants, virtual agents, virtual assistants, and chatbots. More specifically, this application relates to the development and implementation of a high-level dialogue interface that utilizes large language models (LLMs) to infer the conversational intent of spoken language. Background Technology

[0003] Automated agents or assistants, exemplified by widely recognized platforms such as Siri®, Amazon Alexa®, and Google Assistant®, represent a significant advancement in human-computer interaction. These automated agents or assistants are designed to understand and respond to natural, conversational language, enabling users to interact with technology in a conversational manner. Unlike traditional interfaces that rely on text input (such as from keyboards, touchscreen interfaces, or similar input devices), these automated agents and assistants receive audible input and utilize sophisticated natural language processing (NLP) algorithms and speech recognition technology to interpret user commands and queries and provide relevant responses. A significant advantage of this type of interface is its hands-free nature—that is, an individual can interact with the automated agent without using their hands to provide input and control.

[0004] The development of these automated agents and assistants has been driven by technological advancements and the growing need for intuitive and efficient ways to access information, perform tasks, and control various devices and services. By integrating voice and audio-based command interfaces, these automated agents and assistants provide users with a seamless hands-free experience, enabling them to easily interact with technology in a variety of contexts, including smartphones, smart speakers, headsets, infotainment interfaces for automobiles, and other devices. Attached Figure Description

[0005] In the accompanying drawings (which are not necessarily drawn to scale), the same reference numerals may describe similar parts in different views. To facilitate identification of any particular element or action being discussed, one or more of the highest-order digits in the reference numerals refer to the reference numeral where that element was first introduced. Some non-limiting examples are shown in the figures in the accompanying drawings:

[0006] Figure 1This is a diagram illustrating an example of a conventional automation agent that relies on a wake word detection model to identify when a user utters a specific word or phrase (such as a wake word) that indicates a clear intent to invoke certain functions of the automation agent.

[0007] Figure 2 This is a diagram illustrating an example of an automated agent that utilizes a combination of an automatic speech recognition model or speech-to-text model and a large language model (LLM) to infer the user's intent based on captured spoken words.

[0008] Figure 3 This is a diagram illustrating an example of the interaction between a user and an automated agent, which utilizes LLM to infer the user's intent based on the words spoken by the user.

[0009] Figure 4 This is a flowchart illustrating methodological operations for determining a user's intent to interact with a domain-specific automated agent, based on some examples.

[0010] Figure 5 This is a diagram illustrating an alternative system architecture for an automated agent, consistent with some examples, where the automated agent is integrated to operate with various client applications.

[0011] Figure 6 It is a graphical representation of a machine in the form of a computer system, based on some examples, within which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein.

[0012] Figure 7 It is a block diagram showing a software architecture in which examples can be implemented. Detailed Implementation

[0013] This application relates to the technical field of artificial intelligence (AI), and more specifically, to automated, AI-based agents, sometimes referred to as digital agents, digital assistants, virtual agents, virtual assistants, or chatbots. More specifically, this application relates to advanced natural language dialogue interfaces that utilize large language models (LLMs) to infer the user's intent when interacting with an automated agent based on the user's spoken vocabulary. In the following description, numerous specific details are set forth for illustrative purposes to provide a thorough understanding of various aspects of different embodiments of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without all of these specific details.

[0014] Figure 1A typical automation agent 104 is illustrated, which is often integrated into various computing devices, such as smart speakers 102-A, smart glasses 102-B, laptops 102-C, smartphones 102-D, and hands-free car systems 102-E. For the typical automation agent 104, the computing device executing the software-based automation agent 104 is equipped with an "always-on" wake word detection model 106, which is in a continuous passive listening state specifically for predetermined words or phrases, commonly referred to as "wake words" or "wake phrases." By way of example, some common automation agents use "Alexa" or "Hey Siri," or other similar words or phrases as wake words. The wake word detection model 106 acts as a gatekeeper, ensuring that the automation agent 104 remains dormant until the specific wake word is detected in the ambient audio captured by the microphone of the device executing the automation agent 104.

[0015] Once the wake word is recognized by the wake word detection model 106, the automation agent 104 begins to actively capture and process ambient audio via an automatic speech recognition (ASR) model or a speech-to-text model 108. The model 108 is responsible for converting subsequent spoken words into text, which can then be further processed or interpreted by the client application 110 to understand and execute user commands. The client application 110 may optionally communicate via network 112 with the server-based automation agent service 114 to perform requested actions or retrieve information as indicated or otherwise influenced by the user's spoken commands.

[0016] The conventional use of wake words to explicitly invoke the automation agent presents several technical challenges. First, the requirement for a wake word can disrupt the natural flow of conversation, as users must remember and use specific phrases to interact with their device. This can result in a less intuitive user experience, especially for new users or those who don't use the device frequently and may not remember the exact wake word or phrase. Second, the wake word approach can lead to false activations when the wake word detection model 106 incorrectly identifies similar-sounding words or phrases as wake words, resulting in unintended recording and processing of audio. Finally, in noisy environments or during overlapping conversations, the wake word detection model 106 may fail to accurately detect wake words, causing user frustration when the automation agent does not activate as expected. Conversely, background conversations that inadvertently contain wake words may unintentionally trigger the automation agent 104, leading to interruptions and potential privacy breaches.

[0017] As described herein, examples of various embodiments of the present invention provide significant advancements in the field of AI-based dialogue interfaces by introducing techniques that utilize generative language models, such as LLMs, to continuously analyze spoken words detected in ambient audio to determine user intent without requiring a specific wake word. Consistent with some examples, AI-based automated agents include automatic speech recognition (ASR) models or speech-to-text models that capture spoken words and convert them into text. The converted text representing the spoken words is then processed by a cue generator that formulates structured cues to be processed by the LLM as input. The LLM (which may be fine-tuned in some cases for the specific domain served by the automated agent) interprets these cues to infer whether the spoken words are part of the ambient dialogue or intended for the automated agent. Additionally, consistent with some examples, the LLM is instructed to formulate output specifically indicative of user intent and formatted in a structured manner (e.g., as JavaScript Object Notation (JSON) objects).

[0018] In some examples, structured prompts include not only transformed text but also, in certain cases, contextual information, enabling LLMs to discern user intent with greater accuracy. This approach allows automated agents to understand and respond to user commands in a more natural and conversational manner. By eliminating the need for wake words, AI-based automated agents can be seamlessly integrated into user conversations, thereby enhancing the overall user experience.

[0019] One of the technical advantages of the claimed invention is the reduction of false activations, a common problem with wake word detection models. Because the automated agent does not rely on a specific wake word or trigger phrase, the likelihood of it falsely activating in response to similar-sounding words or background noise is significantly reduced. This results in more reliable and user-friendly interactions, as the automated agent only responds when the user's intent is explicitly directed towards it.

[0020] Another advantage is the improved ability of automated agents to handle noisy environments and overlapping conversations more effectively. ASR (Audio-Reference Speech) or speech-to-text models work in conjunction with LLM (Low-Level Microspeech) to filter out irrelevant ambient noise and conversation, focusing on the user's voice, thus improving intent detection accuracy even under challenging audio conditions. This ensures that the automated agent remains responsive and accurate, giving users confidence that their commands will be correctly understood and executed.

[0021] Another advantage of the AI-based automated agent described in this paper is that LLM can correct inaccuracies in text generated by ASR or speech-to-text models. For example, where ASR or speech-to-text models may misinterpret spoken words due to various factors such as speech clarity, accent, or background noise, LLM can correct these errors. LLM analyzes a series of cues within the context of the entire dialogue history maintained in its context window. This comprehensive perspective allows LLM to identify inconsistencies or potential errors in the transcribed text.

[0022] For example, if the ASR model transcribes the phrase "I need a cab" as "I need a cap" due to background noise or speech ambiguity, the LLM can use the surrounding dialogue context to identify that the user's intent is more likely related to transportation than to headwear. With its extensive language understanding and dialogue history, the LLM can infer that "cab" is the correct word and adjust the transcribed text accordingly. This correction not only affects the determination of the user's intent but is also reflected in the LLM's structured output, ensuring that the automated agent accurately captures the user's actual intent. This self-correcting mechanism of the LLM not only improves the accuracy of user intent inference but also reduces the need for the user to repeat themselves or make manual corrections, thus simplifying the interaction process. By providing a more accurate representation of the user's spoken vocabulary, the system ensures that the automated agent's responses and actions are more aligned with the user's actual requests, further enhancing the overall user experience. Other aspects and advantages of various embodiments of the invention are described below in conjunction with several accompanying drawings.

[0023] Figure 2 This is a schematic diagram illustrating an example of an automated agent 204 consistent with some embodiments of the present invention, which utilizes a combination of an automatic speech recognition (ASR) model or speech-to-text model 206 and an LLM (e.g., 214-A or 214-B) to infer the user's intent in interacting with the automated agent 204 based on captured spoken vocabulary. Figure 1 Compared to the conventional system shown, Figure 2 The automated agent 204 described in the document is designed to provide a more natural and intuitive user experience. Specifically, unlike conventional automated agents that rely on wake words, Figure 2 The automated agent described does not rely on a wake word detection model to initiate or invoke user interactions. Instead, the automated agent 204 employs an ASR model or a speech-to-text model 206, which continuously processes captured ambient audio to convert spoken words within the ambient audio into text without requiring a specific wake word or trigger phrase.

[0024] In some examples, the speech-to-text model 206 has or uses pause detection logic, which enables the identification of natural breaks in the user's speech. This pause detection logic operates by analyzing silent periods in the audio stream that exceed a predefined threshold, indicating the end of phrases or sentences. The duration of these silences is carefully calibrated to distinguish between natural pauses occurring during speech (such as pauses for breathing or thinking) and the end of statements or commands. By detecting these pauses, the automation agent can segment the continuous speech stream into coherent and discrete units of text. This allows the automation agent to discern when the user has completed a statement or command, thus segmenting the continuous speech stream into coherent and discrete units of text. This segmentation is helpful in subsequent processing stages because it helps maintain the natural flow of the conversation and ensures that the context of the user's speech is preserved.

[0025] Pause detection logic can leverage various acoustic signals and linguistic cues to improve its accuracy. For example, it can analyze the length of silences, intonation at the end of words, and pause probabilities based on the syntactic structure of the spoken sentence. Additionally, the logic can be trained to recognize filler sounds frequently used by the speaker, such as "uh" or "um," which typically do not indicate the end of a statement. By combining these sophisticated methods, pause detection logic ensures that the speech-to-text model of the automated agent maintains the natural flow of the conversation, accurately reflects the user's intent, and preserves the context required for the LLM to generate relevant and accurate responses.

[0026] Once spoken words are converted into text by the speech-to-text model 206, the prompt generator 208 creates a structured prompt, which includes at least two key components: an instruction portion and context. The instruction portion is designed to instruct the LLM to analyze the provided text and determine whether it represents a command or request intended for an automated agent, or whether it is merely contextual dialogue not intended for an agent to respond to. The context—typically consisting of the converted text and possibly additional dialogue history—provides the necessary background information for the LLM to make this determination.

[0027] Consistent with some examples, the ASR model, or speech-to-text model 206, includes advanced speech recognition capabilities to distinguish spoken words and assign them to the correct individuals, which is particularly advantageous in environments with multiple speakers. This process, known as speaker diarization, involves analyzing various characteristics of the speaker's speech to identify and segment speech fragments corresponding to each individual.

[0028] The speech-to-text model 208 employs machine learning algorithms trained on diverse datasets of speech samples to identify distinct speech features such as pitch, intonation, speech rate, and accent. These speech features are unique to each individual, similar to a speech fingerprint, and enable the speech-to-text model 208 to create a profile for each speaker. During the conversation, the model 208 continuously compares incoming audio with these established profiles to determine the likelihood that a particular speech segment belongs to a specific speaker.

[0029] Furthermore, when the computing device has multiple microphones, the ASR model or speech-to-text model 208 can utilize spatial information. By evaluating the directionality of sound and the time difference of spoken words arriving at different microphones, the system can infer the speaker's position relative to the device. This spatial analysis enhances the device's ability to assign speech segments to the correct individuals, especially when the speech features of two speakers may be similar.

[0030] The combination of speech feature recognition and spatial analysis enables the speech-to-text model 208 to construct more accurate transcriptions of multi-person dialogues. Each speaker's vocabulary is transcribed individually, and speaker tags are attached to the corresponding text segments. This precise attribution is beneficial for subsequent processing stages, as it allows the LLM to accurately infer the context of the dialogue and determine whether spoken words are intended for the automated agent. By maintaining the integrity of the dialogue structure, the ASR model or speech-to-text model 208 ensures that the automated agent can interact with the user in a conversational manner that simulates natural human-to-human communication.

[0031] Consistent with some examples, the prompt generator 208 employs various strategies to generate prompts that are provided as input to the LLM. One approach is to use template-based prompts, where a portion of the prompt is predefined and includes static elements that outline the general structure and objectives of the prompt. These static elements are consistent across different instances and provide a structure that ensures the LLM receives the necessary instructions in a familiar format. Dynamic elements are then inserted into the prompt template in real-time based on the specific context of the user's current interaction. These dynamic elements may include the latest converted text snippets from the user's speech, relevant metadata (such as the time of the interaction, the user's location), or any other relevant information that may affect the LLM analysis.

[0032] Beyond template-based generation, the cue generator 208 can also utilize more sophisticated methods, such as conditional logic, where the content of the cue is further customized based on certain conditions or triggers identified in the user's speech. Alternatively, machine learning algorithms can be employed to learn from past interactions and progressively optimize the structure and content of the cue over time, making it more effective in deriving the desired output from the LLM. Another approach may involve heuristic methods, where the cue generator selects or generates cuees based on heuristic rules or patterns identified in the user's speech, aiming to optimize the performance of the LLM for each unique interaction. These various methods can be used individually or in combination to create a robust and adaptive cue generator 504 that enhances the LLM's ability to accurately discern and respond to user intent.

[0033] Consistent with some examples, the LLMs (such as 214-A and 214-B) are accessed via network 212 and through an external LLM service 214. This LLM service 214 provides an LLM with function call capabilities, enabling the LLM to efficiently handle structured prompts. To enhance the LLM's ability to discern user intent, consistent with some examples, the LLM is fine-tuned using system prompts incorporating multi-shot fine-tuning examples. These examples demonstrate a range of scenarios that help the LLM distinguish between commands designed for an automation agent and merely background dialogue. For example, system prompts used for fine-tuning may include various instances that clearly describe the user intent as directed towards an automation agent or as part of ambient noise. For example, a system prompt with a fine-tuning example might look like this: The system prompts: "You will receive a stream of text. Your task is to determine if someone is talking to you, or if it's an ambient conversation, and then extract the user's intent. Do not answer their questions and always respond in valid JSON format." Example input: "The weather is nice today, but how will it be tomorrow?" Example output: {"intent":"Inquire about tomorrow's weather"} Example input: "I had a long day at work. Oh, and I need to set an alarm for 6 AM." Example output: { "intent": "Set an alarm for 6 AM" ("intent": "Set an alarm for 6 AM")} Example input: "I can't believe how well the team played last night!" Example output: {"intent":"NONE" ("intent": "None")}

[0034] These examples demonstrate the ability of LLM to accurately extract user intent and act accordingly, distinguishing between direct interaction and contextual dialogue. By leveraging these advanced LLM capabilities, Figure 2 The automated agent 204 shown offers significant improvements over traditional wake word-based systems, providing a more seamless and pleasant user experience.

[0035] Although Figure 2 While the LLM is presented as being hosted by an external service provider, an alternative implementation of the present invention allows the LLM to be executed directly on the device of the automation agent 204. In such a configuration, the device will contain the necessary computing resources to execute the LLM locally, thereby potentially reducing latency and reliance on external service connections for processing user commands and queries.

[0036] like Figure 2 The depicted automated agent 204 is multifunctional and can be integrated into a wide variety of devices, each designed to meet the unique needs of different environments and user interactions. These devices range from smart speakers 202-A, which can be used in homes and offices for tasks such as controlling smart home devices or providing information, to smart glasses 202-B, which provide hands-free assistance and augmented reality experiences. Laptops 202-C and smartphones 202-D are ubiquitous devices that benefit from the integration of the automated agent, improving productivity and providing assistance on the go. Additionally, hands-free car systems 202-E can significantly improve the driving experience by allowing the driver to focus on the road while issuing voice commands for navigation, entertainment, or certain vehicle controls.

[0037] Automated agents can be general-purpose, designed to handle a wide variety of tasks and queries from users. These agents are equipped to leverage LLMs with broad language understanding and the ability to process common instructions across multiple domains. On the other hand, domain-specific automated agents are tailored to provide specialized assistance within a specific domain or context. For example, automated agents integrated into medical devices can be fine-tuned to understand and process healthcare-related queries, while automated agents in financial applications can be specialized for handling banking and investment issues.

[0038] For domain-specific automated agents, the LLM fine-tuning process helps ensure high accuracy and relevance of their responses. System cues used to fine-tune the LLM include domain-specific examples that guide the LLM to identify and interpret the intent behind user input within that specific domain. Each example provided to the LLM includes input (such as a stream of text that can be captured from the user's speech) and a corresponding expected output (which can be the user's intent or an indication that the input does not represent an intent toward the automated agent).

[0039] For example, in a domain-specific system designed for cooking assistance, system prompts may include the following examples: The system prompts: "Identify if the following text is a culinary-related request for the automated agent." Example input: "I'm wondering how many teaspoons are in a tablespoon." Example output: { "intent": "Convert teaspoons to tablespoons" ("intent": "Convert teaspoons to tablespoons")}

[0040] Alternative locations for inputs unrelated to the cooking field: Example input: "I'm going to wear my new shoes tonight." Example output: {"intent":"none" ("intent": "none")}

[0041] These fine-tuning examples enable LLM to develop a nuanced understanding of domain-specific language and user requests, allowing Automation Agent 204 to provide targeted and accurate assistance. By combining this domain-specific knowledge, Automation Agent 204 becomes a powerful tool for enhancing user experience and efficiency within its area of ​​expertise.

[0042] Upon receiving structured output from the LLM, client application 210 can take various specific actions based on the inferred user intent. The nature of these actions is highly dependent on the context of the request and the capabilities of the device on which the automation agent runs. For example, if the structured output from LLM 214-A indicates an intent to obtain weather information, client application 210 can direct the request to an external weather service (e.g., automation agent service 216) to retrieve the latest forecast information. The request will be formatted according to the specifications of the weather service's application programming interface (API), ensuring that the user's need for weather-related information is met with accurate and timely data. It is also worth noting here that, in some cases, based on the output received from the LLM (e.g., 2140A or 214B) and the specific intent of the user to invoke the automation agent, a subsequent LLM prompt can be generated and sent to another LLM, different from the one used to infer the user intent. This subsequent prompt allows the second LLM to handle the user's query in a specialized context, such as when the user's intent involves a domain-specific query (e.g., requesting financial news updates), where an LLM specifically tuned to provide financial information will be best suited to respond.

[0043] By way of example, if the structured output indicates that the user intends to schedule a meeting, the client application 210 can interact with the user's calendar system to create a new event. If the intent is to play a specific song or genre, the client application 210 can interface with a multimedia system to start playback. In the case of smart home devices, if the user's intent is to adjust the temperature, the client application 210 can send a command to the home's thermostat system.

[0044] Here are some examples of actions that client application 210 might take: • For the Smart Speaker 202-A, actions could include providing weather updates or setting a cooking timer based on the user's request. • In the case of smart glasses 202-B, actions may involve displaying navigation directions or translating signs or menus from another language into the user's field of vision. • On the 202-C laptop, actions can include opening a document or sending an email based on the user's verbal instructions. • For the 202-D smartphone, the client application may respond to the user's command by initiating a call, sending a text message, or opening a mobile application. • Within the hands-free car system 202-E, actions can be based on the driver's request to find the nearest gas station, change the audio track, or adjust the cabin lighting.

[0045] Additionally, client application 210 can use the output received from LLM 214-A or LLM 214-B to make subsequent calls or queries to the remote, server-based automation agent service 216. This is particularly useful when requests require additional processing power, access to large datasets, or expertise not locally available on the computing device of automation agent 204. For example: • If the user's intent involves travel planning, the client application can query remote services for real-time traffic updates or public transportation timetables. • When a user expresses their intention to dine out, the client application can access server-based services to make restaurant reservations or order takeout. • If a user's request involves banking activities, the client application can contact financial services to check the account balance or initiate a transaction.

[0046] These examples demonstrate the versatility of client application 210 in responding to structured output from LLM, enabling a wide range of actions and interactions to meet user needs and enhance the overall experience of using automation agent 204.

[0047] Figure 3 This is a diagram illustrating an example of user-automated agent interaction based on some examples of automated agents implemented using smart glasses 300, leveraging LLM to infer user intent based on spoken words. In this example, the user interacts with a device equipped with an automated agent, such as “smart” glasses 300, which is designed to capture spoken words via an integrated microphone. The spoken words are then processed by a speech-to-text model 302, which converts the audio input into a text representation. This text is then passed to a cue generator 304, which formulates structured cues that encapsulate the user’s spoken words along with instructions for LLM.

[0048] The structured prompt 308 is then transmitted over the network to the LLM service. The LLM, hosted by the LLM service, analyzes the text and determines the user intent. The LLM does this by processing the instructions provided in the prompt 308, which instruct the LLM to differentiate between commands and contextual dialogue intended for an automation agent. The LLM processes the prompt 308 to interpret the text and generate structured output 310 that reflects the user intent.

[0049] exist Figure 3 In the example shown, the user's spoken words are "I need directions to the library." The prompt generator 304 creates a structured prompt 308 that includes these words and transmits it to the LLM service. Upon receiving prompt 308, the LLM evaluates the text and identifies that the user is requesting directions—a task within the domain of automated agent capabilities. The LLM then generates a structured output in the form of a JavaScript Object Notation (JSON) object, which includes fields such as "intent" and "target." The "intent" field is populated with the user's request ("Asking for directions"), and the "target" field specifies the destination mentioned by the user ("library").

[0050] The client application 306 on the user's device receives the structured output and takes appropriate action. In this case, the client application 306 can interact with a map application or service to provide the user with the requested route to the library. This interaction demonstrates a seamless process of intent inference using LLM, which enables the automated agent 300 to provide relevant and timely assistance to the user.

[0051] Figure 4 This is a flowchart illustrating the operational steps involved in determining a user's intent to interact with a domain-specific automated agent based on processed spoken language, based on some examples. The process begins with the continuous capture of ambient audio 402 via a microphone integrated with a device. This device can be any of the aforementioned examples, such as smart glasses or a smartphone. The ambient audio is expected to contain spoken words from the user, which may or may not be directed at the automated agent.

[0052] Once the audio is captured, spoken words are converted into 404 text using a speech-to-text recognition algorithm or process. This conversion transforms the user's spoken language into a format that can be processed by an LLM (Language Modeling Language). Speech-to-text models can include high-level features such as noise cancellation and language model adaptation to improve accuracy.

[0053] After the transformation, an LLM prompt is created to serve as input 406 for the LLM. This prompt includes the transformed text and instructions that guide the LLM to determine whether the text represents a command or request intended for a domain-specific automation agent, or whether the text is part of an contextual dialogue. The prompt may also include additional context (such as previous user interactions or commands) to assist the LLM in making more informed decisions.

[0054] The prompt is then transmitted to an LLM 408, which resides on a server accessible over the network. The LLM analyzes the prompt and generates structured output as a response. This response indicates whether the spoken vocabulary is intended for a domain-specific automated agent, and if so, indicates the user's intent.

[0055] Upon receiving structured output from the LLM, the automation agent executes an action 412 corresponding to the intent. If the structured output indicates that the vocabulary is intended for a domain-specific automation agent, the client application or the automation agent itself will respond or act appropriately. This may involve querying a remote server-based automation agent service for additional information or performing a local action on the device itself.

[0056] The end of a flowchart indicates the completion of a process. Figure 4 The structured and systematic approach outlined in the document ensures that user intent is accurately captured and responded to, thereby enhancing the user experience when using automated agents.

[0057] In the various accompanying figures, the LLM is depicted and described as being hosted by a remote LLM service, accessible via a network to the automation agent. This configuration enables the utilization of powerful cloud-based computing resources to process and analyze spoken language input, providing the necessary computing power and data access that might not be available locally on the user's device. However, it is important to note that this is not the only possible configuration. In other cases, depending on the performance of the device executing the automation agent, the LLM can be hosted locally on the device itself. This on-device hosting offers advantages such as reduced latency, enhanced privacy, and the ability to operate without a persistent network connection. Devices with sufficient processing power and storage, such as high-end smartphones, laptops, or dedicated AI hardware, can support on-device LLM, allowing the automation agent to process input and infer user intent directly on the device. This flexibility of LLM hosting allows for a range of implementations tailored to the specific requirements and constraints of different devices and use cases.

[0058] Figure 5This diagram illustrates various implementations of automated agents, which can be general-purpose agents serving on dedicated devices such as smart speakers, phones, laptops, glasses, etc., or domain-specific agents customized to provide information and perform tasks within a specific domain typically associated with a particular application. The diagram shows a user interacting with "smart" glasses 500 used as a client device. Client applications (506, 508, 510, 512, 514) represent different domain-specific applications that the user can participate in via smart glasses 500. Each application is associated with its own LLM (520, 522, 524, 526, 528), which is fine-tuned to handle queries and commands relevant to its corresponding domain. The fine-tuning process for each LLM involves using multiple samples of system prompts, which trains the LLM to accurately identify and process domain-specific user intents.

[0059] For example, if a user is using a map application 506 on smart glasses, the prompt generator 504 will guide the user's spoken vocabulary towards an LLM 520 that has been fine-tuned for geography and navigation-related queries. This LLM 520 has been trained with examples such as requesting routes, querying traffic conditions, or asking for the location of nearby points of interest.

[0060] Similarly, if a user opens the calendar app 508, spoken commands related to scheduling (such as setting up meetings, reminders, or checking availability) will be directed to a different LLM 522 specifically designed for scheduling and time management tasks. Another example could be the weather app 510, where a user's inquiry about temperature, forecast, or weather conditions will prompt generator 504 to select LLM 524, which has been trained or fine-tuned with meteorological data and can provide up-to-date weather information.

[0061] In each case, the LLM associated with the active client application processes structured cues to infer user intent. These cues include the user's spoken vocabulary and specific instructions. The LLM then generates structured output, which the client application uses to provide a response or perform actions relevant to the user's request. This architecture enables seamless and intuitive interaction between the user and the automation agent, whose responses are context-appropriate for the application in use.

[0062] The structured JSON object provided by the LLM as output can be customized to include a variety of structured data fields, depending on the specific domain of the automation agent and the instructions described in the prompts. These fields are designed to encapsulate all the necessary information that the client application might need to perform the intended action. For example, in healthcare, the JSON object might include fields for symptoms, medication requests, or appointment scheduling. Conversely, in home automation, fields might relate to device control commands, such as adjusting a thermostat or turning lights on and off. The flexibility of the JSON format allows for the inclusion of a wide range of data types and structures, making it a versatile medium for communication between the LLM and client applications.

[0063] In the various examples and figures provided, the LLM cue is typically described as a single structured cue designed to derive a single output from the LLM. This output is structured to identify whether the spoken words are intended for an automation agent, and if so, to clarify the intent of those spoken words. Therefore, the structured output will typically include at least two data fields: one indicating the presence of an intent for the automation agent, and another detailing the specific intent of the spoken words.

[0064] However, in some cases, particularly for domain-specific client applications, prompts can be designed to generate more complex structured outputs that include additional data fields. For example, in the domain of map applications or services, LLM can produce structured outputs that not only indicate whether the spoken words are intended to target the agent and the intent of the spoken words, but also provide additional data such as the specific location or destination mentioned by the user.

[0065] Consider a scenario where a user using a map application says, "How do I get to the nearest gas station?" In this case, the prompt sent to the LLM (Local Management Module) will be structured to extract multiple pieces of information from the query. The LLM's output can include a data field confirming that the query was indeed intended for the map application ("intent_detected": true), a second field identifying the user's intent ("intent": "request_directions"), and a third field specifying the location-related aspect of the intent ("location": "nearest gas station"). This structured output allows the map application to understand that the user is requesting a route, and is particularly interested in locating the nearest gas station, enabling the application to respond accurately and efficiently by providing the requested route.

[0066] In some cases, this process can involve multiple prompts to accomplish different tasks. For example, an initial prompt might only be used to determine whether spoken vocabulary is intended for an automated agent. Subsequently, a second prompt can be used to pinpoint the user's specific intent. This second stage can involve generating additional data related to the user's intent, the overall task, the domain, the client application, etc. This bifurcated approach enables more granular and precise information extraction, where the first prompt acts as a relevance filter, and the second prompt delves into the details of the user's request, customizing the response based on the specific needs and context of the interaction.

[0067] Within the scope of domain-specific applications, the structured cues provided to the LLM can be enriched with additional data that enhances the LLM's ability to accurately infer user intent. This supplementary data serves as contextual clues to inform the LLM's interpretation of spoken vocabulary, resulting in more accurate and relevant responses from the automated agent.

[0068] For example, in the context of a car, prompts sent to the LLM can include not only the speech-to-text dialogue but also data indicating the vehicle's current state. This could include the vehicle's location, whether it is moving or stationary, and, if moving, its speed. Such information can be crucial for understanding the user's intent. For instance, a request for a nearby gas station would imply a different level of urgency depending on whether the vehicle is stationary or traveling at highway speeds; the latter might indicate a need for immediate assistance due to low fuel.

[0069] Similarly, in a smart home environment, prompts can include data about the time of day, the status of various connected devices, or even the user's calendar information. If a user talks about adjusting the temperature, an LLM (Local Management Module) can make a more informed decision based on the user's intentions, knowing the current indoor temperature and the user's typical preferences for the time of day.

[0070] In healthcare applications, prompts can be enhanced with data such as a user's medical history or current biometric data. This allows LLMs to interpret statements about feeling unwell within the context of a user's known health condition, potentially identifying emergencies requiring immediate attention.

[0071] By tailoring additional data included in prompts based on the specific domain of the application, LLMs can leverage this context to provide responses that are not only accurate but also consistent with the user's immediate needs and the nuances of the environmental context. This approach underscores the adaptability and potential of LLMs in delivering complex, context-aware interactions across a wide range of professional domains. Machine architecture

[0072] Figure 6This is a schematic representation of machine 600, within which instructions 602 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause machine 600 to perform any or more of the methods discussed herein. For example, instruction 602 can cause machine 600 to perform any or more of the methods described herein. Instruction 602 transforms a general, unprogrammed machine 600 into a specific machine 600 programmed to perform the described and illustrated functions in the manner described. Machine 600 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 600 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 600 may include, but is not limited to, server computers, client computers, personal computers (PCs), tablet computers, laptops, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches or smart glasses), wearable augmented / virtual / mixed reality devices, smart home devices (e.g., smart appliances), other smart devices, network devices, network routers, network switches, bridges, or any machine capable of executing instructions 602 (sequentially or otherwise) specifying actions to be taken by machine 600. Furthermore, while a single machine 600 is shown, the term "machine" should also be understood to include a collection of machines that individually or jointly execute instructions 602 to perform any one or more of the methods discussed herein. Machine 600 may, for example, include any one of a user system or a plurality of server devices forming part of an interactive server system for publishing and sharing messages and other content. In some examples, machine 600 may also include both client and server systems, wherein certain operations of a particular method or algorithm are performed on the server side, while certain operations of a particular method or algorithm are performed on the client side.

[0073] Machine 600 may include processor 604, memory 606, and input / output (I / O) components 608, which may be configured to communicate with each other via bus 610. In the example, processor 604 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include processors 612 and 614, for example, executing instruction 602. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although Figure 6 Multiple processors 604 are shown, but machine 600 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0074] Memory 606 includes main memory 616, static memory 618, and memory cell 620, both of which are accessible by processor 604 via bus 610. Main memory 606, static memory 618, and memory cell 620 store instructions 602 embodying any one or more of the methods or functions described herein. Instructions 602 may also reside wholly or partially in main memory 616, static memory 618, machine-readable medium 622 within memory cell 620, at least one of processor 604 (e.g., within the processor's cache memory), or any suitable combination thereof, during execution by machine 600.

[0075] I / O component 608 may include a wide variety of components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, etc. The specific I / O component 608 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine is unlikely to include such a touch input device. It should be understood that I / O component 608 may include... Figure 6Many other components are not shown. In various examples, I / O component 608 may include user output component 624 and user input component 626. User output component 624 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistive mechanisms), other signal generators, etc. User input component 626 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens providing position and force for touch or touch gestures, or other haptic input components), audio input components (e.g., microphones), etc.

[0076] In a further example, I / O component 608 may include biometric component 628, motion component 630, environmental component 632, or position component 634, as well as a variety of other components. For example, biometric component 628 includes components for detecting facial expressions (e.g., hand gestures, facial expressions, vocal expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brainwaves), and recognizing a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Biometric components may include brain-computer interface (BMI) systems that enable communication between the brain and external devices or machines. This can be achieved by recording brain activity data, translating that data into a computer-understandable format, and then using the resulting signals to control the device or machine.

[0077] Examples of BMI technology types include: • BMI based on electroencephalography (EEG) uses electrodes placed on the scalp to record electrical activity in the brain. • Invasive BMI, which uses electrodes implanted into the brain through surgery. • Optogenetics BMI uses light to control the activity of specific nerve cells in the brain.

[0078] Any biometric data collected by biometric components is captured and stored only with the user's approval and deleted upon the user's request. Furthermore, such biometric data may be used for very limited purposes, such as authentication. To ensure limited and authorized use of biometric information and other personally identifiable information (PII), access to this data is restricted to authorized personnel (if applicable). Any use of biometric data may be strictly limited to authentication purposes, and the data will not be shared or sold to any third party without the user's explicit consent. Additionally, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of this sensitive information.

[0079] The moving part 630 includes an acceleration sensor part (e.g., an accelerometer), a gravity sensor part, and a rotation sensor part (e.g., a gyroscope).

[0080] The environmental component 632 includes, for example, one or more camera devices (with still image / photograph and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors for detecting hazardous gas concentrations to ensure safety or measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0081] Regarding the camera device, the user system may have a camera device system, including, for example, a front-facing camera on the front surface of the user system and a rear-facing camera on the rear surface of the user system. The front-facing camera may be used, for example, to capture still images and videos of the user of the user system (e.g., "selfies"), which can then be enhanced using the enhancement data (e.g., filters) described above. The rear-facing camera may be used, for example, to capture still images and videos in a more conventional camera device mode, which can also be enhanced using the enhancement data. In addition to the front and rear-facing cameras, the user system may also include a 360° camera device for capturing 360° photos and videos.

[0082] Furthermore, the user system's camera system may include dual rear cameras (e.g., a main camera and a depth-sensing camera), or even a triple, quad, or penta-camera rear camera configuration on the front and rear of the user system. These multi-camera systems may include, for example, wide-angle cameras, ultra-wide-angle cameras, telephoto cameras, macro cameras, and depth sensors.

[0083] The position component 634 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure and from which altitude can be determined), an orientation sensor component (e.g., a magnetometer), and the like.

[0084] Communication can be implemented using a variety of technologies. I / O component 608 also includes communication component 636, operable to couple machine 600 to network 638 or device 640 via a suitable coupling or connection. For example, communication component 636 may include a network interface component or another suitable device for interfacing with network 638. In further examples, communication component 636 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth components, etc. ® Components (e.g., Bluetooth) ® Low energy consumption), Wi-Fi ® Components, and other communication components for providing communication via other modes. Device 640 can be another machine or any of a wide variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0085] Furthermore, communication component 636 may detect identifiers or include components operable to detect identifiers. For example, communication component 636 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes), and multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix codes, and Dataglyph codes. TM The communication component 636 can be used to obtain various information, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via detection of NFC beacon signals that can indicate a specific location, etc.

[0086] Various memories (e.g., main memory 616, static memory 618, and memory of processor 604) and storage units 620 may store one or more sets of instructions and data structures (e.g., software) embodying or using any one or more of the methods or functions described herein. These instructions (e.g., instruction 602), when executed by processor 604, cause various operations to implement the disclosed examples.

[0087] Instruction 602 may be sent or received via network 638 using a transmission medium, through a network interface device (e.g., a network interface component included in communication component 636) and using any of several well-known transport protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instruction 602 may be sent or received via a transmission medium using a coupling to device 640 (e.g., peer-to-peer coupling). Software Architecture

[0088] Figure 7 This is a block diagram 700 illustrating a software architecture 702 that can be installed on any one or more of the devices described herein. The software architecture 702 is supported by hardware, such as a machine 704 including a processor 706, memory 708, and I / O components 710. In this example, the software architecture 702 can be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 702 includes layers such as an operating system 712, libraries 714, frameworks 716, and applications 718. Operationally, application 718 invokes API calls 720 through the software stack and receives messages 722 in response to API calls 720.

[0089] Operating system 712 manages hardware resources and provides public services. Operating system 712 includes, for example, a core 724, services 726, and drivers 728. Core 724 serves as an abstraction layer between hardware and other software layers. For example, core 724 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 726 can provide other public services to other software layers. Drivers 728 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 728 may include display drivers, camera drivers, Bluetooth® or Bluetooth® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., USB drivers), Wi-Fi® drivers, audio drivers, power management drivers, etc.

[0090] Library 714 provides common low-level infrastructure used by application 718. Library 714 may include system libraries 730 (e.g., the C standard library), which provide functions such as memory allocation, string manipulation, and mathematical functions. Additionally, library 714 may include API libraries 732, such as media libraries (e.g., libraries supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codecs, Joint Picture Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for rendering graphic content on a display in two-dimensional (2D) and three-dimensional (3D) formats), database libraries (e.g., SQLite, providing various relational database functionalities), web libraries (e.g., WebKit, providing web browsing functionality), etc. Library 714 may also include a wide variety of other libraries 734 to provide many other APIs to application 718.

[0091] Framework 716 provides common high-level infrastructure for use by application 718. For example, framework 716 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. Framework 716 can provide a wide range of other APIs that can be used by application 718, some of which may be specific to a particular operating system or platform.

[0092] In the example, application 718 may include a home application 736, a contacts application 738, a browser application 740, a book reader application 742, a location application 744, a media application 746, a messaging application 748, a game application 750, and a wide variety of other applications, such as third-party applications 752. Application 718 is a program that performs the functions defined in the program. Various programming languages ​​can be used to create one or more applications 718, structured in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a specific example, third-party application 752 (e.g., an application developed by an entity other than a platform-specific vendor using the Android™ or iOS™ Software Development Kit (SDK)) may be mobile software running on a mobile operating system (such as iOS™, Android™, Windows® Phone, or other mobile operating systems). In this example, third-party application 752 may invoke API calls 720 provided by the operating system 712 to facilitate the functions described herein. Example

[0093] Example 1 is a system for processing spoken language to determine a user's intent to interact with an automated agent. The system includes: at least one processor; at least one memory unit storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations including: continuously monitoring ambient audio via a microphone integrated with a device; converting captured spoken words from the ambient audio into text using a speech-to-text conversion process; generating a structured cue that includes at least the converted text and instructions for a Large Language Model (LLM), wherein the instructions are configured to request the LLM to infer whether the spoken words are intended for an automated agent; transmitting the structured cue to the LLM; receiving a structured output in a standardized format from the LLM, wherein the structured output includes an inference result indicating whether the spoken words are intended for an automated agent, and if so, identifying the user's intent; and when the inference result indicates that the spoken words are intended for an automated agent, having the automated agent perform an action based on the identified user intent.

[0094] In Example 2, the subject of Example 1 includes, wherein the automated agent is a domain-specific automated agent, and the operation also includes providing system prompts, different from structured prompts, as input to the LLM. The system prompts provide multi-sample fine-tuning examples for the domain-specific automated agent's domain to the LLM, each example including a sample of spoken vocabulary and a corresponding structured output indicating a specific intent or missing intent.

[0095] In Example 3, the subject of Example 2 includes, wherein the operation further includes providing system prompts, different from structured prompts, as input to the LLM, the system prompts including instructions for causing the LLM to: analyze the text stream transformed from spoken vocabulary to determine whether the spoken vocabulary within the stream is for a domain-specific automated agent or constitutes an contextual dialogue, to extract the user's intent from the spoken vocabulary intended for a domain-specific automated agent, and to generate a response in a valid JavaScript Object Notation (JSON) format indicating the extracted user intent without answering any questions posed within the text stream.

[0096] In Example 4, the subject of Example 3 includes a structured output in JSON format, and the structured output includes a field for the intent of the identified user, which is populated when the inference is positive.

[0097] In Example 5, the subject matter of Examples 1 through 4 includes, wherein the structured prompt further includes instructions for instructing the LLM to use contextual information from previous interactions to correct errors in the transformed text, and the operation further includes receiving the corrected text from the LLM as part of the structured output before the action is performed by the automation agent.

[0098] In Example 6, the subject matter of Examples 1 through 5 includes an automated agent being integrated into an augmented reality (AR) device, and actions performed by the automated agent including displaying relevant information via an application running within the AR environment of the AR device.

[0099] In Example 7, the topics of Examples 1 through 6 include a speech-to-text conversion process that includes a pause detection feature that identifies the end of a spoken sentence and triggers the transmission of structured cues to the LLM.

[0100] In Example 8, the subject matter of Examples 1 through 7 includes an LLM configured to utilize the ability to call functions to ensure that structured output is provided in a standardized format. This ability enables the LLM to execute predefined functions within structured prompts that correspond to specific tasks, including correcting transcription errors caused by converting captured spoken words from ambient audio into text using a speech-to-text conversion process, and inferring user intent from the text corresponding to the captured spoken words.

[0101] In Example 9, the subject matter of Examples 1 through 8 includes an automation agent integrated into the vehicle's infotainment system, and actions performed by the automation agent include receiving spoken commands related to vehicle control functions, such as adjusting climate settings, setting a navigation destination, or activating windshield wipers, and wherein the LLM is fine-tuned to recognize and process vehicle-specific commands.

[0102] Example 10 is a computer-implemented method for processing spoken language to determine a user's intent to interact with an automated agent, the method comprising: continuously monitoring ambient audio via a microphone integrated with a device; converting captured spoken words from the ambient audio into text using a speech-to-text conversion process; generating a structured cue that includes at least the converted text and instructions for a Large Language Model (LLM), wherein the instructions are configured to request the LLM to infer whether the spoken words are intended for the automated agent; transmitting the structured cue to the LLM; receiving a structured output in a standardized format from the LLM, wherein the structured output includes an inference result indicating whether the spoken words are intended for the automated agent, and if so, identifying the user's intent; and having the automated agent perform an action based on the identified user intent when the inference result indicates that the spoken words are intended for the automated agent.

[0103] In Example 11, the subject of Example 10 includes the following: the automated agent is a domain-specific automated agent, and the method further includes providing system prompts, different from structured prompts, as input to the LLM. These system prompts provide multi-sample fine-tuning examples for the domain of the domain-specific automated agent to the LLM, each example including a sample of spoken vocabulary and a corresponding structured output indicating a specific intent or missing intent.

[0104] In Example 12, the subject of Example 11 includes, wherein the method further includes providing a system prompt, distinct from the structured prompt, as input to the LLM, the system prompt including instructions for causing the LLM to: analyze the text stream transformed from spoken vocabulary to determine whether the spoken vocabulary within the stream is for a domain-specific automated agent or constitutes an contextual dialogue, to extract the user's intent from the spoken vocabulary intended for a domain-specific automated agent, and to generate a response in a valid JavaScript Object Notation (JSON) format indicative of the extracted user intent without answering any questions posed within the text stream.

[0105] In Example 13, the subject of Example 12 includes a structured output in JSON format, and the structured output includes a field for the intent of the identified user, which is populated when the inference is positive.

[0106] In Example 14, the subject matter of Examples 10 through 13 includes, wherein the structured prompt also includes instructions for instructing the LLM to use contextual information from previous interactions to correct errors in the transformed text, and the method also includes receiving the corrected text from the LLM as part of the structured output before the action is performed by the automation agent.

[0107] In Example 15, the subject matter of Examples 10 to 14 includes an automated agent being integrated into an augmented reality (AR) device, and actions performed by the automated agent including displaying relevant information via an application running within the AR environment of the AR device.

[0108] In Example 16, the subject matter of Examples 10 to 15 includes a speech-to-text conversion process that includes a pause detection feature that identifies the end of a spoken sentence and triggers the transmission of structured cues to the LLM.

[0109] In Example 17, the subject matter of Examples 10 through 16 includes an LLM configured to utilize a function call capability that ensures structured output is provided in a standardized format. This function call capability enables the LLM to execute predefined functions within structured prompts that correspond to specific tasks, including correcting transcription errors caused by converting captured spoken words from ambient audio into text using a speech-to-text conversion process, and inferring user intent from the text corresponding to the captured spoken words.

[0110] In Example 18, the subject matter of Examples 10 to 17 includes an automation agent integrated into the vehicle's infotainment system, and actions performed by the automation agent include receiving spoken commands related to vehicle control functions, such as adjusting climate settings, setting a navigation destination, or activating windshield wipers, and wherein the LLM is fine-tuned to recognize and process vehicle-specific commands.

[0111] Example 19 is a non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the processor to perform operations for processing spoken language to determine the intent of a user to interact with a domain-specific automated agent. The operations include: continuously monitoring ambient audio via a microphone integrated with a device; converting captured spoken words from the ambient audio into text using a speech-to-text conversion process; generating a structured cue that includes at least the converted text and instructions for a Large Language Model (LLM), wherein the instructions are configured to request the LLM to infer whether the spoken words are intended for a domain-specific automated agent; transmitting the structured cue to the LLM, wherein the LLM is fine-tuned to process domain-related cue for the domain-specific automated agent; receiving a structured output in a standardized format from the LLM, wherein the structured output includes an inference result indicating whether the spoken words are intended for a domain-specific automated agent, and if so, identifying the user's intent; and when the inference result indicates that the spoken words are intended for a domain-specific automated agent, having the domain-specific automated agent perform an action based on the identified user intent.

[0112] In Example 20, the subject of Example 19 includes, wherein the operation further includes providing system prompts, different from structured prompts, as input to the LLM. The system prompts provide the LLM with multi-sample fine-tuning examples, each example including a sample of spoken vocabulary and a corresponding structured output indicating a specific intent or missing intent.

[0113] Example 21 is at least one machine-readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement any one of Examples 1 to 20.

[0114] Example 22 is a device that includes means for implementing any one of Examples 1 to 20.

[0115] Example 23 is a system for implementing any of Examples 1 through 20.

[0116] Example 24 is a method for implementing any of Examples 1 through 20.

Claims

1. A system for processing spoken language to determine a user's intent to interact with an automated agent, the system comprising: At least one processor; At least one memory component storing instructions, which, when executed by the at least one processor, cause the at least one processor to perform operations, the operations including: Continuous monitoring of ambient audio via a microphone integrated with the device; The captured spoken words from the ambient audio are converted into text using a speech-to-text conversion process. Generate structured prompts, the structured prompts comprising at least transformed text and instructions for a large language model (LLM), wherein the instructions are configured to request the LLM to infer whether the spoken vocabulary is intended for the automated agent; The structured prompt is transmitted to the LLM; Receives structured output in a standardized format from the LLM, wherein the structured output includes inference results indicating whether the spoken vocabulary is intended for the automated agent, and if so, identifies the user's intent; and When the inference result indicates that the spoken vocabulary is intended for the automated agent, the automated agent performs an action based on the identified intent of the user.

2. The system according to claim 1, wherein, The automated agent is a domain-specific automated agent, and the operation further includes providing system prompts, different from the structured prompts, as input to the LLM. The system prompts provide multi-sample fine-tuning examples for the LLM based on the domain of the domain-specific automated agent. Each example includes a sample of spoken vocabulary and a corresponding structured output indicating a specific intent or missing intent.

3. The system according to claim 2, wherein, The operation also includes providing system prompts, different from the structured prompts, as input to the LLM, the system prompts including instructions for the LLM to: analyze the text stream transformed from spoken vocabulary to determine whether the spoken vocabulary within the stream is for a domain-specific automated agent or constitutes an contextual dialogue, to extract the user's intent from the spoken vocabulary intended for a domain-specific automated agent, and to generate a response in valid JavaScript Object Notation (JSON) format indicating the extracted user intent without answering any questions posed within the text stream.

4. The system according to claim 3, wherein, The structured output is in JSON format and includes fields for the identified user's intent, which are populated when the inference is positive.

5. The system according to claim 1, wherein, The structured prompt also includes instructions for the LLM to use contextual information from previous interactions to correct errors in the transformed text, and the operation further includes receiving the corrected text from the LLM as part of the structured output before the action is performed by the automation agent.

6. The system according to claim 1, wherein, The automated agent is integrated into the augmented reality (AR) device, and the actions performed by the automated agent include displaying relevant information via an application executed within the AR environment of the AR device.

7. The system according to claim 1, wherein, The speech-to-text conversion process includes a pause detection feature, which identifies the end of a spoken sentence and triggers the transmission of the structured prompt to the LLM.

8. The system according to claim 1, wherein, The LLM is configured to utilize function call capabilities that ensure the structured output is provided in a standardized format. These capabilities enable the LLM to execute predefined functions within the structured prompts that correspond to specific tasks, including correcting transcription errors caused by converting captured spoken words from the ambient audio into text using the speech-to-text conversion process, and inferring user intent from the text corresponding to the captured spoken words.

9. The system according to claim 1, wherein, The automated agent is integrated into the vehicle's infotainment system, and the actions performed by the automated agent include receiving spoken commands related to vehicle control functions, such as adjusting climate settings, setting a navigation destination, or activating windshield wipers, wherein the LLM is fine-tuned to recognize and process vehicle-specific commands.

10. A computer-implemented method for processing spoken language to determine a user's intent to interact with an automated agent, the method comprising: Continuous monitoring of ambient audio via a microphone integrated with the device; The captured spoken words from the ambient audio are converted into text using a speech-to-text conversion process. Generate structured prompts, the structured prompts comprising at least transformed text and instructions for a large language model (LLM), wherein the instructions are configured to request the LLM to infer whether the spoken vocabulary is intended for the automated agent; The structured prompt is transmitted to the LLM; Receives structured output in a standardized format from the LLM, wherein the structured output includes inference results indicating whether the spoken vocabulary is intended for the automated agent, and if so, identifies the user's intent; and When the inference result indicates that the spoken vocabulary is intended for the automated agent, the automated agent performs an action based on the identified intent of the user.

11. The computer-implemented method according to claim 10, wherein, The automated agent is a domain-specific automated agent, and the method further includes providing system prompts, different from the structured prompts, as input to the LLM. The system prompts provide multi-sample fine-tuning examples for the LLM based on the domain of the domain-specific automated agent, each example including a sample of spoken vocabulary and a corresponding structured output indicating a specific intent or missing intent.

12. The computer-implemented method according to claim 11, wherein, The method further includes providing system prompts, distinct from the structured prompts, as input to the LLM. These system prompts include instructions for the LLM to: analyze a text stream derived from spoken vocabulary to determine whether the spoken vocabulary within the stream is for a domain-specific automated agent or constitutes an contextual dialogue; extract the user's intent from the spoken vocabulary intended for a domain-specific automated agent; and generate a response in valid JavaScript Object Notation (JSON) format indicative of the extracted user intent, without answering any questions posed within the text stream.

13. The computer-implemented method according to claim 12, wherein, The structured output is in JSON format and includes fields for the identified user's intent, which are populated when the inference is positive.

14. The computer-implemented method according to claim 10, wherein, The structured prompts also include instructions for the LLM to use contextual information from previous interactions to correct errors in the transformed text, and the method further includes receiving the corrected text from the LLM as part of the structured output before the automated agent performs the action.

15. The computer-implemented method according to claim 10, wherein, The automated agent is integrated into the augmented reality (AR) device, and the actions performed by the automated agent include displaying relevant information via an application executed within the AR environment of the AR device.

16. The computer-implemented method according to claim 10, wherein, The speech-to-text conversion process includes a pause detection feature, which identifies the end of a spoken sentence and triggers the transmission of the structured prompt to the LLM.

17. The computer-implemented method according to claim 10, wherein, The LLM is configured to utilize function call capabilities that ensure the structured output is provided in a standardized format. These capabilities enable the LLM to execute predefined functions within the structured prompts that correspond to specific tasks, including correcting transcription errors caused by converting captured spoken words from the ambient audio into text using the speech-to-text conversion process, and inferring the user's intent from the text corresponding to the captured spoken words.

18. The computer-implemented method according to claim 10, wherein, The automated agent is integrated into the vehicle's infotainment system, and the actions performed by the automated agent include receiving spoken commands related to vehicle control functions, such as adjusting climate settings, setting a navigation destination, or activating windshield wipers, wherein the LLM is fine-tuned to recognize and process vehicle-specific commands.

19. A non-transitory computer-readable storage medium storing instructions, which, when executed by at least one processor, cause the at least one processor to perform operations for processing spoken language to determine the intent of a user to interact with a domain-specific automated agent, the operations comprising: Continuous monitoring of ambient audio via a microphone integrated with the device; The captured spoken words from the ambient audio are converted into text using a speech-to-text conversion process. Generate structured prompts, which include at least transformed text and instructions for a large language model (LLM), wherein the instructions are configured to request the LLM to infer whether the spoken vocabulary is intended for a domain-specific automated agent; The structured hints are transmitted to the LLM, wherein the LLM is fine-tuned to process domain-specific hints for the domain-specific automated agent; Receives structured output in a standardized format from the LLM, wherein the structured output includes inference results indicating whether the spoken vocabulary is intended for a domain-specific automated agent, and if so, identifies the user's intent; and When the inference result indicates that the spoken vocabulary is intended for a domain-specific automated agent, the domain-specific automated agent performs an action based on the identified intent of the user.

20. The non-transitory computer-readable medium according to claim 19, wherein, The operation also includes providing system prompts, different from the structured prompts, as input to the LLM. The system prompts provide the LLM with multi-sample fine-tuning examples, each example including a sample of spoken vocabulary and a corresponding structured output indicating a specific intent or missing intent.

Images