EVENT-DRIVEN ARCHITECTURE FOR INTERACTIVE SYSTEMS AND APPLICATIONS

Abstract

In various examples, an interactive agent platform hosting an interactive agent may represent and / or communicate human-machine interactions and associated events using a standardized interaction modeling API and / or an event-driven architecture. In one example implementation, a standardized interaction modeling API serves as a common protocol in which components use a standardized interaction categorization scheme to represent all agent and user activities as actions in a standardized form, represent states of multimodal user and agent actions as events in a standardized form, implement standardized, mutually exclusive modalities that define how conflicts between standardized categories of actions are resolved (e.g.,it is not possible to say two things at once, while it is possible to say something and make a gesture at the same time), and / or to implement standardized protocols for any number of standardized modalities and actions regardless of the implementation.

Description

BACKGROUND

[0001] Conversational artificial intelligence (AI) enables computers to converse with users using natural language, enabling human-like interactions and communication. Multimodal conversational AI integrates multiple communication modes (e.g., text, speech, gestures, emotions, visuals, etc.), enabling richer and more natural interactions between users and AI systems. Multimodal conversational AI is being integrated into an increasing number of applications, from simple chatbots and voicebots to fully interactive avatars (also known as digital characters or digital avatars) and robots. However, designing these applications to interact with the user in a flexible and robust manner is a challenging task.

[0002] First of all, designing convincing avatar interactions is quite challenging. Interactions with an avatar are increasingly perceived as interactions with another person, but interactions with other people can be complex, nuanced, multimodal, and non-sequential. Because we humans have communicated with other people throughout our lives, we can often quickly recognize when there is a sense of dissonance, discomfort, or incongruity in a conversation, or when our counterpart is reacting in an unusual or unnatural way. Such nuances are not easy to model, and the shortcomings of modeling human interactions can become more apparent the closer they are to reality.A similar effect is known as the "uncanny valley" in three-dimensional (3D) graphics, where 3D models of people that are highly lifelike but still have subtle imperfections or discrepancies with real people can appear uncanny and evoke feelings of unease or discomfort.

[0003] To illustrate the complexity of many of these design challenges, consider what would be required to transform a chatbot that can only interact through a loop-based text dialog into a multimodal interactive avatar that can be seen and spoken to. Moving from a single interaction modality (e.g., text dialog) to supporting multiple user input interaction modalities (e.g., text, touch, voice, gestures, emotions, etc.) and / or supporting multiple output interaction modalities to respond to the user (e.g., through text / voice, graphical user interface (GUI), animations, sound effects, lights, etc.) increases the complexity enormously. Moving from a loop-based interactive system to one that supports non-sequential interactions (e.g., multiple simultaneous, potentially higher-order inputs and / or outputs) increases the complexity even further.In many cases, interactive systems that offer only a single interaction modality or use loop-based interactions simply do not work for multimodal and / or non-sequential interactive systems.

[0004] For some interactive systems, such as those that provide interactive avatars, it may be desirable to support speech input and output and also utilize screen space by displaying dynamic information on the screen and allowing the user to interact with that information. For example, it may be desirable to dynamically adapt the visual representation on the screen to the content of the conversation in order to provide useful contextual information (e.g., by displaying a visual representation of some options verbally offered to the user by an avatar). Current AI models are tailored to process verbal input and output (e.g., speech in the form of text) but lack the capability to directly generate corresponding visual elements or graphical user interfaces.This is just one example of how traditional tools are limited in their ability to support multimodal interactions.

[0005] Furthermore, there are many different forms of AI systems that provide multimodal conversational experiences, and different systems rely on a variety of different technologies. This means that most interactive systems connect their individual components (e.g., a decision-making engine, AI models such as deep neural networks (DNNs) and machine learning models, cameras, user interfaces, etc.) in application-specific ways, using bespoke application programming interfaces (APIs) and architectures tailored to each individual interactive system. Today, there is a wealth of toolkits and frameworks for modeling conversational interactions, and many different applications are being developed based on these technologies.As a result, components cannot be easily replaced or updated based on the latest technologies, increasing the time from research to product. Furthermore, heterogeneous systems that represent multimodal interactions in different ways make it difficult to train AI models on historical multimodal interactions, thereby limiting the ability to improve the user experience over time. Furthermore, in many systems, the interaction data is closely tied to the specific implementation of the interactive system. For example, the specific format a particular interactive system uses to encode or represent interaction data (e.g., how a human speaks to a bot) typically depends on the specific implementation.This makes it difficult to think about multimodal interactions without knowing the technical intricacies of a particular interactive system, limiting the ability to leverage existing frameworks or extend existing technologies.

[0006] There is therefore a need for improved systems that enable and support the development and / or deployment of multimodal interactive systems. SUMMARY

[0007] The invention is defined by the claims. To illustrate the invention, aspects and embodiments are described herein, which may or may not fall within the scope of the claims.

[0008] Various examples are presented. An interactive agent platform hosting an interactive agent can represent and / or communicate human-machine interactions and associated events using a standardized interaction modeling API and / or an event-driven architecture. In one example implementation, a standardized interaction modeling API serves as a common protocol in which components use a standardized interaction categorization scheme to represent all agent and user activities as actions in a standardized form, represent states of multimodal user and agent actions as events in a standardized form, implement standardized, mutually exclusive modalities that define how conflicts between standardized categories of actions are resolved (e.g.,it is not possible to say two things at once, while it is possible to say something and make a gesture at the same time), and / or to implement standardized protocols for any number of standardized modalities and actions regardless of the implementation.

[0009] Embodiments of the present disclosure relate to the development and deployment of interactive systems, such as those implementing an interactive agent (e.g., bot, avatar, digital human, or robot). For example, systems and methods are disclosed that implement or support an interaction modeling language and / or an interaction modeling application programming interface (API) that utilizes a standardized interaction categorization scheme, multimodal human-machine interactions, backchanneling, an event-driven architecture, interaction flow management, provisioning using one or more large-scale language models, sensory processing and action execution, interactive visual content, interactive agent (e.g., bot) animations, expected actions and signaling, and / or other features.

[0010] For example, an interactive agent platform hosting the development and / or deployment of an interactive agent (e.g., a bot or robot) may provide an interpreter or compiler that interprets or executes code written in the interaction modeling language, and a designer may provide customized code written in the interaction modeling language for the interpreter to execute. The interaction modeling language may be used to define a sequence of interactions that instruct the interpreter (e.g., an event-driven state machine) which actions or events to generate in response to a sequence of detected and / or executed human-machine interactions. The interaction categorization scheme may categorize interactions according to standardized interaction modalities (e.g., BotUpperBodyMotion) and / or corresponding standardized action categories or types (e.g.,The interaction modeling language can classify bot actions (e.g., BotPose, BotGesture) using standardized action keywords, and the interaction modeling language can use keywords, commands, and / or syntax that incorporates or categorizes standardized modalities, action types, and / or event syntax defined by the interaction categorization scheme. Thus, a flow can be used to model a bot intent or inferred user intent, which a designer can use to create more complex interaction patterns with the interactive agent.

[0011] In some embodiments, one or more flows may implement the logic of an interactive agent and specify a sequence of multimodal interactions. For example, an interactive avatar (e.g., an animated digital character) or other bot may support any number of simultaneous interaction modalities and corresponding interaction channels to engage with the user, such as channels for character or bot actions (e.g., speech, gestures, posture, movement, voice breaks, etc.), scene actions (e.g., two-dimensional (2D) GUI overlays, 3D scene interactions, visual effects, music, etc.), and user actions (e.g., speech, gestures, posture, movement, etc.). Actions based on different modalities may occur sequentially or in parallel (e.g., waving and greeting). Thus, the interactive agent may execute any number of flows that specify a sequence of multimodal actions (e.g.,different types of bot or user actions) using any number of supported interaction modalities and corresponding interaction channels.

[0012] To make conversations with an avatar or other interactive agent feel more natural, some embodiments use backchanneling to provide feedback to the user while they are speaking or doing something recognizable. For example, backchanneling can be implemented by triggering interactive agent postures (e.g., based on the user or avatar speaking, or based on the avatar waiting for a user response), such as mirroring postures (e.g., when an interactive avatar essentially mirrors the user's body posture), short vocal bursts such as "yes," "aha," or "hmm" while the user is speaking (e.g., to signal to the user that the interactive agent is listening), gestures (e.g., shaking the head of the interactive bot or robot), and / or others.This allows a designer to define various feedback channel techniques that make conversations with an interactive agent seem more natural.

[0013] In some embodiments, a platform hosting the development and / or deployment of an interactive system may represent and / or communicate human-machine interactions and associated events using a standardized interaction modeling API, a plug-in, and / or an event-driven architecture. In an example implementation, a standardized interaction modeling API serves as a common protocol in which components of the interactive system use a standardized interaction categorization scheme to represent all bot and user activities as actions in a standardized form, represent states of multimodal bot and user actions as events in a standardized form, implement standardized mutually exclusive modalities that define how conflicts between actions in standardized action categories are resolved (e.g.,say two things at once). For example, it is not possible to say two things at once, while it may be possible to say something and make a gesture at the same time), and / or implement standardized protocols for any number of standardized modalities and actions regardless of the implementation.

[0014] In some embodiments, an interpreter for an interactive agent may be programmed to traverse one or more flows until it reaches an event matcher. A top-level flow may specify instructions to activate any number of flows with any number of event matchers. The interpreter may use any suitable data structure to track active flows and corresponding event matchers (e.g., The interpreter may use an event-driven state machine that listens for various events and triggers corresponding actions specified in matching flows (with event matchers corresponding to an incoming interaction modeling API event).As such, the interpreter can execute a primary processing loop that processes incoming interaction modeling API events and generates outgoing interaction modeling API events that implement the interactive agent.

[0015] In some embodiments, an interaction modeling language and corresponding interpreter may support the use of natural language descriptions and one or more language models (e.g., large language models (LLMs), visual language models (VLMs), multimodal language models, etc.) to reduce the cognitive load on the programmer and facilitate the development and deployment of more complex and nuanced human-machine interactions. For example, the interpreter may analyze one or more specified flows (e.g., at design time) that define the logic of an interactive agent and determine if the specified flows lack a corresponding flow description, and, if so, request a language model to create one based on the flow name and / or instructions.Additionally or alternatively, the interpreter may determine whether a command sequence is missing from certain flows and, if so, request a language model to generate one. In some embodiments, the interpreter may determine whether an event matches an active event matcher for any of the active flows using target event parameters and / or parameter values ​​generated using the language model, request the language model to determine whether an event matches the flow description, request the language model to determine whether a non-matching event matches the name and / or instruction(s) of an active flow, request the language model to generate a flow in response to a non-matching event, and / or in other ways.

[0016] In general, an interactive agent platform hosting the development and / or deployment of an interactive agent (e.g., chatbot, voicebot, digital assistant, interactive avatar, non-player character (NPC), digital human, robot, etc.) may support any number of input and output interaction channels. In some embodiments where sensory processing, interaction decision, and action execution are decoupled, the interactive agent platform may support a sensory server for each input interaction channel and an action server for each output interaction channel.Sensory servers for corresponding input interaction channels can translate inputs or non-standardized technical events into the standardized format and generate corresponding Interaction Modeling API events. An Interaction Manager can process these incoming Interaction Modeling API events and generate outgoing Interaction Modeling API events that represent commands to execute an action. Action servers for corresponding output interaction channels can interpret these outgoing Interaction Modeling API events and execute the corresponding commands. Sensor and / or action servers can be implemented using a combination of asynchronous event loops and processes to ensure that multiple user sessions and system pipelines can be served in parallel.To handle all supported actions for at least one interaction modality, an action server can be equipped with an action handler for each standardized category or type of action and / or action event supported by the interaction modeling language and / or defined by the interaction categorization schema for a given interaction modality. Each action server can manage the lifecycle of all actions under its purview and synchronize action state changes with specific conditions (e.g., wait to start an action until a previous action of the same modality completes, synchronize the completion of two different actions in different modalities, synchronize the start of one action with the end of another action, etc.).

[0017] In some embodiments, an interactive agent platform hosting the development and / or deployment of an interactive agent may use a graphical user interface (GUI)—or more generally, a UI service—to perform interactive visual content actions and generate corresponding GUIs. For example, an interaction modeling API may use a standardized interaction categorization scheme that defines a standardized format (e.g., standardized and semantically meaningful keywords) for specifying events related to interactive visual content actions (e.g., actions that instruct an overlay or other arrangement of visual content that complements a conversation with an interactive agent), such as a visual information scene (e.g., displaying non-interactive content such as images, text, and videos alongside the interaction), visual choices (e.g.,Presenting a visual choice to the user in the form of multiple buttons or a list of options) and / or visual forms (e.g., presenting a visual web form to the user for entering user information). A sensor server can convert detected interactions with GUI interaction elements into standardized interaction modeling API events that represent possible interactions with those elements in a standardized format. Standardized interaction modeling API events can be processed by an interpreter that implements the interactive agent's logic to generate outgoing interaction modeling API events containing commands to update the graphical user interface.An action server implementing a GUI service can translate a standardized representation of a particular GUI, specified by a particular Interaction Modeling API event, into a representation (e.g., JavaScript Object Notation (JSON)) of a modular GUI configuration that defines blocks of visual content specified or otherwise presented by the Interaction Modeling API event, such as paragraphs, images, buttons, multiple-select fields, and / or other types. The GUI service can use these blocks to populate a visual layout (e.g., a template or shell) for a GUI overlay (e.g., a HyperText Markup Language (HTML) page renderable in a web browser) with visual content specified by the Interaction Modeling API event.For example, a visual layout representing a graphical user interface specified by an interaction modeling API event can be generated and presented to the user (e.g., via a user interface server).

[0018] In some embodiments, interaction modeling API events specifying commands to execute bot expressions, poses, gestures, or other interactions or movements may be generated (e.g., by an interpreter executing code written in an interaction modeling language) and translated into corresponding bot animations. More specifically, an interpreter implementing the logic of an interactive agent may generate an interaction modeling API event representing a target bot expression, pose, gesture, or other interaction or movement using a standardized interaction categorization scheme, and an action server implementing an animation service may use the standardized representation of the target bot movement to identify a corresponding supported animation or generate a matching animation on the fly.The animation service may implement an action state machine and an action stack for all events related to a particular interaction modality or action category (e.g., bot gestures), connect to an animation graph that implements a state machine of animation states and transitions between animations, and instruct the animation graph to set a corresponding state variable based on a command to change the state (e.g., initialize, stop, or resume) of a bot movement represented by an interaction modeling API event.

[0019] In some embodiments, an interpreter associated with an interactive agent may generate Interaction Modeling API events that communicate an expectation that certain events will occur and that command or otherwise trigger corresponding preparatory actions, such as turning down the speaker volume in anticipation of user speech, activating computer vision and / or machine learning algorithms in anticipation of vision events, and / or signaling to the user that the interactive agent is waiting for input (e.g., for a particular user interaction modality). Interaction Modeling API events may include one or more fields representing an expectation that a particular target event will occur, using a standardized interaction categorization scheme that identifies expectations as a supported action type (e.g.,ExpectationBotAction, ExpectationSignalingAction) and represents corresponding expectation events (e.g., indicating the state of the expectation such as start, stop, and finished), expected target events (e.g., utteranceUserActionStarted), and / or expected input interaction modalities (e.g., userSpeech) using standardized (e.g., natural language, semantically meaningful) keywords and / or commands.

[0020] Thus, the present techniques can be used to develop and / or deploy interactive bots or robots (e.g., chatbots, voicebots, digital assistants, interactive avatars, non-player characters (NPCs), digital humans, etc.) that perform more complex, nuanced, multimodal, non-sequential, and / or realistic conversational AI and / or other types of human-machine interactions than previous techniques.

[0021] The disclosure extends to all novel aspects or features described and / or illustrated herein.

[0022] Further features of the disclosure are characterized by the independent and dependent claims.

[0023] Any feature of one aspect of the disclosure may be applied to other aspects of the disclosure, in any suitable combination. In particular, method aspects may be applied to device or system aspects, and vice versa.

[0024] Furthermore, functions implemented in hardware may also be implemented in software, and vice versa. Any reference to software and hardware features in this document should be interpreted accordingly.

[0025] Any system or device feature described herein may also be provided as a method feature, and vice versa. System and / or device aspects described functionally (including means-plus-functional features) may alternatively be expressed in terms of their corresponding structure, e.g., an appropriately programmed processor and associated memory.

[0026] It should also be appreciated that certain combinations of the various features described and defined in all aspects of the disclosure may be implemented and / or provided and / or used independently of one another.

[0027] The disclosure also provides computer programs and computer program products comprising software code that, when executed on a data processing device, is capable of performing any of the methods described herein and / or embodying any of the device and system features described herein, including all or part of the component steps of a method.

[0028] The disclosure also provides a computer or computer system (including networked or distributed systems) having an operating system that supports a computer program for performing any of the methods described herein and / or for embodying any of the device or system features described herein.

[0029] The disclosure also provides a computer-readable medium on which one or more of the aforementioned computer programs are stored.

[0030] The disclosure also provides a signal carrying one or more of the above-mentioned computer programs.

[0031] The disclosure extends to methods and / or devices and / or systems as described herein with reference to the accompanying drawings.

[0032] Aspects and embodiments of the disclosure will now be described, by way of example only, with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present systems and methods for developing and deploying interactive systems are described in detail below with reference to the attached figures, where: Fig. 1 shows an example of an interactive system in accordance with some embodiments of the present disclosure; Fig.2 shows an example of an interaction modeling API according to some embodiments of the present disclosure; Fig. 3 illustrates an example of interactive systems that may be supported by an example interaction modeling API and / or an example interaction modeling language in accordance with some embodiments of the present disclosure; Fig. 4 shows an example of modality policies in accordance with some embodiments of the present disclosure; Fig. 5 shows an example of an interaction categorization scheme according to some embodiments of the present disclosure; Fig. 6 shows an example of an event-driven interactive system according to some embodiments of the present disclosure; Fig. 7 shows an example of an interaction manager according to some embodiments of the present disclosure; Fig. 8 is a flow diagram illustrating an exemplary event-driven state machine for an interaction manager in accordance with some embodiments of the present disclosure; Fig. 9 shows an example of an action server according to some embodiments of the present disclosure; Fig. 10 shows an example flow of events through an example action server in accordance with some embodiments of the present disclosure; Fig. 11 shows an example of the life cycle of an action in accordance with some embodiments of the present disclosure; Fig. 12A-12D illustrate some example action handlers for an example GUI service and an example animation service according to some embodiments of the present disclosure; Fig.13A-13F illustrate some example interactions with a visual selection in accordance with some embodiments of the present disclosure; Fig. 14A illustrates an example graphical user interface displaying an interactive avatar and interactive visual content, and Fig. 14B-14L illustrate some example layouts of visual elements for interactive visual content, in accordance with some embodiments of the present disclosure; Fig. 15 illustrates an example event flow for a user utterance action in an implementation in which a user speaks with an interactive avatar, in accordance with some embodiments of the present disclosure; Fig.16 shows an example event flow for a user utterance action in an implementation in which a user speaks with a chatbot, in accordance with some embodiments of the present disclosure; Fig. 17 shows an example event flow for a bot expectation action in an implementation in which a user speaks with an interactive avatar, in accordance with some embodiments of the present disclosure; Fig. 18 is a flow diagram illustrating a method for generating a representation of a responsive agent action categorized using an interaction categorization scheme, in accordance with some embodiments of the present disclosure; Fig.19 is a flow diagram illustrating a method for generating a representation of an addressable agent action based at least on the execution of one or more interaction flows, in accordance with some embodiments of the present disclosure; Fig. 20 is a flowchart illustrating a method for triggering an interactive avatar to provide backchannel feedback in accordance with some embodiments of the present disclosure; Fig. 21 is a flow diagram illustrating a method for generating an interaction modeling event that commands an interactive agent to perform an addressable agent or scene action, in accordance with some embodiments of the present disclosure; Fig.22 is a flow diagram illustrating a method for triggering one or more responsive agent or scene actions specified by one or more matching interaction flows, in accordance with some embodiments of the present disclosure; Fig. 23 is a flow diagram illustrating a method for generating an approachable agent or scene action based at least on prompting from one or more large language models, in accordance with some embodiments of the present disclosure; Fig. 24 is a flow diagram illustrating a method for generating one or more outgoing interaction modeling events that instruct one or more action servers to perform one or more addressable agent or scene actions, in accordance with some embodiments of the present disclosure; Fig.25 is a flow diagram illustrating a method for generating a visual layout representing an update specified by an event, in accordance with some embodiments of the present disclosure; Fig. 26 is a flowchart illustrating a method for triggering an animation state of an interactive agent according to some embodiments of the present disclosure; Fig. 27 is a flowchart illustrating a method for performing one or more preparatory actions according to some embodiments of the present disclosure; Fig. 28A is a block diagram of an exemplary generative language model system suitable for use in implementing at least some embodiments of the present disclosure; Fig.28B is a block diagram of an exemplary generative language model including a transformer encoder-decoder suitable for use in implementing at least some embodiments of the present disclosure; Fig. 28C is a block diagram of an exemplary generative language model including a pure decoder-transformer architecture suitable for use in implementing at least some embodiments of the present disclosure; Fig. 29 is a block diagram of an example content streaming system suitable for implementing some embodiments of the present disclosure; Fig. 30 is a block diagram of an example computing device suitable for use in implementing some embodiments of the present disclosure; and Fig.31 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0034] Systems and methods are disclosed that relate to the development and deployment of interactive systems, such as those that implement an interactive agent (e.g., bot, non-player character, digital avatar, digital human, robot, etc.). For example, systems and methods are disclosed that implement or support an interaction modeling language and / or an interaction modeling API that employs a standardized interaction categorization scheme, multimodal human-machine interactions, backchanneling, an event-driven architecture, interaction flow management, deployment using one or more language models (e.g., LLMs, VLMs, multimodal language models, etc.), sensory processing and action execution, interactive visual content, interactive agent animations, expected actions and signaling, and / or other features.

[0035] Introduction. At a high level, an interactive agent (e.g., chatbot, voicebot, digital assistant, interactive avatar, non-player character, robot, etc.) that participates in AI conversations or other types of human-machine interactions can be created and / or executed using an interactive agent platform. When designing such a platform and / or an interactive system implementing an interactive agent, it can be instructive to consider some possible features that can contribute to convincing human-machine interactions and interaction flows.

[0036] Multimodality is a factor that can contribute to compelling human-computer interactions. For example, when designing an interactive avatar, a designer may want to support many different modalities of interaction with the user. A designer may want their avatar to speak, make gestures, point in a graphical user interface, make sounds, or interact in some other way. Likewise, a designer may want to support different types of input interaction modalities or ways a user can interact with the system. For example, a designer may support detection and response when a user verbally answers a question, selects an item on the screen, or makes a gesture such as a thumbs-up to confirm a selection.One possible implication of multimodality is that a designer may desire flexibility in how interactions are temporarily aligned. For example, a designer may want an avatar to say something while performing a gesture, or they may want to trigger a gesture at a specific time when the avatar says something specific. Therefore, it may be desirable to support different types of independently controllable interaction modalities.

[0037] Backchanneling is a useful tool for promoting effective human communication. It helps convey active listening and engagement, signaling to the speaker that their message is being heard and understood. This feedback loop makes conversation flow more smoothly, helps build connection, and encourages people to continue talking and sharing their thoughts. If a designer wants their avatar to attempt to utilize backchanneling to make the avatar appear more human and facilitate more natural interaction, it may be desirable to support backchannels.

[0038] Some designers may want to support nonlinear interactions. Designers often try to avoid the impression of predictable, guided, or simplified interactions, which can make the user feel like they're following a predetermined or set course, lacking spontaneity or freedom. Even if a desired customer experience inherently involves some degree of linearity, it may be desirable to support interactions in a way that allows users to break out of a strict logic.

[0039] Proactivity can be a helpful feature to implement. Many users are now accustomed to voice assistants, but the conversation pattern with these digital assistants is often very simple. The user initiates a conversation with a trigger word and asks a question or gives a command. The voice assistant responds to this prompt by directly performing the action, answering the question, or following up with a clarifying question. While this interaction pattern may be efficient for retrieving information or setting a timer, it is not very engaging and often unsuitable for more complex use cases. Instead, avatars should be proactive and rephrase questions if the user does not understand them, guide them back to a specific process if they abandon the conversation, or offer alternative ways to complete a specific task.Proactivity can be very helpful in preventing the interaction from becoming monotonous and users becoming disengaged or unsure how to proceed in the conversation.

[0040] Some designers may want to harness the power of language models (e.g., LLMs, VLMs, etc.). For example, a designer might want an avatar or chatbot to use an LLM to make its interactions with the user more natural and adapt to the current interaction context. Some LLM uses could help avoid common pitfalls in avatar or chatbot experiences, such as a bot repeating the same answers over and over again or simple questions not eliciting the intended response. In an interactive avatar environment, a designer may want to use LLMs to generate verbal and / or nonverbal responses such as gestures or facial expressions, or they may even use LLMs to provide useful information in the user interface. As such, it may be desirable to support a variety of LLM uses.

[0041] Interaction modeling language and interaction categorization scheme. In general, human-machine interactions and associated events can be represented and communicated in various ways within an interactive system or an interactive agent platform hosting the development and / or deployment of an interactive system.

[0042] One way to represent and / or communicate interactions is to use an interaction modeling language that specifies user and / or bot interactions and associated events using a standardized interaction categorization scheme. Existing dialogue management techniques such as flowcharts, state machines, and frame-based systems are unable to model highly flexible conversational flows, such as those expected from a realistic interactive avatar. In contrast, a standardized interaction categorization scheme can provide a semantically meaningful way to categorize, specify, and communicate desired interactions and interaction flows.For example, an interactive agent platform may provide an interpreter or compiler that interprets or executes code written in the interaction modeling language, and a designer may provide custom code written in the interaction modeling language to the interpreter for execution. An interaction modeling language that uses a standardized categorization scheme for interactions offers a number of technological advantages, from easing the designer's work by reducing the cognitive load of developing an interactive system, to supporting a variety of interactions or features (such as those described above) that a designer can draw on to customize an interactive system, to promoting interoperability by standardizing the representation of interactions.

[0043] One possible goal is to reduce the cognitive load of a developer when writing code that implements an interactive system. Existing programming languages ​​would require the developer to write functions that implement interactions using generic keywords and commands. However, some embodiments abstract from lower-level programming in favor of a semantically more intuitive representation of an interaction: an interaction flow. Interactions typically occur in flows, so an interaction modeling language can be used to define a flow of interactions. A flow can be viewed as analogous to a function but may consist of primitives that include semantically meaningful (e.g., natural language) keywords and commands that represent events (e.g., something happened) and actions (e.g.,something must happen) using an interaction categorization scheme. An interaction flow can be used as a mechanism to instruct an interpreter (e.g., an event-driven state machine) which actions or events should be generated in response to a sequence of detected and / or executed human-machine interactions.

[0044] In some embodiments, the interaction categorization scheme may classify interactions by standardized interaction modality (e.g., BotUpperBodyMotion) and / or corresponding standardized action category or type (e.g., BotPose, BotGesture) using standardized action keywords. The scheme may support any number and type of interactions or communication methods (e.g., user interactions with the system, bot interactions with the user, bot expectation actions and expectation signaling, scene actions, etc.). The state of an action (e.g., an observed state of a user action, a current state of a bot or scene action) and / or a command to change the state of a bot or scene action may be represented using standardized event keywords, commands, and / or syntax. For example, action events (e.g.,a started or stopped user or bot action) be represented by an event specifier with standardized syntax (e.g., event name and / or identifier with a keyword identifying the standardized category or type of action and a specifier for the user or bot action state).

[0045] In some embodiments, the interaction modeling language may use keywords, commands, and / or syntax that incorporates or categorizes standardized modalities, action types, and / or event syntax defined by an interaction categorization scheme. For example, a line of instructions in a flow may include an event trigger (e.g., using a keyword such as "send") that causes the interpreter to generate a specific event when a specific condition is met (e.g., events representing commands to perform a bot action may trigger the action to be performed, or events representing a change in the state of a detected user action may trigger a corresponding bot action), or an event matcher (e.g.,using a keyword such as "match") that causes the interpreter to pause the flow and wait for a specific event before resuming the flow. The event trigger and event match may specify corresponding trigger and match conditions using an event specifier that includes a standardized event name or identifier (e.g., a keyword identifying a standardized category or type of action, paired with a corresponding action state specifier or a command to change an action state) and an argument that specifies one or more conditions that the specified event must satisfy (e.g., using predefined parameters and supported values ​​or a natural language description). In some embodiments, the interpreter may infer a specific action state (e.g.,"Completed") when an event specifier contains an action but does not specify the status (e.g., the name of the action can be specified as an abbreviation to specify the completion of the action).

[0046] Consider the UserSpeech modality and its corresponding user action, Utterance. Suppose a user makes an utterance that is recognized by an interactive system. Possible examples of this type of action include the user typing into a text interface to interact with a bot or the user speaking to an interactive avatar. This action can be classified as a User Utterance, and the supported action events for this action can include UtteranceUserActionStarted (the user has started producing an utterance) or UtteranceUserActionFinished (the user utterance has finished). An example of a flow statement waiting for the user to say something specific could be: "match UtteranceUserActionFinish(text="How are you?", speed="slow", volume="normal")".In this example, the event identifier is a keyword in bold that links the standardized action category (UtteranceUserAction) to a representation of the specified action status (Completed).

[0047] In some embodiments, an interaction modeling language and its associated interpreter may support any number of keywords that serve to parallelize the execution and matching of actions and flows (e.g., send, match, start, stop, await, activate). In contrast to conventional dialog modeling languages, in which instructions are always considered in a sequential order, some embodiments may support a keyword (e.g., start) that instructs the interpreter to start a specific action in a specific (e.g., standardized) action category or flow and to continue iterating the parent flow without waiting for the started action or child flow to finish. Some may support a keyword (e.g.,stop) that instructs the interpreter to stop the started action or child flow, and some support a keyword (e.g., await) that instructs the interpreter to wait for the started action or child flow to end before continuing with the parent flow. In some embodiments, the interpreter may implement some keywords (e.g., start, await) with other keywords (e.g., send, match) to send or wait for an event to occur. In some implementations, after starting a flow, the interpreter executes all actions in the specified flow up to the first matching statement. Then, if the statement matches, the interpreter may execute subsequent actions in the specified flow up to the next matching statement or the end of the flow, and repeat until the flow ends.

[0048] In some scenarios, a developer may want a child flow to automatically restart upon completion. This can be useful for certain types of flows, such as those attempting to trigger an action that depends on a recurring event. Therefore, some embodiments may support a keyword (e.g., activate) that instructs the interpreter to automatically restart a flow upon its completion. If an activated flow does not contain event matching, in some embodiments the interpreter executes the flow only once but keeps it in an active state so that all child flows also remain active.

[0049] Some embodiments may support a keyword that instructs the interpreter to exit a flow (e.g., return) or abort a flow (e.g., abort), and a flow may instruct the interpreter to determine and return a value. Because some embodiments support multiple active flows, some implementations of the interpreter launch a top-level, root, or main flow (e.g., at startup) that serves as the parent of all other flows. This hierarchy allows for greater abstraction and encapsulation capabilities than previous techniques. In some embodiments, the event matching command may accept a particular flow name or identifier and a particular flow event (e.g., start, exit, failure, interrupt, resume) as an argument, which the interpreter can use as an instruction to match against a corresponding flow event.

[0050] Therefore, in some embodiments, all flows represent a corresponding interaction pattern. In some of these embodiments, a flow can be used to model a bot intent or an inferred user intent, which a designer can use to create more complex interaction patterns. In some of these implementations, a flow effectively describes an expected interaction pattern. When the interpreter starts a flow, it can mark that flow as active and attempt to match the contained pattern of event matching statements with events that represent the ongoing interaction. If the interpreter determines that a matching statement is satisfied by an event, it can advance a corresponding flow header to the next matching statement and execute any non-matching statements in between.The interpreter can be programmed to execute the instructions specified in a flow sequentially, generate any events specified by an event trigger, and stop when a flow header reaches an event handler, an exception, or the end of a flow. To illustrate how a flow can be used to implement different types of interaction patterns and features, consider the following application examples.

[0051] Multimodal interactions. In some embodiments, one or more flows may specify a sequence of multimodal interactions. While traditional chatbots use a loop-based dialogue, an interactive avatar (e.g., an animated digital character) or other bot may support any number of interaction modalities and corresponding interaction channels to engage with the user, such as channels for character or bot actions (e.g., speech, gestures, posture, movement, voice breaks, etc.), scene actions (e.g., two-dimensional (2D) GUI overlays, 3D scene interactions, visual effects, music, etc.), and user actions (e.g., speech, gestures, posture, movement, etc.). Thus, a flow may specify a sequence of multimodal actions (e.g.,different types of bot or user actions) using any number of supported interaction modalities and corresponding interaction channels.

[0052] For example, consider the following example flow, which wraps an action command to start a bot utterance for readability and ease of programming: Flow bot says $statement Send BotUtterStartAction(action_id, "$utterance") Match with BotUtterActionEnded(action_id)

[0053] The command to start the bot utterance action can now be triggered with a statement that simply writes its name and specifies the desired utterance: bot say "Hello world." In this example, when the interpreter executes this statement, it looks for the flow "bot say," which defines an event trigger that creates an event that, when executed, starts the bot utterance action with the specified text (in this case, "Hello world"), and an event handler that waits for the bot utterance action to end. Here is a sample flow that similarly wraps a command to start a bot gesture action: Flow Bot Gesture $gesture send BotGestureStartAction(action_id, "$gesture") Match with BotGestureActionEnded(action_id)

[0054] By defining such a wrapper flow, a designer can instruct the bot to perform a gesture (e.g., trigger the action command "Start bot gesture") by simply writing the wrapper flow name and specifying the desired gesture, e.g., the bot gesture "Wave with both hands."

[0055] Conceptually, actions based on different modalities can occur sequentially or in parallel (e.g., waving and saying hello). Therefore, it may be desirable to give the designer precise temporal control over the supported actions and their coordination. Consider, for example, bot actions such as bot utterances and bot gestures. In some embodiments, a flow can specify the invocation of these actions sequentially, as follows: Bot says "Hello world" Bot gesture "Wave with both hands"

[0056] In this example, because the wrapper flow for the "say bot" action contains an event handler that tells the interpreter to wait for the action to finish before continuing, the bot gesture doesn't start until the "say bot" (or "wait bot say") action finishes.

[0057] Because these two actions are two different modalities, they can be executed simultaneously in some implementations. One way to trigger the simultaneous execution of these two actions is to combine them in an "and" group (e.g., defined by a keyword like "and") to launch them in parallel: Bot says "Hello world" and bot gesture "Wave with both hands."

[0058] More complex action groups can be defined with an "or" group (e.g., using a keyword like "or"), as in the following example: Bot says "Hello World" and (Bot gesture "Wave with both hands" or Bot gesture "Smile")

[0059] For example, a designer can use an "or" action group to specify alternative actions. In this example, the resulting action is either the bot saying "Hello world" and waving its hands, or it saying "Hello world" and smiling. Another example of parallel execution of two actions is using a keyword like "start" to initiate two actions in parallel: Start the bot and say "Hello world"; Start the bot gesture "Wave both hands."

[0060] In some implementations of these examples, the interpreter does not wait for the end of an action before proceeding to the next instruction. To explicitly wait for the completion of a started action, a flow can specify a "match" statement on the finished event of the previously started action, as in the following example: Start bot and say "Hello World" as $Action Match against $Action.Finished()

[0061] Using a sample flow like this, a designer can limit the lifetime of an action by linking it to the end of another action and using a keyword like "stop" to end an action. The following example would stop the bot's gestures when it's finished speaking. Launch the bot and say "Hello World" as $Action_1, and perform the bot gesture "Wave with both hands" as $Action_2. Match with $Action_1.Finished(). Stop $Action_2.

[0062] The previous examples focused on actions initiated by a bot. However, to enable meaningful interaction with a user, it may be desirable to respond to user actions. For example, consider the following example flow, which wraps an event handler for an event indicating that a user utterance has ended: Flow User said $text Match UtteranceUserAction.Beended(final_transcript=$text)

[0063] This flow wrapper (or the wrapped event matcher) can now be used to instruct a flow to wait for a specific user utterance: The user said, "Hello!"

[0064] As with some of the previous example bot actions, in some embodiments, a flow that includes an event matcher for this user action will not proceed to the next instruction until the user says the specified parameter value ("Hello"). In some embodiments, an event matcher can trigger groups of user actions, as in the following example: User said: "Hello!" or User said "Hello" or User said "Hello"

[0065] In some embodiments, a flow including this example of event matching waits for one of these user actions to occur before proceeding to the next statement in the flow. Flows may additionally or alternatively use "and" groups to wait for multiple user actions to occur before proceeding.

[0066] In some embodiments, a flow can be defined with a statement that includes a keyword (e.g., "flow"), a name or identifier for the flow (e.g., "How are you responding?"), and a parameter (e.g., denoted by a leading $ sign) for which the value can be specified and passed when the flow is invoked, as shown in the following example: flow How are you responding $text The user said, "How are you?" Bot says $text

[0067] In some embodiments, each flow defines an action area. For example, if the interpreter triggers the initiation of actions during a flow and those active actions have not finished by the time the interpreter completes execution of the flow, the interpreter can pause those active actions. Returning to the "Hello World" example, in some embodiments, the gesture action does not need to be stopped because it automatically ends when the flow completes: Flow Hello World Example Launch bot and say "Hello World" as $Action_1 and bot gesture "Wave both hands" $Action_2 Match with $Action_1.Finished()

[0068] Backchanneling. Conversations with traditional chatbots or avatars can often feel stiff or unnatural, as they often enforce strict turn-taking behavior. To make conversations with an avatar feel more natural, some embodiments employ a technique called "backchanneling," in which an interactive system (e.g., an interactive avatar) provides feedback to the user while they speak or do something recognizable.

[0069] One way to implement backchanneling is through body postures. For example, a designer can position an avatar in a specific posture depending on whether the user or the avatar is speaking, or when the avatar is waiting for a user response. Here is an example flow that can be used to implement a listening posture: flow Bot listening posture while True as user began to speak Start bot posture "listening" as $listening User said something Sending $listening.Stop()

[0070] In this example, "user started talking" is a flow wrapper for an event matcher for an event indicating the start of a user utterance, and "user said something" is a flow wrapper for an event matcher for an event indicating the end of a user utterance. With an example flow like this, enabled in some implementations, the avatar listens (e.g., shows a listening animation) when the user starts talking and stops the animation when the user stops talking.

[0071] Another example could include various other postures such as "speaking", "attentive", and / or "idle" to give the user feedback about the avatar's current state, as in the following example: flow manage bot postures Start bot posture "idle" as $current_posture while True when the user started speaking Sending $current_posture.Stop() Start bot posture "listening" as $current_posture or when the bot started speaking Sending $current_posture.Stop() Start bot posture "speaking" as $current_posture or when the bot said something Sending $current_posture.Stop() Start bot posture "attentive" as $current_posture

[0072] In one such example flow, which is enabled in some implementations, the avatar assumes an idle posture until the user starts speaking (in which case, it assumes a listening posture), the avatar starts speaking (in which case, it assumes a speaking posture), or the avatar has just spoken something (in which case, it assumes an attentive posture).

[0073] In some embodiments, backchanneling can be achieved with short bursts of vocal sounds like "yes," "aha," or "hmm" while the user is speaking. This can signal to the user that the avatar is listening and makes the interaction seem much more natural. In some embodiments, this effect can be enhanced by nonverbal backchanneling, where the avatar responds to certain things the user says, for example, with gestures. Here is an example of implementing backchanneling using voice breaks and gestures: flow Bot reacts to sad things while True user mentions something sad. Bot gestures "shake head" and Bot says "Ohh." flow Bot reacts to nice things while True user mentions something nice. Bot gestures "celebrate something that's going well," and Bot says "Nice."

[0074] In some implementations, these two flows produce a short burst of voice and a small gesture when the user mentions something nice or something sad. In this example, the "User said something" flow, unlike the "User has said something" flow, which waits for a completed utterance, can be defined to match (and therefore respond to) partial transcripts of what the user says while they are still speaking.

[0075] Here's an example that uses these two bot backchanneling streams in a sequence of interactions: Bot launch and react to sad things Bot launch and react to nice things Start managing bot attitudes Bot says "How was your day?" if user says "*terribllhorribllbad.*" Bot says "I hope you're well?" and bot gesture "looks at user with concern" otherwise Bot says "Great. And do you have any plans for the rest of the day?" User said something Bot says "Thanks for sharing".

[0076] Here, after activating the example bot feedback channel, the bot asks the user how their day was. If the user tells the bot that something bad or nice happened, the bot immediately responds with a speech output and a short animation. These are just a few examples based on a sample implementation of an interpreter, and other variations can be implemented within the scope of this disclosure. Other examples and features of possible interaction modeling languages ​​and interaction categorization schemes are described in more detail below.

[0077] Event-driven architecture and interaction modeling API. In some embodiments, a development and / or deployment platform for an interactive system (e.g., an interactive agent platform) may represent and / or communicate human-machine interactions and associated events using a standardized interaction modeling API and / or an event-driven architecture. In some embodiments, the standardized interaction modeling API standardizes how components represent multimodal interactions, thus enabling a high degree of interoperability between components and the applications that use them.In an example implementation, a standardized interaction modeling API serves as a common protocol in which components use a standardized interaction categorization scheme to represent all bot and user activities as actions in a standardized form, represent states of multimodal user and bot actions as events in a standardized form, implement standardized mutually exclusive modalities that define how conflicts between standardized categories or types of actions are resolved (e.g., it is not possible to say two things at once, while it may be possible to say something and make a gesture at the same time), and / or implement standardized protocols for any number of standardized modalities and types of actions, regardless of the implementation.

[0078] In some embodiments, an interactive agent platform hosting the development and / or deployment of an interactive system may implement an architectural pattern that separates the component(s) implementing decision logic (e.g., the interpreter) from the component(s) performing (e.g., multimodal) interactions. For example, an interaction manager may implement an interpreter for an interaction modeling language as a standalone event-driven component (e.g., an event-driven state machine). The interface for the interaction manager may use a standardized interaction modeling API that defines standardized forms for representing action categories, instances of actions in specific action categories, events, and context.Sensory servers for corresponding inbound interaction channels can translate inputs or non-standardized technical events into the standardized format and generate corresponding Interaction Modeling API events (also called Interaction Modeling events). The Interaction Manager can process these inbound Interaction Modeling API events, determine what actions should be taken (e.g., based on code written in the Interaction Modeling language for execution by the interpreter), and generate outbound Interaction Modeling API events (e.g., in response to an Interaction Modeling language statement such as "send") that represent commands to perform specific actions. Action servers for corresponding outbound interaction channels can interpret these outbound Interaction Modeling API events and execute the appropriate commands.Decoupling these components enables interchangeability and interoperability, thus promoting development and innovation. For example, a component can be replaced with a different design or a different interaction channel can be connected without affecting the functionality of the existing system.

[0079] This architectural pattern and API design can provide a purely event-driven, asynchronous way to handle multimodal interactions. Compared to previous solutions, in some embodiments, there is no strict sequence of conversations (e.g., bot conversations, user conversations, bot conversations). Instead, participants in the interaction can conduct multimodal interactions concurrently and respond to incoming events independently and simultaneously, thereby enhancing the realism of human-machine interaction.

[0080] In some embodiments using this architectural pattern, the interaction manager does not need to know which specific action servers are available within an interactive system. It may be sufficient for the interaction manager to know the supported modalities. Similarly, action and / or sensing servers may be independent of the interaction manager. Thus, any of these components can be upgraded or replaced. As a result, the same platform and / or interaction manager can support different types of interactive systems, all controlled by the same API and interchangeable or customized for a specific use. For example, one implementation might provide a text-based user interface, another a voice-based system, and a third a 2D / 3D avatar.

[0081] Managing multiple flows. The examples above explained how a sample interpreter might be programmed to traverse a specific flow until it reaches an event matcher. In some embodiments, a top-level flow may contain instructions to activate any number of flows with any number of event matchers. The interpreter may use any suitable data structure to track active flows and corresponding event matchers (e.g., using a tree or other representation of nested flow relationships), and it may use an event-driven state machine that listens for various events and triggers corresponding actions specified in matching flows (with event matchers corresponding to an incoming Interaction Modeling API event).

[0082] Because flows can specify human-machine interactions, a developer may want to enable multiple flows that specify conflicting interactions to be triggered under different conditions and / or multiple flows that specify the same interaction—or different but compatible interactions—to be triggered based on the same or similar conditions. In some scenarios, multiple active moves that specify different interactions may be triggered by different conditions that can be satisfied by the same event. Therefore, the interpreter can process incoming Interaction Modeling API events (e.g., from a queue) sequentially, testing the event matcher specified by each active flow for a match against the event.If there is an event matcher from an active flow that matches the event (a matching flow), the interpreter can advance that flow (e.g., generate outbound Interaction Modeling API events to trigger actions). If there are multiple matching flows, the interpreter can determine whether the matching flows agree or disagree on an action. If they agree, the interpreter can forward both matching flows. If they disagree, the interpreter can apply conflict resolution to determine which action should take precedence, advance the matching move with the prioritized action, and abort the other matching move(s) (e.g., because the interaction pattern represented by those moves no longer applies).If there are no active flows that match an event, the interpreter can generate an internal event that matches and triggers a specific flow to handle unmatched or unprocessed events, execute one or more handlers for unprocessed events, and / or use another technique to handle an unprocessed event. After checking for matches and advancing flows, the interpreter can check the flow states for any flows that have terminated or aborted and can stop any active flows that were activated by those terminated or aborted flows (e.g., because the interaction pattern represented by those flows should no longer apply).This allows the interpreter to traverse the events in the queue, advance the flow, perform conflict management to determine which interactions should be performed, and generate outbound interaction modeling API events to trigger those interactions.

[0083] For example, the interpreter can execute a primary processing loop that processes incoming Interaction Modeling API events and generates outgoing Interaction Modeling API events. Unlike simple event-driven state machines, the interpreter can use a series of flow headers. A flow can be thought of as a program containing a sequence of instructions, and the flow headers can be thought of as instruction pointers that traverse the instructions and indicate the current position within the corresponding flows. Depending on the instruction, the interpreter can advance a given flow header to the next instruction, jump to another flow referenced by a label or other flow identifier, split into multiple flow headers, merge multiple flow headers, and / or other actions.Thus, the interpreter can use flow heads to establish and maintain a hierarchy of flow heads. When a parent flow in a branch of a hierarchy of flows or flow heads is paused, interrupted, or resumed, the interpreter can pause, interrupt, or resume all child flow heads of that parent flow head or branch. In some embodiments, each flow can specify any number of scopes, which the interpreter can use to generate events that instruct a corresponding action server to limit the lifetime of launched actions and flows within the corresponding scopes.

[0084] In some embodiments, advancing flows may instruct the interpreter to generate Interaction Modeling API events that instruct a particular action. Additionally or alternatively, advancing flows may instruct the interpreter to generate Interaction Modeling API events that notify listeners that an event has occurred. The interpreter may dispatch these events and / or maintain an internal event queue, place these events in the internal event queue, and process all internal events in the internal event queue sequentially (e.g., check active flows for matches to internal events) before moving on to processing the next incoming Interaction Modeling API event.

[0085] Example of interpreters using language models. In some embodiments, an interaction modeling language and corresponding interpreter may support the use of natural language descriptions and the use of one or more language models (e.g., LLMs, VLMs, multimodal LLMs, etc.) to reduce the cognitive load on the programmer and facilitate the development and deployment of more complex and nuanced human-machine interactions.

[0086] For example, each flow can be specified with a corresponding natural language description that summarizes the interaction pattern represented by the flow. In some embodiments, the interpreter does not require a designer to specify these flow descriptions, but may rely on a flow description under certain circumstances (e.g., by an unknown event handler requesting an LLM to determine whether a mismatched event representing an unrecognized user intent semantically matches the natural language description of an active flow representing a target user intent). Thus, in some embodiments, the interpreter can analyze one or more specified flows (e.g.,at design time), determine whether any specified flows lack a corresponding flow description and, if so, request an LLM to generate one based on the flow name and / or instructions. Additionally or alternatively, the interpreter may determine (e.g., request an LLM to) whether any specified flow descriptions are inconsistent with their corresponding flow descriptions and, if so, request an LLM to generate a new one (e.g., as a suggestion or for automatic replacement) based on the flow name and / or instructions.

[0087] In some embodiments, a designer may specify a flow description (e.g., a natural language description of what the flow should do) without the command sequence, or may name a flow without defining it. In some embodiments, the interpreter may analyze one or more specified flows (e.g., at design time), determine if a command sequence is missing from the specified flows, and, if so, request an LLM to generate one (e.g., based on the name and / or description of the flow). For example, the interpreter may provide the LLM with one or more example flows, the specified name and / or description of the flow, and a request to complete the flow based on its name and / or description. These are just a few examples of how an interpreter may invoke an LLM.

[0088] In an example implementation, flow statements (e.g., including any encountered event triggers) may be executed until an event matcher is reached, at which time a flow may be interrupted. When there are no more flows, an incoming or internal event may be processed by executing the event matcher in each interrupted flow and comparing the event with the target event parameters and parameter values ​​specified by the event matcher's event specifier. In general, any suitable matching technique may be used to determine whether an event matches an active event matcher for one of the active streams (e.g., comparing target event parameters and parameter values ​​with those of the incoming or internal event to produce a representation of whether the event is a match).

[0089] In general, a designer can specify an event to be matched or triggered using a name or identifier for the event and one or more target event parameters and / or parameter values. Target event parameters and / or parameter values ​​can be specified explicitly using positional or named parameters, or as a natural language description (NLD) (e.g., docstring) from which the interpreter can derive the target event parameters and / or values ​​(e.g., from a single NLD of all target event parameters and values, or from an NLD of a single parameter value). Here are some example event specifiers for: • An event with named parameters and explicit values: StartUtteranceBotAction(text="How are you?", volume="whisper", speed="slow") • An event with position parameters: StartUtteranceBotAction("How are you?", "whisper", "slowly") • An event with parameters specified with an NLD value: StartUtteranceBotAction(text="""Ask how it's going""", volume="whisper", speed="slow") • An event with a single NLD parameter: StartUtteranceBotAction("""Ask the user how he is, very slowly and in a whisper""")

[0090] In some embodiments that support event specifiers with NLDs, before executing an instruction (e.g., an event matcher or event trigger) that includes an event specifier, the interpreter may determine (e.g., at runtime) whether the instruction includes an NLD parameter and, if so, request an LLM to generate appropriate target event parameters and / or parameter values. As such, the interpreter may execute the instruction (e.g., an event trigger or event matcher) using the generated target event parameters and / or parameter values.

[0091] Additionally or alternatively, the interpreter may (e.g., at runtime) request an LLM to determine whether an event (e.g., an interaction modeling API event) matches the flow description of an active flow. In general, an interaction modeling API event may represent a user interaction or intent, a bot interaction or intent, a scene interaction, or another type of event using a standardized interaction categorization scheme that classifies actions, action events, event parameters, and / or parameter values ​​using standardized (e.g., natural language, semantically meaningful) keywords and / or commands.As such, the interpreter can execute an event matcher by determining whether the received action, action event, event parameters, and / or parameter values ​​of an incoming or internal event and the event specified by the event matcher match (e.g., exactly or fuzzy). Additionally or alternatively, the interpreter can request an LLM to determine whether the representation of an incoming or internal event matches the (e.g., specified or generated) flow description of an active flow. Depending on the implementation, an LLM can provide a more nuanced or semantic understanding of the match than traditional express or fuzzy matching algorithms.

[0092] For example, suppose the user makes a gesture indicating agreement, such as a thumbs up, a nod, or something informal like "yes." A designer might have written a flow intended to represent the scenario when the user expresses agreement, but provided only a few examples of spoken responses for explicit matching. In this scenario, even if there are no explicit matches, an LLM can determine that the standardized and semantically meaningful representation of a recognized user response (e.g., GestureUserActionCompleted("thumbs up")) is a semantic match to a flow description such as "user expresses agreement."Here's another example where a designer specifies a flow that (via the "User made a selection" and "User said" flow bindings) should match an event where the user selects option B from a list of options: multimodal showcase the flow user """The user selected showcase (B).""" the user selected the "multimodality" option or a user said "Show me the multimodal showcase". or user said "multimodality" or user said "showcase B" or user said "second showcase" or user said "(?i)multimodality".

[0093] If the user selects option B through an interaction not intended by the designer, an LLM can be used to determine whether a standardized representation of a recognized user gesture matches the flow (e.g., a natural language description of a flow, a natural language description of a parameter, a natural language description of a parameter value, etc.). In this way, a specified flow can match multiple gestures, text responses, or other events that may not have been explicitly specified by the designer.

[0094] In some implementations (e.g., in some embodiments where the interpreter checks the event matcher for all active (e.g., suspended) flows for a match and determines that there are no active flows that match the incoming or internal event), the interpreter may (e.g., at runtime) request an LLM to determine whether an incoming or internal event and / or a representation of recent interaction history matches the name and / or instructions of an active flow.For example, some flows may represent target user intentions, and the interpreter can implement an event handler for an unknown user action by providing an LLM with a sample interaction between the user and the bot, some possible target flows listed for target user intentions, a corresponding list of target user intentions, the recent interaction history, the unknown user action, and a request to the LLM to predict whether the unknown user action matches one of the target user intentions. Thus, the interpreter can use an LLM to implement an unknown event handler that provides a more granular or semantic understanding of the match to specific target user intentions.

[0095] In some scenarios, there may not be a matching flow that defines a bot response to a particular user interaction. Therefore, in some implementations (e.g., in some embodiments where the interpreter determines that there are no active flows matching an incoming or internal event representing a user interaction), the interpreter may request an LLM to generate a flow (e.g., at runtime).For example, in some embodiments, the interpreter may first use an LLM to attempt to match an unknown incoming or internal event to the name, instruction(s), and / or other representation of one or more active flows that await corresponding target user intents (and define corresponding bot responses). If the LLM determines that there are no matching flows (target user intents), the interpreter may request (the same or a different) LLM to generate an approachable agent (e.g., bot) flow. In some embodiments, the interpreter may request the LLM to generate one or more intents as an intermediate step.For example, if the unknown event is a user action, the interpreter may apply any number of prompts to instruct the LLM to classify the unknown user action into a user intent, generate an agent intent, and / or generate a flow that implements the agent intent. As a non-limiting example, the interpreter may implement an event handler for an unknown user action by providing an LLM with a sample interaction between the user and the bot, the recent interaction history, the unknown user action, and a prompt for the LLM to predict one or more intents (e.g., user, bot) and / or a prompt for the LLM to generate a corresponding flow.This allows the interpreter to use an LLM to implement an unknown event handler that responds intelligently to unknown events without the designer having to specify the code for the addressable flow.

[0096] Neural networks typically operate like a black box, making it difficult to control the generated responses. The lack of transparency makes it difficult to ensure that the generated content is accurate, appropriate, and ethical. However, using an LLM to autocomplete event parameters or parameter values, perform event matching, or generate flows using a standardized and structured interaction modeling language and / or interaction categorization scheme helps to provide structure and explainability to the LLM, thereby improving the ability to control the LLM output.Embodiments that use an LLM to auto-complete event parameters or parameter values, perform event matching, or generate flows make life easier for the designer by providing an intuitive way to specify human-machine interactions and events to be matched or triggered, thus reducing the cognitive load of developing an interactive system while protecting against unintentional content generation.

[0097] Sensory processing and action execution. Depending on the embodiment and configuration, an interactive agent platform hosting the development and / or deployment of an interactive agent (e.g., chatbot, voicebot, digital assistant, interactive avatar, non-player character (NPC), digital human, robot, etc.) may support any number of input and output interaction channels. In some embodiments where sensory processing, interaction decision, and action execution are decoupled, the interactive agent platform may support a sensory server for each input interaction channel and an action server for each output interaction channel.Sensory servers for corresponding input interaction channels can translate inputs or non-standardized technical events into the standardized format and generate corresponding Interaction Modeling API events. An Interaction Manager can process these incoming Interaction Modeling API events and generate outgoing Interaction Modeling API events that represent commands to execute an action. Action servers for corresponding output interaction channels can interpret these outgoing Interaction Modeling API events and execute the corresponding commands. Using an Interaction Modeling API for communication between these components allows for the distribution of responsibilities for processing different types of inputs among different types of sensor servers and responsibilities for different types of actions among different types of action servers.For example, each action server can be responsible for corresponding groups of actions and action events (e.g., related to a common interaction modality), thus avoiding the complexity of managing events related to different interaction modalities.

[0098] Sensor and / or action servers can be implemented with a combination of asynchronous event loops and processes to ensure that multiple user sessions and system pipelines can be served in parallel. This architecture allows programmers to add different services that can handle different types of actions and events (corresponding to different types of interaction modalities) supported by the interaction modeling API actions. In some embodiments, an event gateway can be used to communicate and distribute events to the appropriate components, whether through synchronous interactions (e.g., via a REST API, Google Remote Procedure Call (RPC), etc.) or asynchronous interactions (e.g., via a message or event broker).For example, each sensor server can send interaction modeling API events to the event gateway for incoming input or non-standard technical events, and the interaction manager can be subscribed or otherwise configured to receive these events from the event gateway. The interaction manager can generate and forward outgoing interaction modeling API events to the event gateway, and each action server can be subscribed or otherwise configured to receive the events it is responsible for executing (e.g., one interaction modality per action server).

[0099] To process all supported actions for at least one interaction modality, an action server may be equipped with an action handler for each standardized action category and / or action event supported by the interaction modeling language and / or defined by the interaction categorization schema for a given interaction modality.For example, an action server may implement a chat service that handles all interaction modeling API events for bot utterance actions; an animation service that handles all interaction modeling API events for bot gesture actions; a graphical user interface (GUI) service that handles all interaction modeling API events that instruct arrangements of visual information such as visual information scene actions, visual selection actions, and / or visual form actions; and / or a timer service that handles all interaction modeling API events for timer actions, to name a few examples.

[0100] Each action server can manage the lifecycle of all actions under its purview. Interaction Modeling API events can contain commands for an action server to initiate, modify, or terminate actions. All events related to the same action can be represented by a common action identifier (e.g., action_uid), so that individual events associated with the same action identifier can represent different states in the lifecycle of a corresponding action. For example, an action server for a given interaction modality can initiate a specific action (e.g., a bot gesture or an utterance) and track active actions and their corresponding states. Each action server can implement a modality policy that specifies how to handle actions triggered during the execution of another action of the same interaction modality (e.g.,Multiple sound effects can be running simultaneously, but a new body animation can replace or temporarily override an active body animation. Some implementations may support commands that modify an ongoing action, which can be useful for longer-running actions (e.g., an avatar animation) that can dynamically adjust their behavior. For example, a nodding animation can be modified to change its speed depending on detected voice activity. Some implementations may support commands that stop an ongoing action, which can be used to proactively stop an action, such as a gesture, that could potentially run for an extended period of time. In some embodiments, an action server may synchronize changes in action state with certain conditions (e.g.,(wait to start an action until a previous action of the same modality has completed, synchronize the completion of two different actions in different modalities, synchronize the start of one action with the end of another action, etc.). When an action server makes a change in action state, it can generate and forward an Interaction Modeling API event to the event gateway reflecting the update, so that any component waiting for that state change can react to it.

[0101] Interactive visual GUI elements. In some scenarios, a designer may want to customize an interactive system, such as one with an interactive avatar that synchronizes conversational AI with complementary visual content, such as a visual representation of related information (e.g., text, images), a choice the user is prompted to make, or a field or form the user is asked to complete.

[0102] In some embodiments, an interaction modeling API may use a standardized interaction categorization scheme that defines a standardized format (e.g., standardized and semantically meaningful keywords) for specifying events associated with a standardized category of interactive visual content actions (e.g., actions that instruct an overlay or other arrangement of visual content that complements a conversation with an interactive agent), such as visual information scene actions, visual selection actions, and / or visual form actions. Some embodiments may include an interaction modeling language that supports the specification of visual designs using natural language descriptions (e.g.,"attention-grabbing, bold, and professional" for an alert message), and a corresponding interpreter can translate a specified description into a standardized representation of corresponding design elements (e.g., color schemes, typography, layout, images) and generate outbound Interaction Modeling API events using the standardized format for interactive visual content action events. Thus, an action server can implement a graphical user interface service that generates robust and visually appealing graphical user interfaces that can be synchronized with verbal responses by conversational AI or otherwise contribute to human-machine interaction.

[0103] In some embodiments, an interaction modeling API defines a way of representing a particular GUI (e.g., a configuration or arrangement of visual elements) using an interaction categorization scheme that defines a standardized category of interactive visual content actions and corresponding events with payloads that specify standardized GUI elements. For example, the interaction categorization scheme may categorize interactive visual content actions and / or GUI elements into semantically meaningful groups so that an interpreter or action server can generate the content of a given GUI element (e.g., using an LLM to generate a text block, retrieving it, or generating an image based on a specified description) based on the current context of the interaction.Each group of interactive visual content actions and / or GUI elements can be used to define a corresponding subspace of possible GUIs, representing different ways a bot can visualize information for the user and / or how the user can interact with that information. An example interaction categorization scheme might categorize interactive visual content actions as visual information scene actions, visual selection actions, and / or visual form actions.

[0104] Visual information scenes may involve displaying information to the user for informational purposes (e.g., text providing background information on a topic or product, an image illustrating a situation or issue), without the expectation that the user might interact with the information in any way other than by reading it. Visual choice actions may involve displaying or interacting with visual elements that present the user with a choice and / or describe the nature of the choice (e.g., multiple choice vs. single choice, small or limited number of options vs. large number of options). Visual form actions may involve displaying or interacting with visual elements that prompt the user to enter a form or field (e.g., the avatar might ask the user to provide their email address) and / or describe the nature of the prompt (e.g.,Email, address, signature).

[0105] In some embodiments, the interaction categorization scheme may define a standardized format for specifying supported GUI interaction elements (e.g., button list, selectable options grid, input text field, hint carousel) such that a sensor server's (e.g., a corresponding action handler of a) sensory server converts detected interactions with these interaction elements (e.g., a state when a button list item is released, e.g., after a click or touch, a state when a user types a character into an input field, a state when a user presses the Enter key or clicks away from a text field) into standardized interaction modeling API events that represent possible interactions with these elements in a standardized format. In some embodiments, it may be used for each of several different input interaction channels (e.g., GUI interactions, user gestures, voice input, etc.).) a sensory server, each sensory server configured to generate standardized interaction modeling API events representing detected interaction events in a standardized format. In some embodiments, a sensory server may convert detected interaction events (e.g., "User clicked the 'Chai Latte' button, scrolled down, and clicked the 'Confirm' button") into a corresponding standardized interaction-level event (e.g., "User selected the 'Chai Latte' option"). Standardized interaction-level events may depend on the type of interactive visual content action defined by the schema. Examples of standardized interaction-level events may include an event representing an update to a user's confirmation status and / or when the update was detected (e.g.,when a single input is requested as part of a Visual Form, an "Enter" keyboard event can be translated into a "confirmed" status update), an event representing an update to a user's selection and / or when the update was detected (e.g., a detected selection of a "Chai Latte" item from a list with multiple choices can be translated into an update to the selection), an event representing an update to a user's form input and / or when the update was detected, and / or others.Thus, standardized interaction modeling API events can be generated and forwarded to an event gateway and processed by the interpreter to generate outgoing interaction modeling API events that can specify commands to perform responsive GUI updates, and the outgoing interaction modeling API events can be forwarded to the event gateway for execution by an appropriate action server.

[0106] In some embodiments, interaction modeling API events specifying commands to update the graphical user interface may be translated into corresponding graphical user interfaces and displayed to the user. To achieve this, in some embodiments, an action server implementing a GUI service may translate a standardized representation of a particular GUI specified by a particular interaction modeling API event into a representation (e.g., JavaScript Object Notation (JSON)) of a modular GUI configuration specifying content blocks such as paragraphs, images, buttons, multiple-select fields, and / or other types. The GUI service may use these content blocks to create a visual layout for a GUI overlay (e.g., a HyperText Markup Language (HTML) layout renderable in any modern web browser).For example, any number of visual template or shell layouts may define corresponding arrangements of various content blocks, and the UI service may select a visual template or shell layout (e.g., based on which content blocks were generated or specified by the Interaction Modeling API event) and populate placeholders for those blocks in the template or shell with the corresponding generated content. In some embodiments, various features of the visual layout of the template or framework (e.g., the size or arrangement of blocks, look-and-feel options such as a color palette for a GUI overlay, etc.) may be customized. Thus, a visual layout representing a graphical user interface specified by an Interaction Modeling API event may be generated and presented to the user (e.g., via a user interface server).

[0107] Using an interactive avatar as an example, an animation service can be used to animate the avatar (as described in more detail below), and a GUI service can be used to synchronize the presentation of related visual elements (e.g., visual information scene, visual choices, visual shapes). For example, the user's screen can contain a region that displays the avatar on an entire web page (e.g., utilizing as much of the height and width of a browser window as possible while maintaining the aspect ratio of the avatar stream), and the visual elements generated by the GUI service can be presented in an overlay over the avatar stream. In one embodiment, the avatar stream can remain at a fixed aspect ratio (e.g., 16:9), with padding around the stream required to maintain the aspect ratio.In some embodiments, the overlay may be maintained at the same relative position on the screen regardless of the size of the stream. In some embodiments, the overlay may be scaled with the size of the avatar. In some embodiments, the overlay may be maintained at a fixed, configurable size relative to the size of the avatar (e.g., 10% of the width and 10% of the height of the avatar).

[0108] In some embodiments, various GUIs (e.g., pages of visual elements) can be configured as part of a stack from which GUI pages can be pushed and pulled out. This configuration can be particularly useful in the context of AI-driven interactions, as the context can change in a non-linear manner during a series of interactions. A GUI stack overlay can be used to ensure that the visual content of the GUI remains relevant throughout the entire series of interactions. These stacked GUIs can be made at least partially transparent to facilitate the visualization of stacked information, allowing the conversational AI to combine GUIs or reshuffle the stack at different stages of a conversation (e.g.,For example, the header of a stacked overlay may describe an entire customer experience, such as "Support Ticket XYZ," while stacked pages within the overlay may represent different steps in the experience, such as "Please enter your email address." In some embodiments, the graphical user interface may be part of a rendered 3D scene (e.g., a tablet computer held by the avatar), the graphical user interface may be 3D (e.g., the buttons may be rendered with appropriate depths), and / or others. These are just a few examples, and other variations may be implemented within the scope of the present disclosure. Although the above example is described in the context of a 2D GUI, those skilled in the art will know how to adapt the above instructions to represent an avatar and / or overlay in augmented and / or virtual reality (AR / VR).

[0109] Animations for interactive agents. In some embodiments, interaction modeling API events specifying commands to execute bot expressions, poses, gestures, or other interactions or movements may be generated (e.g., by the interpreter executing code written in an interaction modeling language) and translated into corresponding bot animations, and the bot animations may be presented to the user. More specifically, in some embodiments, an action server implementing an animation service may use a standardized representation of a target bot expression, pose, gesture, or other interaction or movement specified by a particular interaction modeling API event to identify and trigger or generate a corresponding animation.

[0110] Taking a standardized bot gesture action category (e.g., GestureBotAction) as an example of a bot action type, in some embodiments the animation service may process all events related to actions in the GestureBotAction category, apply a modality policy that overrides active gestures with any subsequently instructed gesture, and use incoming StartGestureBotAction events to create a stack of actions if active GestureBotActions exist. Thus, the animation service may implement an action state machine and an action stack for all GestureBotActions, connect to an animation graph that implements a state machine for animation states and transitions between animations, and instruct the animation graph to invoke a corresponding state variable based on a command to change the state of an instance of a GestureBotAction (e.g.,initialize, pause, or resume a gesture) represented by an Interaction Modeling API event.

[0111] In some embodiments, the animation graph may support a number of clips that animate an avatar or other bot in various expressions, poses, gestures, or other interactions or movements. For example, the animation service may receive a command to change a GestureBotAction state (e.g., initialize, stop, or resume a gesture) represented in a standardized interaction categorization scheme to identify a corresponding supported animation clip. In some scenarios, a designer may want to specify a bot expression, pose, gesture, or other interaction or movement using a natural language description. In some embodiments, the animation service may support a natural language description (e.g., hand-typed or interpreted by the interpreter using an LLM / VLM / etc.generated, which is used as an argument to describe an instance of a standardized type of bot action in an Interaction Modeling API event) to select or generate the best animation clip. For example, the animation service may generate or access a sentence embedding for the natural language description of the bot action (e.g., bot gesture), use it to perform a similarity search on sentence embeddings for descriptions of the available animations, and select an animation based on a similarity measure (e.g., nearest neighbor, within a threshold). In some embodiments, the animation may be played if the best match is within a similarity threshold (e.g., if the distance is below a certain threshold). If no animation matches within the specified threshold, a replacement animation (e.g.,a less specific version of the animation that best matches). If the animation service cannot find a suitable match, the animation service can generate and forward an Interaction Modeling API event to the event gateway indicating that the gesture failed (e.g., ActionBeended(is_success=False, failure_reason="gesture not supported")).

[0112] Expectation-driven actions and expectation signaling. In various scenarios, it can be beneficial to inform an interactive system or one of its components (e.g., a sensory server that controls input processing, an action server that implements bot actions) about what events an interaction manager (e.g., the interpreter) expects next from the user or the system. For example, if the interaction manager expects the user to start speaking (e.g., an utteranceUserActionStarted event), the interactive system can configure itself to listen or enhance its listening capabilities (e.g., by decreasing the speaker volume, increasing the microphone sensitivity, etc.). In noisy environments, an interactive system can be configured to disable the listening features (e.g.,automatic speech recognition) and enables listening only when the interaction manager expects the user to speak. In a chatbot system, a designer can display a thinking indicator when the chatbot (e.g., the interaction manager) is processing a request, and once it expects a response (e.g., a text reply), the interaction manager can communicate this expectation to an action server to update the indicator with a visual cue that the chatbot is waiting for a response. In addition, computer vision algorithms are typically resource-intensive to run. Therefore, the interaction manager can communicate what types of image processing events it expects at a particular time during an interaction, and the interactive system can disable or enable image processing algorithms on the fly.Some example scenarios where enabling and disabling image processing algorithms can be useful include reading QR codes (quick response codes), object detection, user movement detection, and more.

[0113] To facilitate these preparatory actions, expectations can be represented as instances of a standardized action type (ExpectationActions) with corresponding expected states, and an Interaction Modeling API event associated with a particular instance of an ExpectationAction can contain one or more fields representing an expectation that a particular target event occurs using a standardized interaction categorization scheme that identifies the expectation as a supported action type (e.g., ExpectationBotAction) and that represents a corresponding expectation event (e.g., specifying the state of the expectation such as start, stop, and finished) and the expected target event (e.g., UtteranceUserActionStarted) using standardized (e.g., natural language, semantically meaningful) keywords and / or commands.Standardized expectation events can, for example, be an event indicating that a bot expects a specific event on the event gateway in the near future (e.g., StartExpectationBotAction), which can instruct a sensor or action server to optimize its functionality (e.g., a sensor server responsible for processing camera images can enable or disable certain image processing algorithms depending on what the interaction manager expects); an event indicating that the sensor or action server confirms the bot expectation, or that the sensor or action server has updated its functionality in response to the expectation (e.g., ExpectationBotActionStarted); an event indicating that the expectation has been stopped (e.g., StopExpectationBotAction), which can occur when the expectation has been fulfilled (e.g.,the event was received) or something else happened to change the course of the interaction; an event indicating that the sensor or action server acknowledged that the bot expectation has ended (e.g., ExpectationBotActionEnded), and / or others.

[0114] In addition to, or alternatively, communicating (e.g., to a sensor or action server) that the interaction manager (e.g., the interpreter) expects certain events, some embodiments signal to the user that the bot is waiting for input (e.g., for a particular user interaction modality). As such, the standardized interaction categorization scheme may classify this expectation signaling as a supported action type (e.g., ExpectationSignalingAction). This action may allow an interactive system to communicate to the user in subtle (e.g., nonverbal) ways what the bot expects from the user (e.g., the avatar's ears might grow or the avatar might adopt a listening posture when waiting for user input).

[0115] For example, in a chatbot system, the user might be prompted to enter certain information before the interaction is considered complete (e.g., "Please enter your date of birth to confirm the order"). In such a situation, the designer might want the chatbot to signal to the user that it is actively waiting for a user response. In this case, the designer can specify code that triggers the generation of a StartExpectationSignalingBotAction(modality=UserSpeech) event. In another example, an interactive avatar might wait for a specific gesture from the user. In such a situation, the designer might want the avatar to actively communicate this to the user (e.g., by showing a specific animation). In this case, the designer can specify code that triggers the generation of a StartExpectationSignalingBotAction(modality=UserGesture) event.If there is a conflict with another ongoing action in a corresponding output interaction channel (e.g., an active upper body animation), an action server can resolve the conflict based on a specified modality policy.

[0116] To facilitate these expectation signaling actions, an interaction modeling API event can represent expectation signaling events using a standardized interaction categorization scheme that classifies the expectation signaling as a supported action type (e.g., ExpectationSignalingBotAction) and represents corresponding expectation signaling events (e.g., indicating the state of the expectation such as start, stop, finished) and target or input interaction modalities that the bot expects (e.g., user language) using standardized (e.g., natural language, semantically meaningful) keywords and / or commands. Examples of standardized expectation signaling events can include an event indicating that a bot expects an event for a particular interaction modality at the event gateway in the near future (e.g.,StartExpectationSignalingBotAction); an event indicating that the sensor or action server acknowledges the expectation signaling event or confirms that the sensor or action server has started actively waiting for an event for a particular interaction modality (e.g., ExpectationSignalingBotActionStarted); an event indicating that the expectation has stopped (e.g., StopExpectationSignalingBotAction); an event indicating that the sensor or action server confirms that the expectation has ended or that active waiting has stopped (e.g., ExpectationSignalingBotActionEnded), and / or others.

[0117] As such, the present techniques can be used to develop and / or deploy interactive agents such as bots or robots (e.g., chatbots, voicebots, digital assistants, interactive avatars, non-player characters, etc.) that perform more complex, nuanced, multimodal, non-sequential, and / or realistic conversational AI and / or other types of human-machine interactions than previous techniques.Furthermore, various embodiments that implement or support an interaction modeling language and / or an interaction modeling API that uses a standardized interaction categorization scheme promote a number of technological benefits, from making the designer's life easier by reducing their cognitive load when developing an interactive system, to supporting a variety of interactions or features from which a designer can draw to customize an interactive system, to promoting interoperability by standardizing the representation of interactions.

[0118] Fig.1 shows an example of an interactive system 100 in accordance with some embodiments of the present disclosure. It should be understood that these and other arrangements described herein are presented as examples only. Other arrangements and elements (e.g., engines, interfaces, functions, arrangements, groupings of functions, etc.) may be used in addition to or in place of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional units that may be implemented as individual or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions performed by units described herein may be performed by hardware, firmware, and / or software.For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, the system and methods described herein may be implemented, for example, using one or more generative language models (e.g., as described in the . Fig. 28A-28C), one or more computing devices or components thereof (e.g. as described in Fig. 30) and / or one or more data centers or components thereof (e.g. as described in Fig. 31) can be implemented.

[0119] At a high level, the interactive system 100 may execute, control, or otherwise provide an interactive agent (e.g., chatbot, voicebot, digital assistant, interactive avatar, non-player character (NPC), digital human, interactive television or other device, another type of interactive robot, etc.). Some examples of interactive systems that may provide an interactive agent include a digital kiosk, an automotive infotainment system, a digital assistant platform, a smart television or other smart device, a video game or animation environment, a virtual or augmented reality environment, a video conferencing system, and / or others. Fig.1 shows an example implementation in which a client device 101 (e.g., a smartphone, a tablet, a smart TV, a game console, a digital kiosk, etc.) provides an interface for human-machine interaction over any number and type of interaction channels, wherein one or more sensory servers 160 translate inputs into (e.g., standardized interaction modeling API) events representing detected interaction states, an interaction manager 190 determines which actions the interactive agent should perform and generates events (e.g., standardized interaction modeling API) representing corresponding commands, and one or more action servers 170 interpret these commands and cause the interactive agent to perform corresponding actions over the corresponding interaction channel(s).

[0120] Depending on the version, the components can be Fig. 1 may be implemented on any number of physical machines (e.g., having similar components, features, and / or functions as the example computing device 3000 of Fig. 30). Consider a digital kiosk as an example. In some embodiments, the physical kiosk may correspond to client device 101, which connects to one or more remotely controlled components. In some embodiments, some or all of the components may be Fig. 1 be implemented as corresponding microservices and / or physical devices that reside in a cluster of nodes in a data center (e.g., having similar components, features, and / or functionality as the example data center 3100 of Fig.31), on one or more edge devices, on dedicated hardware, and / or in other ways. In some implementations, some or all of the components run locally on a physical machine (e.g., on a digital kiosk, a robot, or other interactive system) with various types of interface hardware managed by an operating system, firmware, and / or other software. In some of these embodiments, the client device 101 corresponds to the various hardware interfaces, and some or all of the other components in Fig. 1 (e.g., the sensor server(s) 160, the action server(s) 170, the interaction manager 190, etc.) represent functions of the operating system, firmware, and / or other software that send commands or requests to the various hardware interfaces.

[0121] In an exemplary virtual or augmented reality environment, the Fig.1 may be implemented on a local device (e.g., an AR / VR headset, a smartphone running a VR / AR application), a cloud server, an edge computing device, dedicated hardware, and / or otherwise. In some embodiments, there is one sensor server per input interaction channel (e.g., one sensor server for processing video input, one for processing audio input, one for processing touch input) and / or one action server per output interaction channel (e.g., one action server for processing bot animations, one for processing bot speech, one for processing interactive visual content). In some implementations, some or all of the sensor servers 160 and / or action servers 170 are combined into a single engine and / or microservice that processes the corresponding interaction channels using the corresponding services.These are just some examples, and other configurations and implementations are possible within the scope of this disclosure.

[0122] In some embodiments, some or all of Fig.1 are part of, or at least partially hosted by, a development and / or deployment platform for an interactive system (e.g., an interactive agent platform). For example, a platform such as NVIDIA® OMNIVERSE® (and / or another platform or system, e.g., those using Universal Scene Descriptor (USD) data formats - e.g., OpenUSD) may provide an infrastructure and various functionalities that provide a framework for developing and / or deploying interactive agents. The platform may provide various authoring tools that allow users to create and customize interactive agents, a real-time rendering engine, integration with various services (e.g., computer vision, speech recognition, natural language understanding, avatar animation, speech generation, simulation software, recommendation engines), and / or other components.In some embodiments, some or all of these tools and / or the information contained in . Fig. 1 are integrated into an application and processed in real time (e.g., using a framework for developing and deploying cloud-native applications, such as NVIDIA® Unified Cloud Services Tools). Some or all of these tools and / or the components described in Fig. The components illustrated in Figure 1 may be deployed as microservices and managed with a containerized application orchestration platform (e.g., NVIDIA FLEET COMMAND™). Accordingly, in some embodiments, these tools and / or components may be used to customize and / or deploy the interactive system 100.

[0123] For example, in some embodiments, interaction manager 190 may implement an interpreter for an interaction modeling language, and code implementing decision logic for an interactive agent may be written in the interaction modeling language, loaded onto interaction manager 190, or otherwise accessed and executed by interaction manager 190. Depending on the desired interactive agent, any number and type of interaction channel may be connected, configured, and supported by appropriate sensory servers 160 and / or action servers 170. Thus, in some embodiments, a development and / or deployment platform may be used to host interactive system 100, and interactive system 100 may implement a (e.g., customizable) interactive agent.

[0124] At a high level, a user may operate or otherwise interact with client device 101 or another interactive system that includes any number of input and / or output interaction channels. As a non-limiting example, Fig.1 a video input interaction channel comprising a camera (not shown) and an image processing microservice 110 that uses any known computer vision technique to recognize user gestures; an audio input interaction channel comprising a microphone (not shown) and a speech recognition microservice 120 that uses any known speech recognition and / or capture technique to recognize the user's speech; a video output interaction channel comprising a screen (not shown) and an animation microservice 140 that uses any known animation technique to animate a bot (e.g.,, bot poses, bot gestures, blends, text-to-motion, text-to-animation); an audio output interaction channel comprising a speaker (not shown) and a speech generation microservice 150 that uses any known speech synthesis technique to synthesize bot speech; a graphical user interface (GUI) having a GUI input interaction channel that accepts GUI user input (e.g., touch, click), a GUI output interaction channel that displays interactive visual content, and a user interface server 130 that uses any known technique to manage and / or provide user interfaces for the GUI.

[0125] In an example process through the interactive system 100 of Fig.1, a representation of user input, such as a gesture recognized by the vision microservice 110, a voice command recognized by the speech recognition microservice 120, or a touch or click input recognized by the UI server 130, may be forwarded to a corresponding sensor server 160 responsible for a corresponding interaction channel. The sensor server(s) 160 may translate this user input into a standardized representation of a corresponding event and forward the event to an event gateway 180. The event gateway 180 may be used to communicate and distribute events to corresponding components, whether through synchronous interactions (e.g., via a REST API, Google Remote Procedure Call (RPC), etc.) and / or asynchronous interactions (e.g., via a message or event broker).The interaction manager 190 may be subscribed to or otherwise configured to ingest or receive these events from the event gateway 180. As such, the interaction manager 190 may process the events (e.g., using an event-driven state machine), determine which interactions should be performed, and generate commands as corresponding events in a standardized representation and forward them to the event gateway 180. The action server(s) 170 responsible for the corresponding interaction channel(s) may be subscribed to or otherwise configured to ingest the events it is responsible for executing from the event gateway 180.As such, the action server(s) 170 may execute, schedule, and / or otherwise handle events for the corresponding interaction modality(ies) by interfacing with a corresponding service that controls a corresponding output interface. For example, depending on the commanded action, a corresponding action server 170 may schedule and trigger bot speech via an audio interface (e.g., speech generation microservice 150), generate a bot animation on a screen or headset (e.g., animation microservice 140), present interactive visual content on a screen or headset (e.g., UI server 130), and / or in other ways.

[0126] In some embodiments, the interactive system 100 represents and / or communicates human-machine interactions and associated events using a standardized interaction modeling API and / or an event-driven architecture. In some embodiments, the standardized interaction modeling API standardizes how components (e.g., the sensor server(s) 160, the action server(s) 170, the interaction manager 190) represent multimodal interactions. In an example implementation, a standardized interaction modeling API serves as a common protocol in which various components of the interactive system 100 use a standardized interaction categorization scheme to represent all activities of bots, users, and / or the interactive system 100 as actions in a standardized form, states (e.g.,, of multimodal actions of users and bots) as events in a standardized form, support standardized, mutually exclusive interaction modalities and define how conflicts between standardized categories or action types are resolved, and / or implement standardized protocols for any number of standardized modalities and action categories, regardless of implementation.

[0127] Fig.2 shows an example of an interaction modeling API 220 according to some embodiments of the present disclosure. In general, different types of interactive systems may include different types of interaction channels 230. For example, a chatbot may use a text interface that supports an input interaction channel for input text and an output channel for output text. A voice assistant may use an audio interface that supports an interaction channel for input speech and an output channel for output speech. An interactive avatar may use a video input interface that supports an input interaction channel for recognized gestures, an audio input interface that supports an input interaction channel for recognized speech, a video output interface that supports an output interaction channel for avatar animations (e.g.,Postures, gestures), an audio output interface supporting an output interaction channel for avatar output speech, and / or a graphical user interface supporting an input interaction channel for touch input and / or an output channel for interactive visual content. A non-player character may use a game controller interface supporting an input interaction channel for controller input, a video output interface supporting an output interaction channel for non-player character animations, and an audio output interface supporting an output interaction channel for non-player character speech output. These are only examples, and other types of interactive systems, interactive agents, and / or interaction channels may be implemented within the scope of the present disclosure.

[0128] Fig.2 shows an example of an interaction modeling API 220 between the interaction manager 190 and the interaction channel(s) 230. In some embodiments, the interaction modeling API 220 defines a standardized format for specifying user and / or bot interactions, system events, and related events using a standardized interaction categorization scheme. The interaction categorization scheme may use standardized (e.g., semantically meaningful) keywords, commands, and / or syntax that includes or categorizes standardized interaction modalities, action types, and / or event syntax. Using the example of standardized interaction modalities, the interaction categorization scheme may be used to categorize interactions (e.g., bot actions) by standardized interaction modality and / or corresponding standardized action category (e.g.,Bot utterance, bot posture, bot gesture, bot gaze) using standardized action keywords. Fig. Figure 2 illustrates this by using separate rows to represent events for different interaction modalities (e.g., bot utterance events, bot posture events, bot gesture events, bot gaze events, scene or interactive visual content events). Furthermore, in some embodiments, the interaction modeling API 220 defines a standardized format for indicating changes in action states as corresponding events to support an event-driven architecture. Fig. Figure 2 illustrates this by using different start and stop times for different actions (e.g., the bot starts with a tense body posture before making an utterance, and initiates a gesture and a gaze action before completing the utterance, etc.).

[0129] In some embodiments, to promote configurability and interoperability, the interaction modeling API 220, a corresponding interaction modeling language supported by the interaction manager 190, and / or a corresponding interaction categorization scheme supported by (e.g., sensory and / or action servers in) the interaction channel(s) 230 may provide a way to categorize, specify, and represent interactions for a variety of different interactive systems and corresponding interaction channels, which may enable designers to customize an interactive system using standardized components. Fig.Figure 3 illustrates some features of example interactive systems that may be supported by an example interaction modeling API and / or an example interaction modeling language in accordance with some embodiments of the present disclosure. In some embodiments, an interactive system relies on an interaction modeling API and / or an interaction modeling language that supports more interactions and action keywords than the interactive system itself uses. For example, an interaction modeling API and / or an interaction modeling language may support a keyword for a bot gesture (e.g., MakeGesture), even though an interactive system using the API and / or modeling language (e.g., a chatbot) may not utilize that type of interaction.However, by supporting a variety of multimodal interactions, an interaction modeling API and / or an interaction modeling language can support a variety of interactions or features that a designer can draw on to customize an interactive system, promote interoperability by standardizing the representation of interactions, and make the designer's life easier by reducing their cognitive load when developing an interactive system.

[0130] In some embodiments, an interaction modeling API and / or an interaction modeling language may support standardized representations of actions and events for interaction modalities such as speech, gestures, emotions, movements, scenes, and / or others. In some embodiments, the interaction modeling API and / or language may define mutually exclusive interaction modalities such that actions in different interaction modalities can be performed independently of each other (e.g., by corresponding action servers) (e.g., a bot can say something independently of gestures). The possibility of simultaneous or conflicting actions in the same interaction modality may be accounted for by (e.g., a corresponding action server) implementing a modality policy for that interaction modality.Thus, an action server implementing this interaction modality can use a specific modality policy to determine how events for this interaction modality should be executed, scheduled, and / or otherwise handled. Fig. Figure 4 shows some example modality policies in accordance with some embodiments of the present disclosure. Accordingly, an interaction modeling API and / or an interaction modeling language may support an interaction categorization scheme that defines standardized representations of supported interaction modalities and corresponding actions and events, such as the one described in Fig.5. As illustrated in this example, some modality groups (e.g., motion) can be divided into groups of interaction modalities that can be executed independently (e.g., a Bot expression on a Bot face modality can be animated independently of a Bot pose on a Bot upper body modality).

[0131] In interactive systems that support multimodal interactions, information may be exchanged between a user and an interactive system across multiple interaction modalities. Each interaction modality may be implemented via a corresponding interaction channel between the interactive system and the user. In some embodiments, an interaction categorization scheme may categorize any action as part of a single interaction modality, although, depending on the interactive system, an action server for that interaction modality may map that action to multiple output interfaces (e.g., audio, video, GUI, etc.). For example, a BotUtterance action (instructing a bot to verbally communicate with a user) may be classified as part of a BotVoice modality. In an interactive system that represents the bot as a 3D avatar (e.g.,On a 2D screen, on an AR or VR device, the BotVoice modality and / or the BotUtterance action can trigger various types of output, such as audio output (e.g., synthesized speech), lip movement (e.g., lip syncing to speech), and / or text on a user interface (e.g., closed captions for utterances). In another example, a BotMovement action can be classified as part of a BotLowerBody modality and trigger a lower body animation (e.g., a walking animation) and audio output (e.g., the sound of footsteps).

[0132] Fig. 6 shows an example of an event-driven interactive system 600 according to some embodiments of the present disclosure. Fig.Figure 6 shows an example implementation of an architectural pattern that separates the component(s) that implement the decision logic that determines which actions to perform (e.g., the interaction manager 640) from the component(s) that handle(s) the interactions (e.g., the sensor server 620 and the action server 670).

[0133] At a high level, detected input events 610 (e.g., the representation of user input such as a detected gesture, voice command, or touch or click input; the representation of a detected feature or event associated with user input such as a detected presence or absence of voice activity, a detected presence or absence of tapping, a detected transcribed speech, a detected change in the volume or speed of tapping, etc.) may be forwarded to a sensory server 620, and the sensory server 620 may translate the detected input events 610 into standardized input events 630.An interaction manager 640 may process the standardized input events 630 and generate events representing instructed bot actions (instructed bot action events 650), and an action server 670 may execute the actions represented by the instructed bot action events 650. In some embodiments, the interaction manager 640 may generate internal events 660 representing internal state changes (e.g., expiration state changes) or instructed bot actions, and / or the action server 670 may generate events 665 representing confirmation of action state changes, which may be evaluated by the interaction manager 640 in determining the actions to take.

[0134] The interaction manager 640 (which corresponds to the interaction manager 190 of Fig. 1 and / or Fig.2) may be responsible for deciding which actions the interactive system 600 should perform in response to user actions or other events (e.g., the standardized input events 630, the internal events 660, the events 665 representing confirmation of changes in action status). The interaction manager 640 may (but need not) interact with the rest of the interactive system 600 (e.g., exclusively) via an event-driven mechanism. In practice, it may happen that other parts of the interactive system 600 generate other events while the interaction manager 640 is busy processing an event (e.g., deciding on the next action). Thus, the interaction manager 640 may process multiple events sequentially or all at once, depending on the implementation.In a stateful approach, interaction manager 640 may maintain the state or context of a user's interaction with interactive system 600 across multiple interactions within a given session. In a stateless approach, the history of the state or context may be presented along with each new event. In some embodiments, there is no shared state between interaction manager 640 and the rest of interactive system 600.

[0135] In general, the interactive system 600 may include any number of interaction managers (e.g., interaction manager 640). In some implementations, the interactive system 600 may include a primary interaction manager with internal or secondary interaction managers. In an example involving an interactive avatar experience, a primary interaction manager may manage the high-level flow for a human-machine interaction (e.g., the various phases such as greeting, data collection, data provision, confirmation, etc.), and the primary interaction manager may delegate decision-making to one or more secondary interaction managers as appropriate (e.g., for a complex authentication flow, for an interactive question-and-answer scenario, etc.). In some implementations, the interactive system 600 may include multiple peer interaction managers, each handling different types of events.For example, one interaction manager can manage the conversation logic (e.g., what the bot should say), and a second interaction manager can manage the animation of an avatar based on what it says.

[0136] In some embodiments, interactions between the interaction manager 640 and the rest of the interactive system 600 occur via various types of (e.g., standardized) events, such as those representing detected input events (e.g., detected input events 630), instructed bot action events (e.g., instructed bot action events 650), and system or context events. In general, detected input events can be used to represent any event that may be relevant to an interaction, such as the user saying something (e.g., UserSays), making a gesture (e.g., UserGesture), or clicking a GUI element (e.g., UserSelect). Bot action events can specify what the interactive system 600 should do, such as saying something, playing a sound, displaying something on a screen, changing the appearance or pose of an avatar, calling a third-party API, etc.Bot action events may represent transitions in the lifecycle of an action, e.g., an instruction to do something (e.g., StartAction), an indication of when the action began (e.g., ActionStarted) or ended (e.g., ActionEnded), etc. System or context events may represent changes to related interaction data contained in the interactive system 600 (e.g., ContextUpdate), such as user name, user rights, selected product, device information, etc.

[0137] Thus, the interaction manager 640 may evaluate various types of events (e.g., the standardized input events 630, internal events 660, events 665 representing confirmation of changes in action state), determine which actions should be performed, and generate corresponding instructed bot action events 650. As such, the action server 670 may execute the actions represented by the instructed bot action events 650. For example, the interaction manager 640 may decide that the interactive system 600 should say "Hello!" and, upon completing that utterance (e.g., Say), perform a particular gesture (e.g., point to a screen and ask something). In some examples, the interaction manager 640 may generate an event indicating that the gesture should begin (e.g., with keywords such as StartAction(MakeGesture)) when the interactive system 600 finishes the greeting (e.g.,via a specific condition such as ActionFinishes(Says)). As another example, the interaction manager 640 may decide to start a wave animation when a Say(Hello) action has begun and to stop the animation when Say(Hello) has finished. In some of these examples, the interaction manager 640 may specify conditions (e.g., ActionStarted(Says) and ActionFinishes(Says)) when specifying corresponding instructions to start and stop the gesture (e.g., StartAction(MakeGesture(Wink)) and StopAction(MakeGesture(Wink))).

[0138] In some embodiments, the interaction manager 640 implements an interpreter or compiler that interprets or executes code written in an interaction modeling language that categorizes user and / or bot interactions and associated events using a standardized interaction categorization scheme (such as that described in Fig.5). In general, the interpreter and interaction modeling language may support any number of keywords used to parallelize action and flow execution and matching (e.g., send, match, start, stop, wait, activate). The interaction modeling language may be used to define a flow of interactions using primitives that include semantically meaningful (e.g., natural language) keywords and commands that specify events (e.g., something happened) and actions (e.g., something must happen) using the interaction categorization scheme. In some embodiments, events (e.g.,The event (e.g., the standardized input events 630, internal events 660, events 665 representing the confirmation of action state changes) may be represented using an event specifier with standardized syntax defined by the interaction categorization schema, the interaction modeling language, and / or the interaction modeling API and supported by the interpreter. In some embodiments, an event may include a representation of corresponding (e.g., standardized) fields and values ​​(e.g., payloads specifying them) that the interpreter (and other components) may be able to understand. As such, the interpreter may execute code that implements an interaction flow in the interaction modeling language, where the interaction flow may instruct the interpreter which actions or events to generate in response to which events.

[0139] For example, an event within the interactive system 600 may be represented and / or communicated in various ways. As a non-limiting example, an event (e.g., payload) may include fields that specify or encode values ​​representing an action type (e.g., identification of a standardized interaction modality or corresponding action type such as UserSays), an action state (e.g., an observed state of a user action such as Finished, a current or confirmed state of a bot or scene action such as Started, a commanded state of a bot or scene action such as Start), recognized or commanded action content (e.g., transcribed or commanded speech such as "Hello," a description of a recognized or commanded gesture, a description of a recognized or commanded pose or expression, etc.), a unique identifier (UID) to identify the event, a timestamp (e.g.,, indicating when the event was created, when an action was updated), a unique source identifier that identifies the source of the event, one or more tags (indicating, for example, that the event was generated as part of a particular stream or session, or is associated with a particular user or account), context, and / or other properties or information.

[0140] In some embodiments, each action can be identified by a unique identifier (action_uid), and all events related to the same action can reference the same action_uid. Thus, individual events referencing the same action_uid can serve to represent the lifecycle of a corresponding action from start to finish (e.g., including updated action states in between). In some embodiments, the component that sends the StartAction and ActionStarted events can generate the action_uid for new instances of an action, and the specific component involved can depend on the type of action (e.g., bot or user action).For example, the interaction manager 640 may be responsible for generating the action_uid for new instances of bot actions initiated by the interaction manager 640, and the sensor server 620 may be responsible for generating the action_uid for new instances of observed user actions. Thus, individual events can be linked to a corresponding instance of a particular action type.

[0141] Using an example of an interaction categorization scheme as described in Fig.5, actions can be classified into corresponding interaction modalities such as speech, gestures, emotions, movements, scenes, and / or others. Using speech as an example, the interactive system 600 can support various events and actions related to dialogue management by using the speech modality. For example, the user can use a user speech modality (e.g., via a user utterance action), or a bot provided by the interactive system 600 can use a bot speech modality (e.g., via a bot utterance action). In an example of a user utterance action, the user can make an utterance that is recognized by the interactive system 600. Examples of this action include the user typing into a text interface to interact with the bot or the user speaking to an interactive avatar.Examples of possible events associated with this action are UtteranceUserActionStarted, StopUtteranceUserAction (e.g., instructing the action server 670 to decrease the hold time of automatic speech recognition), UtteranceUserActionTranscriptUpdated (e.g., providing updated transcripts during an UtteranceUserAction), UtteranceUserActionIntensityUpdated (e.g., providing the recognized speech intensity, typing speed, change in volume or pitch, etc.), UtteranceUserActionEnded (e.g., providing a final transcript), and / or others.

[0142] In an example bot utterance action, a bot may deliver an utterance (e.g., say something) to the user via a form of verbal communication (e.g., via a chat interface, a voice interface, brain-to-machine communication, etc.). Examples of possible events associated with this action are StartUtteranceBotAction (e.g., instructing the bot to produce an utterance, with payload that may include a transcript of the bot's instructed utterance, a representation of intensity, such as speech intensity, output text rate, change in volume or pitch, etc.), UtteranceBotActionStarted (e.g., indicating that the bot has begun producing an utterance), ChangeUtteranceBotAction (e.g., instructing an adjustment of volume or other property if the action has already begun), UtteranceBotActionScriptUpdated (e.g.,Providing updated transcripts during an utterance, bot action, stop utterance, bot action (e.g., instructing the bot utterance to end), utterance, bot action, finished (e.g., confirming or notifying that the bot utterance has ended, e.g., because it has been completed or because the user has ended the utterance), and / or others.

[0143] Using motion as an example, interactive system 600 may support various events and actions related to the motion modality. Motion actions may represent movements or sets of movements that have a specific meaning. For example, the user may make a gesture or pose recognized using computer vision, or a bot provided by interactive system 600 may make a gesture or pose. In some embodiments, the user and / or bot may use any suitable motion modality (e.g., face, upper body, lower body). In some embodiments, these modalities may be governed by an "override" modality policy, which action server 670 may interpret as an instruction to handle concurrent actions by temporarily overriding a currently running action with a newly initiated action.As a non-limiting example, if the interaction manager 640 initiates a BotPosture ("crossed arms") action that instructs an avatar to keep its arm crossed until the action is stopped, and two seconds later the interaction manager 640 initiates a BotGesture ("wave") action, the action server 670 may perform the "wave" action by overriding the "crossed arms" posture with the "wave" action (e.g., so that the avatar waves to the user). Once the wave action is complete, the action server 670 may return the avatar to the "crossed arms" posture (e.g., resume the overridden action).

[0144] For a bot action with a facial expression, a corresponding event can instruct a bot to make a facial expression (e.g., a smiley face in a chatbot text message, a digital avatar facial expression in an interactive avatar experience) using a specific expression or emotion (e.g., joy, surprise, contempt, sadness, fear, disgust, anger, etc.). Examples of possible events related to this action include StartExpressionBotAction (e.g., instructing the bot to change the expression and specifying the type of expression), ExpressionBotActionStarted (e.g., indicating that the bot has started the action), StopExpressionBotAction (e.g., instructing the bot to stop the expression), ExpressionBotActionEnded (e.g., indicating that the bot has finished the expression), and / or others.

[0145] In some embodiments, the interactive system 600 may support user actions involving facial expressions and corresponding events representing recognized user expressions. Examples of possible events associated with this action include ExpressionUserActionStarted (e.g., indicating that a user's facial expression has been recognized, including a representation of the expression's content, such as joy, surprise, contempt, sadness, fear, disgust, anger, etc.) and ExpressionUserActionEnded (e.g., indicating that the recognized user's facial expression has returned to a neutral expression).

[0146] In an example of a gesture bot action, a corresponding event may instruct a bot to perform a specific gesture. In some embodiments, events associated with this action may include payload data comprising a natural language description of the gesture, which may include a base gesture, one or more gesture modifiers, and / or other features. Examples of base gestures include speaking, idle (e.g., a spontaneous physical movement or action during a period of inactivity), affirm (e.g., a nonverbal cue or action indicating agreement, confirmation, or affirmation), negate (e.g., a nonverbal cue or action indicating disagreement, opposition, or refusal), attract (e.g., a specific movement, action, or behavior intended to draw a user's or audience's attention to a particular object, location, or activity), and / or others.Example hierarchies for some basic gestures include: speaking with emotion (e.g., "excited speaking"), idling with arousal level (e.g., "nervous idle"), and attracting with intensity (e.g., "subtle attracting"). Examples of possible events associated with this action might include StartGestureBotAction, GestureBotActionStarted, StopGestureBotAction, GestureBotActionEnded, and / or others.

[0147] In some embodiments, the interactive system 600 may support gesture user actions and corresponding events representing recognized user gestures. Examples of possible events associated with this action include GestureUserActionStarted (e.g., indicating that a user gesture has been recognized, including a representation of the content of the gesture) and GestureUserActionCompleted (e.g., indicating that completion of the user gesture has been detected).

[0148] For an example bot position change or bot movement action (e.g., in the BotUnderBody movement modality), a corresponding event may instruct the bot to move to a specific position (e.g., on a screen, in a simulated, or virtual environment). A specific position may include a base position, one or more position modifiers, and / or other characteristics. In an example implementation, the supported base positions may be front and back, and the supported position modifiers may be left and right. Examples of possible events related to this action include StartPositionChangeBotAction (e.g., identifying a specific position for the bot to move to) and PositionChangeBotActionCompleted.

[0149] For an example action involving a change in user position or user movement (e.g., in the BotLowerBody movement modality), a corresponding event may indicate a detected change in the position of the user's lower body. Examples of possible events related to this action include: PositionChangeUserAction (e.g., indicating that a detected user movement has begun, including a representation of the direction or nature of the detected movement, such as active, approaching, passive, walking away, sideways, etc.); PositionChangeUserDirectionUpdated (e.g., indicating when the user changes direction during a detected movement); PositionChangeUserActionCompleted (e.g., indicating that the detected movement is complete).

[0150] In some embodiments, interactive system 600 supports interactive visual content actions and events that represent the presentation and / or interaction with various types of visual information (e.g., in a 2D or 3D interface). Examples of interactive visual content actions (also referred to as visual actions) include visual selection actions, visual information scene actions, and visual form actions.

[0151] In one example of a visual selection action, a corresponding event may instruct the visualization of a selection with which the user can interact. The interactive system 600 may support various types of interactions with a visual selection (e.g., by presenting a website on a display that accepts a touch or click option, or by accepting voice input to select an option). For example, a StartVisualSelectionSceneAction event may include a payload with a prompt describing the choice to be presented to the user; an image describing what to show the user; one or more assistance prompts to assist or guide the user in making a choice (e.g., "Just say 'Yes' or 'No' to continue"), or a recommendation ("I can recommend the Cheeseburger"); a list of options from which the user can choose (e.g.,each option can have a corresponding image); a selection type ("selection", "search", etc.); and / or an indication of whether multiple selections should be allowed. Other examples of possible events related to this action include a VisualSelectionSceneActionUpdated event (e.g., to indicate detected user interaction with a selection presented in the scene when the user has not yet confirmed the selection), StopVisualSelectionSceneAction (e.g., to instruct the user to remove the visual selection), VisualSelectionSceneActionFinish (e.g., to indicate a finally confirmed selection), and / or others.

[0152] In an example of a visual information scene action, a corresponding event can instruct the visualization of specific information for the user. The visual information scene action can be used to show the user detailed information about a specific topic related to the interaction. For example, if the user is interested in details about a specific or displayed product or service, a visual information scene action can instruct the presentation of information about that product or service. Examples of possible events related to this type of action are VisualInformationSceneAction (e.g., instruction to visualize; specifying a description of the content to be visualized; specifying one or more blocks of the content to be visualized, such asa title, a summary of the content and / or a description of one or more images to be visualized; one or more prompts for assistance, etc.); VisualInformationSceneActionStarted (indicates, for example, that the visual information scene action has started), Stop VisualInformationSceneAction (instructs, for example, the visualization to stop), VisualInformationSceneActionFinish (indicates, for example, that the user has closed the visualization or the visual information scene action has stopped), and / or others.

[0153] In an example of a visual form action, a corresponding event can instruct the visualization of a specific visual form with one or more form fields (e.g., email, address, name, etc.) that the user should fill out. Examples of possible events related to this type of action are StartVisualFormSceneAction (e.g., instructing the visualization; specifying one or more inputs, a prompt for the user, one or more help options, one or more images, etc.), StartVisualFormSceneActionStarted (e.g., indicating that the user has started entering information into the form), StartVisualFormSceneInputUpdated (indicating, for example, that the user has entered information into the form but has not yet confirmed the selection), StopVisualFormSceneAction (indicating, for example,indicates that the visualization of the form should be paused), VisualFormSceneActionCompleted (indicates, for example, that the user has confirmed or canceled the form input) and / or others.

[0154] In some embodiments, the interactive system 600 may support actions and events that represent various aspects of the scene in which the human-machine interaction takes place. For example, the interactive system 600 may support actions related to a sound modality (e.g., indicating sound effects or background noise), an object interaction modality (e.g., indicating interactions between a bot and virtual objects in the environment), a camera modality (e.g., indicating camera cuts, actions, transitions, etc.), a visual effects modality (e.g., indicating visual effects), a user presence modality (e.g., indicating whether or not the user's presence is detected), and / or other examples. Examples such as these and others are further described in U.S. Provisional Application No. 63 / 604,721, filed November 30, 2023, the contents of which are incorporated by reference in their entirety.

[0155] After describing some example events associated with standardized action types and interaction modalities, and some possible ways of representing such events and actions, the following discussion turns to some possible ways in which (e.g., an interpreter of) interaction manager 640 may use a designated interaction flow (or simply a flow) (e.g., written in an interaction modeling language) to evaluate such events (e.g., incoming and / or queued instances of standardized input events 630, internal events 660, events 665 representing confirmation of action state changes), to determine what actions or events to generate in response thereto, and to generate corresponding events (e.g., outgoing instances of instructed bot action events 650, internal events 660).

[0156] In general, a flow can specify instructions using primitives from an interaction modeling language that includes semantically meaningful (e.g., natural language) keywords and commands that specify events (e.g., something happened) and actions (e.g., something must happen) using an interaction categorization scheme. The state of an action (e.g., an observed state of a user action, a current state of a bot or scene action) and / or a command to change the state of a bot or scene action can be represented using standardized event keywords, commands, and / or syntax. For example, action events (e.g., a user or bot action started or stopped) can be represented by an event specifier with standardized syntax (e.g.,Event name and / or identifier with a keyword identifying the standardized action category and a specifier for the user or bot action state). A command line in a flow may contain an event trigger (e.g., using a keyword such as send) that causes the interpreter to generate a specific event when a certain condition is met (e.g., events representing commands to perform a bot action can trigger the action to be performed, or events representing a change in user state can trigger a corresponding bot action), or an event matcher (e.g., using a keyword such as match) that causes the interpreter to pause the flow and wait for a specific event before resuming the flow.The event trigger and event match can specify corresponding trigger and match conditions by using an event specifier that includes a standardized event name or identifier (e.g., a keyword identifying a standardized action category paired with a corresponding action state specifier or a command to change an action state) and an argument that specifies one or more conditions that the specified event must satisfy (e.g., using predefined parameters and supported values ​​or a natural language description).

[0157] For example, (e.g., the interpreter of) interaction manager 640 may be equipped with logic that interprets corresponding keywords, commands, and / or syntax such as those mentioned. In some embodiments, interaction manager 640 may support any number of keywords used to parallelize action and flow execution and reconciliation (e.g., any of the keywords described above such as send, match, start, stop, wait, activate, return, abort, and / or others). Interaction manager 640 may be programmed to sequentially execute the instructions specified in a particular flow, generate any events specified by an event trigger, and stop when a flow header reaches an event reconciler, an exception, or the end of a flow. In some embodiments, interaction manager 640 may support and track multiple active flows (e.g.,interrupted at corresponding event matchers), (e.g., using an event-driven state machine to) wait for incoming events that match an event matcher of an active flow and trigger corresponding events and actions specified in matching flows.

[0158] Fig.7 shows an example of an interaction manager 700 in accordance with some embodiments of the present disclosure. In this example, the interaction manager 700 includes an interpreter 710, interaction flow(s) 780, and an internal event queue 790. At a high level, the interaction flow(s) 780 specify corresponding sequences of instructions in an interaction modeling language and may be loaded or otherwise made available to the interpreter 710. The interpreter 710 may include an event processing component 730 that sequentially executes the instructions specified in the interaction flow(s) 780 to process incoming events and generate outgoing events (e.g., in a standardized form).

[0159] The event processing component 730 may execute a primary processing loop that processes incoming events and generates outgoing events. At a high level, the event processing component 730 includes a flow execution component 750 and a flow matcher 740. The flow execution component 750 may sequentially execute the instructions specified in a flow (e.g., a parent flow, a matching flow) of the interaction flow(s) 780, generate any events specified by an event trigger, and stop when a flow header reaches an event matcher, an exception, or the end of a flow.The flow adaptor 740 may evaluate incoming events to determine whether they match a flow adaptor of an active flow, instruct an action conflict resolver 760 to resolve any conflicts between multiple matching flows, and instruct the flow execution component 750 to forward matching flows (e.g., conflict-free).

[0160] In one embodiment, the flow execution component 750 may perform lexical analysis of the instructions specified in the interaction flow(s) 780 (e.g., tokenization; identifying keywords, identifiers, arguments, and other elements), iterate over the flow instructions, executing each one in turn, and include mechanisms for handling exceptions. In some embodiments, the flow execution component 750 uses a different flow header for each of the (e.g., active) interaction flows 780 to indicate the current position and advance through the instructions in a corresponding interaction flow.Depending on the instruction, the flow execution component 750 may advance any given flow header to the next instruction, jump to another flow referenced by a particular label or other flow identifier, split into multiple heads, merge multiple flow heads, and / or other. As such, the flow execution component 750 may coordinate with a trace and control component 770 to establish and maintain a hierarchy of flow heads. When a parent flow header in a branch of a hierarchy of flows or flow headers is stopped, paused, or resumed, the flow execution component 750 may coordinate with the trace and control component 770 to stop, pause, or resume all child flow heads of that parent flow header or branch.In some embodiments, each flow may specify any number of scopes that the flow execution component 750 may use to generate stop events that instruct a corresponding action server to stop previously initiated actions within the corresponding scopes.

[0161] For example (e.g., at startup), the flow execution component 750 may execute a top-level flow (e.g., of the interaction flow(s) 780) that contains instructions to activate any number of flows (e.g., of the interaction flow(s) 780) with any number of event matchers. The trace and control component 770 may use any suitable data structure to track active flows and corresponding event matchers (e.g., using a tree or other representation of nested flow relationships), and it may use an event-driven state machine that listens for various events and triggers corresponding actions specified in matching flows (with event matchers corresponding to an incoming event).As such, the flow execution component 750 may iterate through active flows, generate any events specified by an event trigger, and stop when a flow header reaches an event handler, an exception, or the end of a flow.

[0162] In some embodiments, ongoing flows may instruct the flow execution component 750 to generate outgoing events that instruct an action. Additionally or alternatively, ongoing flows may instruct the flow execution component 750 to generate events that notify listeners (such as the flow execution component 750 itself) that an event has occurred. As such, the flow execution component 750 may dispatch these events, and / or the interpreter 710 may maintain an internal event queue 790 and place these events into the internal event queue 790 (e.g., in case another flow is waiting for a generated event).

[0163] Once the flow heads for all advanced flows have reached an event matcher, an exception, or the end of a flow, the flow matcher 740 can sequentially process incoming events (e.g., from the internal event queue 790, from another queue, or an event gateway, such as the event gateway 180 of Fig.1) and, for each event, test the event matcher specified by each active flow for a match to the event. In some embodiments, the flow matcher 740 sequentially processes all internal events in the internal event queue 790 (e.g., testing active flows for matches to internal events) before moving on to processing the next incoming event (e.g., from the event gateway). Internal events may represent updated states of interaction flows 780 (e.g., indicating that a particular flow has started, finished, aborted, etc.) that have progressed in response to a particular incoming event. This allows a designer to create a flow that depends on the evolution or state of other flows.

[0164] When processing an event, the event matching function 740 may compare the event with the event matching function for each active (e.g., suspended) flow to determine if the event matches any of the active flows (e.g., using a known matching technique and / or as described in more detail below). In some scenarios, multiple active flows specifying different interactions may be triggered by different conditions that can be satisfied by the same event. If there is an event matcher from an active flow that matches the event (a matching flow), the flow matcher 740 may instruct the flow execution component 750 to forward that flow (e.g., and generate outbound events to trigger any actions specified by the forwarded flow).

[0165] If there are multiple matching moves, the move reconciler 740 may instruct the action conflict resolver 760 to determine whether the matching moves agree on an action or not. If they agree, the action conflict resolver 760 (or the flow reconciler 740) may instruct the flow execution component 750 to forward both matching flows. If they disagree, the action conflict resolver 760 may apply conflict resolution to determine which action should take precedence and instruct the flow execution component 750 to advance the matching flow with the prioritized action and abort the other matching flow(s) (e.g., because the interaction pattern represented by those flows no longer applies).If there are no active flows that match an event, the flow matcher may generate an internal event that matches a particular flow to handle unmatched or unprocessed events, may execute one or more unprocessed event handlers (e.g., unprocessed event handler 744), and / or use another technique to handle an unprocessed event.

[0166] After checking for matches and advancing flows, the tracing and control component 770 can check the flow states for any flows that have completed or aborted and stop any active flows that were activated by these completed or aborted flows (e.g., because the interaction pattern represented by these flows should no longer apply). This allows the interpreter 710 to iterate through events, advance flows, perform conflict management to determine which actions to perform, and generate outgoing events to trigger these actions.

[0167] By way of illustration, in some embodiments, the interpreter 710 uses an event-driven state machine such as the event-driven state machine 800 of Fig.8 to process incoming action events 805 and internal events 820. In some embodiments, the event-driven state machine 800 may process incoming action events 805 (e.g., corresponding to the standardized input events 630 of Fig. 3 and are connected via an event gateway such as the Event Gateway 180 of Fig. 1) into an interaction event queue 810. The event-driven state machine 800 may place internal events 820 (e.g., corresponding to the internal events 660 of Fig. 6) into an internal event queue 815 (which may correspond, for example, to the internal event queue 790 of Fig. 7) and may give priority to processing events from the internal event queue 815 over events from the interaction event queue 810.

[0168] For each event, the event-driven state machine 800 may perform at least some of the steps illustrated in block 825. For example, in block 830, the event-driven state machine 800 may check the event matcher specified by each active flow for a match to the event. If there is an event matcher from an active flow that matches the event (a matching flow), the event-driven state machine 800 may proceed to block 835 and advance that flow (e.g., generate outbound interaction events 870 to trigger actions). If there are multiple matching flows, the event-driven state machine 800 may proceed to block 840 and determine whether or not the matching flows agree to an action. If they agree, the event-driven state machine 800 may proceed to block 850 and advance both matching streams.If they do not match, the event-driven state machine 800 may proceed to block 855 and apply conflict resolution to determine which action should take precedence, advance the matching flow with the prioritized action, and abort the other matching flow(s). If there are no active flows that match an event, the event-driven state machine 800 may proceed to block 835 and execute one or more unprocessed event handlers (or generate an internal event that matches a particular flow for handling unmatched or unprocessed events).After checking for matches and advancing flows, the event-driven state machine 800 may proceed to block 860, check the flow states for completed or aborted flows, stop any active flows activated by these completed or aborted flows, and proceed to the next event in block 865. Thus, the event-driven state machine 800 may iterate through the internal events 820 in the internal event queue 815 and / or the incoming action events 805 in the interaction event queue 810, advance flows, perform conflict management to determine which interactions to execute, and generate outgoing interaction events 870 to trigger these interactions.

[0169] Back to Fig.7: In some embodiments, the interpreter 710 may support the use of natural language descriptions and the use of one or more LLMs, such as the generative LLM system 2800 of Fig. 28A or the generative LLM 2830 from the Fig. 4A, Fig. 4B, or 4C.

[0170] For example, each of the interaction flows 780 may be specified with a corresponding natural language description that summarizes the interaction pattern represented by the flow, and the interpreter 710 uses such flow descriptions under certain circumstances (e.g., a designated flow for handling unknown events and / or the unknown event handler 744 may request an LLM to determine whether an unmatched event representing an unrecognized user intent semantically matches the natural language description of an active flow representing a target user intent). In some embodiments, the interpreter 710 may include a flow description generator 720 that analyzes one or more specified interaction flows 780 (e.g.,at design time), performs a lexical analysis to determine if any specified flows lack a corresponding flow description, and if so, requests an LLM to generate one (e.g., based on the name and / or instructions of the flow). Additionally or alternatively, the flow description generator 720 may determine (e.g., request an LLM to) if any specified flow descriptions are inconsistent with their corresponding flow descriptions, and if so, request an LLM to generate a new one (e.g., as a suggestion or for automatic replacement) (e.g., from the name and / or instructions of the flow). Thus, the flow description generator 720 may determine if a description is to be created for any of the interaction flows 780 and may generate corresponding flow descriptions.

[0171] In some embodiments, a designer may specify a flow description for the interaction flow(s) 780 (e.g., a natural language description of what the flow should do) without the command sequence, or may name one of the interaction flows 780 without defining it. Thus, in some embodiments, the interpreter 710 may include a flow auto-completion component 725 that analyzes the interaction flow(s) 780 (e.g., at design time, at runtime), identifies whether the interaction flow(s) 780 is missing a sequence of instructions, and, if so, prompts an LLM to generate one (e.g., based on the name and / or description of the flow). For example, the auto-completion component 725 may provide the LLM with one or more prompts that include one or more example flows, a specified name of the interaction flow(s) 780, and / or a (e.g.,specified or generated) natural description of the interaction flow(s) 780 and a request to complete the interaction flow(s) 780.

[0172] For example, the auto-completion component 725 may create a prompt using a template with placeholders, such as the following: Content: |- # Example flows: {{ examples}} # Complete the following flow based on its statement: Flow {{ flow_name}} """ {{ natural language description of the flow}}""

[0173] This example template contains placeholders for example flows, a specific flow name, and a specific natural language description of the flow. The flow auto-completion component 725 may generate one or more prompts by populating the placeholders with appropriate content (e.g., designated example flow(s), a specific flow name, a specific natural language description of the flow, and / or other content) and may provide these constructed prompt(s) to an LLM (e.g., via an API request). As such, the LLM may generate and return an auto-completed flow with generated instructions that the flow auto-completion component 725 may insert into or otherwise link to the corresponding interaction flow(s) 780.

[0174] In an example implementation, the flow execution component 750 may execute instructions specified in the interaction flow(s) 780 (e.g., including any encountered event triggers) until it reaches an event matcher, at which point the flow execution component 750 may interrupt the interaction flow(s) 780. The flow matcher 740 may process each event by executing the event matcher in each interrupted flow and comparing the event to the target event parameters and the parameter values ​​specified by the flow matcher's event specifier. Depending on the implementation, the flow matcher 740 may support a variety of matching techniques to determine whether an event matches an active event matcher for one of the active flows.In general, the trace matcher 740 may use any known technique to compare the parameters and parameter values ​​of the target event with those of the event to generate a representation of whether the event matches (e.g., a binary indication or a match score quantifying an explicit or fuzzy match).

[0175] However, in some embodiments, an event trigger or event matcher in one of the interaction flows 780 may specify target event parameters and / or parameter values ​​using natural language descriptions. Thus, in some embodiments, a syntax generator 752 may derive the target event parameters and / or values ​​from a specified natural language description in the interaction flow(s) 780 (e.g., a description of all target event parameters and values, a description of a single parameter value), and the syntax generator 752 may insert (or otherwise link) the generated target event parameters and values ​​into the corresponding event specifier in the interaction flow(s) 780. Before the flow execution component 750 executes an instruction (e.g., an event trigger) that includes an event specifier, the flow execution component 750 (e.g.,at runtime) instruct the syntax generator 752 to determine whether the statement contains a parameter specified with a natural language description (e.g., with a lexical analysis). Additionally or alternatively, prior to executing a statement (e.g., an event matcher) that contains an event specifier, the flow matcher 740 may instruct the syntax generator 752 (e.g., at runtime) to determine whether the statement contains a parameter specified by a natural language description (e.g., by a lexical analysis). If so, the syntax generator 752 may request an LLM to generate corresponding target event parameters and / or parameter values ​​for the event specifier and to update the event specifier in a corresponding one of the interaction flows 780 with the generated target event parameters and / or parameter values.

[0176] As an example of a prompt for generating a target event parameter value (or other variable value), the syntax generator 752 may construct a prompt using a template prompt with placeholders, such as the following: Content: |- """ {{ general_instructions}} """ # This is what a conversation between a user and the bot might look like: {{ pattern_conversation}} # This is the current conversation between the user and the bot: {{ history | colang}} # {{ natural language description of the parameter value}} ${{ var_name}} =

[0177] This example template contains placeholders for general instructions, a conversation example (or a series of interactions), a history of the current conversation (or series of interactions), the name of the variable with the value to be generated, and a request to generate the value ("${{ Var_Name}} ="). The syntax generator 752 can generate one or more prompts by filling the placeholders with appropriate content (e.g., specific instructions, a specific conversation example or interaction history, a recorded history of the current conversation or series of interactions, the extracted natural language description of the parameter value to be generated for a corresponding variable, the name of the variable, and / or other content), and can provide these constructed prompt(s) to an LLM (e.g., via an API request).As such, the LLM can generate and return the requested value, which the syntax generator 752 can insert into the event specifier in the corresponding statement. This example is intended to illustrate only one way an LLM can be used to generate a target event parameter value from a specified natural language description of the value. Other types of prompts and prompt content may be implemented within the scope of the present disclosure. Furthermore, those skilled in the art will know how to adapt the above example prompt to generate other content described herein (e.g., generating the name of a target event parameter from a natural language description of the target event parameter(s) and / or parameter value(s), a list of auxiliary variable names, etc.).Thus, the flow execution component 750 may execute an event trigger and / or the flow matcher 740 may execute an event matcher using target event parameters and parameter values ​​generated by an LLM.

[0178] Accordingly, in some embodiments, the flow matcher 740 generates and / or quantifies a representation of whether an event is a match (e.g., explicit or fuzzy) by comparing specified or generated target event parameters / parameter values ​​of an event matcher (e.g., keyword(s) or command(s) representing a target interaction modality, an action, an action state, and / or other event parameter values) with the corresponding parameters / parameter values ​​of the tested event (e.g., keyword(s) or command(s) representing a commanded or recognized interaction modality, an action, an action state, and / or other event parameter values). Additionally or alternatively, the flow matcher 740 may include a flow description matcher 742 that (e.g.,at runtime) requests an LLM to determine whether an event matches the flow description of one of the interaction flows 780 and / or a specified natural language description of one or more parameters or parameter values.

[0179] At a high level, an event may represent a user action or intent, a bot action or intent, a scene interaction, or another type of event using a standardized interaction categorization scheme that classifies actions, action events, event parameters, and / or parameter values ​​using (e.g., standardized, natural language, semantically meaningful) keywords and / or commands and / or a natural language description (e.g., GestureUserActionCompleted("ThumbsUp")). As such, a flow description matcher 742 of flow matcher 740 may perform an event matcher by requesting an LLM to determine whether the keywords, commands, and / or natural language description of the incoming or internal event match the (e.g., specified or generated) flow description of one of the interaction flows 780.For example, the flow description matcher 742 may construct a prompt using a prompt template containing a prompt to determine whether an event matches a flow description, populating placeholders with appropriate content (e.g., labeled instructions, a labeled example conversation or interaction history, a recorded history of the current conversation or interaction series(s), specified or generated flow description(s) of the interaction flow(s) 780, keywords and / or commands represented by the incoming or internal event, and / or other content), and may provide these constructed prompt(s) to an LLM (e.g., via an API request). As such, the LLM may return an indication of whether the event matches a flow description of the interaction flow(s) 780.In many cases, an LLM can provide a more nuanced or semantic understanding of the matching than traditional express or fuzzy matching algorithms.

[0180] Additionally or alternatively, the flow matcher 740 may include a flow instruction matcher 746 that requests an LLM to determine whether an incoming or internal event matches the instructions of an active flow of the interaction flow(s) 780. For example, the flow matcher 740 may trigger the execution of a designated flow (e.g., for handling unknown events) or an unprocessed event handler 744 that includes the flow instruction matcher 746 in response to the flow matcher 740 applying one or more matching techniques (e.g., using express matching, fuzzy matching, flow description matching, and / or others) and determining that there are no active flows that match the incoming or internal event.) In an example implementation, the raw event handler 744 includes the flow statement matcher 746 and a bot interaction flow generator 748, but this is only an example. In general, any number of customization techniques may be applied in any order, whether as an initial test, as part of the raw event handler 744, and / or otherwise.

[0181] In one embodiment, the flow instruction matcher 746 may request an LLM to determine whether an incoming or internal event and / or a representation of recent interaction history matches the specified content of an active flow of the interaction flow(s) 780. The flow instruction matcher 746 may accomplish this by inferring a user intent (e.g., by matching the incoming or internal event with the instructions of a flow waiting for a corresponding user intent). In one embodiment, the flow instruction matching function 746 may perform an event matching function by requesting an LLM to determine whether the keywords, commands, and / or natural language description of the incoming or internal event match the instructions (e.g., specified or generated) of one of the interaction flows 780.

[0182] For example, the flow instruction matcher 746 can create a prompt using a prompt template with placeholders, such as the following: Content: |- """ {{ general_instructions}} """ # This is how a conversation between a user and the bot might go: {{ sample_conversation}} # These are the most likely user intents: {{ sample flows and / or intents}} # This is the current conversation between the user and the bot: {{ history}} User action: {{ incoming or internal event}} User intent:

[0183] This example prompt template includes placeholders for general instructions, a conversation example (or series of interactions), some possible flows representing the user intents to be matched (or a corresponding list of possible user intents), a history of the current conversation (or series of interactions), keywords and / or commands represented by the incoming or internal event, and a prompt predicting the matching flow or user intent ("User Intent:"). The flow instruction matcher 746 may generate one or more prompts by filling placeholders with appropriate content (e.g.,specific instructions, a specific conversation pattern or interaction history, names and / or instructions of specific interaction flows 780 representing possible user intents, a corresponding list of the possible user intents, a recorded history of the current conversation or series(s) of interactions, keywords and / or commands represented by the incoming or internal event, and / or other content), and may provide these constructed prompt(s) to an LLM (e.g., via an API request). As such, the LLM may return an indication of whether the event matches one of the interaction flows 780 and / or a corresponding user intent, and if so, which one.

[0184] In some scenarios, there may not be a matching flow that defines a bot response to a particular user interaction, or the flow matcher 740 may not identify one. As such (e.g., in some embodiments where the flow matching function 740 determines that there are no active flows that match an incoming or internal event that represents a user interaction), the bot interaction flow generator 748 may request an LLM to generate a flow (e.g., at runtime). For example, in some embodiments, the flow matcher 740 (e.g.,the flow instruction matcher 746) may first use an LLM to attempt to match an unknown incoming or internal event with the name, instructions, and / or other representation of one or more designated flows that await corresponding target user intents (and define bot responses), and if the LLM determines that there are no matching flows or target user intents, the bot interaction flow generator 748 may request (the same or a different) LLM to predict the user intent represented by an unknown incoming or internal event, generate an agent response intent, and / or generate a response flow.For example, if the unknown event represents a user action, the bot interaction flow generator 748 may apply any number of prompts to instruct the LLM to classify the unknown user action into a user intent, generate an agent intent in response, and / or generate a flow that implements the agent intent in response.

[0185] For example, the Bot Interaction Flow Generator 748 can create an initial prompt using a prompt template with placeholders, such as the following: Content: |- """ {{ general_instructions}} """ # This is how a conversation between a user and the bot might proceed: {{ pattern_conversation}} # This is the current conversation between the user and the bot: {{ history}} Bot intent:

[0186] and create a second command prompt like this: Bot action:

[0187] These example templates include placeholders for general instructions, a conversation example (or series of interactions), a history of the current conversation (or series of interactions) including keywords and / or commands represented by the incoming or internal event, a prompt to predict the intent of the responding agent ("Bot Intent:"), and a prompt to predict the flow of an agent response ("Bot Action:"). The bot interaction flow generator 748 can generate one or more prompts by filling placeholders with appropriate content (e.g.,specific instructions, a specific conversation pattern or interaction history, a recorded history of the current conversation or series of interactions, keywords and / or commands represented by the incoming or internal event, and / or other content), and may provide these constructed prompts to an LLM (e.g., via an API request). As such, the LLM may generate and return an agent flow in response, which the bot interaction flow generator 748 may determine to be a match and provide to the flow execution component 750 for execution.

[0188] The following example can be used to illustrate some possible prompt contents. For example, the interpreter 715 can create a prompt (e.g., by populating a template prompt with a placeholder for certain general statements) that contains general statements, such as: Below you can see a conversation between Emma, ​​a helpful AI assistant (bot), and a user. The bot is designed to generate human-like actions based on the user actions it receives. The bot is chatty and provides many specific details. The bot enjoys chatting with the user about topics such as sports, music, leisure activities, NVIDIA, technology, food, weather, and animals. When the user remains silent, the bot encourages the user to try out the different storefronts by pointing out the different options they can select by voice or by clicking on the presented options. When the user asks a question, the bot responds with an appropriate answer. When the user gives an instruction, the bot follows that instruction. Appearance of the bot:

[0189] Emma is wearing a dark green skirt and a white blouse. On the blouse is a small card with the logo of the AI ​​company NVIDIA. Emma is wearing glasses, white earrings, and has medium-length brown hair. Her eyes are green-brown.

[0190] These are the available showcases: A) A simple number guessing game B) The example of multimodality shows how an interaction modeling language can handle multiple parallel actions C) Shows how the bot can communicate with the user via back channels D) A showcase to present different bot poses depending on the current interaction status E) Shows how the bot can be proactive by repeating a question if the user does not answer

[0191] This is important: - The bot uses the “Bot Gesture” action as often as possible. - The bot must not repeat itself if the user was silent User actions:

[0192] User said "text" Bot actions: Bot says "text" Bot informs "text" Bot asks "text" Bot expression "text" Bot answers "text" Bot clarifies "text" Bot suggests "text" Bot gesture "gesture"

[0193] In some embodiments, the interpreter 715 may create a prompt (e.g., by populating a prompt template with a placeholder for a particular conversation example or set of interactions) using a conversation example or set of interactions, such as:

[0194] In some embodiments, the interpreter 715 may construct a prompt (e.g., by populating a prompt template with a placeholder for a recorded history of the current conversation or series of interactions) by using a recorded history of the current conversation or series of interactions, such as: # This is the current conversation between the user and the bot: User intent: User was silent 8.0

[0195] The prompt content described here is intended as an example only, and other variations may be implemented within the scope of this disclosure.

[0196] Back to Fig.6: The sensory server 620 may translate detected input events 610 (e.g., a detected user input such as a detected gesture, voice command, or touch or click input; a detected feature or event associated with a user input such as a detected presence or absence of voice activity, a detected presence or absence of tapping, a detected transcribed speech, a detected change in tapping volume or speed; etc.) into standardized input events 630. In some embodiments, different sensory servers may process the detected input events 610 for different interaction modalities (e.g., one sensory server for translating detected gestures, one sensory server for translating detected voice commands, one sensory server for translating detected touch inputs, etc.).Thus, any sensor server 620 can act as an event transponder, acting as an intermediary between a corresponding input source and one or more downstream components (e.g., the event gateway 180 of . Fig. 1) functions by, for example, converting input events into a standardized format.

[0197] Taking a sensory server for GUI input events as an example, the sensory server can effectively translate GUI input events (e.g., "User clicked the 'Chai Latte' button, scrolled down, and clicked the 'Confirm' button") into standardized interaction-level events (e.g., "User selected the 'Chai Latte' option"). One possible example of a standardized interaction-level event is a confirmation status update event (e.g., an event indicating a detected state or a state change of a presented confirmation status, such as "confirmed," "cancelled," or "unknown"). For example, the sensory server can translate different types of GUI input into corresponding confirmation status update events, and the translation logic may differ depending on the type of interaction element being presented or interacted with.For example, a key press can be translated into a "confirmed" status update event, or if a visual form contains a single form field input, the sensor server can translate an "Enter" keyboard event into a "confirmed" status update event. Another possible standardized interaction-level event is a selection update event (e.g., to indicate a detected change to the user's current selection). For example, if a user selects a "Chai Latte" item from a list of multiple choices, the sensor server can translate a corresponding detected GUI input event (e.g., clicking or tapping a button or icon) into a standardized choice update event indicating a detected change to the user's current selection.Another example of a possible standardized interaction-level event is a form input update event, which indicates an update to a requested form input. These are just a few examples, and more are conceivable within the scope of the present disclosure. Other examples of standardized interaction-level GUI input events (e.g., for recognized GUI gestures such as swipes, pinches, or rotations for touchscreen devices), standardized interaction-level video input events (e.g., presentation of recognized visual gestures such as face recognition, pose detection, object detection, presence detection, or motion tracking), standardized interaction-level audio input events (e.g., presentation of recognized speech, recognized voice commands, keyword detection, other audio events, etc.), and / or others are contemplated within the scope of the present disclosure.

[0198] Fig. 9 shows an example of an action server 930 according to some embodiments of the present disclosure. The action server 930 may be connected to the action server(s) 170 of Fig. 1 and / or the action server 670 of Fig. 6. At a high level, the action server 930 may be subscribed or otherwise configured to receive the events that the action server 930 is responsible for executing (e.g., one interaction modality per action server) from an event bus 910 (e.g., provided to the event gateway 180 of Fig. 1) and executes. In the Fig. 9, the action server 900 includes one or more event workers 960 that push incoming events to corresponding modality services, and an event interface manager 940 that manages the event worker(s) 960.

[0199] For example, the event interface manager 940 may subscribe to a global event channel of the event bus 910, which transmits (e.g., standardized) events indicating when an interaction channel connecting an interaction manager to an end-user device has been acquired (e.g., PipelineAcquired) or released (e.g., PipelineReleased). As such, the event interface manager 940 may create a new event worker (e.g., event worker 960) in response to an event indicating that a new interaction channel has been acquired and / or delete an event worker in response to an event indicating that a corresponding interaction channel has been released. In some embodiments, the event interface manager 940 performs periodic health checks (e.g.,using any known technique such as interprocess communication) to ensure that the event worker(s) 960 are healthy and running. If the event interface manager 940 determines that one of the event workers 960 is unresponsive, the event interface manager 940 may restart the event worker.

[0200] Event worker(s) 960 may register with one or more stream-related event channels of event bus 910 (e.g., a stream-related event channel dedicated to a particular interaction modality for which action server 930 is responsible) and may forward incoming events to the various modality services that have registered for corresponding events. In some embodiments, event workers may run in separate (e.g., multi-process) processes (e.g., process 950) and manage incoming and outgoing events (e.g., using an Asycnio event loop).

[0201] Modality services (e.g. modality services A and B in Fig.9) can implement action-specific logic for each standardized action category and / or action event supported by an interaction modeling language and / or defined by an interaction categorization schema for a particular interaction modality. Thus, a particular modality service can be used to map actions of a corresponding interaction modality to a particular implementation within an interactive system. In an example implementation, all supported actions in a single interaction modality are handled by a single modality service. In some embodiments, a modality service can support multiple interaction modalities, but different actions for the same interaction modality are not handled by different modality services.For example, in some embodiments that include an interactive visual content modality, different actions in that interaction modality (e.g., VisualFormSceneAction, VisualSelectionSceneAction, VisualInformationSceneAction) are processed by the same GUI modality service.

[0202] Fig. Figure 10 shows an example event flow through an example action server 1000 in accordance with some embodiments of the present disclosure. In this example, stream XY (e.g., associated with one of the stream-related event channels of event bus 910 of Fig. 9) contain various events (e.g., events 1-7), and an event worker 1010 (e.g., which corresponds to the event worker(s) 960 of Fig.9) can subscribe to stream XY and provide a subscribed modality service (e.g., modality service A) with a corresponding event view (e.g., event view A, event view) populated with a subset of the events from stream XY that the modality is subscribed to receive. Thus, the modality service can perform instructed actions represented by its subscribed events, apply a corresponding modality policy to maintain a corresponding action stack, and invoke corresponding action handlers to execute the actions. As such, the action handlers can execute the actions, generate internal events (e.g., to indicate timeouts) and place them in a corresponding event view (so that the modality service can take appropriate action and maintain the action stack and lifecycle) and / or (e.g.,Generate standardized) interaction modality events (e.g., to indicate that certain actions have been started, completed, or updated) and place them in the data stream XY.

[0203] In some embodiments, each modality service registers with an event worker (e.g., event worker 1010) with a list of (e.g., types of) events of interest (e.g., handled by that modality service). Thus, event worker 1010 may provide an event view (e.g., event view A) that represents a subset of all events in that service's data stream. The modality service may process the events within a corresponding event view sequentially. In some embodiments, where action server 1000 includes multiple modality services, different modality services may process events in parallel (e.g., using an asynchronous event loop).

[0204] In some embodiments, each modality service implements a particular modality policy (e.g., the one described in Fig. 4). In an example of a modality policy that allows parallel actions in a particular modality, a corresponding modality service may trigger, track, and / or otherwise manage parallel actions, and any number of actions may be executed concurrently. The modality service may assign a common action identifier (e.g., action_uid) that uniquely identifies a particular instance of an action and may track the lifecycle of that instance of the action in response to action events generated by corresponding action handlers that reference the same action identifier.

[0205] In an example of a modality policy that overrides overlapping actions in a given modality, a corresponding modality service may maintain a stack of actions, and the modality service may suspend or hide a currently executing action in response to a subsequently instructed action. Once an action is finished, and a corresponding (e.g., internal) action event representing that event is associated with the modality service, the modality service may trigger the topmost of the remaining actions in the stack to resume or make it visible again.For example, a modality service for animations may first dispatch a StartGestureBotAction(gesture="talk") event, and if it subsequently receives a StartGestureBotAction(gesture="point down") event before the speaking gesture (animation) is complete, the modality service may pause the speaking gesture, dispatch the point down gesture, and resume the speaking gesture when the point down gesture is complete.

[0206] In some embodiments, a modality service may synchronize changes in action state with certain conditions (e.g., wait to start an action until a previous action of the same modality has completed, synchronize the completion of two different actions in different modalities, synchronize the start of one action with the end of another action, etc.). For illustration, Fig.11 illustrates an example of an action lifecycle 1100. For example, a modality service may receive a StartAction event 1110 indicating that an action should begin on a corresponding interaction modality handled by that modality service. Fig. 1, the modality service may determine whether the modality is available in decision block 1120. For example, the modality service may implement a modality policy that waits for ongoing actions on that modality to complete before starting a new action on that modality. The modality service may track the lifecycle of initiated actions on that modality and thus determine that there is another pending action that has begun and not yet completed. Accordingly, the modality service may wait until it receives an event (e.g., the one in Fig.11) indicating that the action is complete, after which the modality service may proceed to decision block 1130. In decision block 1130, the modality service may determine whether a particular start condition is met (e.g., an instruction to synchronize the start of the new action with the start or completion of another action). As such, the modality service may wait until the intended start condition occurs (e.g., indicated by the synchronization event in Fig. 11) and the interaction modality remains free before initiating the action, and generate an event in block 1140 indicating that the action has begun.

[0207] In block 1150, the modality service may determine whether a certain termination condition is met before completing the action (e.g., canceling a waving gesture when the bot has finished saying goodbye). Thus, the modality service may terminate the action when the designated termination condition occurs, or the action may last for a specified duration and reach its natural conclusion. Once the action is completed or otherwise stopped, the modality service may generate an event indicating that the action has stopped in block 1160. This allows the modality service to manage and track the lifecycle of the action and generate events that represent changes in the state of the action during its lifecycle.

[0208] Back to Fig.10: An action handler (e.g., Action1Processor, Action2Processor) may be responsible for executing a single category of supported (e.g., standardized) actions and implement a corresponding action state machine. For example, an action handler may receive events representing instructions to change the action state (e.g., start, stop, change), and it may receive internal events from the modality service or itself (e.g., API callback calls, timeouts, etc.). In some embodiments, action handlers may directly inject events (e.g., standardized interaction modeling) into stream XY (e.g., a change in action state such as "started," "finished," or "updated").

[0209] The following section describes some example implementations of modality services, namely a GUI service that handles interactive visual content and an animation service that handles bot gesture actions.

[0210] Example GUI service. In some embodiments, a GUI service (e.g., assigned to modality service B in Fig.9) interactive visual content actions (e.g., VisualInformationSceneAction, VisualSelectionSceneAction, VisualFormSceneAction) and corresponding events. In an example implementation, the GUI service translates standardized events representing instructed interactive visual content actions (e.g., instructed GUI updates) into calls to a user interface server API, applies a modality policy that overrides active actions with subsequently instructed ones, and maintains a corresponding stack of visual information scene actions (e.g., in response to receiving an event instructing a new interactive visual content action when there is at least one ongoing interactive visual content action). Thus, the GUI service can implement GUI updates that represent interactive visual content (e.g.,synchronize visual information, a choice the user is prompted to make, or a field or form the user is asked to fill out) with the current state of an interaction with a conversational AI.

[0211] In some embodiments, the GUI service may operate in coordination with a user interface server (e.g., on the same physical device, on an attached or networked physical device, etc.), such as the user interface server 130 of Fig. 1. In general, the user interface server may be responsible for managing and providing user interfaces to a client device (e.g., client device 101 of Fig.1) be responsible for providing front-end components (e.g., HTML files for structuring content, cascading style sheets for styling, JavaScript files for interactivity) that form the user interface of a web application. The user interface server may provide static elements such as images, fonts, and other resources for the user interface and / or use any known user interface provisioning technique. The user interface server may act as an intermediary between the client device and the GUI service, converting GUI input into standardized GUI input events and standardized GUI output events into corresponding GUI outputs.

[0212] The GUI service can manage an action state machine and / or an action stack for all interactive visual content actions. In a sample implementation, the GUI service contains an action handler for each supported event for each supported interactive visual content action. Fig. 12A-12C illustrate some example action handlers for some example events for interactive visual content, in accordance with some embodiments of the present disclosure. More specifically, Fig. 12A some example action processors for some example action events of a visual information scene, Fig. 12B illustrates some example action handlers for some example action events of a visual selection, and Fig. Figure 12C illustrates some example action handlers for some example action events of a visual form.

[0213] For example, an interactive visual content event (e.g., generated by an interaction manager such as the interaction manager 190 of Fig. 1 or the interaction manager 700 of Fig. 7) Instruct the visualization of various types of visual information (e.g., in a 2D or 3D interface). In some embodiments, an interactive visual content event (e.g., payload) includes fields that specify or encode values ​​representing a supported action type categorizing a directed action (e.g., VisualInformationSceneAction, VisualSelectionSceneAction, VisualFormSceneAction), an action status (e.g., "initiated," "scheduled," "starting," "running," "paused," "resuming," "stopping," or "finished"), a representation of the directed visual content, and / or other properties or information. The type of visual content indicated by an event may depend on the type of action.

[0214] For example, (e.g., the payload) of an event (e.g., a start event) for a visual information scene action may contain fields that specify corresponding values, such as a particular title, a particular summary of the information to be presented, particular content to be presented (e.g., a list of information blocks to be shown to the user, where each block may contain particular text, a particular image (e.g., a description or an identifier such as a Uniform Resource Locator), or both), one or more particular assistance prompts that assist or guide the user in making a selection, and / or others. Thus, the action handler for a corresponding event (e.g., a start event) for a visual information scene action may convert the event into a (e.g.,Translate a JSON representation of a modular GUI configuration that specifies content blocks such as a hint carousel block for one or more specified support blocks, a header block for a specified title, image and / or text blocks for the specified content, a button(s) (e.g., continue, cancel), and / or other elements. Accordingly, the action handler can use these content blocks to generate a custom page by populating a visual layout (e.g., a specified template or a visual frame layout with appropriate placeholders) for a GUI overlay layout (e.g., an HTML layout), and it can call the user interface server's endpoint with the custom page to cause the user interface server to render the custom page.

[0215] In some embodiments, an event (e.g., a launch event) for a visual selection action (e.g., the payload) may include fields that specify corresponding values, such as a specified prompt (e.g., describing a choice to be presented to the user), a specified image (e.g., a description or identifier such as a Unified Source Placer of an image to be presented with the selection), one or more specified assistance prompts that assist or guide the user in making the selection, one or more specified options from which the user can select (e.g., text, image, and / or other content for each option), a specified selection type (e.g., configuring the type of selection the user can make, such as a picker, search bar, etc.), an indication of whether multiple selections are allowed, and / or other.Thus, for a corresponding event (e.g., a start event) for a visual selection action, the action handler can translate the event into a (e.g., JSON) representation of a modular GUI configuration that specifies content blocks, such as a hint carousel block for one or more specified support blocks, a header block for a specified title, an image block for a specified image, a selectable options grid block for specified options, a (e.g., cancel) button(s), and / or other elements. Accordingly, the action handler can use these content blocks to generate a custom page by providing a visual layout (e.g., a specific template or a visual frame layout with appropriate placeholders) for a GUI overlay layout (e.g.,an HTML layout), and it can call the endpoint of the user interface server with the custom page to cause the user interface server to render the custom page.

[0216] The Fig. 13A-13F illustrate some example interactions with a visual selection, in accordance with some embodiments of the present disclosure. For example, Fig. 13A and Fig. 13D, for example, show the representation of a visual selection between four labeled images with an interactive avatar asking which image the user likes best. Fig. 13B shows a scenario in which the user indicates the third image with a touch input, and Fig.Figure 13E shows the same selection with a verbal input. In these scenarios, the verbal input can be detected, forwarded to an appropriate sensory server, and translated into a corresponding standardized event, which is forwarded to an interaction manager and used to generate an event that instructs a corresponding GUI update, an event that instructs a verbal bot response, and / or an event that instructs a corresponding agent gesture. As such, the events can be forwarded to appropriate action servers and executed. Fig. 13C and Fig. 13F show an example of a bot response (e.g., visual highlighting of the selected choice and verbal confirmation).

[0217] In some embodiments, the payload of an event (e.g., a start event) for a visual form action may include fields that specify corresponding values, such as a particular prompt (e.g., a description of the user's desired input), a particular image (e.g., a description or identifier such as a Uniform Resource Locator of an image to be presented with the selection), one or more specified assistance prompts that assist or guide the user in making the selection, one or more specified user inputs (e.g., where each specified user input may include a specified input type such as numeric or date, a specified description such as "personal email address" or "place of birth," etc.), and / or other. Thus, for a corresponding event (e.g., a start event) for a visual form action, the action handler may convert the event to a (e.g.,B. JSON) representation of a modular GUI configuration that defines content blocks represented by (e.g., corresponding fields of the event), such as a hint carousel block for one or more specified support blocks, a header block for a specified prompt, an image block for a specified image, a list of input blocks representing corresponding form fields for specified inputs, a button (e.g., Cancel), and / or other elements. Accordingly, the action handler can use these content blocks to generate a custom layout or page by providing a visual layout (e.g., a specific template or a visual shell layout with placeholders for corresponding content blocks) for a GUI overlay page (e.g.,an HTML page) and can call the user interface server's endpoint with the custom layout or page to cause the user interface server to present the custom layout or page.

[0218] In some embodiments, where an event (e.g., a launch event) for an interactive visual content action specifies an image with a natural language description (e.g., "image of summer mountains"), a corresponding action handler for the event may trigger or perform an image search for a matching image. For example, an action handler may extract the natural language description of the desired image, interface with a suitable image search tool (e.g., via a suitable API), and send the natural language description of the desired image to the search tool. In some embodiments, the search tool returns an identifier, e.g., a Uniform Resource Locator for a matching image, and the action handler may insert the identifier into a corresponding block on a custom page.The action control can provide the custom page to the user interface server (which can retrieve the specified image using the inserted identifier) ​​for presentation.

[0219] The Fig. 14A-14L show example layouts of visual elements for interactive visual content in accordance with some embodiments of the present disclosure. For example, Fig. 14A shows an example of a GUI overlay 1420 displayed over a scene with an interactive avatar. The Fig. 14B-14L show some example layouts of blocks of visual elements that could be used as corresponding GUI overlays. These are intended merely as examples, and other layouts may be implemented within the scope of the present disclosure.

[0220] Example of an animation service. In some embodiments, an animation service (e.g., associated with modality service A in Fig. 9) process bot gesture actions (e.g., GestureBotAction) and corresponding events. In an example implementation, the animation service applies a modality policy that overrides active actions with subsequently instructed actions and creates a corresponding stack of actions in response to incoming StartGestureBotAction events when there are one or more ongoing GestureBotAction actions. The animation service may maintain an action state machine and an action stack for all GestureBotActions, connect to an animation graph that implements a state machine with animation states and transitions between animations, and instruct the animation graph to set a corresponding state variable.

[0221] In a sample implementation, the animation service contains an action handler for each supported bot gesture action event. Fig. 12D shows some example action handlers for some example GestureBotAction events according to some embodiments of the present disclosure.

[0222] For example, a bot gesture action event (e.g., generated by an interaction manager such as the interaction manager 190 of Fig. 1 or the interaction manager 700 of Fig.7 is generated) instruct a particular animation (e.g., in a 2D or 3D interface). In some embodiments, a bot gesture action event (e.g., payload) may include fields that indicate or encode values ​​representing a supported action type categorizing a commanded action (e.g., GestureBotAction), an action status (e.g., "initiated," "scheduled," "start," "running," "paused," "resume," "stop," or "completed"), a representation of a commanded bot gesture, and / or other properties or information. For example, (e.g., the payload) of an event (e.g., a start event) for a bot gesture action may include a field (or fields) indicating (e.g., a natural language description or other identifier) ​​a bot gesture. Depending on the implementation, one or more categories or types of actions (e.g.,Bot gestures (e.g., bot expression, pose, gesture, or other interaction or movement) may be standardized for corresponding bot functions, and a corresponding action event may specify the desired action. Taking a bot gesture specified as a natural language description for a bot gesture action category as an example, for a corresponding (e.g., start) event for the bot gesture action category, the action processor may extract the natural language description from the event, generate or access a sentence embedding for the natural language description of the bot gesture, use it to perform a similarity search on sentence embeddings for descriptions of the available animations, and select an animation using a similarity measure (e.g., nearest neighbor, within a threshold).If the best match is above a certain threshold, the action processor may trigger the animation graph to play a corresponding animation for the user. In some embodiments, for a corresponding event (e.g., launch) for the bot gesture action category, the action processor may extract the natural language description from the event and use any known generative technique to generate an animation from the natural language description (e.g., text-to-motion models, text-to-animation techniques, any other suitable animation technique).

[0223] Example event flows. The following explains some possible event flows in example implementations. For example, the following table shows a set of events that can be generated and distributed in an implementation where a bot has a dialogue with a user: history Next action UtteranceUserActionEnded final_transcript="Hello!" StartUtteranceBotAction script="Hello everyone!" UtteranceBotActionStarted between_script="Hello everyone!" UtteranceBotActionEnded final_script="Hello everyone!"

[0224] In this example, the event in the first row represents a detected completion of a user utterance ("Hello!"), which triggers an event that instructs the bot to begin a response utterance ("Hello!"). The event in the second row indicates that the bot started the utterance, and the event in the third row indicates that the bot finished the utterance.

[0225] The following table shows a set of events that can be generated and distributed in an implementation where a bot interacts with a user via gestures, emotions, and a display: history Further ... UtteranceBotActionEnded final_script="Which option?" StartVisualSelectionScenesAction content=[...] StartTimerBotAction Duration= 2sec Visual SelectionScenesActionStarted TimerBotActionStarted Duration-- 2sec StartGestureBotAction gesture="point at the display" GestureBotActionStarted GestureBotActionCompleted TimerBotActionEnded Duration= 2sec StartUtteranceBotAction script="Do you need more time?" UtteranceBotActionStarted script="Do you need more time?" GestureUserActionCompleted gesture="nod" StartGestureBotAction move="lean forward" GestureBotActionStarted UtteranceBotActionEnded final_script="Do you need more time?" GeseBotActionEnded ExpressionUserActionStarted expression="happy"

[0226] In this example, the event in the first row indicates that the bot has finished asking "Which option?", which triggers an event that instructs the graphical user interface to present a visual choice. The event in the second row instructs a two-second timer to start. The event in the third row indicates that the visual choice was presented, and the event in the fourth row indicates that the timer started, which triggers an event that instructs the bot to point to the visual choice display. The event in the fifth row indicates that the pointing gesture began, and the event in the sixth row indicates that the pointing gesture ended. The event in the seventh row indicates that the two-second timer expired, which triggers a bot utterance ("Do you need more time?"). The event in the eighth row indicates that the bot utterance began.The event in the ninth row indicates that the recognition of a recognized user gesture (nod) is complete, triggering a corresponding agent gesture (lean forward). The event in the tenth row indicates that the bot gesture has begun. The event in the eleventh row indicates that the bot utterance ("Do you need more time?") has ended, the event in the twelfth row indicates that the bot gesture (lean forward) has ended, and the event in the last row indicates that the beginning of a recognized user expression (happy) has been detected.

[0227] In various scenarios, it may be advantageous to instruct an interactive system or one of its components (e.g., a sensory server that controls input processing, an action server that implements bot actions) to perform a specific action, or otherwise signal the expectation, in anticipation of an event that an interaction manager (e.g., the interpreter) expects to occur next from the user or the system. Some possible expectation actions in example implementations, as well as other example functions, are explained below.

[0228] Fig.For example, Figure 15 shows an example event flow 1500 for a user utterance action in an implementation in which a user 1518 speaks with an interactive avatar 1504. In this example, the interactive avatar 1504 is implemented with a user interface 1516 (e.g., a microphone and audio interface), a voice activity detector 1514, an automatic speech recognition system 1512, and an action server 1510 responsible for processing events for user utterance actions (e.g., UtteranceUserAction 1508). In this example, the action server 1510 functions as both a sensory server and an action server, converting sensory input into standardized events and executing standardized events that instruct specific actions. An interaction manager 1506 may handle decision making for the interactive avatar 1504.Although the interaction manager 1506 and the interactive avatar 1504 are depicted as separate components, the interaction manager 1506 could be considered part of the interactive avatar 1504.

[0229] In an example flow, user 1518 begins speaking in step 1520. In step 1522, voice activity detector 1514 captures the speech and sends a speech stream to automatic speech recognition system 1512. In step 1524, voice activity detector 1514 notifies action server 1510 that voice activity has been detected, and in step 1526, automatic speech recognition system 1512 transmits the transcribed speech to action server 1510. In this sense, action server 1510 generates a standardized event indicating that a recognized user utterance has begun (e.g., including the transcribed speech) in step 1528 and sends the event (e.g., UtteranceUserActionStarted) to event gateway 1502, which interaction manager 1506 picks up in step 1530.

[0230] The following steps 1532-1546 may be executed in a loop. In step 1532, the user has said a few words, and in step 1534, the automatic speech recognition system 1512 sends a partial transcript to the action server 1510. In step 1536, the action server 1510 generates a standardized event indicating a detected update to a recognized user utterance (e.g., including the transcribed speech) and sends the event (e.g., UtteranceUserActionTranscriptUpdated) to the event gateway 1502, which the interaction manager 1506 picks up in step 1538. In step 1540, the user speaks louder, and in step 1542, the voice activity detector 1514 detects the increase in volume and notifies the action server 1510 of the detected change in volume.In step 1544, the action server 1510 generates a standardized event indicating a detected update to a detected intensity of the user utterance (e.g., including a detected intensity or volume) and sends the event (e.g., UtteranceUserActionIntensityUpdated) to the event gateway 1502, which the interaction manager 1506 picks up in step 1546.

[0231] In some embodiments, in step 1548, the interaction manager 1506 generates a standardized event indicating the expectation that the user is about to stop speaking and / or instructing the interactive avatar 1504 to take a preparatory action, and the interaction manager 1506 sends the event (e.g., StopUtteranceUserAction) to the event gateway 1502, which the action server 1510 picks up in step 1550. In response, in step 1552, the action server 1510 instructs the voice activity detector 1514 to decrease the audio hold time (e.g., the amount of time a detected speech signal is maintained before being considered inactive or muted).

[0232] In step 1554, the user stops speaking. In step 1556, the voice activity detector 1514 detects the voice inactivity and stops the voice stream to the automatic speech recognition system 1512, and in step 1558, the automatic speech recognition system 1512 stops streaming a transcript to the action server 1510. In step 1560, the hold time expires, and in step 1562, the voice activity detector 1514 notifies the action server 1510 that voice inactivity has been detected. Therefore, in step 1564, the action server 1510 generates a standardized event indicating a detected completion of a recognized user utterance and sends the event (e.g., UtteranceUserActionCompleted) to the event gateway 1502, which the interaction manager 1506 picks up in step 1566.

[0233] Fig.16 shows an example event flow 1600 for a user utterance action in an implementation in which a user 1618 speaks with a chatbot 1604. In this example, the chatbot 1604 is implemented with a user interface 1616 (e.g., a hardware or software keyboard and driver), a timer 1612, and an action server 1610 responsible for processing events for user utterance actions (e.g., UtteranceUserAction 1608). In this example, the action server 1610 acts as both a sensory server and an action server, converting sensory input (e.g., recognized text, typing speed) into standardized events and executing standardized events that instruct specific actions. An interaction manager 1606 may handle decision making for the chatbot 1604.Although the interaction manager 1606 and the chatbot 1604 are depicted as separate components, the interaction manager 1606 could be considered part of the chatbot 1604.

[0234] In an example flow, user 1618 begins typing in step 1620. In step 1622, user interface 1616 notifies action server 1610 that typing has begun, and in step 1624, action server 1610 generates a standardized event indicating that a recognized user utterance has begun and sends the event (e.g., UtteranceUserActionStarted) to event gateway 1602, which interaction manager 1606 picks up in step 1626.

[0235] The following steps 1628-1640 may be executed in a loop. In step 1628, user interface 1616 sends the typed text to action server 1610, and in step 1630, action server 1610 generates a standardized event indicating a detected update to a detected user utterance (e.g., including the typed text) and sends the event (e.g., UtteranceUserActionTranscriptUpdated) to event gateway 1602, which interaction manager 1606 picks up in step 1634. In step 1632, the user begins typing faster, and in step 1636, user interface 1616 detects the increase in typing speed and notifies action server 1610 of the detected speed change. In step 1638, the action server 1610 generates a standardized event indicating a detected update of a detected intensity of the user utterance (e.g.,including a detected intensity or typing speed), and sends the event (e.g., UtteranceUserActionIntensityUpdated) to the event gateway 1602, which the interaction manager 1606 receives in step 1640.

[0236] In some embodiments, in step 1642, the interaction manager 1606 generates a standardized event indicating that the user is about to stop typing and / or instructing the chatbot 1604 to take a preparatory action, and the interaction manager 1606 sends the event (e.g., StopUtteranceUserAction) to the event gateway 1602, which the action server 1610 picks up in step 1644. In response, in step 1646, the action server 1610 decreases the post-keypress timeout (e.g., the amount of time during which detected inactivity or delay in typing is interpreted as completion of the utterance).

[0237] In step 1648, the user stops typing. In step 1650, user interface 1616 sends action server 1610 a notification that typing has ended, and in step 1652, action server 1610 instructs timer 1612 to start. In step 1654, timer 1612 notifies action server 1610 that the timer has expired, and action server 1610 informs user interface 1616 to lock the input field for further input. In step 1658, the user interface 1616 sends the completed text input to the action server 1610. In this sense, in step 1660, the action server 1610 generates a standardized event indicating the detected completion of a recognized user utterance (e.g., including the completed text input) and sends the event (e.g., UtteranceUserActionCompleted) to the event gateway 1602, which the interaction manager 1606 picks up in step 1662.

[0238] Fig.17 shows an example event flow 1700 for a bot expectation action in an implementation in which a user 1718 speaks with an interactive avatar 1704. In this example, the interactive avatar 1704 is implemented with a client device 1716 (e.g., with a microphone and an audio interface), an automatic speech recognition system 1714, and an action server 1712 responsible for processing events for user utterance actions (e.g., UtteranceUserAction 1710) and bot expectation actions for user utterance actions (e.g., BotExecutionAction 1708). In this example, the action server 1712 functions as both a sensory server and an action server, converting sensory input into standardized events and executing standardized events that instruct specific actions. An interaction manager 1706 can take over the decision making for the interactive avatar 1704.Although the interaction manager 1706 and the interactive avatar 1704 are depicted as separate components, the interaction manager 1706 could be considered part of the interactive avatar 1704.

[0239] In step 1720, the interaction manager 1706 creates a standardized event indicating that the start of a user utterance is expected shortly and representing an instruction to perform a preparatory action in anticipation of the user utterance, and sends the event (e.g., StartBotExpectationAction(UtteranceUserActionCompleted)) to the event gateway 1702, which the action server 1712 picks up in step 1722. Note that in this example, the argument of the keyword used to identify the expectation is an expected target event (e.g., the completion of the user utterance), which may trigger a corresponding stop action indicating that the expectation of the interaction manager 1706 has been met or is no longer relevant, which in turn may trigger the inversion of the preparatory action, but this syntax is intended only as an example and need not be used.In response, in step 1724, the action server 1712 informs the client device 1716 to disable its audio output, in step 1726, it informs the client device 1716 to enable its microphone, and in step 1728, it informs the automatic speech recognition system 1714 to enable automatic speech recognition. In step 1730, the action server 1712 generates a standardized event confirming that the bot await action has begun and / or indicating that the preparatory action has been initiated and sends the event (e.g., BotAwaitingActionStarted(UtteranceUserActionEnded)) to the event gateway 1702, which the interaction manager 1706 picks up in step 1732.

[0240] In some embodiments, when user 1718 begins speaking, speech is detected (not shown), and in step 1734, action server 1712 generates a standardized event indicating that a detected user utterance has begun and sends the event (e.g., UtteranceUserActionStarted) to event gateway 1702, which interaction manager 1706 picks up in step 1736. Once user 1718 stops speaking, the end of the utterance is detected (not shown), action server 1712 generates a standardized event indicating a detected completion of a detected user utterance and sends the event (e.g., UtteranceUserActionEnded) to event gateway 1702 (not shown), which interaction manager 1706 picks up in step 1738.In this example, the interaction manager 1706 is programmed to stop the bot awaiting action in response to receiving an event indicating the detected completion of a detected user utterance. Therefore, in step 1740, the interaction manager 1706 generates a standardized event indicating that the awaiting user utterance is complete and instructing the reversal of the preparatory action, and sends the event (e.g., StopBotAwaitingAction(UtteranceUserActionCompleted)) to the event gateway 1702, which the action server 1712 receives in step 1742. The action server 1712 then instructs the automatic speech recognition system 1714 to terminate automatic speech recognition in step 1744 and instructs the client device 1716 to disable its microphone in step 1746.In step 1748, the action server 1712 generates a standardized event confirming that the bot await action has ended and / or indicating that the preparatory action has been undone and sends the event (e.g., BotAwaitActionEnded(UtteranceUserActionEnded)) to the event gateway 1702, which the action server 1712 picks up in step 1750.

[0241] Flowcharts. With reference to the Fig.18-27, each block of the methods 1800-2700 described herein comprises a computational process that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. The method 1800-2700 may also be embodied in the form of computer-usable instructions stored on computer storage media. The method 1800-2700 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in for another product, to name a few. Furthermore, the methods 1800-2700 are described with reference to an example system, such as the interactive system 100 of Fig.1. However, these methods may additionally or alternatively be performed by any system or combination of systems, including, but not limited to, the systems described herein.

[0242] Fig. 18 is a flowchart illustrating a method 1800 for generating a representation of an addressable agent action categorized using an interaction categorization scheme, in accordance with some embodiments of the present disclosure. The method 1800 includes, at block B1802, receiving, by an interpreter of an interactive agent platform associated with an interactive agent, one or more representations of one or more recognized user actions categorized using an interaction categorization scheme. With respect to the interactive system 100 of Fig.1, for example, a representation of user input, such as a gesture captured by vision microservice 110, a voice command captured by speech recognition microservice 120, or a touch or click input captured by UI server 130, may be forwarded to a corresponding one of the sensory servers 160 responsible for a corresponding interaction channel. The sensory server(s) 160 may translate this user input into a standardized representation of a corresponding event defined by an interaction categorization schema and forward the event to an event gateway 180. The interaction manager 190 may implement an interpreter that is subscribed to or otherwise configured to ingest or receive these events from the event gateway 180.

[0243] The method 1800 includes, at block B1804, generating one or more representations of one or more responsive agent actions categorized using the interaction categorization scheme, at least based on the interpreter executing one or more command lines of one or more interaction flows written in an interaction modeling language that instruct one or more agent actions to be generated in response to the one or more recognized user actions. With respect to the interactive system 100 of Fig.1, for example, an interpreter implemented by interaction manager 190 may support an interaction modeling language, and code implementing decision logic for an interactive agent may be written in the interaction modeling language, loaded into, or otherwise retrieved from interaction manager 190, and executed by interaction manager 190. As such, interaction manager 190 may process events from event gateway 180 (e.g., using an event-driven state machine), determine which interactions should be performed, and generate commands as corresponding events in a standardized representation and forward them to event gateway 180.

[0244] Fig.19 is a flowchart illustrating a method 1900 for generating a representation of an addressable agent action based at least on the execution of one or more interaction sequences, in accordance with some embodiments of the present disclosure. The method 1900 includes, at block B1902, receiving one or more representations of one or more recognized user actions by an interpreter of an interactive agent platform that supports the concurrent execution of agent actions in different interaction modalities. With respect to the interactive system 100 of Fig.1, the interaction manager 190 may, for example, implement an interpreter that is subscribed to or otherwise configured to ingest or receive events representing detected user actions from the event gateway 180. The interaction manager 190 may implement decision logic for an interactive agent written in an interaction modeling language, and the interaction modeling API and / or language used by the interaction manager 190 may define mutually exclusive interaction modalities such that events instructing actions in different interaction modalities may be executed independently (e.g., concurrently) by corresponding action servers 170 designated for corresponding interaction modalities.

[0245] The method 1900 includes, at block B1904, generating one or more representations of one or more addressable agent actions, at least based on the interpreter executing one or more command lines of one or more interaction flows in response to the one or more detected user actions. With respect to the interactive system 100 of Fig.1, for example, code that implements the decision logic for an interactive agent and defines one or more interaction flows may be written in the interaction modeling language, loaded into an interpreter implemented by the interaction manager 190, or otherwise accessed and executed by the interpreter. Thus, the interaction manager 190 may process events (e.g., detected user actions) from the event gateway 180 (e.g., using an event-driven state machine), determine which interactions should be performed, and generate commands as corresponding events in a standardized representation and forward them to the event gateway 180.

[0246] Fig.20 is a flowchart illustrating a method 2000 for triggering an interactive avatar to provide backchannel feedback, in accordance with some embodiments of the present disclosure. The method 2000 includes, at block B2002, receiving, by an interpreter associated with an interactive avatar that supports non-sequential human-machine interactions, one or more representations of one or more detected initiations of one or more user actions, and includes, at block B2004, triggering the interactive avatar, at least based on the interpreter executing one or more command lines of one or more interaction flows in response to the one or more detected initiations, to provide backchannel feedback during the one or more user actions. For example, the interactive system 100 of Fig.1 support non-sequential interactions by utilizing various features described herein, such as an event-driven architecture, an interpreter that supports various keywords, and / or the decoupling of sensory processing, interaction decision, and action execution. For example, to support the execution of an addressable agent action prior to the completion of a triggering user action (e.g., an utterance), the sensory server(s) 160 may generate an event representing a detected initiation of the user action and provide the event (via the event gateway 180) to the interaction manager 190, and the interaction manager 190 may check the event for a matching active (e.g., suspended) flow waiting for such an event.Because the sensor server(s) 160, the interaction manager 190, and the action server(s) 170 can operate independently of each other, the sensor server(s) 160 can continue to process the user input while the interaction manager 190 generates an event representing a response action and causes the action server(s) 170 to perform the response action (e.g., backchanneling feedback).

[0247] Fig.21 is a flowchart illustrating a method 2100 for generating an interaction modeling event instructing an interactive agent to perform a responsive agent or scene action, in accordance with some embodiments of the present disclosure. The method 2100 includes, at block B2102, receiving one or more first interaction modeling events via one or more event gateways and by an interaction manager associated with an interactive agent, wherein these events represent at least one of the following actions: one or more recognized user actions, one or more instructed agent actions, or one or more instructed scene actions. With respect to the interactive system 100 of Fig.1, the sensor server(s) 160 can, for example, translate recognized user inputs into a standardized representation of a corresponding event and place the event on an event gateway 180. In addition, the interaction manager 640 can, with respect to Fig. 6 generate internal events 660 representing internal state changes (e.g., expiration state changes) or instructed bot actions, and / or the action server 670 may generate events 665 representing the confirmation of action state changes. As such, the interaction manager 190 of Fig. 1 and / or the interaction manager 640 of Fig. 6 be subscribed or otherwise configured to receive the events from the event gateway 180.

[0248] The method 2100 includes, at block B2104, generating one or more second interaction modeling events that command the interactive agent to perform at least one of the following actions: one or more addressable agent actions or one or more addressable scene actions, based at least on the processing of the one or more first interaction modeling events by the interaction manager using an event-driven state machine. With respect to the event-driven interactive system 600 of Fig. 6, the interaction manager 640 (which is assigned to the interaction manager 190 of Fig. 1 and / or Fig.2), for example, be responsible for deciding which actions the interactive system 600 should perform in response to user actions or other events (e.g., the standardized input events 630, the internal events 660, the events 665 representing confirmation of action state changes). The interaction manager 640 may interact with the rest of the interactive system 600 via an event-driven mechanism. As such, the interaction manager 640 may evaluate various types of events (e.g., the standardized input events 630, the internal events 660, the events 665 representing confirmation of action state changes), determine which actions to perform, and generate corresponding instructed bot action events 650 or event instruction updates for another aspect of the scene (e.g., interactive visual content actions).

[0249] Fig. 22 is a flowchart illustrating a method 2200 for triggering one or more responsive agent or scene actions specified by one or more matching interaction flows, in accordance with some embodiments of the present disclosure. The method 2200 includes, at block B2202, tracing one or more interrupted interaction flows representing one or more human-machine interactions. With respect to the event-driven interactive system 600 of Fig. 6, the interaction manager 640 can, for example, support and track multiple active processes (e.g., interrupted by corresponding event synchronizations).

[0250] The method 2200 includes, at block B2204, checking one or more incoming interaction events for one or more matching interaction flows of the one or more interrupted interaction flows. With respect to the event-driven interactive system 600 of Fig. 6, the interaction manager 640 may, for example, use an event-driven state machine to wait for events that match an event matcher of an active flow. For example, the flow matcher 740 of Fig. 7 Evaluate incoming events to determine if they match an event matcher of an active flow, where incoming events are processed sequentially (e.g., from the internal event queue 790, from another queue, or an event gateway, such as the event gateway 180 of Fig.1) and for each event, the event matcher specified by each active flow is tested for a match to the event.

[0251] The method 2200 includes, at block B2206, triggering one or more responsive agent or scene actions specified by the one or more matching interaction flows in response to identifying the one or more matching interaction flows. With respect to the event-driven interactive system 600 of Fig. 6, the interaction manager 640 may, for example, trigger corresponding events and actions specified in flows corresponding to the event being tested. For example, the flow aligner 740 of Fig.7 instruct the flow execution component 750 to forward matching (e.g., non-conflicting) flows, and the forwarding flows may instruct the flow execution component 750 to generate outgoing events that instruct an action.

[0252] Fig. 23 is a flowchart illustrating a method 2300 for generating an responsive agent or scene action based at least on the prompting of one or more large language models, in accordance with some embodiments of the present disclosure. The method 2300 includes, at block B2302, receiving one or more representations of one or more recognized user actions by an interpreter of an interactive agent platform. With respect to the interactive system 100 of Fig.1, for example, the sensor server(s) 160 may translate detected user input into a standardized representation of a corresponding event and place the event on an event gateway 180, and the interaction manager 190 may implement an interpreter that is subscribed to or otherwise configured to retrieve or receive the events from the event gateway 180.

[0253] The method 2300 includes, in block B2304, generating one or more representations of one or more actions of the recognized user(s) for one or more matches with one or more interrupted interaction sequences, at least based on the request of the interpreter to create one or more large language models (LLMs) and evaluate the one or more representations of one or more actions of the agent(s) or the scene. With respect to Fig.7, the interpreter 710 may, for example, support the use of natural language descriptions and the use of one or more LLMs. For example, the interpreter 710 may request an LLM to generate a natural language description of one or more lines of instructions defining a flow, generate one or more lines of instructions for a particular flow, determine whether an event matches a flow description of an active flow, determine whether a non-matching event matches the name and / or instruction(s) of an active flow, generate a flow in response to a non-matching event, and / or other.

[0254] Fig.24 is a flowchart illustrating a method 2400 for generating one or more outgoing interaction modeling events instructing one or more action servers to perform one or more addressable agent or scene actions, in accordance with some embodiments of the present disclosure. The method 2400 includes, at block B2402, generating, by one or more sensory servers in one or more input interaction channels, one or more incoming interaction modeling events representing one or more detected user actions. With respect to the interactive system 100 of Fig. 1, the sensor server(s) 160 can, for example, translate recognized user inputs into a standardized representation of a corresponding event and place the event on an event gateway 180.

[0255] The method 2400 includes, at block B2404, generating one or more outbound interaction modeling events by an interaction manager based on at least one or more inbound interaction modeling events that instruct one or more action servers in one or more outbound interaction channels to execute one or more addressable agent actions or scene actions associated with an interactive agent. With respect to the interactive system 100 of Fig.1, the interaction manager 190 may, for example, implement an interpreter that is subscribed to or otherwise configured to ingest or receive the events from the event gateway 180, process the events (e.g., using an event-driven state machine), determine which interactions should be performed, and generate commands as corresponding events in a standardized representation and forward them to the event gateway 180. The action server(s) 170 responsible for the corresponding interaction channel(s) may be subscribed to or otherwise configured to ingest the events it is responsible for executing from the event gateway 180.The action server(s) 170 may execute, schedule, and / or otherwise handle events for the corresponding interaction modality(ies) by interfacing with a corresponding service that controls a corresponding output interface.

[0256] Fig.25 is a flowchart illustrating a method 2500 for generating a visual layout representing an update specified by an event, in accordance with some embodiments of the present disclosure. The method 2500 includes, at block B2502, receiving from one or more action servers handling one or more visual content overlays that augment one or more conversations with an interactive agent, one or more events representing one or more visual content actions categorized using an interaction categorization scheme, and instructing one or more updates to the one or more overlays in one or more GUIs. With respect to Fig.9, the action server 930 may, for example, include a GUI service (e.g., Modality Service B) that processes interactive visual content and corresponding events. An interactive visual content event (e.g., generated by an interaction manager such as the interaction manager 190 of Fig. 1 or the interaction manager 700 of Fig.7) may instruct the visualization of various types of visual information (e.g., in a 2D or 3D interface). In some embodiments, an interactive visual content event (e.g., payload) includes fields that specify or encode values ​​representing a supported action type categorizing a instructed action (e.g., VisualInformationSceneAction, VisualSelectionSceneAction, VisualFormSceneAction), an action status (e.g., "initiated," "scheduled," "starting," "running," "paused," "resuming," "stopping," or "finished"), a representation of the instructed visual content, and / or other properties or information.

[0257] The method 2500 includes, at block B2504, generating, by the one or more action servers, one or more visual layouts representing the one or more updates specified by the one or more events. With respect to Fig.9, the action server 930 may, for example, include a GUI service (e.g., Modality Service B) that includes an action handler for each supported event for each supported interactive visual content action, and the action handler for a corresponding (e.g., a start) event for a visual information scene action may translate the event into a (e.g., JSON) representation of a modular GUI configuration specifying content blocks such as a hint carousel block for one or more specified support blocks, a header block for a specified title, image and / or text blocks for the specified content, a (e.g., continue, cancel) button(s), and / or other elements. Accordingly, the action handler may use these content blocks to generate a custom page by creating a visual layout (e.g.,a specific template or a visual shell layout with appropriate placeholders) for a GUI overlay layout (e.g., an HTML layout), and it can call the user interface server endpoint with the custom page to cause the user interface server to render the custom page.

[0258] Fig. 26 is a flowchart illustrating a method 2600 for triggering an animation state of an interactive agent, according to some embodiments of the present disclosure. The method 2600 includes, at block B2602, receiving from one or more action servers handling the animation of gestures of an interactive agent, one or more first interaction modeling events instructing one or more target states of one or more agent gestures represented using an interaction categorization scheme. With respect to Fig.9, the action server 930 may, for example, include an animation service (e.g., Modality Service A) that processes bot movements and / or gesture actions (e.g., GestureBotAction) and corresponding events. For example, a bot gesture action event (e.g., generated by an interaction manager such as the interaction manager 190 of Fig. 1 or the interaction manager 700 of Fig. 7) a particular animation (e.g., in a 2D or 3D interface) using fields that specify or encode values ​​representing a supported action type categorizing a commanded action (e.g., GestureBotAction), an action state (e.g., start, started, updated, stopped, finished), a representation of a commanded bot gesture, and / or other properties or information.

[0259] The method 2600 includes, at block B2604, triggering, by the one or more action servers, one or more animation states of the interactive agent corresponding to the one or more target states of the one or more agent gestures commanded by the one or more first interaction modeling events. With respect to Fig. 9, the action server 930 may include, for example, an animation service (e.g., modality service A) that includes an action handler for each supported event for each supported bot gesture action. Fig.12D shows some example action processors for some example gestureBotAction events according to some embodiments of the present disclosure. Taking a bot gesture specified as a natural language description for a bot gesture action as an example, the action processor for a corresponding event (e.g., start) for the bot gesture action may extract the natural language description from the event, generate or access a sentence embedding for the natural language description of the bot gesture, use it to perform a similarity search on sentence embeddings for descriptions of the available animations, and select an animation using a similarity measure (e.g., nearest neighbor, within a threshold).

[0260] Fig.27 is a flowchart illustrating a method 2700 for performing one or more preparatory actions in accordance with some embodiments of the present disclosure. The method 2700 includes, at block B2702, receiving, by one or more servers associated with an interactive agent, one or more first interaction modeling events instructing one or more preparatory actions associated with the expectation that one or more specified events will occur and represented using an interaction categorization scheme. With respect to Fig.17, for example, in step 1720, the interaction manager 1706 generates a standardized event indicating that a user utterance is expected to begin soon and instructing a preparatory action, and sends the event (e.g., StartBotExpectationAction(UtteranceUserActionEnded)) to the event gateway 1702, which the action server 1712 picks up in step 1722.

[0261] The method 2700 includes, in block B2704, the execution of the one or more preparatory actions by the first server. With respect to Fig.17, for example, in step 1724, the action server 1712 informs the client device 1716 to disable its audio output, in step 1726, it informs the client device 1716 to enable its microphone, and in step 1728, it informs the automatic speech recognition system 1714 to enable automatic speech recognition. In step 1730, the action server 1712 generates a standardized event confirming that the bot awaiting action has begun and / or indicating that the preparatory action has been initiated and sends the event (e.g., BotAwaitingActionStarted(UtteranceUserActionEnded)) to the event gateway 1702, which the interaction manager 1706 picks up in step 1732.

[0262] The systems and methods described herein may be used for a variety of purposes, including, but not limited to, controlling machines (e.g., robots, vehicles, construction equipment, warehouse vehicles / machines, autonomous, semi-autonomous, and / or other types of machines), machine locomotion, machine driving, synthetic data generation, model training (e.g., using real-world, augmented, and / or synthetic data, such as synthetic data generated using a simulation platform or system, synthetic data generation techniques such as, but not limited to, those described herein, etc.), perception, augmented reality (AR), virtual reality (VR), mixed reality (MR), robotics, security, and surveillance (e.g.,, in a smart cities implementation), autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twinning, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), distributed or collaborative content creation for 3D goods (e.g., using universal scene descriptor (USD) data such as OpenUSD and / or other data types), cloud computing, generative artificial intelligence (e.g., using one or more diffusion models, transformer models, etc.), and / or other suitable applications.

[0263] Included embodiments may be included in a variety of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented using a robot or robotic platform, aviation systems, media systems, boat systems, smart area surveillance systems, systems for performing deep learning operations, systems for performing simulation operations (e.g., in a driving or vehicle simulation, in a robot simulation, in a smart city or surveillance simulation, etc.), systems for performing digital twin operations (e.g., in conjunction with a platform or system for collaborative content creation, such asOMNIVERSE by NVIDIA and / or any other platform, system, or service that uses USD or OpenUSD data types), systems implemented using an edge device, systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations (e.g., using one or more neural replay fields (NERFs), Gaussian splat techniques, diffusion models, transformer models, etc.), systems implemented at least partially in a data center, systems for performing conversational AI operations, systems implementing one or more language models - such as one or more large-scale language models (LLMs), one or more visual language models (VLMs), one or more multimodal language models, etc., systems for performing light transport simulations, systems for collaborative content creation for 3D goods (e.g.,using universal scene description data (USD) such as OpenUSD, computer-aided design (CAD) data, 2D and / or 3D graphics or design data, and / or other data types), systems implemented at least in part using cloud computing resources, and / or other types of systems.

[0264] In some embodiments, the systems and methods described herein may be performed within a 3D content collaboration platform (e.g., NVIDIA's OMNIVERSE) for 3D rendering, industrial digitization, generative physical AI, and / or other use cases, applications, or services. For example, the content collaboration platform may host a framework for developing and / or deploying interactive agents (e.g., an interactive avatar) and include a system for using or developing universal scene descriptors (USD) (e.g., OpenUSD) for managing objects, features, scenes, etc., in a digital environment, a simulated environment, etc. The platform may include a real physics simulation, e.g.,using NVIDIA's PhysX SDK to simulate real-world physics and physical interactions with virtual objects, characters, simulations, or other types of 3D content hosted by the platform. The platform may integrate OpenUSD along with ray tracing / path tracing / light transport simulation (e.g., NVIDIA's RTX rendering technologies) into software tools and rendering workflows. In some embodiments, the development and / or deployment of an interactive agent (e.g., an interactive bot or robot) may leverage one or more cloud services and / or machine learning models (e.g., neural networks, large language models).For example, NVIDIA's Avatar Cloud Engine (ACE) is a suite of cloud-based AI models and services designed to create and manage interactive, lifelike avatars using hosted natural language processing, speech recognition, computer vision, and / or conversational AI services. In some embodiments, an interactive agent may be developed and / or deployed as part of an application hosted by a (e.g., streaming) platform such as a cloud-based gaming platform (e.g., NVIDIA GeForce NOW). Thus, interactive agents such as digital avatars may be developed and / or deployed for various applications, such as customer service, virtual assistants, interactive entertainment or gaming, digital twins (e.g.,for video conference participants), education or training, healthcare, virtual or augmented reality experiences, social media interactions, marketing and advertising, and / or other applications. EXAMPLE LANGUAGE MODELS

[0265] At least in some embodiments, language models such as large language models (LLMs), vision language models (VLMs), multimodal language models (MMLMs), and / or other types of generative artificial intelligence (AI) may be implemented. These models may be capable of understanding, summarizing, translating, and / or otherwise generating text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., in USD format, such as OpenUSD), and / or the like, based on the context provided in prompts or queries. These language models may be considered "large" in embodiments because the models are trained on large-scale datasets and have architectures with a large number of learnable network parameters (weights and biases)—such as millions or billions of parameters. The LLMs / VLMs / MMLMs / etc.can be used to summarize text data, analyze and gain insights from data (e.g., text, image, video, etc.), and generate new text / images / videos / etc. in user-specified styles, sounds, and / or formats. The LLMs / VLMs / MMLMs / etc. of the present disclosure may, in certain embodiments, be used exclusively for text processing, while in other embodiments, multimodal LLMs may be implemented to accept, understand, and / or generate text and / or other types of content such as images, audio, 2D and / or 3D data (e.g., in USD formats), and / or video. For example, vision language models (VLMs) or, more generally, multimodal language models (MMLMs) may be implemented to understand image, video, audio, text, 3D design (e.g.,CAD) and / or other input data types and / or to generate or output image, video, audio, text, 3D design and / or other output data types.

[0266] Different types of LLMs / VLMs / MMLMs / etc. architectures can be implemented in different embodiments. For example, different architectures can be implemented that use different techniques for understanding and generating outputs—such as text, audio, video, images, 2D and / or 3D design or asset data, etc. In some embodiments, LLMs / VLMs / MMLMs / etc. architectures such as recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) can be used, while in other embodiments, transformer architectures—such as those relying on self-attention and / or cross-attention (e.g., between context data and text data)—are used to understand and recognize relationships between words or tokens and / or context data (e.g., other text, video, image, design data, USD, etc.). One or more generative processing pipelines that use LLMs / VLMs / MMLMs / etc.may also include one or more diffusion blocks (e.g., denozer). The LLMs / VLMs / MMLMs / etc. of the present disclosure may include one or more encoder and / or decoder blocks. For example, discriminative or encoder-only models such as BERT (Bidirectional Encoder Representations from Transformers) may be implemented for tasks that involve language understanding, such as classification, sentiment analysis, question answering, and named entity recognition. Another example is generative or decoder-only models such as GPT (Generative Pretrained Transformer), which may be used for tasks that involve speech and content generation, such as text completion, story generation, and dialogue generation. LLMs / VLMs / MMLMs / etc., which contain both encoder and decoder components, such as T5 (Text-to-Text Transformer), can be used for content understanding and generation, for example, for translation and summarization. These examples are not intended to be limiting, and any type of architecture—including but not limited to those described here—can be implemented depending on the particular embodiment and the task(s) being performed with the LLMs / VLMs / MMLMs / etc.

[0267] In various embodiments, the LLMs / VLMs / MMLMs / etc. can be trained through unsupervised learning, in which an LLMs / VLMs / MMLMs / etc. learns patterns from large amounts of unlabeled text / audio / video / image / design / USD, etc. data. Due to the large-scale training, the models may not require task-specific or domain-specific training. LLMs / VLMs / MMLMs / etc. that have undergone extensive pre-training on massive amounts of unlabeled data can be referred to as baseline models and are suitable for a variety of tasks, such as question answering, summarization, filling in missing information, translation, image / video / design / USD data generation. Some LLMs / VLMs / MMLMs / etc. can be tailored to a specific use case by using techniques such as prompt tuning, fine-tuning, Retrieval Augmented Generation (RAG), adding adapters (e.g.custom neural networks and / or neural network layers that tune or adapt prompts or tokens to target the language model to a specific task or domain) and / or other fine-tuning or tailoring techniques that optimize the models for use on specific tasks and / or within specific domains.

[0268] In some embodiments, the LLMs / VLMs / MMLMs / etc. of the present disclosure may be implemented using various model matching techniques. For example, in some embodiments, guardrails may be implemented to identify invalid or undesirable inputs (e.g., prompts) and / or outputs of the models. In doing so, the system may use the guardrails and / or other model matching techniques to either prevent a particular undesirable input from being processed with the LLMs / VLMs / MMLMs / etc. and / or to prevent the output or presentation (e.g., display, audio output, etc.) of information generated with the LLMs / VLMs / MMLMs / etc. In some embodiments, one or more additional models—or layers thereof—may be implemented to detect problems with inputs and / or outputs of the models.For example, these "protection" models can be trained to identify inputs and / or outputs that are "safe" or otherwise acceptable or desirable, and / or that are "unsafe" or otherwise undesirable for the particular application / implementation. As a result, the LLMs / VLMs / MMLMs / etc. of the present disclosure are less likely to output speech / text / audio / video / design data / LJSD data / etc. that might be offensive, vulgar, inappropriate, unsafe, out of range, and / or otherwise undesirable for the particular application / implementation.

[0269] In some embodiments, the LLMs / VLMs / etc. may be configured or capable of accessing or using one or more plug-ins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations for which it is not ideally suited, the model may have instructions (e.g., as a result of training and / or based on instructions in a given prompt) to access one or more plug-ins (e.g., third-party plug-ins) to assist in processing the current input. In such an example, where at least part of a prompt relates to restaurants or the weather, the model may access one or more restaurant or weather plug-ins (e.g., via one or more APIs) to retrieve the relevant information.Another example: If at least part of an answer requires a mathematical calculation, the model can access one or more mathematical plugins or APIs for help solving the problem(s) and can then use the plug-in and / or API's answer in the model's output. This process can be repeated—e.g., recursively—for any number of iterations and using any number of plugins and / or APIs until a response to the prompt can be generated that addresses each question / problem / process / operation / etc. This allows the model(s) to rely not only on their own knowledge gained from training with a large dataset(s), but also on the expertise or optimized nature of one or more external resources—such as APIs, plugins, and / or the like.

[0270] In some embodiments, multiple language models (e.g., LLMs / VLMs / MMLMs / etc.), multiple instances of the same language model, and / or multiple prompts submitted to the same language model or instance of the same language model may be implemented, executed, or retrieved (e.g., using one or more plug-ins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide output responsive to the same query or to separate parts of a query. In at least one embodiment, multiple language models, e.g., language models with different architectures, language models trained on different (e.g., updated) datasets, may be provided with the same input query and prompt (e.g., a set of constraints, conditions, etc.).In one or more embodiments, the language models may be different versions of the same basic model. In one or more embodiments, at least one language model may be instantiated as a multi-agent—e.g., more than one prompt may be provided to constrain, control, or otherwise influence a style, content, character, etc., of the provided output. In one or more non-limited examples, the same language model may be requested to provide output corresponding to a different role, perspective, character, or knowledge base, etc.—as defined by a predetermined prompt.

[0271] In each of these embodiments, the output of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiated agents of at least one language model, and / or two further requests provided to at least one language model may be further processed, e.g., aggregated, compared, or filtered, or used to determine (and provide) a consensus answer. In one or more embodiments, the output of one language model—or version, instance, or agent—may be provided as input to another language model for further processing and / or validation. In one or more embodiments, a language model may be requested to generate or otherwise obtain output related to input source material, linking the output to the input source material.Such linking may, for example, involve generating a caption or a portion of text embedded (e.g., as metadata) in input source text or an image. In one or more embodiments, the output of a language model may be used to determine the validity of input source material for further processing or inclusion in a dataset. For example, a language model may be used to assess the presence (or absence) of a target word in a portion of text or an object in an image, with the text or image being annotated to note the presence (or absence). Alternatively, the language model determination may be used to determine whether the source material should be included in a curated dataset, for example, and without limitation.

[0272] Fig.28A is a block diagram of an exemplary generative language model system 2800 suitable for use in implementing at least some embodiments of the present disclosure. In the Fig. 28A, the generative language model system 2800 includes a RAG component 2892, an input processor 2805, a tokenizer 2810, an embedding component 2820, plug-ins / APIs 2895, and a generative language model (LM) 2830 (which may include an LLM, a VLM, a multimodal LM, etc.).

[0273] At a high level, the input processor 2805 may receive an input 2801 comprising text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, Universal Scene Descriptor (USD) data - such as OpenUSD, etc.), depending on the architecture of the generative LM 2830 (e.g., LLM / VLM / MMLM, etc.). In some embodiments, the input 2801 comprises plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, the input 2801 may include numeric sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., in tabular formats, JSON, or XML).In some implementations where the generative LM 2830 is capable of processing multimodal inputs, the input 2801 may combine text with (or omit text from) image data, audio data, video data, design data, USD data, and / or other types of input data, such as, but not limited to, those described herein. Using raw input text as an example, the input processor 2805 may process raw input text in various ways. For example, the input processor 2805 may perform various types of text filtering to remove noise (e.g., special characters, punctuation, HTML tags, stop words, portions of an image or portions of audio, etc.) from the relevant text content. In an example involving stop words (common words that typically have little semantic meaning), the input processor 2805 may remove stop words to reduce the noise and focus the generative LM 2830 on more meaningful content.Input processor 2805 may perform text normalization, such as converting all characters to lowercase, removing accents, and / or handling special cases such as contractions or abbreviations to ensure consistency. These are just a few examples, and other types of input processing may also be applied.

[0274] In some embodiments, a RAG component 2892 (which may contain one or more RAG models and / or be executed using the generative LM 2830 itself) may be used to retrieve additional information to be used as part of the input 2801 or prompt. RAG may be used to augment the input to the LLM / VLM / MMLM / etc. with external knowledge so that the answers to specific questions, queries, or requests are more relevant—for example, in a case where specific knowledge is required. The RAG component 2892 may retrieve this additional information (e.g., base information such as basic text / image / video / audio / USD / CAD / etc.) from one or more external sources, which may then be fed into the LLM / VLM / MMLM / etc. along with the prompt to improve the accuracy of the model's answers or outputs.

[0275] For example, in some embodiments, input 2801 may be generated using the query or input to the model (e.g., a question, a request, etc.) in addition to the data retrieved with RAG component 2892. In some embodiments, input processor 2805 may parse input 2801 and communicate with RAG component 2892 (or RAG component 2892 may be part of input processor 2805 in embodiments) to identify relevant text and / or other data to provide to generative LM 2830 as additional context or sources of information from which the answer, response, or output 2890 generally is identified.For example, if the input indicates that the user is interested in a desired tire pressure for a particular vehicle make and model, the RAG component 2892 may—for example, using a RAG model that performs a vector search in an embedding space—retrieve the tire pressure information or the corresponding text from a digital (embedded) version of the user's manual for that particular vehicle make and model. Similarly, upon a revisit to a chatbot related to a particular product offering or service, the RAG component 2892 may retrieve a previous stored conversation history—or at least a summary thereof—and include the previous conversation history, along with the current question / request, as part of the input 2801 in the generative LM 2830.

[0276] The RAG component 2892 may use various RAG techniques. For example, naive RAG may be used, in which documents are indexed, divided into chunks, and applied to an embedding model to generate embeddings corresponding to the chunks. A user query may also be applied to the embedding model and / or another embedding model of the RAG component 2892, and the chunk embeddings along with the query embeddings may be compared to identify the most similar / related query embeddings, which may be fed to the generative LM 2830 to generate an output.

[0277] In some embodiments, more advanced RAG techniques may be used. For example, the chunks may undergo pre-retrieval processes (e.g., routing, rewriting, metadata analysis, expansion, etc.) before being passed to the embedding model. Furthermore, post-retrieval processes (e.g., re-ranking, prompt compression, etc.) may be performed on the outputs of the embedding model before generating the final embeddings. These post-retrieval processes may be performed on the outputs of the embedding model before the final embeddings are used for comparison against an input query.

[0278] As another example, modular RAG techniques can be used, which are similar to naive and / or advanced RAG, but also include features such as hybrid search, recursive retrieval and query engines, stepback approaches, subqueries, and hypothetical document embedding.

[0279] Another example: Graph RAG can use knowledge graphs as a source of contextual or factual information. Graph RAG can be implemented using a graph database as the source of contextual information, which is sent to the LLM / VLM / MMLM / etc. Instead of (or in addition to) feeding the model with pieces of data extracted from larger documents—which can lead to a lack of context, factual accuracy, linguistic precision, etc.—Graph RAG can also provide structured entity information to the LLM / VLM / MMLM / etc. by combining the structured text description of the entity with its many properties and relationships, allowing the model to gain deeper insights. When implementing Graph RAG, the systems and methods described here use a graph as a content store, extracting relevant pieces of documents, and prompting the LLM / VLM / MMLM / etc. to respond based on them.In such embodiments, the knowledge graph may contain relevant textual content and metadata about the knowledge graph and may be integrated with a vector database. In some embodiments, the Graph RAG may use a graph as a domain expert, where descriptions of concepts and entities relevant to a query / prompt may be extracted and passed to the model as semantic context. These descriptions may include relationships between the concepts. In other examples, the graph may be used as a database, where part of a query / prompt may be mapped to a graph query, the graph query may be executed, and the LLM / VLM / MMLM / etc. may summarize the results.In such an example, the graph may contain relevant factual information, and a query (natural language query) in a graph query tool (NL-to-graph query tool) and entity linking may be used. In some embodiments, graph RAG (e.g., using a graph database) may be combined with standard RAG (e.g., vector database) and / or other RAG types to benefit from multiple approaches.

[0280] In all embodiments, the RAG component 2892 may implement a plugin, an API, a user interface, and / or other functionality for performing RAG. For example, a graph RAG plugin may be used by the LLM / VLM / MMLM / etc. to execute queries against the knowledge graph to extract relevant information for feeding into the model, and a standard or vector RAG plugin may be used to execute queries against a vector database. For example, the graph database may interact with the REST interface of a plugin so that the graph database is decoupled from the vector database and / or the embedding models.

[0281] The Tokenizer 2810 can segment the (e.g., processed) text data into smaller units (tokens) for subsequent analysis and processing. Depending on the implementation, the tokens can represent individual words, subwords, characters, parts of audio / video / images, etc. Word-based tokenization divides the text into individual words, with each word being processed as a separate token. Partial word tokenization breaks words into smaller, meaningful units (e.g., prefixes, suffixes, root words) so that the Generative LM 2830 can understand morphological variations and better handle words that are not in the vocabulary. Character-based tokenization represents each character as a separate token, allowing the Generative LM 2830 to process text at a fine-grained level.The choice of tokenization strategy may depend on factors such as the language to be processed, the task to be performed, and / or the characteristics of the training dataset. For example, tokenizer 2810 may convert the (e.g., processed) text into a structured format according to the tokenization scheme implemented in the respective embodiment.

[0282] Embedding component 2820 may use any known embedding technique to convert discrete tokens into (e.g., dense, continuous vector) representations of semantic meaning. For example, embedding component 2820 may use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more neural network embedding layers, and / or other methods.

[0283] In some implementations where input 2801 contains image data / video data, etc., input processor 2801 may scale the data to a standard size compatible with the format of a corresponding input channel and / or normalize the pixel values ​​to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 2820 may encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations where input 2801 contains audio data, input processor 2801 may resample an audio file to a consistent sampling rate for uniform processing, and embedding component 2820 may use any known technique to extract and encode audio features—for example, in the form of a spectrogram (e.g.,a Mel spectrogram). In some implementations where input 2801 contains video data, input processor 2801 may extract frame embeddings or apply resizing to extracted frame embeddings, and embedding component 2820 may extract features such as optical flow embeddings or video embeddings and / or encode temporal information or sequences of frame embeddings. In some implementations where input 2801 contains multimodal data, embedding component 2820 may fuse representations of the various data types (e.g., text, image, audio, USD, video, design, etc.) using techniques such as early fusion (concatenation), late fusion (sequential processing), attention-based fusion (e.g., self-attention, cross-attention), etc.

[0284] The generative LM 2830 and / or other components of the generative LM system 2800 may use various types of neural network architectures depending on the implementation. For example, transformer-based architectures, as used in models such as GPT, may be implemented and may include self-attention mechanisms that weigh the importance of different words or tokens in the input sequence and / or feedforward networks that process the output of the self-attention layers by applying nonlinear transformations to the input representations and extracting higher-level features. Some non-limiting example architectures include transformers (e.g.,Encoder-decoder, decoder-only, multimodal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn joint embedding spaces, graph neural networks (GNNs), hybrid architectures that combine different types of architectures, adversary networks such as generative adversary networks or GANs or adversary autocoders (AAEs) for joint distribution learning, and others. Depending on the implementation and architecture, the embedding component 2820 may apply an encoded representation of the input 2801 to the generative LM 2830, and the generative LM 2830 may process the encoded representation of the input 2801 to produce an output 2890, which may include responsive text and / or other types of data.

[0285] As described herein, in some embodiments, the generative LM 2830 may be configured to access or use—or be capable of accessing or using—plug-ins / APIs 2895 (which may include one or more plug-ins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations for which the generative LM 2830 is not ideally suited, the model may have instructions (e.g., as a result of training and / or based on instructions in a given prompt, such as those retrieved with the RAG component 2892) to access one or more plug-ins / APIs 2895 (e.g., third-party plug-ins) for assistance in processing the current input.In such an example, where at least part of a prompt relates to restaurants or the weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs), send at least part of the prompt relating to the particular plug-in / API 2895 to the plug-in / API 2895, the plug-in / API 2895 may process the information and return a response to the generative LM 2830, and the generative LM 2830 may use the response to generate the output 2890. This process may be repeated—e.g., recursively—for any number of iterations and using any number of plug-ins / APIs 2895 until an output 2890 can be generated that addresses each question / request / query / process / operation / etc. from the input 2801.As such, the model(s) may rely not only on its own knowledge gained from training with a large dataset(s) and / or from data retrieved using the RAG component 2892, but also on the expertise or optimized nature of one or more external resources—such as the plug-ins / APIs 2895.

[0286] Fig. Figure 28B is a block diagram of an example implementation where the generative LM 2830 includes a transformer encoder-decoder. Suppose an input text such as "Who discovered gravity?" is tokenized (e.g., by the tokenizer 2810 of Fig.28A), and each token is encoded (e.g., by the embedding component 2820 of FIG. 28A) into a corresponding embedding (e.g., of size 512). Since these token embeddings typically do not represent the position of the token in the input sequence, any known technique may be used to add a positional encoding to each token embedding to encode the sequential relationships and context of the tokens in the input sequence. As such, the (e.g., resulting) embeddings may be applied to one or more encoders 2835 of the generative LM 2830.

[0287] In an example implementation, the encoder(s) 2835 form an encoder stack, with each encoder comprising an introspection layer and a feedforward network. In an example transformer architecture, each token (e.g., a word) traverses a separate path. Thus, each encoder can accept a sequence of vectors, with each vector passed through the self-attention layer, then the feedforward network, and then upstream to the next encoder in the stack. Any known self-attention technique can be used. For example, to calculate a self-attention score for each token (word), a query vector, a key vector, and a value vector can be created for each token.A self-attention score can be calculated for token pairs by taking the dot product of the query vector with the corresponding key vectors, normalizing the resulting scores, multiplying them by the corresponding value vectors, and summing the weighted value vectors. The encoder can apply multi-headed attention, in which the attention mechanism is applied multiple times in parallel with different learned weight matrices. Any number of encoders can be cascaded to generate a context vector that encodes the input. An attention projection layer 2840 can convert the context vector into attention vectors (keys and values) for the decoder(s) 2845.

[0288] In an example implementation, the decoders 2845 form a decoder stack, with each decoder including a self-attention layer, an encoder-decoder self-attention layer that uses the encoder's attention vectors (keys and values) to focus on relevant parts of the input sequence, and a feedforward network. As with the encoder(s) 2835, in an example transformer architecture, each token (e.g., a word) traverses a separate path in the decoder(s) 2845. During a first pass, the decoder(s) 2845, a classifier 2850, and a generation mechanism 2855 may generate a first token, and the generation mechanism 2855 may use the generated token as input during a second pass. The process may repeat in a loop, with successive tokens (e.g.,words) are generated and added to the output from the previous pass, and the token embeddings of the composite sequence with position encodings are used as input to the decoder(s) 2845 during a subsequent pass, sequentially generating one token at a time (known as autoregression) until a symbol or token is predicted that represents the end of the response. Within each decoder, the introspection layer is typically forced to attend only to previous positions in the output sequence by applying a masking technique before the softmax operation (e.g., by setting future positions to negative infinity). In an example implementation, the encoder-decoder attention layer operates similarly to the (e.g.,multi-headed) self-attention in the encoder(s) 2835, except that it constructs its queries from the underlying layer and takes the keys and values ​​(e.g., the matrix) from the output of the encoder(s) 2835.

[0289] As such, the decoder(s) 2845 may output a decoded (e.g., vector) representation of the input applied during a particular pass. The classifier 2850 may include a multi-class classifier comprising one or more neural network layers that project the decoded (e.g., vector) representation into an appropriate dimensionality (e.g., one dimension for each supported word or token in the output vocabulary) and a softmax operation that converts logits to probabilities. As such, the generation mechanism 2855 may select or query for a word or token based on a corresponding predicted probability (e.g., selecting the word with the highest predicted probability) and append it to the output from a previous pass, generating each word or token in turn.The generation mechanism 2855 may repeat the process by triggering successive decoder inputs and corresponding predictions until it selects or samples a symbol or token that represents the end of the response, at which point the generation mechanism 2855 may output the generated response.

[0290] Fig. Figure 28C is a block diagram of an example implementation where the generative LM 2830 includes a pure decoder-transformer architecture. For example, the decoder(s) 2860 of Fig. 28C similar to the decoder(s) 2845 from Fig. 28B, except that each of the decoders 2860 of Fig.28C omits the encoder-decoder introspection layer (since there is no encoder in this implementation). As such, the decoders 2860 may form a decoder stack, with each decoder including an introspection layer and a feedforward network. Furthermore, instead of encoding the input sequence, a symbol or token representing the end of the input sequence (or the beginning of the output sequence) may be appended to the input sequence, and the resulting sequence (e.g., corresponding embeddings with position encodings) may be applied to the decoder(s) 2860. As with the decoder(s) 2845 of Fig.28B, each token (e.g., word) may traverse a separate path in the decoder(s) 2860, and the decoder(s) 2860, a classifier 2865, and a generation mechanism 2870 may use autoregression to sequentially generate one token after another until a symbol or token is predicted that represents the end of the response. The classifier 2865 and the generation mechanism 2870 may operate similarly to the classifier 2850 and the generation mechanism 2855 of Fig.28B, wherein the generation mechanism 2870 selects or samples each successive output token based on a corresponding predicted probability and appends it to the output from a previous pass, with each token being generated sequentially until a symbol or token is selected or sampled that represents the end of the response. These and other architectures described herein are merely examples, and other suitable architectures may be implemented within the scope of the present disclosure. EXAMPLE OF A CONTENT STREAMING SYSTEM

[0291] Fig. 29 is an example system diagram for a content streaming system 2900 according to some embodiments of the present disclosure. Fig. 29 includes one or more application servers 2902 (which have similar components, features, and / or functions as the example computing device 3000 of Fig.30), one or more client devices 2904 (which may include similar components, features, and / or functions as the example computing device 3000 of Fig. 30) and one or more networks 2906 (which may be similar to the networks described herein). In some embodiments of the present disclosure, system 2900 may support an application session corresponding to a game streaming application (e.g., NVIDIA GeForce NOW), a remote desktop application, a simulation application (e.g., autonomous or semi-autonomous vehicle simulation), computer-aided design (CAD) applications, virtual reality (VR) and / or augmented reality (AR) streaming applications, deep learning applications, and / or other application types.

[0292] In system 2900, the client device(s) 2904 for an application session may only receive input data in response to inputs to the input device(s), transmit the input data to the application server(s) 2902, receive encoded display data from the application server(s) 2902, and display the display data on the display 2924. Thus, the more computationally intensive calculations and processing may be offloaded to the application server(s) 2902 (e.g., rendering—particularly ray or path tracing—for the graphical output of the application session may be performed by the GPU(s) of the application server(s) 2902, such as a game server). In other words, the application session may be streamed from the application server(s) 2902 to the client device(s) 2904, thereby reducing the graphics processing and rendering requirements of the client device(s) 2904.

[0293] For example, with respect to instantiation of an application session, a client device 2904 may display a frame of the application session on display 2924 after receiving display data from the application server(s) 2902. The client device 2904 may receive input to one of the input devices and generate input data in response. The client device 2904 may transmit the input data to the application server(s) 2902 via communication interface 2920 and over the network(s) 2906 (e.g., the Internet), and the application server(s) 2902 may receive the input data via communication interface 2918. The CPU(s) may receive the input data, process the input data, and transmit data to the GPU(s), causing the GPU(s) to generate a rendering of the application session.For example, the input data may represent movement of a user character in a gaming session of a gaming application, firing a weapon, reloading, passing a ball, turning a vehicle, etc. The rendering component 2912 may render the application session (e.g., representative of the result of the input data), and the rendering capture component 2914 may capture the rendering of the application session as display data (e.g., as image data capturing the rendered image of the application session). The rendering of the application session may include ray- or path-traced lighting and / or shadow effects computed using one or more parallel processing units—such as GPUs, which may further utilize one or more dedicated hardware accelerators or processor cores to perform ray- or path-tracing techniques—of the application server(s) 2902.In some embodiments, one or more virtual machines (VMs)—e.g., including one or more virtual components such as vGPUs, vCPUs, etc.—may be used by the application server(s) 2902 to support the application sessions. The encoder 2916 may then encode the display data to generate encoded display data, and the encoded display data may be transmitted over the network(s) 2906 to the client device 2904 via the communication interface 2918. The client device 2904 may receive the encoded display data via the communication interface 2920, and the decoder 2922 may decode the encoded display data to generate the display data. The client device 2904 may then display the display data via the display 2924. EXAMPLE OF A DATA PROCESSING SYSTEM

[0294] Fig.30 is a block diagram of an example computing device(s) 3000 suitable for use in implementing some embodiments of the present disclosure. Computing device 3000 may include an interconnect system 3002 that directly or indirectly couples the following devices: memory 3004, one or more central processing units (CPUs) 3006, one or more graphics processing units (GPUs) 3008, a communications interface 3010, input / output (I / O) ports 3012, input / output components 3014, a power supply 3016, one or more presentation components 3018 (e.g., display(s)), and one or more logic units 3020. In at least one embodiment, computing device(s) 3000 may include one or more virtual machines (VMs), and / or each of its components may include virtual components (e.g., virtual hardware components).As non-limiting examples, one or more of the GPUs 3008 may include one or more vGPUs, one or more of the CPUs 3006 may include one or more vCPUs, and / or one or more of the logic units 3020 may include one or more virtual logic units. Thus, a compute unit 3000 may include discrete components (e.g., a full GPU associated with the compute unit 3000), virtual components (e.g., a portion of a GPU associated with the compute unit 3000), or a combination thereof.

[0295] Although the different blocks in Fig.30 are shown as being connected to wires via the interconnect system 3002, this is not intended as a limitation and is for clarity only. For example, in some embodiments, a presentation component 3018, such as a display device, may be considered an I / O component 3014 (e.g., if the display is a touchscreen). Another example is that the CPUs 3006 and / or GPUs 3008 may include memory (e.g., the memory 3004 may represent a storage device in addition to the memory of the GPUs 3008, the CPUs 3006, and / or other components). In other words, the computing device of Fig.30 is merely illustrative. No distinction is made between categories such as ‘workstation’, ‘server’, ‘laptop’, ‘desktop’, ‘tablet’, ‘client device’, ‘mobile device’, ‘handheld device’, ‘game console’, ‘electronic control unit (ECU)’, ‘virtual reality system’ and / or other device or system types, as they all fall within the scope of the computing device of Fig. 30 fall.

[0296] The interconnect system 3002 may represent one or more connections or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 3002 may include one or more types of buses or connections, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or connection. In some embodiments, there are direct connections between the components. For example, the CPU 3006 may be directly connected to the memory 3004. Further, the CPU 3006 may be directly connected to the GPU 3008. For direct or point-to-point connections between components, the interconnect system 3002 may include a PCIe connection to establish the connection.In these examples, a PCI bus does not need to be integrated into the 3000 computer device.

[0297] The storage 3004 may consist of a variety of computer-readable media. The computer-readable media may be any available media accessible by the computing device 3000. The computer-readable media may include both volatile and non-volatile media, as well as removable and non-removable media. By way of example and without limitation, the computer-readable media may include computer storage media and communication media.

[0298] The computer storage media may include both volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 3004 may store computer-readable instructions (e.g., those representing a program(s) and / or a program element(s), such as an operating system.Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 3000. The term "computer storage medium" as used herein does not per se include signals.

[0299] The computer storage media may embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and include any information transmission media. The term "modulated data signal" may refer to a signal having one or more of its characteristics adjusted or altered to encode information in the signal. The computer storage media may include, for example, wired media, such as a wired network or a direct cable connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of the above media should also be considered computer-readable media.

[0300] The CPU(s) 3006 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 3000 and to perform one or more of the methods and / or processes described herein. The CPU(s) 3006 may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a plurality of software threads concurrently. The CPU(s) 3006 may include any type of processor and may include different types of processors depending on the type of computing device 3000 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers).Depending on the type of computing device 3000, the processor may be, for example, an Advanced RISC Machines (ARM) processor using Reduced Instruction Set Computing (RISC) or an x86 processor using Complex Instruction Set Computing (CISC). The computing device 3000 may include one or more CPUs 3006 in addition to one or more microprocessors or additional coprocessors, such as math coprocessors.

[0301] In addition to or alternatively to the CPU(s) 3006, the GPU(s) 3008 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 3000 to perform one or more of the methods and / or processes described herein. One or more of the GPU(s) 3008 may be an integrated GPU (e.g., with one or more of the CPU(s) 3006 and / or one or more of the GPU(s) 3008 may be a discrete GPU. In embodiments, one or more of the GPU(s) 3008 may be a coprocessor of one or more of the CPU(s) 3006. The graphics processor(s) 3008 may be used by the computing device 3000 to render graphics (e.g., 3D graph...

Claims

A processor or multiple processors having processing circuitry for:receiving one or more first interaction modeling events representing at least one of: one or more detected user actions, one or more instructed agent actions, or one or more instructed scene actions, via one or more event gateways and by an interaction manager associated with an interactive agent,generating one or more second interaction modeling events instructing the interactive agent to perform at least one of: one or more addressable agent actions or one or more addressable scene actions, based on at least one processing of the one or more first interaction modeling events by the interaction manager using an event-driven state machine;andcausing a representation to play the one or more addressable agent actions or the one or more addressable scene actions based on at least the one or more second interaction modeling events; The one or more processors of claim 1, wherein the processing circuitry is further operable to generate the one or more first interaction modeling events representing the one or more detected user actions based on at least one or more sensory servers in one or more corresponding input interaction channels translating one or more user inputs into one or more standardized formats. The one or more processors of any preceding claim, wherein the processing circuitry is further operable to execute the one or more addressable agent actions or the one or more addressable scene actions through one or more action servers in one or more corresponding output interaction channels. The one or more processors of any preceding claim, wherein the interaction manager accepts standardized input interaction modeling events representing captured user actions and outputs standardized output interaction modeling events representing instructed agent actions using an interaction categorization scheme. The one or more processors of any preceding claim, wherein the interaction manager is independent of one or more sensory servers that translate user inputs into standardized interaction modeling input events and one or more action servers that execute commands represented by standardized interaction modeling output events issued by the interaction manager. The one or more processors of any preceding claim, wherein the processing circuitry is further operable to transmit, through the interaction manager to the one or more event gateways, the one or more second interaction modeling events instructing the interactive agent to perform the one or more addressable agent actions or the one or more addressable scene actions. The one or more processors according to any one of the preceding claims, wherein the interaction manager implements an interpreter as an event-driven component that supports the execution of one or more interaction flows created in an interaction modeling language and representing one or more flows of one or more human-machine interactions. The one or more processors of any preceding claim, wherein the one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D goods; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more augmented reality, virtual reality, or mixed reality content; a system implemented with an edge device;a system implemented using a robot; a system for performing conversational AI operations; a system for implementing one or more language models; a system for implementing one or more large language models (LLMs); a system for implementing one or more visual language models (VLMs); a system for implementing one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources. A system comprising one or more processors for generating, based on at least one processing of one or more first interaction modeling events using an interaction manager associated with an interactive agent, one or more second interaction modeling events that command the interactive agent to perform one or more addressable agent actions or one or more addressable scene actions. The system of claim 9, wherein the one or more processors are further operable to generate the one or more first interaction modeling events representing one or more detected user actions based on at least one or more sensory servers in one or more corresponding input interaction channels translating one or more user inputs into one or more standardized formats. The system of claim 9 or 10, wherein the one or more processors are further operable to execute the one or more addressable agent actions or the one or more addressable scene actions through one or more action servers in one or more corresponding output interaction channels. The system of any one of claims 9 to 11, wherein the interaction manager accepts standardized interaction modeling input events representing captured user actions and outputs standardized output interaction modeling events representing instructed agent actions using an interaction categorization scheme. A system according to any one of claims 9 to 12, wherein the interaction manager is independent of one or more sensory servers that translate user inputs into standardized interaction modeling input events, and of one or more action servers that execute commands represented by standardized interaction modeling output events issued by the interaction manager. The system of any one of claims 9 to 13, wherein the one or more processors are further operable to transmit, through the interaction manager, to one or more event gateways the one or more second interaction modeling events instructing the interactive agent to perform the one or more addressable agent actions or the one or more addressable scene actions. System according to one of claims 9 to 14, wherein the interaction manager implements an interpreter as an event-driven component that supports the execution of one or more interaction flows created in an interaction modeling language and representing one or more flows of one or more human-machine interactions. The system of any of claims 9 to 15, wherein the system consists of at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D goods; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or displaying one or more augmented reality, virtual reality, or mixed reality contents; a system implemented with an edge device; a system implemented with the aid of a robot;a system for performing conversational AI operations; a system for implementing one or more language models; a system for implementing one or more large language models (LLMs); a system for implementing one or more visual language models (VLMs); a system for implementing one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources. A method comprising:receiving one or more first interaction modeling events representing at least one of: one or more detected user actions, one or more instructed agent actions, or one or more instructed scene actions; andgenerating one or more second interaction modeling events commanding an interactive agent to perform at least one of: one or more addressable agent actions or one or more addressable scene actions based on at least processing the one or more first interaction modeling events. The method of claim 17, further comprising generating the one or more first interaction modeling events representing the one or more captured user actions based on at least translating one or more user inputs into one or more standardized formats. The method of claim 17 or 18, further comprising executing the one or more addressable agent actions or the one or more addressable scene actions in one or more corresponding output interaction channels. A method according to any one of claims 17 to 19, wherein the method is performed by at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D goods; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or displaying one or more augmented reality, virtual reality, or mixed reality contents; a system implemented with an edge device; a system implemented with the aid of a robot;a system for performing conversational AI operations; a system for implementing one or more language models; a system for implementing one or more large language models (LLMs); a system for implementing one or more visual language models (VLMs); a system for implementing one or more multimodal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Images