MULTIMODAL HUMAN-MACHINE INTERACTIONS FOR INTERACTIVE SYSTEMS AND APPLICATIONS

Abstract

In various examples, an interactive agent platform hosting an interactive agent may execute one or more flows that implement the logic of the interactive agent and specify a sequence of multimodal interactions. For example, an interactive avatar may support any number of simultaneous interaction modalities and corresponding interaction channels to engage with the user, such as channels for character or bot actions (e.g., speech, gestures, posture, movement, voice breaks, etc.), scene actions (e.g., two-dimensional (2D) GUI overlays, 3D scene interactions, visual effects, music, etc.), and user actions (e.g., speech, gestures, posture, movement, etc.). Actions relying on different modalities may occur sequentially or in parallel (e.g., waving and greeting).The interactive agent can therefore execute any number of flows that specify a sequence of multimodal actions (e.g., different types of bot or user actions) using any number of supported interaction modalities and corresponding interaction channels.

Description

BACKGROUNDCall-carrying artificial intelligence (AI) allows computers to sustain themselves with users in natural language, thus enabling human-like interactions and communication. Multi-mode conversational AI integrates multiple types of communication (e.g., text, speech, gestures, emotions, visual elements, etc.) and thus allows for more comprehensive and natural interaction between users and AI systems. Multi-mode call-carrying AI is being integrated into more and more applications, from simple chat messages via voice messages to fully interactive avatars (also known as digital characters or digital avatars) and robots. However, it is a difficult task to design these applications so that they can interact flexibly and robust with the user.First, the design of convincing avatar interactions is a rather challenging challenge. Interactions with an avatar are increasingly perceived as interactions with another person, but interactions with other humans may be complex, nuancing, multimodal and non-sequential. Since we have communicated humans our full lives with other humans, we can often quickly recognize when there is a feeling of incongruity, discomfort or incongruity in a conversation or when our opposite responds in an unusual or unnatural manner. Such nuances are not easy to model, and the defects in modeling human interactions may become more pronounced the closer they are to reality. A similar effect is known as "false valley" in three-dimensional (3D) graphics, where 3D models of humans, while very lifelike, nevertheless have subtilous imperfections or discrepancies from real humans, may be unheimious and cause discomfort or discomfort.To clarify the complexity of many of these design challenges, one should look ahead of the eye which would be necessary to make a chatbot that can interact only through a loop-based text dialog into a multi-modal interactive avatar that can be seen and spoken with. The conversion from a single interaction modality (e.g., text dialog) to the support of multiple user input interaction modalities (e.g., text, touch, speech, gestures, emotions, etc.) and / or the support of multiple output interaction modalities to respond to the user (e.g., by text / voice, graphical user interface (GUI), animations, sound effects, lights, etc.) greatly increases complexity. Transitioning from a loop-based interactive system to a system that supports non-sequential interactions (e.g., multiple concurrent, potentially higher level inputs and / or outputs) further increases complexity. In many cases, interactive systems that provide only a single interaction modality or use loop-based interactions simply do not work for multi-modal and / or non-sequential interactive systems.In some interactive systems, such as those that provide interactive avatars, it may be desirable to support voice input and output and also to use space on the screen by displaying dynamic information on the screen and allowing the user to interact with that information. Thus, it may be desirable to dynamically adapt the visual representation on the screen to the content of the conversation to provide useful contextual information (e.g., by displaying a visual representation of some options that an avatar verbally offers to the user). Present day AI models are tailored to process verbal input and output (e.g., speech in the form of text), but do not have the ability to directly generate corresponding visual elements or graphical user interfaces. This is only an example of conventional tools being limited in their ability to support multimodal interactions.Moreover, there are many different forms of AI systems that provide multi-modal conversation experiences, and different systems rely on a variety of different technologies. That is, most interactive systems connect their individual components (e.g., a decision unit, AI models such as deep neural networks (DNNs), and machine learning models, cameras, user interfaces, etc.) in an application-specific manner by using customized application programming interfaces (APIs) and architectures customized to each individual interactive system. Nowadays, there is a lot of toolkits and frameworks for modeling conversational interactions, and many different applications are developed based on these technologies. As a result, components cannot be easily replaced or updated based on latest technologies, which extends the time from research to product. Moreover, heterogeneous systems that present multi-modal interactions in different ways complicate training AI models from historical multi-modal interactions, thereby limiting the ability to improve the user experience over time. Moreover, in many systems, the interaction data is closely associated with the specific implementation of the interactive system. For example, the specific format that a particular interactive system uses to encode or display interaction data (e.g., how a human is speaking with a bot) typically depends on the particular implementation. This makes it difficult to consult multimodal interactions without knowing the technical details of a particular interactive system, which limits the possibility of using existing frameworks or extending existing technologies.Thus, there is a need for improved systems that enable and support the development and / or deployment of multi-modal interactive systems.SUMMARYThe invention is defined by the claims. To illustrate the invention, aspects and embodiments are described herein which may or may not fall within the scope of the claims.Various examples are presented where an interactive agent platform hosting an interactive agent may perform one or more operations implementing the interactive agent logic and defining a sequence of multimodal interactions. For example, an interactive avatar may support any number of simultaneous interaction modalities and corresponding interaction channels to contact the user, such as channels for character or bot actions (e.g., speech, gestures, posture, motion, voice breaks, etc.), scene actions (e.g., two-dimensional (2D) GUI overlays, 3D scene interactions, visual effects, music, etc.), and user actions (e.g., speech, gestures, posture, motion, etc.). Actions based on different modalities may occur sequentially or in parallel (e.g., swabbing and welcoming). Thus, the interactive agent may perform any number of sequences that specify a sequence of multi-modal actions (e.g., different types of bot or user actions) using any number of supported interaction modalities and corresponding interaction channels.Embodiments of the present disclosure relate to the development and deployment of interactive systems, such as those implementing an interactive agent (e.g., bot, avatar, digital human, or robot). For example, systems and methods are disclosed that implement or support an interaction modeling language and / or an interaction modeling application programming interface (API) that uses a standardized interaction categorization scheme, multimodal human-machine interactions, back-channelization, an event driven architecture, management of interaction flows, provisioning using one or more large language models, sensory processing and action execution, interactive visual content, interactive agent (e.g., bot) animations, expected actions and signaling, and / or other features.For example, an interactive agent platform hosting development and / or deployment of an interactive agent (e.g., a bot or robot) may provide an interpreter or compiler that interprets or executes code written in the interaction modeling language, and a designer may provide customized code written in the interaction modeling language that the interpreter is to execute. The interaction modelling language may be used to define a sequence of interactions that instruct the interpreter (e.g., an event driven state machine) which actions or events are to be generated in response to a sequence of recognized and / or executed human-machine interactions. The interaction categorization scheme may classify interactions according to standardized interaction modalities (e.g., BotUpperBodyMotion) and / or corresponding standardized action categories or types (e.g., BotPose, Bot Gesture) using standardized action keywords, and the interaction modeling language may use keywords, commands, and / or syntax that incorporate or categorize standardized modalities, action types, and / or an event syntax defined by the interaction categorization scheme. Thus, a procedure may be used to model a bot intent or a derived user intent that a designer may use to create more complex interaction patterns with the interactive agent.In some embodiments, one or more flows may implement the logic of an interactive agent and specify a sequence of multi-modal interactions. For example, an interactive avatar (e.g., an animated digital character) or other bot may support any number of simultaneous interaction modalities and corresponding interaction channels to contact the user, such as channels for character or bot actions (e.g., speech, gestures, posture, motion, voice breaks, etc.), scene actions (e.g., two-dimensional (2D) GUI overlays, 3D scene interactions, visual effects, music, etc.), and user actions (e.g., speech, gestures, posture, motion, etc.). Actions based on different modalities may occur sequentially or in parallel (e.g., swabbing and welcoming). Thus, the interactive agent may perform any number of sequences that specify a sequence of multi-modal actions (e.g., different types of bot or user actions) using any number of supported interaction modalities and corresponding interaction channels.In some embodiments, in order for calls to feel more natural with an avatar or other interactive agent, back channelization is employed to provide feedback to the user while he is speaking or doing something recognizable. For example, backhauling may be implemented by triggering interactive agent postures (e.g., based on the user or avatar speaking or based on the avatar waiting for a response of the user), such as postures (e.g., when an interactive avatar substantially reflects the posture of the user), short voiced breaks such as "yes", "aha", or "hmm" while the user is speaking (e.g., to signal to the user that the interactive agent is listening), gestures (e.g., shaking the head of the interactive bot or robot), and / or others. Thus, a designer may specify various back channel techniques that make calls with an interactive agent more natural.In some embodiments, a platform hosting development and / or deployment of an interactive system may represent and / or communicate human-machine interactions and related events using a standardized interaction modelling API, a plug-in, and / or an event driven architecture. In an example implementation, a standardized interaction modelling API serves as a common protocol in which components of the interactive system use a standardized interaction categorization scheme to represent all activities of bots and users as actions in a standardized form, represent states of multimodal actions of bots and users as events in a standardized form, implement standardized mutually exclusive modalities that define how conflicts between actions are resolved in standardized action categories (e.g., say two things simultaneously). e.g., it is not possible to say two things simultaneously while it may be possible to say and make a gesture simultaneously), and / or implement standardized protocols for any number of standardized modalities and actions independent of the implementation.In some embodiments, an interpreter for an interactive agent may be programmed to pass through one or more flows until it reaches an event matcher, an top-level flow may indicate instructions to activate any number of flows with any number of event matchers, the interpreter may use any suitable data structure to track active flows and corresponding event matchers (e.g., the interpreter may use an event driven state machine that waits for various events and triggers corresponding actions specified in corresponding flows (with event matchers corresponding to an incoming interaction modelling API event). As such, the interpreter may execute a primary processing loop that processes incoming interaction modelling API events and generates outgoing interaction modelling API events that implement the interactive agent.In some embodiments, an interaction modeling language and corresponding interpreter may support the use of natural language descriptions and one or more language models (e.g., large language models (LLMs), image language models (VLMs), multi-modal language models, etc.) to reduce programmer cognitive burden and facilitate the development and provision of more complex and nuancing human-machine interactions. For example, the interpreter may analyze one or more specified procedures (e.g., at design time) defining the logic of an interactive agent and determine whether a corresponding procedure description is missing from the specified procedures and, if so, request a language model to create such a language model based on the name and / or instructions of the procedure. Additionally or alternatively, the interpreter may determine whether a sequence of instructions is missing in certain sequences and, if so, request a language model to generate such. In some embodiments, the interpreter may determine whether an event matches an active event matcher for any of the active flows by using target event parameters and / or parameter values generated using the language model, request the language model to determine whether an event matches the flow description, request the language model to determine whether a non-matching event matches the name and / or instruction(s) of an active flow, request the language model to generate a flow in response to a non-matching event, and / or otherwise.In general, an interactive agent platform that hosts the development and / or deployment of an interactive agent (e.g., chatbot, voicebot, digital assistant, interactive avatar, non-player character (NPC), digital human, robot, etc.) may support any number of input and output interaction channels. In some embodiments where sensory processing, interaction decision, and action execution are decoupled, the interactive agent platform may support a sensory server for each input interaction channel and an action server for each output interaction channel. Sensory(s) servers for respective input interaction channels may translate input or non-standardized technical events into the standardized format and generate respective interaction modelling API events, an interaction manager may process these input interaction modelling API events and generate output interaction modelling API events representing commands for execution of an action, and action servers for respective output interaction channels may interpret these output interaction modelling API events and execute the respective commands. Sensor and / or action servers may be implemented with a combination of asynchronous event loops and processes to ensure that multiple user sessions and system pipelines may be served in parallel. In order to be able to handle all supported actions for at least one interaction modality, an action server may be equipped with an action processor for each standardized category or type of action and / or action event supported by the interaction modelling language and / or defined by the interaction categorization scheme for a particular interaction modality. Each action server may manage the life cycle of all actions in its area of responsibility and synchronize changes in action state with certain conditions (e.g., wait for a start of an action until a previous action of the same modality is completed, match the completion of two different actions in different modalities, match the start of an action with the end of another action, etc.).In some embodiments, an interactive agent platform hosting development and / or deployment of an interactive agent may use a graphical user interface (GUI) or, generally, a UI service to perform interactive visual content actions and generate corresponding GUIs. For example, an interaction modelling API may use a standardized interaction categorization scheme defining a standardized format (e.g., standardized and semantically meaningful keywords) for indicating events associated with interactive visual content actions (e.g., actions instructing superposition or other arrangement of visual content complementing conversation with an interactive agent), such as a visual information scene (e.g., displaying non-interactive content such as images, text, and videos adjacent to the interaction), Visual selections (e.g., presentation of a visual selection to the user in the form of multiple buttons or an option list) and / or visual forms (e.g., presentation of a web visual form to the user for input of user information). A sensor server may convert detected interactions with GUI interaction elements into standardized interaction modelling API events representing possible interactions with these elements in a standardized format. Standardized interaction modelling API events can be processed by an interpreter that implements the logic of the interactive agent to generate outgoing interaction modelling API events containing commands for updating the graphical user interface. An action server implementing a GUI service may translate a standardized representation of a particular GUI specified by a particular interaction modelling API event into a representation (e.g., JavaScript Object Notation (JSON)) of a modular GUI configuration defining blocks of visual content specified or otherwise represented by the interaction modelling API event, such as paragraphs, images, buttons, multiple selection fields, and / or other types. The GUI service may use these blocks to fill a visual layout (e.g., template or shell) for a GUI overlay (e.g., a HyperText Markup Language (HTML) page that may be rendered in a web browser) with visual content specified by the interaction modelling API event. Thus, a visual layout representing a graphical user interface specified by an interaction modelling API event may be generated and presented to the user (e.g., via a user interface server).In some embodiments, interaction modelling API events that specify commands to execute bot expressions, poses, gestures, or other interactions or movements may be generated (e.g., by an interpreter executing code written in an interaction modelling language) and translated into corresponding bot animations. More specifically, an interpreter implementing interactive agent logic may generate an interaction modeling API event representing a target bot expression, pose, gesture, or other interaction or motion using a standardized interaction categorization scheme, and an action server implementing an animation service may use the standardized representation of the target bot motion to identify a corresponding supported animation or generate a matching animation in operation. The animation service may implement an action state machine and an action stack for all events related to a particular interaction modality or category of action (e.g., bot gestures), connect to an animation graph that implements a state machine of animation states and transitions between animations, and instruct the animation graph to set a corresponding state variable based on a command to change the state (e.g., initialize, stop, or resume) of a bot motion represented by an interaction modelling API event.In some embodiments, an interpreter connected to an interactive agent may generate interaction modelling API events that communicate an expectation that certain events will occur and command or otherwise trigger appropriate preparatory actions, such as turning down speaker volume in expectation of user speech, enabling computer vision and / or machine learning algorithms in expectation of vision events, and / or signaling the user that the interactive agent is waiting for input (e.g., for a certain user interaction modality). Interaction modelling API events may include one or more fields representing an expectation that a particular target event will occur using a standardized interaction categorization scheme that identifies expectations as being a supported action type (e.g., ExpectationTimeAction, ExpectationTimeAction) and represents corresponding expectation events (e.g., specifying the state of expectation such as start, stop, and end), expected target events (e.g., UtteranceNutzerActionTime), and / or expected input interaction modalities (e.g., UserSprache) using standardized (e.g., natural language, semantic meaning) keywords and / or commands.Thus, the present techniques may be used to develop and / or utilize interactive bots or robots (e.g., chatbotes, voicebotes, digital assistants, interactive avatars, non-player characters (NPCs), digital people, etc.) that perform more complex, shade, multimodal, non-sequential, and / or realistic conversation AIs and / or other types of human-machine interactions than previous techniques.The disclosure extends to all novel aspects or features described and / or illustrated herein.Further features of the disclosure are characterized by the independent and dependent claims.Each feature of an aspect of the disclosure may be applied to other aspects of the disclosure in any suitable combination. In particular, method aspects may be applied to device or system aspects, and vice versa.In addition, functions implemented in hardware may also be implemented in software, and vice versa. Any reference to software and hardware features in this document is to be interpreted accordingly.Any system or device feature described herein may also be provided as a method feature, and vice versa. System and / or device aspects that are functionally described (including means-plus features) may alternatively be expressed in terms of their corresponding structure, e.g., by a correspondingly programmed processor and associated memory.It should also be appreciated that certain combinations of the various features described and defined in all aspects of the disclosure may be implemented and / or provided and / or used independently of each other.The disclosure also provides computer programs and computer program products comprising software code, which when executed on a data processing apparatus, is adapted to perform any of the methods described herein and / or embody any of the apparatus and system features described herein, including all or part of the component steps of a method.The disclosure also provides a computer or computer system (including networked or distributed systems) having an operating system that supports a computer program for performing any of the methods described herein and / or for embodying any of the apparatus or system features described herein.The disclosure also provides a computer readable medium having stored thereon one or more of the aforementioned computer programs.The disclosure also provides a signal carrying one or more of the above mentioned computer programs.The disclosure extends to methods and / or apparatuses and / or systems as described herein with reference to the accompanying drawings.Aspects and embodiments of the disclosure will now be described, purely by way of example, with reference to the accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGSThe present systems and methods for the development and use of interactive systems are described in detail below with reference to the accompanying figures, wherein: FIG. 1 shows an example of an interactive system that corresponds to some embodiments of the present disclosure; FIG. 2 shows an example of an interaction modelling API according to some embodiments of the present disclosure; FIG. 3 illustrates an example interactive systems that may be supported by an example interaction modelling API and / or an example interaction modelling language, in accordance with some embodiments of the present disclosure; FIG. 4 shows an example of modality policies in accordance with some embodiments of the present disclosure; FIG. 5 illustrates an example interaction categorization scheme in accordance with some embodiments of the present disclosure; FIG. 6 illustrates an example of an event driven interactive system in accordance with some embodiments of the present disclosure; FIG. 7 shows an example of an interaction manager, in accordance with some embodiments of the present disclosure; FIG. 8 is a flow diagram illustrating an example event driven state machine for an interaction manager in accordance with some embodiments of the present disclosure; FIG. 9 shows an example of an action server according to some embodiments of the present disclosure; FIG. 10 illustrates an example event flow through an example action server in accordance with some embodiments of the present disclosure; FIG. 11 shows an example of the life cycle of an action in accordance with some embodiments of the present disclosure; FIGS. 12A-12D illustrate some example action processors for an example GUI service and an example animation service, in accordance with some embodiments of the present disclosure; FIGS. 13A-13F illustrate some example interactions with visual selection in accordance with some embodiments of the present disclosure; FIG. 14A illustrates an example graphical user interface illustrating an interactive avatar and interactive visual content, and FIGS. 14B-14L show some example layouts of visual elements for interactive visual content, in accordance with some embodiments of the present disclosure; FIG. 15 illustrates an example event flow for a user utterance action in an implementation in which a user is speaking with an interactive avatar, in accordance with some embodiments of the present disclosure; FIG. 16 illustrates an example event flow for a user utterance action in an implementation in which a user is speaking with a chatbot, in accordance with some embodiments of the present disclosure; FIG. 17 illustrates an example event flow for a bot expectation action in an implementation in which a user is speaking with an interactive avatar, in accordance with some embodiments of the present disclosure; FIG. 18 is a flow diagram illustrating a method for generating a representation of a responding agent's action categorized using an interaction categorization scheme, in accordance with some embodiments of the present disclosure; FIG. 19 is a flow diagram illustrating a method for generating a responsive agent action representation based at least on execution of one or more interaction flows, in accordance with some embodiments of the present disclosure; FIG. 20 is a flow diagram illustrating a method for triggering an interactive avatar to provide back channel feedback in accordance with some embodiments of the present disclosure; FIG. 21 is a flow diagram illustrating a method for generating an interaction modeling event that instructs an interactive agent to execute a responsive agent or scene action, in accordance with some embodiments of the present disclosure; FIG. 22 is a flow diagram illustrating a method for triggering one or more responsive agent or scene actions specified by one or more matching interaction flows, in accordance with some embodiments of the present disclosure; FIG. 23 is a flow diagram illustrating a method for generating a responsive agent or scene action based at least on prompting one or more big language models, in accordance with some embodiments of the present disclosure; FIG. 24 is a flow diagram illustrating a method for generating one or more outbound interaction modelling events that direct one or more action servers to execute one or more responsive agent or scene actions, in accordance with some embodiments of the present disclosure; FIG. 25 is a flow diagram illustrating a method for generating a visual layout representing an update specified by an event, in accordance with some embodiments of the present disclosure; FIG. 26 is a flow diagram illustrating a method for triggering an animation state of an interactive agent in accordance with some embodiments of the present disclosure; FIG. 27 is a flow chart illustrating a method of executing one or more preparatory actions in accordance with some embodiments of the present disclosure; FIG. 28A is a block diagram of an example generative language model system suitable for use in implementing at least some embodiments of the present disclosure; FIG. 28B is a block diagram of an example generative language model including a transformer encoder decoder suitable for use in implementing at least some embodiments of the present disclosure; FIG. 28C is a block diagram of an example generative language model that includes a decoder-only transformer architecture suitable for use in implementing at least some embodiments of the present disclosure; FIG. 29 is a block diagram of an example system for streaming content suitable for implementing some embodiments of the present disclosure; FIG. 30 is a block diagram of an example computing device suitable for use in implementing some embodiments of the present disclosure; and FIG. 31 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure.DETAILED DESCRIPTIONSystems and methods are disclosed that relate to the development and deployment of interactive systems, e.g., those that implement an interactive agent (e.g., bot, non-player character, digital avatar, digital human, robot, etc.). For example, systems and methods are disclosed that implement or support an interaction modeling language and / or an interaction modeling API that utilize a standardized interaction categorization scheme, multi-modal human-machine interactions, back-channelization, an event driven architecture, management of interaction flows, provisioning using one or more language models (e.g., LLMs, VLMs, multi-modal language models, etc.), sensory processing and action execution, interactive visual content, interactive agent animations, expected actions and signaling, and / or other features.Introduction. At a high level, an interactive agent (e.g., chatbot, voicebot, digital assistant, interactive avatar, non-player character, robot, etc.) that participates in AI conversations or other types of human-machine interactions may be created and / or executed using an interactive agent platform. In designing such a platform and / or interactive system implementing an interactive agent, it may be instructive to take into account some possible features that may contribute to convincing human-machine interactions and interaction flows.Multi-modality is a factor that can contribute to convincing human-machine interactions. For example, in the design of an interactive avatar, a designer may want to support many different modalities of interaction with the user. A designer may want his avatar to speak, make gestures, show something in a graphical user interface, give sounds, or otherwise interact. Likewise, a designer may wish to support various types of input interaction modalities or ways of interaction of a user with the system. For example, a designer may assist recognition and response when a user verbally answers a question, selects an element on the screen, or makes a gesture like the thumb high to confirm selection. One possible impact of the multimode is that a designer desires flexibility in the manner in which interactions are temporarily aligned. For example, a designer may want to say that an avatar is speaking something while making a gesture, or may want to trigger a gesture at a certain time when the avatar is speaking something. Therefore, it may be desirable to support different types of independently controllable interaction modalities.Backcanalization is a useful instrument for promoting effective human communication. It helps mediate active listening and engagement and signals the speaker that his message belongs and is understood. This feedback loop makes conversation smoother, helps establish a connection, and encourage people to talk further and communicate their thoughts. If a designer wants his avatar to attempt to use return channels to make the avatar appear more human and allow for a more natural interaction, it may be desirable to support return channels.Some designers may wish to support non-linear interactions. Designers often attempt to avoid the impression of predictable, guided, or simplified interactions that may give the user the feel to follow a predetermined or fixed course that lacks spontaneity or freedom. Even if a desired customer experience inherently involves some level of linearity, it may be desirable to support interactions such that users may abort from stringent logic.Proactivity can be a helpful function to implement. Many users are now accustomed to voice assistants, but the conversation pattern with these digital assistants is often very simple. The user starts a conversation with a stimulus word and makes a question or gives a command. The voice assistant responds to this prompt by directly performing the action, answering the question, or hooking with a clearing question. This interaction pattern may be efficient when it is a matter of retrieving information or setting a timer, but is not very appealing and often unsuitable for more complex use cases. Instead, the avatars should be proactive and reword questions if the user does not understand them, return them to a particular process when he breaks the call, or offer alternative ways to accomplish a particular task. Proactivity may be very helpful in preventing the interaction from becoming personal and the users from getting with or knowing how they should continue in entertainment.Some designers may wish to utilize the performance of language models (e.g., LLMs, VLMs, etc.). For example, a designer might want an avatar or chatbot to use an LLM to make their interactions with the user more natural and adapt to the current interaction context. Some LLM uses could help avoid frequent fall-knitting in avatar or chatbot experiences, e.g., when a bot repeats the same responses again and again or when simple questions do not cause the intended response. In an avatar interactive environment, a designer may want to use LLMs to generate verbal and / or nonverbal responses such as gestures or facial expressions, or even may use LLMs to provide useful information on the user interface. As such, it may be desirable to support a variety of LLM uses.Interaction Modelling Language and Interaction Categorization Scheme. In general, human-machine interactions and related events within an interactive system or an interactive agent platform hosting development and / or deployment of an interactive system may be presented and communicated in a variety of ways.One way to present and / or communicate interactions is to use an interaction modeling language that specifies user and / or bot interactions and associated events using a standardized interaction categorization scheme. Existing dialog management techniques such as flowcharts, state machines, and frame-based systems are unable to model highly flexible conversation procedures that can be expected from a realistic interactive avatar. In contrast, a standardized interaction categorization scheme may provide a semantically reasonable way to categorize, specify, and communicate desired interactions and interaction flows. For example, an interactive agent platform may provide an interpreter or compiler that interprets or executes code written in the interaction modeling language, and a designer may provide user-defined code written in the interaction modeling language to the interpreter for execution. An interaction modeling language that uses a standardized categorization scheme for interactions offers a number of technological advantages, from facilitating the designer's work by reducing cognitive burden on developing an interactive system via supporting a variety of interactions or functions (such as those described above) that a designer can resort to to to adapt an interactive system, to promoting interoperability by standardization of the representation of interactions.One possible goal is to reduce the cognitive burden on a developer in writing code implementing an interactive system. Existing programming languages would require the developer to write functions that implement interactions with generic keywords and instructions. However, some embodiments abstract lower level programming in favor of a more semantically intuitive representation of an interaction: an interaction flow. Interactions typically occur in flows so that an interaction modelling language can be used to define a flow of interactions. A flow may be viewed analogous to a function, but may consist of primitives that include semantically meaningful (e.g., natural language) keywords and commands that specify events (e.g., something is passed) and actions (e.g., something must pass) using an interaction categorization scheme. An interaction flow may be used as a mechanism to instruct an interpreter (e.g., an event driven state machine) which actions or events should be generated in response to a sequence of detected and / or executed human-machine interactions.In some embodiments, the interaction categorization scheme may classify interactions according to standardized interaction modality (e.g., BotUpperBodyMo) and / or corresponding standardized action category or type (e.g., BotPose, Bot Gesture) using standardized action keywords. The scheme may support any number and type of interactions or communication methods (e.g., user interactions with the system, bot interactions with the user, bot expectation actions and expectation signaling, scene actions, etc.). The state of an action (e.g., an observed state of a user action, a current state of a bot or scene action) and / or a command to change the state of a bot or scene action may be represented using standardized event keywords, commands, and / or syntax. For example, action events (e.g., a started or stopped user or bot action) may be represented by a standardized syntax event specifier (e.g., event name and / or identifier having a keyword identifying the standardized category or type of action and a user or bot action state specifier).In some embodiments, the interaction modeling language may use keywords, commands, and / or syntax that involve or categorize standardized modalities, action types, and / or event syntax defined by an interaction categorization scheme. For example, an instruction line in a flow may include an event trigger (e.g., using a keyword such as "send") that causes the interpreter to generate a particular event when a particular condition is met (e.g., events representing instructions to execute a bot action may trigger the action to be executed or events representing a change in state of a detected user action may trigger a corresponding bot action), or an event balancer (e.g., using a keyword such as "match") that causes the interpreter to interrupt the flow and wait for a particular event before resuming the flow. The event trigger and event match may specify corresponding trigger and match conditions using an event specifier that includes a standardized event name or identifier (e.g., a keyword identifying a standardized category or type of action paired with a corresponding action state specifier or an action state change command) and an argument that specifies one or more conditions that the specified event must satisfy (e.g., using predefined parameters and supported values or a natural language description). In some embodiments, the interpreter may derive (e.g., "Completed") a particular action status when an event specifier includes an action but does not indicate the status (e.g., the name of the action may be indicated as an abbreviation to specify the completion of the action).Take as an example the UserSprache modality and the associated UserAction utterance. Suppose a user makes an utterance recognized by an interactive system. Possible examples of this type of action are the user's input into a text interface for interaction with a bot or the user's speaking with an interactive avatar. This action can be classified as a user utterance, and the supported action events for this action can include UtteranceNutzerActionGestartet (the user has started producing an utterance) or UtteranceNutzerActionEnd (the user utterance has been ended). An example of a sequence instruction waiting for the user to tell something about could be: "match ÄußerungNutzerAktionBeended(text="How are you?", speed="slow"volume="normal")". In this example, the event identifier is a keyword in lock caption that associates the standardized action category (UtteranceNutzerAction) with a representation of the specified action status (Ended).In some embodiments, an interaction modeling language and associated interpreter may support any number of keywords that serve to parallelize execution and matching actions and procedures (e.g., send, match, start, stop, await, activate). In contrast to conventional dialog modeling languages in which instructions are always viewed in a sequential order, some embodiments may support a keyword (e.g., launch) that instructs the interpreter to launch a particular action in a particular (e.g., standardized) action category or sequence and continue to iteration the parent sequence without waiting for the action or sequence that was launched to be completed, some may support a keyword (e.g., some may support a keyword (e.g., stop) that instructs the interpreter to stop the launched action or sequence, and some may support a keyword (e.g., await) that instructs the interpreter, waiting for the end of the started action or of the subordinate sequence before it continues the superordinate sequence. In some embodiments, the interpreter may implement some keywords (e.g., start, await) with other keywords (e.g., send, match) to send or wait for an event to occur. In some implementations, after starting a flow, the interpreter performs all actions in the specified flow until the first matching instruction. Subsequently, if the instruction matches, the interpreter may execute the subsequent actions in the specified flow until the next matching instruction or the end of the flow and repeat this until the flow is completed.In some scenarios, a developer may wish to automatically restart subordinate flow after completion. This may be useful for certain types of procedures, such as those that attempt to trigger an action that depends on a recurrent event. Thus, some embodiments may support a keyword (e.g., activate) that instructs the interpreter to automatically restart execution after it is completed. In some embodiments, if an activated flow does not contain event matching, the interpreter executes the flow only once, but maintains it in an active state, so all subordinate flows also remain active.Some embodiments may support a keyword that instructs the interpreter to end (e.g., return) or abort (e.g., abort) a flow, and a flow may instruct the interpreter to determine and return a value. Because some embodiments support multiple active flows, some implementations of the interpreter launch a top-level, root, or main flow (e.g., at launch) that serves as a parent of all other flows. This hierarchy allows for better abstraction and encapsulation capabilities than previous techniques. In some embodiments, the event match command may accept a particular name or identifier of a sequence and a particular sequence event (e.g., start, end, miss, interrupt, resume) as an argument that the interpreter may use as an instruction to match a corresponding sequence event.In some embodiments, therefore, all sequences represent a corresponding interaction pattern. In some of these embodiments, a process may be used to model a bot intent or a derived user intent that a designer may use to create more complex interaction patterns. In some of these implementations, a flow effectively describes an expected interaction pattern. When the interpreter starts a process, it may flag this process as active and attempt to match the contained pattern of event matching instructions with events representing the current interaction. If the interpreter determines that a matching instruction is satisfied by an event, it can pass a corresponding flow header to the next matching instruction and execute all intervening non-matching instructions. The interpreter may be programmed to execute the instructions specified in a flow sequentially, generate all events specified by an event trigger, and stop when a flow head reaches an event matcher, exception, or end of a flow. To illustrate how a flow may be used to implement various types of interaction patterns and functions, consider the following application examples.Multimodal interactions. In some embodiments, one or more flows may specify a sequence of multimodal interactions. While conventional chatbotes use loop-based dialog, an interactive avatar (e.g., an animated digital character) or other bot may support any number of interaction modalities and corresponding interaction channels to contact the user, such as channels for character or bot actions (e.g., speech, gestures, posture, motion, voice breaks, etc.), scene actions (e.g., two-dimensional (2D) GUI overlays, 3D scene interactions, visual effects, music, etc.), and user actions (e.g., speech, gestures, posture, motion, etc.). Thus, a flow may specify a sequence of multi-modal actions (e.g., different types of bot or user actions) using any number of supported interaction modalities and corresponding interaction channels.For example, consider the following example flow that includes an action command to start a bot utterance to facilitate readability and programming: Flow-Bot says $Aussage Send BotÄußerungStartAktion(Aktion_id "$Ä") match BotÄußerungAktionBeended(Aktion_id)The command to start the bot utterance action can now be triggered with an instruction that simply writes its name and indicates the desired utterance: bot say "Hello world". In this example, when the interpreter executes this instruction, it searches for the flow "bot say" defining an event trigger that generates an event that, when executed, starts the bot utterance action with the specified text (in this case, "Hello World"), and an event matcher that waits for the end of the bot utterance action. Here, an example flow that similarly includes a command to start a bot gesture action is: Flow Bot Gesture $ Gesture BotGesteStartAktion(Aktion_id "$ gesture") Match with BotGesteAktionBeended(Aktion_id)By defining such a wrapper flow, a designer may cause the bot to show a gesture (e.g., trigger the action command "start bot gesture") by simply writing the name of the wrapper flow and indicating the desired gesture, e.g., the bot gesture "roll with both hands.".Conceptually, actions based on different modalities may be performed sequentially or in parallel (e.g., say Hamen and Hello). Therefore, it may be desirable to allow the designer to accurately time control the supported actions and their mutual coordination. For example, take bot actions such as bot utterances and bot gestures. In some embodiments, a flow may specify the invocation of these actions in sequence as follows: Bot says "Hello World" Bot gesture "Wake with both hands"In this example, since the wrapper flow for the action "Let Say" includes an event matcher that instructs the interpreter to wait for the end of the action before continuing the flow, the bot gesture does not start until the action "Let Say" (or "Let Wait Bot") is completed.Since these two actions are two different modalities, they may be performed simultaneously in some embodiments. One way to trigger the simultaneous execution of these two actions is to combine them into an "and" group (e.g., defined by a keyword like "and") to start them in parallel: Bot says "Hello World" and Bot gesture "Sleep with both hands"More complex action groups may be defined with an "or" group (e.g., by a keyword such as "or"), as in the following example: Bot says "Hello World" and (Bot Gesture "Winken with both hands" or Bot Gesture "Smile")Thus, a designer may use an "or" action group to indicate alternative actions. In this example, the resulting action is either that the bot says "Hello World" and wins hands or that it says and becomes smudging "Hello World". Another example of performing two actions in parallel is using a keyword such as "start" to start two actions in parallel: Bot start and "Hello World" say Bot gesture "Roll on both hands" startIn some implementations of these examples, the interpreter does not wait for the end of an action before continuing with the next instruction. To explicitly wait for the termination of a begun action, a flow may indicate a "match" instruction to the terminated event of the previously begun action, as in the following example: Bot start and "Hello World" as $Action say matching with $Action.Finished()Using an example flow such as this, a designer may limit the lifetime of an action by associating it with the end of another action and using a keyword such as "stop" to end an action. The following example would stop the gestures of the bot when it is ready to talk. Bot start and say "Hello World" as $Action_1, and Bot gesture "Sleep with both hands" $Action_2 match $Aktion_1.Finished () Stop $Action_2The foregoing examples focused on actions initiated by a bot. However, to enable meaningful interaction with a user, it may be desirable to respond to user actions. For example, consider the following example flow that encloses an event matcher for an event that indicates that a user utterance is completed: Flow UsersLd $text Match ÄußerungNutzerAktion.Beended(final _transscript=$text)This flow wrapper (or wrapped event matcher) can now be used to direct a flow to wait for a particular user utterance: The user says "Hello!"As with some of the previous example bot actions, in some embodiments, a flow that includes an event adjuster for that user action will only proceed to the next instruction when the user says the specified parameter value ("Hello"). In some embodiments, an event matcher may trigger groups of user actions, such as the following example: the user says "Hello!", or the user says "Hello", or the user says "Hello"In some embodiments, a flow including this example of event matching waits for one of these user actions to occur before continuing with the next instruction in the flow. Operations may additionally or alternatively use "and" groups to wait for multiple user actions to occur before continuing.In some embodiments, a flow may be defined with an instruction that includes a keyword (e.g., "flow"), a name or identifier for the flow (e.g., "How to respond?" ) and a parameter (e.g., characterized by a leading $ character) for which the value may be specified and passed upon invoking the flow, as illustrated in the following example: flow How to respond to $text The user says "How to do you?" Bot says $TextIn some embodiments, each flow defines an action range. For example, if the interpreter initiates initiation of actions during a procedure and these active actions have not yet been completed at a time when the interpreter completes execution of the procedure, the interpreter may stop these active actions. Returning to the "Hello World" example: in some embodiments, the gesture action need not be stopped because it is automatically ended when the flow is completed: Flow Hello World example Bot starts and say "Hello World" as $Action_1, and Bot gesture "Sink with both hands" $Action_2 matches $Aktion_1.Finished ()Back channelization. Calls with conventional chatbotes or avatars may often feel stiff or unnatural, as they often force strict alternation behavior. In order for entertainment with an avatar to feel more natural, in some embodiments a technique called "backchanneling" is employed in which an interactive system (e.g., an interactive avatar) provides feedback to the user while the user is speaking or is doing something recognizable.One possibility for implementing back-channelization is postures. For example, a designer may put an avatar in a certain posture depending on whether the user or the avatar is speaking or when the avatar waits for a response of the user. Here, an example of a flow that can be used for implementing listener posture is: flow Bot Listening Posture while True as user began speaking Start Bot Posture "listening" as $ listing user stated some sending of $isting.Stop()In this example, "user started talking" is a flow cover for an event balancer for an event indicating the beginning of a user utterance, and "user say sonic" is a flow cover for an event balancer for an event indicating the end of a user utterance. With an example flow such as this, activated in some implementations, the avatar listens (e.g., shows a listening animation) when the user starts speaking and stops animation when the user stops speaking.Another example could include various other poses such as "talk", "attention", and / or "idle" to provide feedback to the user about the current state of the avatar, as in the following example: flow management bot postures bot pose "idle" start as $current_posture while true as the user began speaking sending $current_posture.Stop () bot pose "listen" as $current_posturestarten or as the bot was, To say, send $current_posture.Stop( ) Bot pose "talk" as $current_posturestarten or if the Bot to say $current_posture.Stop( ) Bot pose "attentive" as $current_posture start sendingIn such an example flow, activated in some implementations, the avatar assumes an idle posture until the user begins speaking (in this case, assumes a listening posture), the avatar begins speaking (in this case, assumes a speaking posture), or the avatar has just stated something (in this case, assumes an alerting posture).In some embodiments, backhauling may be implemented with short vowel sounds such as "yes", "aha", or "hmm" while the user is speaking. This can signal to the user that the avatar is listening and makes the interaction appear much more natural. In some embodiments, this effect may be enhanced by nonverbal backchanneling, where the avatar responds to certain things the user says, e.g., gestures. Here, an example of implementing back channelization using voice break and gestures is: flow Bot responds to rough things while True Users mention some Trace Bot gesture "Shake Head" and Bot says "Ohh" flow Bot responds to fine things while True Users mention some Nettes Bot gesture "Feel Good" and Bot says "Fine"In some implementations, these two sequences produce a short break-out of voice and a small gesture when the user mentions something beauty or progress. In this example, unlike the "user has something stated" flow waiting for a completed utterance, the flow "user has something stated" may be defined to match (and therefore respond to) partial descriptors of what the user is speaking during the time he is still speaking.Here, an example is that these two bot backhauling streams are used in a sequence of interactions: Bot start and respond to rough things Bot start and respond to fine things to begin bot postures Bot: "How was the day?", if the user r".*terribl|horribl|bad.*" says Bot says "I hop, it is you well?", and Bot gesture "looks like the user" else says "big like.". And has du plans for the rest of the day?" User says some Bot says "Thanke for sharing.".Here, the bot asks for the activation of the example bot return channel as the day of the user was. If the user guesses the bot that something bad or beauty has passed, the bot responds immediately with a voice output and a short animation. These are just a few examples based on an example implementation of an interpreter, and other variations may be implemented within the scope of the present disclosure. Other examples and features of possible interaction modeling languages and interaction categorization schemes are described in more detail below.Event Driven Architecture and Interaction Modelling API. In some embodiments, an interactive system development and / or deployment platform (e.g., an interactive agent platform) may represent and / or communicate human-machine interactions and related events using a standardized interaction modelling API and / or an event driven architecture. In some embodiments, the standardized interaction modelling API standardizes the manner in which components represent multimodal interactions, thus allowing a high degree of interoperability between components and the applications they use. In an example implementation, a standardized interaction modelling API serves as a common protocol in which components use a standardized interaction categorization scheme to represent all activities of bots and users as actions in a standardized form, represent states of multimodal actions of users and bots as events in a standardized form, implement standardized mutually exclusive modalities that define how conflicts between standardized categories or types of actions are resolved (e.g., it is not possible to say two things simultaneously, while it may be possible to say something and make a gesture simultaneously), and / or implement standardized protocols for any number of standardized modalities and types of actions independent of the implementation.In some embodiments, an interactive agent platform hosting development and / or deployment of an interactive system may implement an architectural pattern that separates the component(s) implementing the decision logic (e.g., the interpreter) from the component(s) executing the (e.g., multi-modal) interactions. For example, an interaction manager may implement an interpreter for an interaction modelling language as a stand-alone event driven component (e.g., an event driven state machine). The interface for the interaction manager may use a standardized interaction modelling API defining standardized forms for the presentation of action categories, instances of actions in particular action categories, events and context. Sensor servers for respective input interaction channels may translate inputs or non-standardized technical events into the standardized format and generate respective interaction modelling API events (also referred to as interaction modelling events). The interaction manager may process these incoming interaction modelling API events, determine which actions should be taken (e.g., based on code written in the interaction modelling language for execution by the interpreter), and generate outgoing interaction modelling API events (e.g., in response to an instruction in an interaction modelling language such as "send") representing commands to perform particular actions. Action servers for respective egress interaction channels may interpret these egress interaction modelling API events and execute the respective commands. Decoupling these components allows for interoperability and interoperability and thus promotes development and innovation. For example, a component can be replaced by another design or a different interaction channel can be connected without this having an effect on the functionality of the existing system.This architectural pattern and API design can provide a purely event driven, asynchronous way to handle multimodal interactions. Compared to previous solutions, in some embodiments there is no strict sequence of calls (e.g., bot calls, user calls, bot calls). Instead, participants in the interaction may simultaneously perform multi-modal interactions and independently and simultaneously respond to incoming events, thereby improving the reality of the human-machine interaction.In some embodiments using this architectural pattern, the interaction manager does not need to know which specific action servers are available within an interactive system. It may be sufficient for the interaction manager to know the supported modalities. Similarly, action and / or sensor system servers may be independent of the interaction manager. Thus, each of these components can be upgraded or replaced. As a result, one and the same platform and / or one and the same interaction manager may support different types of interactive systems, all of which may be controlled via the same API and may be exchanged and changed or adapted for a particular application. For example, one implementation could provide a text-based user interface, another a voice-based system, and a third a 2D / 3D avatar.Management of Plural Procedures. In the above examples, it has been explained how an example interpreter can be programmed to run through a particular flow until it reaches an event matcher. In some embodiments, a top-level flow may include instructions to activate any number of flows with any number of event matching agents. The interpreter may use any suitable data structure to track active flows and corresponding event matchings (e.g., using a tree or other nested flow relationship representation), and may use an event driven state machine that waits for various events and triggers corresponding actions indicated in matching flows (with event matchings corresponding to an incoming interaction modelling API event).Because procedures may specify human-machine interactions, a developer may wish to activate multiple procedures that specify conflicting interactions to be triggered under different conditions and / or multiple procedures that specify the same interaction - or different but compatible interactions - to be triggered based on the same or similar conditions. In some scenarios, multiple active movements specifying different interactions may be triggered by different conditions that may be met by the same event. Thus, the interpreter may process incoming interaction modelling API events (e.g., from a queue) sequentially and, for each event, test the event matcher specified by each active flow for a match with the event. If there is an event matcher from an active flow that matches the event (a matching flow), the interpreter may advance that flow (e.g., generate outgoing interaction modelling API events to trigger actions). If there are multiple matching procedures, the interpreter may determine whether or not the matching procedures agree with an action. If they match, the interpreter can forward both matching flows. If they do not match, the interpreter may apply the conflict resolution to determine which action should take precedence, advance the matching motion with the prioritized action, and abort the other matching(s) motion(s) (e.g., because the interaction pattern represented by these motions no longer applies). If there are no active threads that match an event, the interpreter may generate an internal event that matches and triggers a particular thread to treat non-matching or non-processed events, execute one or more non-processed event processors, and / or use another technique to treat an non-processed event. After checking for matches and passing operations, the interpreter may check the operation states for all operations that have been completed or aborted, and may stop all active operations that have been activated by those completed or aborted operations (e.g., because the interaction pattern represented by those operations should no longer apply). Thus, the interpreter may run through the events in the queue, drive the flows, operate conflict management to determine which interactions are to be performed, and generate outgoing interaction modelling API events to trigger these interactions.Thus, the interpreter can execute a primary processing loop that processes incoming interaction modelling API events and generates outgoing interaction modelling API events. Unlike simple event driven state machines, the interpreter can use a series of flow headers. A flow may be considered a program including a sequence of instructions, and the flow heads may be considered instruction pointers that pass through the instructions and indicate the current position within the respective flows. Depending on the instruction, the interpreter may pass a given execution header to the next instruction, jump to another execution referenced by a label or other execution identifier, split into multiple execution headers, merge multiple execution headers, and / or others. Thus, the interpreter can use flow headers to build and maintain a hierarchy of flow headers. If a parent sequence is suspended, interrupted or resumed in a branch of a hierarchy of sequences or sequence headers, the interpreter may suspend, interrupt or resume all the parent sequence headers of that parent sequence header or branch. In some embodiments, each flow may indicate any number of ranges that the interpreter may use to generate events that instruct a corresponding action server to limit the lifetime of launched actions and flows within the corresponding ranges.In some embodiments, progressive flows may instruct the interpreter to generate interaction modelling API events that instruct a particular action. Additionally or alternatively, proceeding flows may instruct the interpreter to generate interaction modelling API events that inform listeners that an event has occurred. The interpreter may broadcast these events and / or maintain an internal event queue, place these events into the internal event queue, and sequentially process all internal events in the internal event queue (e.g., check active flows for matches with internal events) before proceeding to processing the next incoming interaction modelling API event.Example of the Use of Language Models by Interpreters. In some embodiments, an interaction modeling language and corresponding interpreter may support the use of natural language descriptions and the use of one or more language models (e.g., LLMs, VLMs, multimodal LLMs, etc.) to reduce programmer cognitive burden and facilitate the development and provision of more complex and nuancing human-machine interactions.For example, each flow may be specified with a corresponding natural language description summarizing the interaction pattern represented by the flow. In some embodiments, the interpreter does not require a designer to specify these flow descriptions, but may, under certain circumstances, rely on a flow description (e.g., by an unknown event handler requesting an LLM to determine whether an unmatched event representing an unrecognized user intent semantically matches the natural language description of an active flow representing a target user intent). Thus, in some embodiments, the interpreter may analyze (e.g., at design time) one or more specified procedures, determine whether any specified procedures fail a corresponding procedure description, and if so, request an LLM to generate such based on the name and / or instructions of the procedure. Additionally or alternatively, the interpreter may determine (e.g., request an LLM for) whether any specified sequence descriptions are inconsistent with their corresponding sequence descriptions, and if so, request an LLM to create a new one (e.g., as a proposal or for automatic replacement) based on the name and / or instructions of the sequence.In some embodiments, a designer may specify a sequence description (e.g., a natural language description of what the sequence is to do) without the command sequence or may call a sequence on behalf without defining it. In some embodiments, the interpreter may analyze (e.g., at design time) one or more specified flows, determine whether a sequence of instructions is missing from the specified flows, and if so, request an LLM to generate such (e.g., based on the name and / or description of the flow). For example, the interpreter may provide the LLM with one or more example flows, the specified name and / or description of the flow, and a request to complete the flow based on its name and / or description. These are just a few examples of how an interpreter can call an LLM.In an example implementation, sequence instructions (e.g., including all event triggers encountered) may be executed until an event matcher is reached, whereupon a sequence may be suspended. If there are no more events, an incoming or internal event may be processed by executing the event matcher in each interrupted event and comparing the event to the target event parameters and the parameter values indicated by the event specifier of the event matcher. In general, any suitable matching technique may be used to determine whether an event matches an active event matcher for one of the active streams (e.g., comparison of target event parameters and parameter values to those of the incoming or internal event to generate a representation of whether the event is a match).In general, a designer may specify an event to be matched or triggered with a name or identifier for the event and one or more target event parameters and / or parameter values. Target event parameters and / or parameter values may be expressly specified using position or named parameters or as a natural language description (NLD) (e.g., docstring) from which the interpreter may derive the target event parameters and / or values (e.g., from a single NLD of all target event parameters and values or from an NLD of a single parameter value). Here, some example event specifiers are:• An event with named parameters and explicit values:StartÄußerungBotAktion(text="Wie goes you?", volume="whisper"speed="slow")• An event with position parameters:StartÄußerungBotAktion("Wie goes you?", "fitren", "slow")• An event with parameters indicated with an NLD value:StartÄußerungBotAktion(text="""Nachfragen as it runs"", volume="whisper", speed="slow")• An event with a single NLD parameter:StartÄußerungBotAktion("""Den users ask how they are, slow and smooth"")In some embodiments supporting event specifiers with NLDs, prior to executing an instruction (e.g., an event matcher or event trigger) containing an event specifier, the interpreter may determine (e.g., at runtime) whether the instruction contains an NLD parameter, and, if so, request an LLM to generate corresponding target event parameters and / or parameter values. As such, the interpreter may execute the instruction (e.g., an event trigger or event matcher) using the generated target event parameters and / or parameter values.Additionally or alternatively, the interpreter may request (e.g., at runtime) an LLM to determine whether an event (e.g., an interaction modelling API event) matches the flow description of an active flow. In general, an interaction modelling API event may represent a user interaction or intent, a bot interaction or intent, a scene interaction, or other type of event using a standardized interaction categorization scheme that classifies actions, action events, event parameters, and / or parameter values using standardized (e.g., natural language, semantically meaningful) keywords and / or commands. As such, the interpreter may execute an event matcher by determining whether the received action, action event, event parameters and / or parameter values of an incoming or internal event and the event specified by the event matcher match (e.g., exactly or blurred). Additionally or alternatively, the interpreter may request an LLM to determine whether the representation of an incoming or internal event matches the (e.g., specified or generated) flow description of an active flow. Depending on the implementation, an LLM may provide a more nuanced or semantic understanding of the matching than traditional express or fuzzy matching algorithms.For example, assume that the user makes a gesture that means approval, such as a thumb up, a pitch, or some informulae such as "Yes.". A designer could have written a flow that is intended to depict the scenario when the user utters consent, but provided only a few examples of spoken answers for explicit matching. In this scenario, even if there are no explicit matches, an LLM may determine that the standardized and semantically meaningful representation of a recognized user response (e.g., GesteNutzerAktionBeended("Daumen high")) is a semantic match with a sequence description such as "user presses consent off.". Here, another example is where a designer specifies a flow to match (via the flow bindings "user has made selection" and "user says") an event where the user selects option B from a list of options: multi-modal show window the flow user """user has selected show window (B).""the user has selected multi-modality" option or a user says "Zig with multi-modal show window". or user says "multi-modality" or user says "show window B" or user says "second show window" or user says r"(?i)Multimodalität"If the user selects option B through an interaction that the designer did not provide, an LLM may be used to determine whether a standardized representation of a recognized user gesture matches the flow (e.g., a natural language description of a flow, a natural language description of a parameter, a natural language description of a parameter value, etc.). In this way, a specified sequence may match multiple gestures, text responses, or other events that may not be expressly specified by the designer.In some implementations (e.g., in some embodiments where the interpreter checks the event matcher for all active (e.g., interrupted) flows and determines that there are no active flows that match the incoming or internal event), the interpreter may request (e.g., at runtime) an LLM to determine whether an incoming or internal event and / or a representation of the most recent interaction flow matches the name and / or instructions of an active flow. For example, some operations may represent target user intentions, and the interpreter may implement an event handler for an unknown user action by providing an LLM with an example interaction between the user and the bot, some possible target operations listed for target user intentions, a corresponding list of the target user intentions, the most recent interaction history, the unknown user action, and a request to the LLM to predict whether the unknown user action matches one of the target user intentions. Thus, the interpreter may use an LLM to implement an unknown event processor that provides a more differentiated or semantic understanding of the match with particular target user intentions.In some scenarios, there may be no appropriate flow defining a bot response to a particular user interaction. Thus, in some implementations (e.g., in some embodiments, where the interpreter determines that there are no active flows that match an incoming or internal event that represents a user interaction), the interpreter may request an LLM to generate a flow (e.g., at runtime). For example, in some embodiments, the interpreter may first use an LLM to attempt to match an unknown incoming or internal event with the name, instruction(s), and / or other representation of one or more active flows waiting for corresponding target user intents (and defining corresponding bot responses), and if the LLM determines that there are no matching flows (target user intents), the interpreter (the same or a different) may request LLM to generate a responsive agent (e.g., bot) flow. In some embodiments, the interpreter may request the LLM to generate one or more intents as an intermediate step. For example, if the unknown event is a user action, the interpreter may apply any number of prompts to direct the LLM to classify the unknown user action into a user intention, generate an agent intention, and / or generate a flow that implements the agent intention. As a non-limiting example, the interpreter may implement an event processor for an unknown user action by providing an LLM with an example interaction between the user and the bot, the most recent interaction history, the unknown user action, and a request to the LLM to predict one or more intents (e.g., user, bot), and / or a request to the LLM to generate a corresponding flow. Thus, the interpreter may use an LLM to implement an unknown event processor that intelligently responds to unknown events without requiring the designer to specify the code for the responsive procedure.Neural networks typically operate like a black box, making it difficult to control the responses generated. The lack of transparency makes it difficult to ensure that the generated contents are correct, reasonable, and ethically acceptable. However, the use of an LLM to automatically complete event parameters or parameter values, perform event matching, or generate flows using a standardized and structured interaction modeling language and / or interaction categorization scheme helps to provide the LLM structure and clarity, thereby improving the ability to control LLM output. Embodiments using an LLM to automatically complete event parameters or parameter values, perform event matching, or generate procedures facilitate the designer living by providing an intuitive way to specify human-machine interactions and events to be matched or triggered, thus reducing cognitive burden in the development of an interactive system while protecting against inadvertent content generation.Sensory Processing and Execution of Actions. Depending on the embodiment and configuration, an interactive agent platform hosting the development and / or deployment of an interactive agent (e.g., chatbot, voicebot, digital assistant, interactive avatar, non-player character (NPC), digital human, robot, etc.) may support any number of input and output interaction channels. In some embodiments where sensory processing, interaction decision, and action execution are decoupled, the interactive agent platform may support a sensory server for each input interaction channel and an action server for each output interaction channel. Sensory(s) servers for respective input interaction channels may translate input or non-standardized technical events into the standardized format and generate respective interaction modelling API events, an interaction manager may process these input interaction modelling API events and generate output interaction modelling API events representing commands for execution of an action, and action servers for respective output interaction channels may interpret these output interaction modelling API events and execute the respective commands. The use of an interaction modelling API for communication between these components allows the responsibilities for processing different types of inputs to different types of sensor servers and the responsibilities for different types of actions to different types of action servers to be distributed. For example, each action server may be responsible for respective groups of actions and action events (e.g., in conjunction with a common interaction modality), thereby avoiding the complexity of managing events in conjunction with different interaction modalities.Sensory and / or action servers can be implemented with a combination of asynchronous event loops and processes to ensure that multiple user sessions and system pipelines can be served in parallel. This architecture allows programmers to add different services that can process different types of actions and events (corresponding to different types of interaction modalities) supported by the interaction modelling API actions. In some embodiments, an event gateway may be used to communicate and distribute events to the respective components, whether through synchronous interactions (e.g., via a REST API, Google Remote Procedure Call (RPC), etc.) or asynchronous interactions (e.g., via a message or event broker). Thus, each sensor server may send API events to the incoming input event gateway for interaction modelling or non-standard technical events, and the interaction manager may be subscribed to or otherwise configured to accept these events from the event gateway. The interaction manager may generate and forward outgoing interaction modelling API events to the event gateway, and each action server may be subscribed to or otherwise configured to accept the events for which it is responsible for executing (e.g., one interaction modality per action server).To process all supported actions for at least one interaction modality, an action server may be equipped with an action processor for each standardized action category and / or action event supported by the interaction modelling language and / or defined by the interaction categorization scheme for a particular interaction modality. For example, an action server may implement a chat service that handles all interaction modelling API events for bot utterance actions; an animation service that handles all interaction modelling API events for bot gesture actions; a graphical user interface (GUI) service that handles all interaction modelling API events that command arrangements of visual information such as visual information scene actions, visual selection actions, and / or visual form actions; and / or a timer service that handles all interaction modelling API events for timer actions, just to name a few examples.Each action server can manage the life cycle of all actions in its responsibility area. Interaction modelling API events may include commands for an action server to initiate, modify or end actions. All events related to the same action may be represented by a common action identifier (e.g., Action_uid), such that individual events associated with the same action identifier may represent different states in the lifecycle of a corresponding action. Thus, for a particular interaction modality, an action server may start a particular action (e.g., a bot gesture or utterance) and track active actions and their corresponding states. Each action server may implement a modality policy that determines how to handle actions triggered during execution of another action of the same interaction modality (e.g., multiple sound effects may be performed simultaneously, but a new body animation may replace or temporarily override an active body animation). Some implementations may support commands that alter a current action, which may be useful for longer current actions (e.g., avatar animation) that can dynamically adapt their behavior. For example, a pitch animation can be modified to change its velocity depending on the detected voice activity. Some implementations may support commands that stop a running action, which may be used to proactively stop an action, such as a gesture, that could potentially run over a longer period of time. In some embodiments, an action server may synchronize changes in action status with certain conditions (e.g., wait for a start of an action until a previous action of the same modality is completed, match the completion of two different actions in different modalities, match the start of an action with the end of another action, etc.). When an action server makes a change in action status, it may generate and forward an interaction modelling API event to the event gateway reflecting the update so that any component waiting for this status change may respond to it.Interactive Visual GUI Elements. In some scenarios, a designer desires to adapt an interactive system, e.g., one with an interactive avatar that synchronizes call-carrying AI with supplemental visual content, e.g., a visual representation of associated information (e.g., text, images), a selection to which the user is prompted, or a field or form that the user is to fill.In some embodiments, an interaction modelling API may use a standardized interaction categorization scheme that defines a standardized format (e.g., standardized and semantically meaningful keywords) for the specification of events associated with a standardized category of interactive content visual actions (e.g., actions that instruct superposition or other arrangement of visual contents that supplement conversation with an interactive agent), such as visual information scene actions, visual selection actions, and / or visual form actions. Some embodiments may include an interaction modeling language that supports specification of visual designs using natural language descriptions (e.g., "attention strong, rich, and professional" for a warning message), and a corresponding interpreter may translate a specified description into a standardized representation of corresponding design items (e.g., color schemes, typography, layout, images) and generate outgoing interaction modeling API events using the standardized format for interactive visual content action events. Thus, an action server may implement a graphical user interface service that generates robust and visually appealing graphical user interfaces that may be synchronized with verbal responses by conversational AI or otherwise contribute to human-machine interaction.In some embodiments, an interaction modelling API defines a manner of presenting a particular GUI (e.g., a configuration or arrangement of visual elements) using an interaction categorization scheme that defines a standardized category of interactive visual content actions and corresponding events with payload data indicative of standardized GUI elements. For example, the interaction categorization scheme may categorize interactive visual content actions and / or GUI elements into semantically meaningful groups such that an interpreter or action server may generate the content of a given GUI element (e.g., using an LLM to generate a text block, retrieve, or generate an image based on a specified description) based on the current context of the interaction. Each set of interactive visual content actions and / or GUI elements may be used to define a corresponding subspace of possible GUIs that represent various ways in which a bot can visualize information for the user and / or how the user can interact with this information. An example of an interaction categorization scheme may categorize interactive visual content actions as visual information scene actions, visual selection actions, and / or visual form actions.Visual information scenes may include the display of information to the user for information purposes (e.g., text with background information on a theme or product, an image to illustrate a situation or problem), without expectation that the user could interact with the information in a manner other than by reading. Visual selections may include the display of or interaction with visual elements that provide an option to the user and / or describe the type of option (e.g., multiple selection vs. single selection, small or limited number of options vs. large number of options). Visual form actions may include the display of or interaction with visual elements prompting the user to enter a form or field (e.g., avatar could prompt the user to specify their email address) and / or describe the type of prompts (e.g., email, address, signature).In some embodiments, the interaction categorization scheme may define a standardized format for specification of supported GUI interaction elements (e.g., button list, selectable option grid, input text box, prompt carousel) such that (e.g., a corresponding action processor of a) sensor system server converts detected interactions with these interaction elements (e.g., a state when a button list element is released, e.g., after a click or a touch, a state when a user enters a character into an input box, a state when a user presses the input button or clicks a text box away) into standardized interaction modelling API events, said possible interactions with said elements are in a standardized format. In some embodiments, there may be a sensory server for each of a plurality of different input interaction channels (e.g., GUI interactions, user gestures, voice inputs, etc.), each sensory server configured to generate standardized interaction modelling API events representing recognized interaction events in the standardized format. In some embodiments, a sensor server may convert detected interaction events (e.g., "user has clicked on the Chain Staff button, scroll down, and click on the Confirm button") to a corresponding standardized interaction level event (e.g., "user has selected the Chain Staff option"). Standardized interaction level events may depend on the type of interactive visual content action defined by the schema. Examples of standardized interaction level events may include an event that represents an update to a user's confirmation status and / or when the update was detected (e.g., when a single input is requested as part of a visual form, an "enter" keyboard event may be "confirmed" to a status update), an event that represents an update to a user's selection and / or when the update was detected (e.g., a detected selection of an element "chain slat" from a list of multiple selection elements may be translated to an update to the selection), an event that represents an update to a user's form input and / or when the update was detected, and / or others. Thus, standardized interaction modelling API events may be generated and passed to an event gateway and processed by the interpreter to generate outgoing interaction modelling API events which may indicate commands for performing responsive GUI updates, and the outgoing interaction modelling API events may be passed to the event gateway for execution by a corresponding action server.In some embodiments, interaction modelling API events that indicate commands to update the graphical user interface may be translated into corresponding graphical user interfaces and displayed to the user. To achieve this, in some embodiments, an action server implementing a GUI service may translate a standardized representation of a particular GUI specified by a particular interaction modelling API event into a representation (e.g., JavaScript Object Notation (JSON)) of a modular GUI configuration that indicates content blocks such as paragraphs, images, buttons, multiple selection fields, and / or other types. The GUI service may use these content blocks to create a visual layout for GUI overlay (e.g., a HyperText Markup Language (HTML) layout that may be rendered in any modern web browser). For example, any number of visual template or shell layouts may define corresponding arrangements of different content blocks, and the UI service may select a visual template or shell layout (e.g., based on which content blocks were generated or specified by the interaction modelling API event) and fill place holders for those blocks in the template or shell with the corresponding generated content. In some embodiments, various features of the visual layout of the template or frame (e.g., size or arrangement of blocks, lookand feel options such as a color palette for GUI overlay, etc.) may be adjusted. Thus, a visual layout representing a graphical user interface specified by an interaction modelling API event may be generated and presented to the user (e.g., via a user interface server).Taking an example of an interactive avatar, an animation service may be used to animate the avatar (as described in more detail below), and a GUI service may be used to synchronize the representation of related visual elements (e.g., visual information scene, visual selections, visual shapes). For example, the user's screen may include an area that displays the avatar on a full web page (e.g., utilizing as much as possible the height and width of a browser window with the avatar stream aspect ratio remaining the same), and the visual elements generated by the GUI service may be displayed in overlay over the avatar stream. In one embodiment, the avatar stream may remain in a fixed aspect ratio (e.g., 16:9), requiring replenishment around the stream to maintain the aspect ratio. In some embodiments, the overlay may be maintained at the same relative position on the screen regardless of the size of the stream. In some embodiments, the overlay may be scaled with the size of the avatar. In some embodiments, the overlay may be maintained at a fixed, configurable size relative to the size of the avatar (e.g., 10% of the width and 10% of the height of the avatar).In some embodiments, different GUIs (e.g., visual element pages) may be configured as part of a stack from which GUI pages may be pushed out and pulled out. This configuration may be particularly useful in connection with K1-controlled interactions, as the context may change in a non-linear manner during a series of interactions. A GUI stack overlay may be used to ensure that the visual content of the GUI remains relevant throughout the interaction series. These stacked GUIs may be made at least partially transparent to facilitate visualization of stacked information such that the conversation AI is able to combine GUIs or re-layer the stack at various stages of a conversation (e.g., the header of a stacked overlay may describe an entire customer experience, such as "Support Ticket XYZ," while stacked pages within the overlay may represent various steps in the experience, such as "Please enter Your email.". In some embodiments, the graphical user interface may be part of a rendered 3D scene (e.g., a tablet computer that the avatar holds in hand), the graphical user interface may be 3D (e.g., the buttons may be rendered at corresponding depths), and / or others. These are just a few examples, and other variations may be implemented within the scope of the present disclosure. Although the above example is described in the context of a 2D GUI, those skilled in the art will know how to adapt the above guidance to represent an avatar and / or an augmented and / or virtual reality (AR / VR) overlay.Animations for interactive agents. In some embodiments, interaction modelling API events that specify commands to execute bot expressions, poses, gestures, or other interactions or movements may be generated (e.g., by the interpreter executing code written in an interaction modelling language) and translated into corresponding bot animations, and the bot animations may be presented to the user. More specifically, in some embodiments, an action server implementing an animation service may use a standardized representation of a target bot expression, pose, gesture, or other interaction or motion specified by a particular interaction modelling API event to identify and trigger or generate a corresponding animation.Taking a standardized bot gesture action category (e.g., GestureBotAction) as an example of a bot action type, in some embodiments, the animation service may process all events associated with actions in the GestureBotAction category, apply a modality policy that overrides active gestures by any subsequently commanded gesture, and use incoming start GestureBotAction events to create a stack of actions when active GestureBotAktions are present. Thus, the animation service may implement an action state machine and an action stack for all GestureBotAktionen, connect to an animation graph that implements a state machine for animation states and transitions between animations, and instruct the animation graph to set a corresponding state variable based on a command to change the state of an instance of a GestureBot action (e.g., initialize, stop, or continue a gesture) represented by an interaction modelling API event.In some embodiments, the animation graph may support a number of clips that an avatar or other bot animated in different expressions, poses, gestures, or other interactions or movements. Thus, the animation service may receive a command to change a GestureBotAction state (e.g., initialize, stop, or continue a gesture) that is represented in a standardized interaction categorization scheme to identify a corresponding supported animation clip. In some scenarios, a designer may wish to specify a bot expression, pose, gesture, or other interaction or movement using a natural language description. In some embodiments, the animation service may use a natural language description (e.g., manually input or generated by the interpreter using an LLM / VLM / etc., which is used as an argument to describe an instance of a standardized type of bot action in an interaction modelling API event) to select or generate the best animation clip. For example, the animation service may generate or access a sentence embedding for the natural language description of the bot action (e.g., bot gesture), use it to perform a similarity search on sentence embeddings for descriptions of the available animations, and select an animation based on a similarity measure (e.g., nearest neighbor, within a threshold). In some embodiments, the animation may be played when the best match is within a threshold for similarity (e.g., when the distance is below a certain threshold). If no animation matches within the set threshold, a substitute animation (e.g., a less specific version of the animation that matches best) may be played. If the animation service cannot determine an appropriate match, the animation service may generate and forward an interaction modelling API event to the event gateway indicating that the gesture has failed (e.g., AktionBeended(is_success=False failure_reason="gesture not supported")).Expectation-Controlled Actions and Expectation Signaling. In various scenarios, it may be advantageous to inform an interactive system or one of its components (e.g., a sensor system server that controls input processing, an action server that implements bot actions) which events an interaction manager (e.g., the interpreter) is next awaiting by the user or system. For example, if the interaction manager awaits the user to begin speaking (e.g., an UtteranceUserActionTime event), the interactive system may configure to listen to or enhance its listening capabilities (e.g., by reducing speaker volume, increasing microphone sensitivity, etc.). In noisy environments, an interactive system can be configured to turn off the listening functions (e.g., automatic speech recognition) and only activate listening when the interaction manager awaits the user to speak. In a chatbot system, a designer may superimpose a dedicative when the chatbot (e.g., the interaction manager) is processing a request, and once expecting a response (e.g., a text response), the interaction manager may communicate this expectation to an action server to update the display with a visual indication that the chatbot is waiting for a response. In addition, execution of computer vision algorithms is typically resource intensive. Therefore, the interaction manager can tell what types of image processing events it is expecting at a particular time during an interaction, and the interactive system can spontaneously disable or enable image processing algorithms. Some example scenarios in which the disabling and enabling of image processing algorithms may be useful are reading quick response (QR) codes, object detection, user motion detection, and many more.To facilitate these preliminary actions, expectation may be represented as instances of a standardized action type (expectation actions) with corresponding expected states, and an interaction modelling API event associated with a particular instance of an expectation action may include one or more fields representing an expectation that a particular target event will occur using a standardized interaction categorization scheme that identifies the expectation as a supported action type (e.g., ExpectationTimeAction) and that a corresponding expectation event (e.g., indicating the state of expectation such as start, stop, and end) and the expected target event (e.g., UtteranceUserActionTime) using standardized (e.g., natural language, This is a semantically meaningful key words and / or commands. Standardized expectation events may be, for example, an event that indicates that a bot is awaiting a particular event on the event gateway in the near future (e.g., StartTimeBotAction), that may instruct a sensor or action server to optimize its functionality (e.g., a sensor system server responsible for processing camera images may enable or disable particular image processing algorithms depending on what the interaction manager is awaiting); an event that indicates that the sensor or action server confirms the bot expectation, or that the sensor or action server has updated its functionality in response to the expectation (e.g., ExpectationTimeActionGestartet); an event indicating that the expectation has been stopped (e.g., StopErwartungBotAction), which may occur when the expectation has been satisfied (e.g., the event has been received) or otherwise has passed to change the progress of the interaction; an event indicating that the sensor or action server confirms that the bot expectation has ended (e.g., ExpectationBotActionEnd), and / or others.Additionally or alternatively to notifying (e.g., to a sensor or action server) that the interaction manager (e.g., the interpreter) is awaiting certain events, some embodiments signal to the user that the bot is waiting for input (e.g., for a certain user interaction modality). As such, the standardized interaction categorization scheme may classify this expectation signaling as a supported action type (e.g., ExpectationTimeAction). This action may allow an interactive system to notify the user in a subtil (e.g., nonverbal) manner what the bot is expecting by the user (e.g., the avatar's ears may grow or the avatar may take a listening posture when waiting for user inputs).For example, in a chatbot system, the user could be prompted to enter certain information before the interaction is deemed complete (e.g., "Please enter her birth date to confirm the order"). In such a situation, the designer may want the chatbot to signal the user to actively wait for a response of the user. In this case, the designer code may indicate that triggers the generation of a StartErwartungSignalisierungBotAktion(modality=NutzerSprache) event. In another example, an interactive avatar may wait for a particular gesture of the user. In such a situation, the designer may want the avatar to actively notify the user (e.g., by showing a particular animation). In this case, the designer code may indicate that triggers the generation of a StartErwartungSignalisierungAction (modality=User Gesture) event. If there is a conflict with another ongoing action in a corresponding output interaction channel (e.g., an active upper body animation), an action server may resolve the conflict based on a fixed modality policy.To facilitate these expectation signaling actions, an interaction modelling API event may represent expectation signaling events using a standardized interaction categorization scheme that classifies the expectation signaling as a supported action type (e.g., ExpectationTimeTimeAction) and represents corresponding expectation signaling events (e.g., specifying the state of expectation such as start, stop, completed) and target or input interaction modalities that the bot is expecting (e.g., user language) using standardized (e.g., natural language, semantically meaningful) keywords and / or commands. Examples of standardized expectation signaling events may include an event that indicates that a bot in the near future would expect an event for a particular interaction modality at the event gateway (e.g., StartTimeTimeTimeAction); an event that indicates that the sensor or action server would confirm or confirm the expectation signaling event that the sensor or action server would have started actively waiting for an event for a particular interaction modality (e.g., ExpectationTimeTimeActionTime); an event that indicates that the expectation was stopped (e.g., StopTimeTimeTimeAction); an event indicating that the sensor or action server confirms that expectation has ended or active waiting has been stopped (e.g., ExpectationTimeActionEnd), and / or others.As such, the present techniques may be used to develop and / or utilize interactive agents such as bots or robots (e.g., chatbotes, voicebotes, digital assistants, interactive avatars, non-player characters, etc.) that perform more complex, nuancing, multimodal, non-sequential, and / or realistic conversation AI and / or other types of human-machine interactions than previous techniques. Moreover, various embodiments implementing or supporting an interaction modelling language and / or an interaction modelling API using a standardized interaction categorization scheme promote a number of technological advantages, from facilitating the designer's life by reducing his cognitive burden on the development of an interactive system via the support of a plurality of interactions or functions from which a designer can exhaust to adapt an interactive system, to promoting interoperability by standardization of the representation of interactions.FIG. 1 shows an example of an interactive system 100, in accordance with some embodiments of the present disclosure. It should be understood that these and other arrangements described herein are presented by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, arrangements, groupings of functions, etc.) may be used in addition to or in place of those illustrated, and some elements may be omitted altogether. In addition, many of the elements described herein are functional units that can be implemented as single or distributed components or in conjunction with other components and in any suitable combination and location. Various functions described herein, which are performed by units, may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, the system and methods described herein may be implemented using, for example, one or more generative language models (e.g., as described in FIGS. 28A-28C ), one or more computing devices or components thereof (e.g., as described in FIG. 30 ), and / or one or more data centers or components thereof (e.g., as described in FIG. 31 ).At a high level, the interactive system 100 may execute, control, or otherwise provide an interactive agent (e.g., chatbot, voicebot, digital assistant, interactive avatar, non-player character (NPC), digital human, interactive television or other device, other type of interactive robot, etc.). Some examples of interactive systems that may provide an interactive agent include a digital slide, an infotainment system for automobiles, a digital assistant platform, a smart television or other smart device, a video game or animation environment, a virtual or augmented reality environment, a video conferencing system, and / or others. FIG. 1 shows an example implementation where a client device 101 (e.g., a smartphone, a tablet, a smart television, a game console, a digital slide, etc.) provides an interface for human-machine interaction over any number and type of interaction channels, where one or more sensory servers 160 translate inputs to (e.g., one or more sensory servers 160 translate inputs to events (e.g., standardized interaction modelling API) representing detected interaction states, an interaction manager 190 determines which actions the interactive agent should perform, and generates events (e.g., standardized interaction modelling API) representing corresponding commands, and one or more action servers 170 interpret these commands and cause the interactive agent, To perform appropriate actions via the corresponding interaction channel / s.Depending on the implementation, the components of FIG. 1 may be implemented on any number of physical machines (e.g., which may include similar components, features, and / or functions as the example computing device 3000 of FIG. 30). Take a digital slide as an example. In some embodiments, the physical slide may correspond to the client device 101 connecting to one or more remote controlled components. In some embodiments, some or all of the components in FIG. 1 may be implemented as corresponding microservices and / or physical devices provided in a cluster of nodes in a data center (e.g., which may include similar components, features, and / or functions as example data center 3100 of FIG. 31 ), on one or more edge devices, on dedicated hardware, and / or in other ways. In some implementations, some or all of the components run locally on a physical machine (e.g., on a digital slide, robot, or other interactive system) with various types of interface hardware managed by an operating system, firmware, and / or other software. In some of these embodiments, the client device 101 corresponds to the various hardware interfaces, and some or all of the other components in FIG. 1 (e.g., the sensor / s servers 160, the action server / s 170, the interaction manager 190, etc.) represent functions of the operating system, firmware, and / or other software that send commands or requests to the various hardware interfaces.In an example virtual or augmented reality environment, the components illustrated in FIG. 1 may be implemented on a local device (e.g., an AR / VR headset, a smartphone running a VR / AR application), a cloud server, an edge computing device, dedicated hardware, and / or otherwise. In some embodiments, there is one sensor server per input interaction channel (e.g., a sensor server for processing video inputs, one for processing audio inputs, one for processing touch inputs), and / or one action server per output interaction channel (e.g., an action server for processing bot animations, one for processing bot speech, one for processing interactive visual content). In some implementations, some or all of the sensor system servers 160 and / or action servers 170 are merged into a single machine and / or microservice that processes the respective interaction channels using the respective services. These are just a few examples, and other configurations and implementations are possible within the scope of the present disclosure.In some embodiments, some or all of the components shown in FIG. 1 are part of or at least partially hosted by an interactive system development and / or deployment platform (e.g., an interactive agent platform). For example, a platform such as NVIDIA® OMNIVERSE® (and / or another platform or system, e.g., those that use universal scene descriptor (USD) data formats-e.g., OpenUSD) may provide infrastructure and various functionalities that provide a framework for the development and / or provision of interactive agents. The platform may provide various creation tools that allow users to create and adjust interactive agents, a real-time rendering engine, integration with various services (e.g., computer vision, speech recognition, natural language understanding, avatar animation, speech generation, simulation software, recommendation engines), and / or other components. In some embodiments, some or all of these tools and / or the components depicted in FIG. 1 are integrated into an application and processed in real-time (e.g., using a framework for development and provision of cloud-native applications, such as NVIDIA® unified cloud services tools). Some or all of these tools and / or the components shown in FIG. 1 may be provided as microservices and managed with a platform for orchestration of containerized applications (e.g., NVIDIA FLEET COMMAND™). Accordingly, in some embodiments, these tools and / or components may be used to customize and / or employ the interactive system 100.For example, in some embodiments, interaction manager 190 may implement an interpreter for an interaction modeling language, and code implementing the decision logic for an interactive agent may be written in the interaction modeling language, loaded or otherwise accessed by interaction manager 190 and executed by interaction manager 190. Depending on the desired interactive agent, any number and type of interaction channel may be connected, configured, and supported by corresponding sensor servers 160 and / or action servers 170. Thus, in some embodiments, a development and / or deployment platform may be used to host the interactive system 100, and the interactive system 100 may implement an interactive agent (e.g., customizable).At a high level, a user may operate or otherwise interact with the client device 101 or other interactive system that includes any number of input and / or output interaction channels. By way of non-limiting example, FIG. 1 shows a video input interaction channel that includes a camera (not shown) and an image processing microservice 110 that uses any known computer vision technique to recognize user gestures; an audio input interaction channel that includes a microphone (not shown) and a speech recognition microservice 120 that uses any known speech recognition and / or detection technique to recognize the user's speech; a video output interaction channel comprising a screen (not shown) and an animation microservice 140 that uses any known animation technique to animate a bot (e.g., bot poses, bot gestures, mixed shapes, text-to-motion, text-to-animation); an audio output interaction channel comprising a speaker (not shown); and a speech generation microservice 150 that uses any known speech synthesis technique to synthesize bot speech; a graphical user interface (GUI) having a GUI input interaction channel that accepts GUI user input (e.g., touch, click), a GUI output interaction channel that displays interactive visual content, and a user interface server 130 that uses any known technique to manage and / or provide user interfaces for the GUI.In an example flow through the interactive system 100 of FIG. 1, a representation of user input, such as a gesture recognized by the vision microservice 110, a voice command recognized by the voice recognition microservice 120, or a touch or click input recognized by the UI server 130, may be forwarded to a corresponding sensor system server 160 responsible for a corresponding interaction channel. The sensor system server / s 160 can / can translate this user input into a standardized representation of a corresponding event and forward the event to an event gateway 180. The event gateway 180 may be used to communicate and distribute events to respective components, whether through synchronous interactions (e.g., via a REST API, Google Remote Procedure Call (RPC), etc.) and / or asynchronous interactions (e.g., via a message or event broker). The interaction manager 190 may be subscribed to or otherwise configured to receive these events from the event gateway 180. As such, the interaction manager 190 may process the events (e.g., using an event driven state machine), determine which interactions are to be performed, and generate commands as corresponding events in a standardized representation, and forward them to the event gateway 180. The action server / s 170 responsible for the corresponding interaction channel / channels may / may be subscribed to or otherwise configured to receive / receive the events for which he / she is responsible for executing from the event gateway 180. As such, the action server or servers 170 may execute, plan, and / or otherwise handle events for the corresponding interaction modality(s) by interfacing with a corresponding service that controls a corresponding output interface. For example, depending on the instructed action, a corresponding action server 170 may plan and trigger the bot language via an audio interface (e.g., the voice generation microservice 150), generate a bot animation on a screen or headset (e.g., the animation microservice 140), present interactive visual content on a screen or headset (e.g., the UI server 130), and / or otherwise.In some embodiments, interactive system 100 represents and / or communicates human-machine interactions and related events using a standardized interaction modelling API and / or an event driven architecture. In some embodiments, the standardized interaction modelling API standardizes the manner in which components (e.g., the sensor system server / servers 160, the action server / servers 170, the interaction manager 190) represent multimodal interactions. In an example implementation, a standardized interaction modelling API serves as a common protocol in which various components of the interactive system 100 use a standardized interaction categorization scheme to represent all activities of bots, users, and / or the interactive system 100 as actions in a standardized form, represent states (e.g., of multimodal actions of users and bots) as events in a standardized form, support standardized mutually exclusive interaction modalities and define how conflicts between standardized categories or action types are resolved, and / or implement standardized protocols for any number of standardized modalities and action categories regardless of the implementation.FIG. 2 shows an example of an interaction modelling API 220 according to some embodiments of the present disclosure. In general, different types of interactive systems may include different types of interaction channels 230. For example, a chatbot may use a text interface that supports an input interaction channel for input text and an output channel for output text. A voice assistant may use an audio interface that supports an interaction channel for input of voice and an output channel for output of voice. An interactive avatar may use a video input interface supporting a recognized gesture input interaction channel, an audio input interface supporting a recognized language input interaction channel, a video output interface supporting an avatar animation output interaction channel (e.g., poses, gestures), an audio output interface supporting an avatar output output interaction channel, and / or a graphical user interface supporting a touch input interaction channel and / or an interactive visual content output channel. A non-player character may use a game controller interface that supports an input interaction channel for controller inputs, a video output interface that supports an output interaction channel for non-player character animations, and an audio output interface that supports an output interaction channel for non-player character voice output. These are only examples, and other types of interactive systems, interactive agents, and / or interaction channels may be implemented within the scope of the present disclosure.FIG. 2 shows an example of an interaction modelling API 220 between the interaction manager 190 and the interaction channel / s 230. In some embodiments, the interaction modelling API 220 defines a standardized format for specifying user and / or bot interactions, system events and associated events using a standardized interaction categorization scheme. The interaction categorization scheme may use standardized (e.g., semantically meaningful) keywords, commands, and / or syntax that includes or categorizes standardized interaction modalities, action types, and / or event syntax. Taking the example of standardized interaction modalities, the interaction categorization scheme can be used to classify interactions (e.g. bot actions) according to standardized interaction modality and / or corresponding standardized action category (e.g. bot utterance, bot pose, bot gesture, bot gaze) using standardized action keywords. FIG. 2 illustrates this by using separate lines to represent events for different interaction modalities (e.g., bot utterance events, bot pose events, bot gesture events, bot gaze events, scene or interactive visual content events). Moreover, in some embodiments, the interaction modelling API 220 defines a standardized format for specifying changes in action states as corresponding events to support an event driven architecture. FIG. 2 illustrates this using different start and stop times for different actions (e.g., the bot begins with a tensed posture before making an utterance, and initiates a gesture and a gaze action before completing the utterance, etc.).In some embodiments, to promote configurability and interoperability, the interaction modelling API 220, a corresponding interaction modelling language supported by the interaction manager 190 and / or a corresponding interaction categorization scheme supported by (e.g., sensory and / or action servers in) the interaction channel / channels 230 may provide a way to categorize, specify and represent interactions for a plurality of different interactive systems and corresponding interaction channels, which may allow designers to adapt an interactive system using standardized components. FIG. 3 illustrates some features of example interactive systems that may be supported by an example interaction modelling API and / or an example interaction modelling language in accordance with some embodiments of the present disclosure. In some embodiments, an interactive system relies on an interaction modelling API and / or an interaction modelling language that supports more interactions and action keywords than the interactive system itself uses. For example, an interaction modeling API and / or language may support a keyword for a bot gesture (e.g., MakeGeste), although an interactive system using the API and / or modeling language (e.g., a chatbot) may not use this type of interaction. However, by supporting a plurality of multi-modal interactions, an interaction modelling API and / or interaction modelling language may support a plurality of interactions or functions that a designer may resort to to to adapt an interactive system, promote interoperability by standardization of the representation of interactions, and facilitate the designer's life by reducing his cognitive burden in developing an interactive system.In some embodiments, an interaction modeling API and / or interaction modeling language may support standardized representations of actions and events for interaction modalities such as speech, gestures, sensations, movements, scenes, and / or others. In some embodiments, the interaction modelling API and / or language may define mutually exclusive interaction modalities such that actions in different interaction modalities may be performed independently of each other (e.g., by corresponding action servers) (e.g., a bot may say something independent of gestures). The possibility of simultaneous or conflicting actions in the same interaction modality may be taken into account by implementing (e.g., a corresponding action server) a modality policy for that interaction modality. Thus, an action server implementing this interaction modality may use a particular modality policy to determine how events are to be executed, scheduled, and / or otherwise handled for that interaction modality. FIG. 4 illustrates some example modality policies in accordance with some embodiments of the present disclosure. Accordingly, an interaction modeling API and / or interaction modeling language may support an interaction categorization scheme that defines standardized representations of supported interaction modalities and corresponding actions and events, such as the interaction categorization scheme illustrated in FIG. 5. As illustrated in this example, some modality groups (e.g., motion) may be divided into groups of interaction modalities that may be performed independently of each other (e.g., a BotExpression on a BotGesicht modality may be animated independently of a BotPose on a BotOberBody modality).In interactive systems that support multi-modal interactions, information may be exchanged between a user and an interactive system via multiple interaction modalities. Each interaction modality may be implemented via a corresponding interaction channel between the interactive system and the user. In some embodiments, an interaction categorization scheme may categorize any action as part of a single interaction modality, although depending on the interactive system, an action server for that interaction modality may map that action to multiple output interfaces (e.g., audio, video, GUI, etc.). For example, a BotUß action (instructing a Bot to verbally communicate with a user) may be classified as part of a BotStimm modality. In an interactive system that represents the bot as a 3D avatar (e.g., on a 2D screen, on an AR or VR device), the BotStimm modality and / or the BotUß action may trigger various types of outputs, such as audio output (e.g., synthesized speech), lip movement (e.g., lip synchronization to speech), and / or text on a user interface (e.g., subtitles for utterances). In another example, a BotMove action may be classified as part of a BotSubBody modality and trigger animation of the subbody (e.g., walking animation) and audio output (e.g., the sound of foot steps).FIG. 6 illustrates an example of an event driven interactive system 600 in accordance with some embodiments of the present disclosure. FIG. 6 illustrates an example implementation of an architectural pattern that separates the component(s) implementing the decision logic that determines which actions to perform (e.g., the interaction manager 640) from the component(s) handling / handling the interactions (e.g., the sensor server 620 and the action server 670).At a high level, detected input events 610 (e.g., the presentation of a user input such as a detected gesture, a voice command, or a touch or click input; the presentation of a detected feature or event associated with a user input such as a detected presence or absence of voice activity, a detected presence or absence of taps, a detected transcribed speech, a detected change in volume or speed of tapping, etc.) may be forwarded to a sensor system server 620, and the sensor system server 620 may translate the detected input events 610 into standardized input events 630. An interaction manager 640 may process the standardized input events 630 and generate events representing commanded bot actions (commanded bot action events 650), and an action server 670 may perform the actions represented by the commanded bot action events 650. In some embodiments, the interaction manager 640 may generate internal events 660 representing internal state changes (e.g., state of progress changes) or commanded bot actions, and / or the action server 670 may generate events 665 representing the confirmation of action state changes that may be evaluated by the interaction manager 640 in determining the actions to take.The interaction manager 640 (which may correspond to the interaction manager 190 of FIGS. 1 and / or 2 ) may be responsible for deciding which actions the interactive system 600 should perform in response to user actions or other events (e.g., the standardized input events 630, the internal events 660, the events 665 representing the confirmation of changes in action status). The interaction manager 640 may (but need not) interact with the rest of the interactive system 600 (e.g., exclusively) via an event driven mechanism. In practice, other portions of the interactive system 600 may generate other events while the interaction manager 640 is engaged in processing an event (e.g., deciding on the next action). Thus, depending on the implementation, the interaction manager 640 may process multiple events one after the other or all at a time. In a state oriented approach, the interaction manager 640 may maintain the state or context of a user's interaction with the interactive system 600 across multiple interactions within a particular session. In a stateless approach, the history of the state or context may be presented along with each new event. In some embodiments, there is no common state between the interaction manager 640 and the rest of the interactive system 600.In general, interactive system 600 may include any number of interaction managers (e.g., interaction managers 640). In some implementations, the interactive system 600 may include a primary interaction manager with internal or secondary interaction managers. In an example that includes an interactive avatar experience, a primary interaction manager may manage the high level flow for a human-machine interaction (e.g., the various phases such as welcoming, data collection, data delivery, confirmation, etc.), and the primary interaction manager may pass the decision making to one or more secondary interaction managers (e.g., for a complex authentication flow, for an interactive question-answer scenario, etc.), as appropriate. In some implementations, interactive system 600 may include multiple peer interaction managers that each handle different types of events. For example, one interaction manager may manage the conversation logic (e.g., what the bot is to say), and a second interaction manager may manage animation of an avatar based on what it says.In some embodiments, interactions between the interaction manager 640 and the rest of the interactive system 600 occur via various types of (e.g., standardized) events, e.g., those representing detected input events (e.g., detected input events 630), commanded bot action events (e.g., commanded bot action events 650), and system or context events. In general, recognized input events can be used to represent any event that may be relevant to interaction, e.g., when the user is speaking something (e.g., UserSagt), making a gesture (e.g., UserGeste), or clicking a GUI element (e.g., UserAuswahl). Bot action events may specify what the interactive system 600 is to do, e.g., say, play a sound, display something on a display, change the avatar's appearance or pose, invoke a third party API, etc. Bot action events may represent transitions in the life cycle of an action, e.g., by an instruction to do something (e.g., start action), an indication of when the action was begun (e.g., actiontime) or ended (e.g., actiontime), etc. System or context events may represent changes to associated interaction data included in the interactive system 600 (e.g., context update), such as username, user rights, etc. The interaction data may be a function of the interaction data that is included in the interactive system 600, selected product, device information, etc.Thus, the interaction manager 640 may evaluate various types of events (e.g., standardized input events 630, internal events 660, events 665 representing confirmation of changes in action state), determine which actions to perform, and generate corresponding commanded bot action events 650. As such, the action server 670 may perform the actions represented by the commanded bot action events 650. For example, the interaction manager 640 may decide that the interactive system 600 should say "Hello!" and, upon completion of this utterance (e.g., says), perform a particular gesture (e.g., point to a screen and query something). In some examples, the interaction manager 640 may generate an event indicating that the gesture is to begin (e.g., with keywords such as StartAction(Machine Gesture)) when the interactive system 600 completes the welcoming (e.g., via a particular condition such as ActionBeendet(Stell)). As another example, the interaction manager 640 may decide to start a wincanimation when a Sag(Hello) action has begun and stop the animation when Sag(Hello) is completed. In some of these examples, the interaction manager 640 may indicate conditions (e.g., ActionGestartet(Speaking) and ActionBeendet(Speaking)) when it indicates corresponding instructions to start and stop the gesture (e.g., StartAction(Machine Gesture(Wink)) and StopAktion(MacheGeste(Wink)).In some embodiments, the interaction manager 640 implements an interpreter or compiler that interprets or executes code written in an interaction modeling language that specifies user and / or bot interactions and associated events using a standardized interaction categorization scheme (such as that illustrated in FIG. 5 ). In general, the interpreter and interaction modelling language may support any number of keywords that serve (e.g., send, adjust, start, stop, wait, activate) to parallelize action and execution and matching. The interaction modeling language may be used to define a flow of interactions using primitives that include semantically meaningful (e.g., natural language) keywords and commands that specify events (e.g., something is passed) and actions (e.g., something must pass) using the interaction categorization scheme. In some embodiments, events (e.g., standardized input events 630, internal events 660, events 665 representing the confirmation of action state changes) may be represented using a standardized syntax event specifier defined by the interaction categorization scheme, interaction modelling language, and / or interaction modelling API and supported by the interpreter. In some embodiments, an event may include a representation of corresponding (e.g., standardized) fields and values (e.g., payload data specifying them) that the interpreter (and other components) may possibly understand. As such, the interpreter may execute code that implements an interaction flow in the interaction modelling language, which interaction flow may instruct the interpreter which actions or events are to be generated in response to which events.For example, an event may be presented and / or communicated within the interactive system 600 in various ways. As a non-limiting example, an event (e.g., payload) may include fields that specify or encode values that specify or encode an action type (e.g., identification of a standardized interaction modality or corresponding action type such as user action), an action state (e.g., an observed state of a user action such as ending, a current or confirmed state of a bot or scene action such as starting, a commanded state of a bot or scene action such as starting), recognized or commanded action content (e.g., transcribed or commanded speech such as "hello", a description of a recognized or commanded gesture, a description of a recognized or commanded pose or mimic, etc.), a unique identifier (UID) for identifying the event, a timestamp (e.g., indicating when the event was created, when an action was updated), a unique source identifier identifying the source of the event, one or more tags (e.g., indicating that the event was created as part of a particular stream or session or is associated with a particular user or account), context, and / or other properties or information.In some embodiments, each action may be identified by a unique identifier (Action_uid), and all events that relate to the same action may refer to the same Action_uid. Thus, individual events that reference the same action_uidmay serve to represent the life cycle of a corresponding action from beginning to end (e.g., including updated action states therebetween). In some embodiments, the component that sends the StartAction and ActionTime events may generate the Action_uid for new instances of an action, and the specific involved component may depend on the type of action (e.g., bot or user action). For example, the interaction manager 640 may be responsible for generating the action_uidfor new instances of bot actions that are launched by the interaction manager 640, and the sensor system server 620 may be responsible for generating the action_uidfor new instances of observed user actions. Thus, individual events can be linked to a corresponding instance of a particular action type.Using an example interaction categorization scheme as illustrated in FIG. 5, actions may be classified into corresponding interaction modalities such as speech, gestures, emotions, movements, scenes, and / or others. Using the example language, interactive system 600 may support various events and actions associated with dialog management using the language modality. For example, the user may use a UserSprach modality (e.g., via a UserUß action), or a bot provided by the interactive system 600 may use a BotSprach modality (e.g., via a BotÄußert action). In an example of a user utterance action, the user may make an utterance that is recognized by the interactive system 600. Examples of this action are the user's input to a text interface for interaction with the bot or the user's speaking with an interactive avatar. Examples of possible events associated with this action include UtteranceNutzerActionGestartet, StopUßUserAction (e.g., instructing action server 670 to shorten the dwell time of automatic speech recognition), UtteranceUserActionTranskriptAktualisiert (e.g., (e.g., providing updated transcripts during an UtteranceUserAction), UtteranceUserActionIntensitätAktualisiert (e.g., providing the detected voice intensity, tap speed, changing volume or pitch, etc.), UtteranceUserActionEnd (e.g., providing a final transcript), and / or others.In an example bot utterance action, a bot may provide an utterance (e.g., say something) to the user via a form of verbal communication (e.g., via a chat interface, a voice interface, brain-to-machine communication, etc.). Examples of possible events associated with this action are StartUßBot action (e.g., instructing the bot to produce an utterance, with payload data that may include a transcript of the instructed utterance of the bot, a representation of intensity such as voice intensity, output text rate, volume or pitch change, etc.), UtteranceBotAktion actionGestartet (e.g., indicating that the bot has begun producing an utterance), ChangeUßBot action (e.g., instructing an adjustment to the volume or other characteristic if the action has already begun), UtteranceBotActionScriptAktualisiert (e.g., providing updated transcripts during an utteranceBotAction), StopÄBot action (e.g., instructing to terminate the bot utterance), UtteranceBotActionBeendet (e.g., confirming or notifying that the bot utterance is terminated, e.g., because it has been completed or because the user has terminated the utterance), and / or others.Using the example of motion, interactive system 600 may support various events and actions associated with the motion modality. Motion actions may represent motions or sets of motions that have a particular meaning. For example, the user may make a gesture or pose recognized using computer vision, or a bot provided by the interactive system 600 may make a gesture or pose. In some embodiments, the user and / or the bot may use any suitable movement modality (e.g., face, upper torso, lower torso). In some embodiments, these modalities may be regulated by a "override" modality policy that the action server 670 may interpret as an instruction to handle concurrent actions by temporarily overriding a currently-occurring action by a restarted action. As a non-limiting example, when the interaction manager 640 starts a BotHaltung action ("nested arms") instructing an avatar to keep its arm nested until the action is stopped, and two seconds later the interaction manager 640 starts a BotGeste action ("ham"), the action server 670 may perform the "ham" action by overwriting the "nested arms" posture by the "ham" action (e.g., to have the avatar press the user). Once the wave action is completed, the action server 670 may return the avatar to the "entangled arms" pose (e.g., the overwritten action resumes).In a bot action with facial expression, a corresponding event may instruct a bot to make a facial expression (e.g., a smiley in a chatbot text message, a digital avatar facial expression in an avatar interactive experience), while using a particular expression or emotion (e.g., friends, overseeing, pasture, punish, anxiety, ekel, wut, etc.). Examples of possible events associated with this action include StartExpressionTimeAction (e.g., the instruction to change the facial expression of the bot and indicate the type of expression), ExpressionTimeActionTime (e.g., the indication that the bot has started the action), StopExpressionTimeAction (e.g., the instruction to the bot to end the facial expression), ExpressionTimeActionBe (e.g., the indication that the bot has ended the facial expression), and / or others.In some embodiments, interactive system 600 may support user actions with facial expression and corresponding events representing recognized user expressions. Examples of possible events associated with this action include ExpressionNutzerGestartet (e.g., indicating that a facial expression of the user has been detected, including a representation of the content of the expression, such as friends, overseeing, pasture, trauer, anxiety, ekel, wut, etc.) and ExpressionNutzerAktion (e.g., indicating that the detected facial expression of the user has returned to a neutral expression).In an example gesture bot action, a corresponding event may instruct a bot to perform a particular gesture. In some embodiments, events associated with this action may include payload data that includes a natural language description of the gesture, which may include a base gesture, one or more gesture modifiers, and / or other features. Examples of base gestures include speaking, idling (e.g., spontaneous physical movement or action during a time of inactivity), affirmative (e.g., a nonverbal hint or action indicating consent, confirmation, or affirmative), declaring (e.g., a nonverbal hint or action indicating inunity, contradiction, or rejection), attracting (e.g., a particular movement, action, or behavior that is to draw a user or audience's attention to a particular object, location, or activity), and / or others. Example hierarchies for some base gestures are: speaking with emotion (e.g., "speaking excited"), idling with excitation level (e.g., "nervous idling"), attracting with intensity (e.g., "subtil attracting"). Examples of possible events associated with this action may include Start GestureBotAction, GestureBotActionGestartet, Stop GestureBotAction, GestureBotActionBeendet, and / or others.In some embodiments, interactive system 600 may support gesture user actions and corresponding events representing recognized user gestures. Examples of possible events associated with this action include GestureUserActionGestartet (e.g., indicating that a user gesture has been detected, including a representation of the content of the gesture) and GestureUserActionBeendet (e.g., indicating that completion of the user gesture has been detected).In an example bot position change or bot motion action (e.g., at the BotSubBody motion modality), a corresponding event may instruct the bot to move to a particular position (e.g., on a screen, in a simulated or virtual environment). A particular location may include a base location, one or more location modifiers, and / or other features. In an example implementation, the supported base positions may be front and back, and the supported position modifiers may be left and right. Examples of possible events associated with this action are StartPositionChangeTimeAction (e.g., identifying a particular position to which the bot is to move) and PositionChangeTimeActionTime.In an example action to change user position or motion (e.g., at the BotSubBody motion modality), a corresponding event may indicate a detected change in position of the user's subbody. Examples of possible events associated with this action are: PositionChangeUserAction (e.g., indicating that a recognized user motion has begun, including a representation of the direction or character of the recognized motion, such as active, approximating, passive, going away, sideways, etc.); PositionChangeUserDirectionTime (e.g., indicating when the user changes direction during a recognized motion), PositionChangeUserActionTime (e.g., indicating that the recognized motion is complete).In some embodiments, interactive system 600 supports interactive visual content actions and events representing presentation and / or interaction with various types of visual information (e.g., in a 2D or 3D interface). Examples of interactive visual content actions (also referred to as visual actions) include visual selection actions, visual information scene actions, and visual form actions.In an example of a visual selection action, a corresponding event may instruct visualization of a selection with which the user may interact. Interactive system 600 may support various types of visual selection interactions (e.g., by presenting a website on a display that accepts a touch or click option, or by accepting a voice input to select an option). For example, a StartVisuelleSelectSzenenAction event may include payload data with a prompt describing the choice to be presented to the user; an image describing what is to be presented to the user, one or more prompts for assistance that assist or direct the user in the selection (e.g., "say Simply 'Yes' or 'No' to proceed"), or a recommendation ("I may recommend the cheese burger"); a list of options from which the user may choose (e.g., each option may have a corresponding image); a selection type ("selection", "search", etc.); and / or an indication of whether multiple selections are to be permitted. Further examples of possible events associated with this action include a VisualAuswahlSzenenAktualisiert event (e.g., to indicate a detected user interaction with a selection presented in the scene when the user has not yet confirmed the selection), StopVisuelleAuswahlSzenen action (e.g., to instruct to remove the visual selection), VisualAuswahlSzenenAktion actionBeendet (e.g., to indicate a final confirmed selection), and / or others.In an example of a visual information scene action, a corresponding event may instruct visualization of certain information for the user. The visual information scene action can be used to show detailed information to the user about a particular theme associated with the interaction. For example, if the user is interested in details about a particular or displayed product or service, a visual information scene action may instruct the presentation of information about that product or service. Examples of possible events associated with this type of action are Visual InformationSzenen action (e.g., instruction to visualize; indication of a description of the content to be visualized; indication of one or more blocks of the content to be visualized, such as a title, summary of the content, and / or description of one or more images to be visualized; one or more prompts for assistance, etc.); Visual InformationSzenenActionGestartet (e.g., indicates that the visual information scene action has been started), Stop Visual InformationSzenenAction (e.g., indicates to stop the visualization), Visual InformationSzenenActionBeendet (e.g., indicates that the user has closed the visualization or the visual information scene action has been stopped), and / or others.In an example of a visual form action, a corresponding event may instruct visualization of a particular visual form with one or more form fields (e.g., email, address, name, etc.) that the user is to fill in. Examples of possible events associated with this type of action are StartVisuelleFormSzenenAction (e.g., instruction for visualization; Indication of one or more inputs, a prompt for the user, one or more help orders, one or more images, etc.), StartVisuelleFormSzenenAktionGestartet (e.g., indicating that the user has begun inputting information into the form), StartVisuelleFormSzenenInputAktualisiert (e.g., indicating that the user has input information into the form, but has not yet confirmed the selection), StopVisuelleFormSzenenAktion action (e.g., indicating that visualization of the form is to be halted), VisualFormSzenenAktion (e.g., indicating that the user has confirmed or aborted the form input), and / or others.In some embodiments, interactive system 600 may support actions and events representing various aspects of the scene in which the human-machine interaction occurs. For example, interactive system 600 may support actions on a sound modality (e.g., indication of sound effects or background sounds), an object interaction modality (e.g., indication of interactions between a bot and virtual objects in the environment), a camera modality (e.g., indication of camera slices, actions, transitions, etc.), a visual effect modality (e.g., indication of visual effects), a user presence modality (e.g., indication of whether or not the presence of the user is detected), and / or other examples. Examples such as these and others are described in more detail in U.S. Provisional Application No. 63 / 604,721 filed Nov. 30, 2023, the contents of which are incorporated herein by reference in their entirety.Having described some example events associated with standardized action types and interaction modalities and some possible ways of presenting such events and actions, the following discussion will apply to some possible ways in which (e.g., an interpreter of) interaction manager 640 may use a designated interaction flow (or simply a flow) (e.g., written in an interaction modeling language) to evaluate such events (e.g., incoming and / or queued instances of standardized input events 630, internal events 660, events 665 representing an acknowledgement of action state changes) to determine which actions or events are to be generated in response thereto, and to generate corresponding events (e.g., outgoing instances of the commanded bot action events 650, the internal events 660).In general, a flow may specify instructions using primitives from an interaction modeling language that includes semantically meaningful (e.g., natural language) keywords and commands that specify events (e.g., something is passed) and actions (e.g., something must pass) using an interaction categorization scheme. The state of an action (e.g., an observed state of a user action, a current state of a bot or scene action) and / or a command to change the state of a bot or scene action may be represented using standardized event keywords, commands, and / or syntax. For example, action events (e.g., a started or stopped user or bot action) may be represented by a standardized syntax event specifier (e.g., event name and / or identifier having a keyword identifying the standardized action category and a user or bot action state specifier). A line of instructions in a flow may include an event trigger (e.g., using a keyword such as send) that causes the interpreter to generate a particular event when a particular condition is met (e.g., events representing instructions for executing a bot action may trigger the action to be executed or events representing a change in user state may trigger a corresponding bot action), or an event balancer (e.g., using a keyword such as match) that causes the interpreter to interrupt the flow and wait for a particular event before resuming the flow. The event trigger and event match may establish corresponding trigger and match conditions using an event specifier that includes a standardized event name or identifier (e.g., a keyword identifying a standardized action category paired with a corresponding action state specifier or an action state change command) and an argument that indicates one or more conditions that the indicated event must satisfy (e.g., using predefined parameters and supported values or a natural language description).Thus, (e.g., the interpreter of) interaction manager 640 may be equipped with logic that interprets corresponding keywords, commands, and / or syntax as mentioned. In some embodiments, the interaction manager 640 may support any number of keywords that are used to parallelize action and flow execution and matching (e.g., any of the keywords described above, such as sending, adjusting, starting, stopping, waiting, activating, returning, aborting, and / or others). The interaction manager 640 may be programmed to execute the instructions specified in a particular flow sequentially, generate all events specified by an event trigger, and stop when a flow head reaches an event matcher, exception, or end of a flow. In some embodiments, interaction manager 640 may support and track multiple active flows (e.g., interrupted at corresponding event balancers) (e.g., using an event-driven state machine to wait for incoming events that match an event balancer of an active flow, and trigger corresponding events and actions indicated in matching flows.FIG. 7 shows an example of an interaction manager 700 in accordance with some embodiments of the present disclosure. In this example, interaction manager 700 includes interpreter 710, interaction flow(s) 780, and internal event queue 790. At a high level, the interaction flow / s 780 specify corresponding sequences of instructions in an interaction modelling language and may be loaded or otherwise made accessible to the interpreter 710, and the interpreter 710 may include an event processing component 730 that sequentially executes the instructions specified in the interaction flow / s 780 to process incoming events and generate outgoing events (e.g., in standardized form).Event processing component 730 may execute a primary processing loop that processes incoming events and generates outgoing events. At a high level, event processing component 730 includes a run execution component 750 and a run balancer 740. The flow execution component 750 may sequentially execute the instructions indicated in a flow (e.g., parent flow, matching flow) of the interaction flow / s 780, generate and stop all events indicated by an event trigger when a flow head reaches an event matcher, exception, or end of a flow. The policy 740 may evaluate incoming events to determine if they match an active policy policy policy, instruct an action conflict resolver 760 to resolve any conflicts between multiple matching policies, and instruct the policy execution component 750 to forward matching policies (e.g., conflict-free).In one embodiment, the flow execution component 750 may perform lexical analysis of the instructions specified in the interaction flow / s 780 (e.g., tokenization; identification of keywords, identifiers, arguments, and other elements) over which flow instructions iterate, execute each one in turn, and include exception handling mechanisms. In some embodiments, the flow execution component 750 uses a different flow header for each of the (e.g., active) interaction flows 780 to indicate the current position and proceed by the instructions in a corresponding interaction flow. Depending on the instruction, the flow execution component 750 may advance each given flow header to the next instruction, skip to another flow referenced by a particular label or flow identifier, split into multiple headers, merge multiple flow headers, and / or others. As such, the execution component 750 may be coordinated with a trace and control component 770 to build and maintain a hierarchy of headers. When a parent header is suspended, paused, or resumed in a branch of a hierarchy of headers, the execution engine 750 may be coordinated with the trace and control engine 770 to suspend, pause, or resume all the children headers of that parent header or branch. In some embodiments, each flow may indicate any number of ranges that the flow execution component 750 may use to generate stop events that instruct a corresponding action server to stop previously started actions within the corresponding ranges.For example (e.g., at start-up), the flow execution component 750 may execute an uppermost level flow (e.g., the interaction flow / flows 780) including instructions to activate any number of flows (e.g., the interaction flow / flows 780) with any number of event matching agents. The trace and control component 770 may use any suitable data structure to trace active flows and corresponding event matchings (e.g., using a tree or other nested flow relationship representation), and may use an event driven state machine that waits for various events and triggers corresponding actions indicated in matching flows (with event matchings corresponding to an incoming event). As such, the flow execution component 750 may iterate through active flows, generate all events specified by an event trigger, and stop when a flow head reaches an event matcher, exception, or end of a flow.In some embodiments, progressive operations may instruct the operation execution component 750 to generate outgoing events that instruct an action. Additionally or alternatively, ongoing operations may instruct the operation execution component 750 to generate events that inform listeners (such as the operation execution component 750 itself) that an event has occurred. As such, the thread execution component 750 may broadcast these events, and / or the interpreter 710 may maintain an internal event queue 790 and place these events into the internal event queue 790 (e.g., in the event another thread waits for a generated event).Once the flow headers for all advanced flows have reached an event matcher, exception, or end of a flow, the flow matcher 740 may process incoming events sequentially (e.g., from the internal event queue 790, from another queue or event gateway, such as the event gateway 180 of FIG. 1 ), and test for each event the event matcher indicated by each active flow for a match with the event. In some embodiments, the flow alignment device 740 sequentially processes all internal events in the internal event queue 790 (e.g., testing active flows for matches with internal events) before transitioning to processing the next incoming event (e.g., from the event gateway). Internal events may represent updated states of interaction flows 780 (e.g., indicating that a particular flow has begun, ended, aborted, etc.) that have advanced in response to a particular incoming event. Thus, a designer can prepare a process depending on the development or state of other processes.In processing an event, the event matching function 740 may compare the event with the event matching function for each active (e.g., interrupted) flow to determine whether the event matches one of the active flows (e.g., using a known matching technique and / or as described in more detail below). In some scenarios, multiple active flows specifying different interactions may be triggered by different conditions that may be met by the same event. If there is an event matcher from an active flow that matches the event (a matching flow), the flow matcher 740 may instruct the flow execution component 750 to forward that flow (e.g., and generate outgoing events to trigger all actions indicated by the driven flow).If there are multiple matching movements, the movement adjuster 740 may instruct the action conflict resolver 760 to determine whether or not the matching movements match in an action. If they match, action conflict resolver 760 (or flow matcher 740) may instruct flow execution component 750 to forward both matching flows. If they do not match, the action conflict resolver 760 may apply the conflict resolution to determine which action should take precedence and instruct the flow execution component 750 to advance the matching flow with the prioritized action and abort the other matching flow / s (e.g., because the interaction pattern represented by these flows no longer applies). If there are no active threads that match an event, the thread matcher may generate an internal event that matches a particular thread for handling non-matching or non-processed events, may execute one or more non-processed event processors (e.g., non-processed event processor 744), and / or may use another technique to handle an non-processed event.After checking for matches and advancing operations, the trace and control component 770 may check the operation states for all operations that have been completed or aborted and stop all active operations that have been activated by those completed or aborted operations (e.g., because the interaction pattern represented by those operations should no longer apply). Thus, interpreter 710 may iterate through events, advance flows, operate conflict management to determine which actions to perform, and generate outgoing events to trigger those actions.To illustrate, in some embodiments, interpreter 710 uses an event driven state machine, such as event driven state machine 800 of FIG. 8, to process incoming action events 805 and internal events 820. In some embodiments, event-driven state machine 800 may place incoming action events 805 (which may correspond to standardized input events 630 of FIG. 3, for example, and may be routed via an event gateway such as event gateway 180 of FIG. 1 ) into interaction event queue 810. Event driven state machine 800 may place internal events 820 (which may correspond to internal events 660 of FIG. 6, for example) into an internal event queue 815 (which may correspond to internal event queue 790 of FIG. 7, for example) and may give priority to the processing of events from internal event queue 815 to events from interaction event queue 810.For each event, event driven state machine 800 may perform at least some of the steps illustrated in block 825. For example, in block 830, event driven state machine 800 may check the event matcher indicated by each active flow for a match with the event. If there is an event matcher from an active flow that matches the event (a matching flow), the event driven state machine 800 may proceed to block 835 and advance that flow (e.g., generate outgoing interaction events 870 to trigger actions). If there are multiple matching sequences, event driven state machine 800 may proceed to block 840 and determine whether or not the matching sequences agree with an action. If they match, event driven state machine 800 may proceed to block 850 and advance both matching streams. If they do not match, event driven state machine 800 may proceed to block 855 and apply conflict resolution to determine which action should take precedence, advance the matching flow with the prioritized action, and abort the other matching flow / s. If there are no active procedures that match an event, event driven state machine 800 may proceed to block 835 and execute one or more unprocessed event processors (or generate an internal event that matches a particular procedure for handling non-matching or unprocessed events). After checking for matches and advancing flows, event driven state machine 800 may proceed to block 860, check flow states for completed or aborted flows, stop all active flows activated by those completed or aborted flows, and proceed to the next event in block 865. Thus, event driven state machine 800 may iterate through internal events 820 in internal event queue 815 and / or incoming action events 805 in interaction event queue 810, advance flows, perform conflict management to determine which interactions are to be carried out, and generate outgoing interaction events 870 to trigger these interactions.Returning to FIG. 7, in some embodiments, the interpreter 710 may support the use of natural language descriptions and the use of one or more LLMs, such as the generative LLM system 2800 of FIG. 28A or the generative LLM 2830 of FIGS. 4A, 4B, or 4C.For example, each of the interaction flows 780 may be specified with a corresponding natural language description summarizing the interaction pattern represented by the flow, and the interpreter 710 uses such flow descriptions under certain circumstances (e.g., a designated flow for handling unknown events and / or the unknown event processor 744 may request an LLM to determine whether an unmatched event representing an unrecognized user intent semantically matches the natural language description of an active flow representing a target user intent). In some embodiments, interpreter 710 may include a sequence description generator 720 that analyzes (e.g., at design time) one or more specified interaction sequences 780, performs lexical analysis to determine whether any specified sequences fail a corresponding sequence description, and if so, requests an LLM to generate such (e.g., based on the name and / or instructions of the sequence). Additionally or alternatively, the sequence description generator 720 may determine (e.g., request an LLM to do so) whether any specified sequence descriptions are inconsistent with their corresponding sequence descriptions, and if so, request an LLM to generate a new one (e.g., as a proposal or for automatic replacement) (e.g., from the name and / or instructions of the sequence). Thus, the sequence description generator 720 may determine whether to create a description for one of the interaction sequences 780 and may generate corresponding sequence descriptions.In some embodiments, a designer may specify a sequence description for / interaction flow / s 780 (e.g., a natural language description of what the flow is to do) without the command sequence or may call one of interaction flows 780 on behalf without defining it. Thus, in some embodiments, the interpreter 710 may include a flow auto-completion component 725 that analyzes (e.g., at design time, runtime) the interaction flow / s 780, identifies whether the interaction flow / s 780 lack an instruction sequence, and, if so, prompts an LLM to generate such (e.g., based on the name and / or description of the flow). For example, auto-completion component 725 may provide the LLM with one or more prompts including one or more example operations, a specified name of interaction operation / operations 780, and / or a (e.g., specified or generated) natural description of interaction operation / operations 780 and a request to complete interaction operation / operations 780.For example, auto-completion component 725 may generate a prompt using a template with place holders, such as the following: content: |-# Example Flows: { { Examples }} # Complete the following flow based on its instruction: Flow {{ Flow_Name }} """ { { { } } "This example template includes place holders for example threads, a particular name of the thread, and a particular description of the natural language thread. The auto-completion of operations component 725 may generate one or more prompt by filling the place holders with corresponding content (e.g., labeled(s) example operation(s), a particular name of the operation, a particular natural language description of the operation, and / or other content), and may provide this constructed prompt(s) to an LLM (e.g., via an API request). As such, the LLM may generate and return an automatically completed flow with generated instructions that may insert or otherwise associate the automatic completion flow component 725 with the corresponding interaction flow or flows 780.In an example implementation, the workflow execution component 750 may execute instructions specified in the interaction workflow / s 780 (e.g., including all event triggers encountered) until it reaches an event matcher, at which point the workflow execution component 750 may interrupt the interaction workflow / s 780. The flow alignment device 740 may process each event by executing the event alignment device in each interrupted flow and comparing the event to the target event parameters and the parameter values indicated by the flow alignment device's event specifier. Depending on the implementation, the flow alignment device 740 may support a variety of alignment techniques to determine whether an event matches an active event alignment device for one of the active flows. In general, the flow adjuster 740 may use any known technique to compare the parameters and parameter values of the target event with those of the event to generate a representation of whether the event matches (e.g., a binary indication or a match score quantifying an explicit or fuzzy match).However, in some embodiments, an event trigger or event matcher in one of the interaction flows 780 may indicate target event parameters and / or parameter values using natural language descriptions. Thus, in some embodiments, syntax generator 752 may derive the target event parameters and / or values from a specified natural language description in / interaction flow / s 780 (e.g., a description of all target event parameters and values, a description of a single parameter value), and syntax generator 752 may insert (or otherwise associate) the generated target event parameters and values into the corresponding event specifier in / interaction flow / s 780. Before the execution component 750 executes an instruction (e.g., an event trigger) that includes an event specifier, the execution component 750 may instruct (e.g., at runtime) the syntax generator 752 to determine whether the instruction includes a parameter indicated with a natural language description (e.g., with lexical analysis). Additionally or alternatively, before executing an instruction (e.g., an event aligner) that includes an event specifier, the flow alignment engine 740 may instruct the syntax generator 752 (e.g., at runtime) to determine whether the instruction includes a parameter specified by a natural language description (e.g., by lexical analysis). If so, syntax generator 752 may request an LLM to generate corresponding target event parameters and / or parameter values for the event specifier and update the event specifier with the generated target event parameter(s) and / or parameter values in a corresponding one of interaction flows 780.As an example of a prompt for generating a target event parameter value (or other variable value), syntax generator 752 may construct a prompt using a template prompt with place holders, such as the following: content: |-""" {{ general_Anweisungen }} """ # So, entertainment may be performed between a user and the bot: {{ Pattern_Unter }} # This is the current entertainment between the user and the bot: {{ History | Coolang }} # { { natural language description of the parameter value }}} ${{ Var_Name }} =This example template includes general instruction place holders, a call instance (or series of interactions), a history of the current call (or series of interactions), the name of the variable with the value to be generated, and a request to generate the value ("${{Var_name }}="). Syntax generator 752 may generate one or more prompt by filling the place holders with corresponding content (e.g., certain instructions, a certain call example or interaction history, a recorded history of the current call or a series of interactions, the extracted natural language description of the parameter value to be generated for a corresponding variable, the name of the variable, and / or other content), and may provide this constructed prompt(s) to an LLM (e.g., via an API request). As such, the LLM may generate and return the queried value that syntax generator 752 may insert into the event specifier in the corresponding instruction. This example is intended merely to demonstrate one way in which an LLM may be used to generate a target event parameter value from a specified natural language description of the value. Other types of prompt and prompt content may be implemented within the scope of the present disclosure. Moreover, those skilled in the art will know how to adjust the above example prompt to generate other content described herein (e.g., generating the name of a target event parameter from a natural language description of the target event parameter / s and / or parameter value(s), a list of names of auxiliary variables, etc.). Thus, the flow execution component 750 may execute an event trigger and / or the flow balancer 740 may execute an event balancer using target event parameters and parameter values generated by an LLM.Accordingly, in some embodiments, the flow adjuster 740 generates and / or quantify a representation of whether an event is a match (e.g., explicit or fuzzy) by comparing specified or generated target event parameters / parameter values of an event adjuster (e.g., keyword(s) or command(s) representing a target interaction modality, an action, an action state, and / or other event parameter values) to the corresponding parameters / parameter values of the tested event (e.g., keyword(s) or command(s) representing a commanded or recognized interaction modality, an action, an action state, and / or other event parameter values). Additionally or alternatively, the flow alignment device 740 may include a flow description alignment device 742 that requests (e.g., at runtime) an LLM to determine whether an event matches the flow description of one of the interaction flows 780 and / or a specified natural language description of one or more parameters or parameter values.At a high level, an event may represent a user action or intent, a bot action or intent, a scene interaction, or other type of event using a standardized interaction categorization scheme that classifies actions, action events, event parameters, and / or parameter values using (e.g., standardized, natural language, semantically meaningful) keywords and / or commands and / or a natural language description (e.g., GestureUserActionBeendet("thumb high")). As such, a sequence description alignmenter 742 of the sequence alignmenter 740 may execute an event alignmenter by requesting an LLM to determine whether the keywords, commands, and / or natural language description of the incoming or internal event match the (e.g., specified or generated) sequence description of one of the interaction sequences 780. For example, the policy description alignmentor 742 may generate a prompt using a prompt template that includes a prompt to determine whether an event matches a policy description, filling place holders with corresponding content (e.g., designated instructions, a designated example conversation or interaction history, a recorded history of the current conversation or interaction series(s), specified or generated policy description(s) of the interaction policy / procedures 780, keywords and / or commands represented by the incoming or internal event, and / or other content), and may provide this constructed request(s) to an LLM (e.g., via an API request). As such, the LLM may return an indication of whether the event matches a flow description of the interaction flow / flows 780. In many cases, an LLM may provide a more nuanced or semantic understanding of the matching than traditional express or fuzzy matching algorithms.Additionally or alternatively, the flow alignment device 740 may include a flow instruction alignment device 746 that requests an LLM to determine whether an incoming or internal event matches the instructions of an active flow of the interaction flow / flows 780. For example, the flow balancer 740 may trigger execution of a designated flow (e.g., to treat unknown events) or an unprocessed event processor 744 that includes the flow instruction balancer 746 in response to the flow balancer 740 applying one or more matching techniques (e.g., using express matching, fuzzy matching, flow description matching, and / or others) and determining that there are no active flows corresponding to the incoming or internal event.) In an example implementation, the unprocessed event handler 744 includes the flow instruction aligner 746 and a Bot interaction flow generator 748, but this is to be understood as an example only. In general, any number of adaptation techniques may be applied in any order, whether as an initial test, as part of the event processor for unprocessed 744, and / or in some other way.In one embodiment, the sequence instruction alignmenter 746 may request an LLM to determine whether an incoming or internal event and / or a representation of the most recent interaction history matches the indicated content of an active sequence of the interaction sequence / sequences 780. The expiration instruction aligner 746 may achieve this by deriving a user intent (e.g., by matching the incoming or internal event with instructions of a flow waiting for a corresponding user intent). In one embodiment, the sequence instruction matching function 746 may perform an event matching function by requesting an LLM to determine whether the keywords, commands, and / or natural language description of the incoming or internal event match the instructions (e.g., specified or generated) of one of the interaction sequences 780.For example, the sequence instruction balancer 746 may generate a prompt using a prompt template with place holders such as the following: content: |-""" {{ general_Anweisungen }} """ # So, a entertainment may be performed between a user and the bot: { { Pattern Entertainment }}} # These are the most likely intents of the users: { { { Example Flows and / or intents }} # This is the current entertainment between the user and the bot: { { { { History }} User Action: { { { Incoming or Internal Event }}User intention:This example prompt template includes general instruction place holders, a conversation example (or series of interactions), some possible flows representing the user intents to match (or a corresponding list of possible user intents), a flow of the current conversation (or series of interactions), keywords and / or commands represented by the incoming or internal event, and a prompt to predict the matching flow or user intent ("user intent:"). The sequence instruction balancer 746 may generate one or more prompts by filling place holders with appropriate content (e.g., particular instructions, a particular call pattern or interaction history, names and / or instructions of particular interaction sequences 780 representing possible user intents, a corresponding list of the possible user intents, a recorded history of the current conversation or interaction series(s), keywords and / or commands represented by the incoming or internal event, and / or other content), and may provide this constructed prompt(s) to an LLM (e.g., via an API request). As such, the LLM may return an indication of whether the event matches one of the interaction flows 780 and / or a corresponding user intent, and if so, which.In some scenarios, there may not be a matching flow that defines a bot response to a particular user interaction, or the flow balancer 740 may not identify. As such (e.g., in some embodiments where the flow alignment function 740 determines that there are no active flows that match an incoming or internal event representing a user interaction), the bot interaction flow generator 748 may request an LLM to generate a flow (e.g., at runtime). For example, in some embodiments, the flow alignmenter 740 (e.g., in some embodiments, the flow alignmenter 740 (e.g., the flow instruction alignmenter 746) may first use an LLM to attempt to align an unknown incoming or internal event with the name, instructions, and / or another representation of one or more designated flows waiting for corresponding target user intents (and defining bot responses), and if the LLM determines that there are no matching flows or target user intents, the bot interaction flow generator 748 may request (the same or a different) LLM to predict the user intent represented by an unknown incoming or internal event, generating an agent response intention and / or generating a response sequence. For example, if the unknown event represents a user action, the bot interaction flow generator 748 may apply any number of prompts to direct the LLM to classify the unknown user action into a user intent, generate an agent intent in response, and / or generate a flow that implements the agent intent in response.For example, the bot interaction flow generator 748 may generate a first prompt by using a prompt template with place holders, such as content: |-""" {{ general_Anweisungen } """ # So, entertainment between a user and the bot may be performed: { { Pattern_Unter } } } # This is the current entertainment between the user and the bot: { { { History }} Bot intent: and generate a second prompt, such as the following: Bot action:These example templates include general instruction place holders, a conversation example (or series of interactions), a history of the current conversation (or series of interactions) including keywords and / or commands represented by the incoming or internal event, a request to predict the intention of the responding agent ("Bot Intention:"), and a request to predict the history of a response of the agent ("Bot Action:"). The bot interaction flow generator 748 may generate one or more prompts by filling place holders with corresponding content (e.g., particular instructions, a particular call pattern or interaction flow, a recorded flow of the current call or a series of interactions, keywords and / or commands represented by the incoming or internal event, and / or other content), and may provide these constructed prompts to an LLM (e.g., via an API request). As such, the LLM may generate and return an agent flow in response that the bot interaction flow generator 748 may rank as matching and provide the flow execution component 750 for execution.The following example may be used to illustrate some possible prompt content. For example, interpreter 715 may generate a prompt (e.g., by padding a template prompt with a place holder for particular general instructions) that includes general instructions, such as:Subsequently, see a conversation between emma, a helpful AI assistant (Bot), and a user.The bot is designed to generate human-like actions based on the user actions it receives.The bot is verbal and provides many specific details.The bot would like to place with the user, including themes such as sport, music, recreational activities, NVIDIA, engineering, eating, weather and animals.When the user is silent, the bot motivates the user to try the different viewports by pointing to the different options that he can select by voice or by clicking on the presented options.If the user makes a question, the bot responds with an appropriate response.When the user gives an instruction, the bot follows this instruction.Appearance of the Bot:Emma carries a dark green rock and a white bluuse. A small card with the logo of the KI company NVIDIA is located on the blouse. Emma wears eyeglasses, white earrings and has brown, medium-length hair. Their eyes are green-brown.These are the available viewports: A) A simple numeral-rate game B) The example of the multi-modality shows how an interaction modelling language can process multiple parallel actions C) Shows how the bot can communicate with the user via return channels D) A showcase for presenting various bot postures that depend on the current interaction status E) Shows how the bot can be proactive by repeating a question if the user does not answerThis is important:The bot uses the action "bot gesture" as often as possible.The bot must not repeat when the user was silentUser actions: User did state "text" Bot actions: Bot does state "text" Bot informs "text Bot asks "text" Bot expression "text" Bot responds "text" Bot clarifys "text" Bot proposes "text" Bot gesture"In some embodiments, interpreter 715 may generate a prompt (e.g., by padding a prompt template with a place holder for a particular call instance or series of interactions) using a call instance or series of interactions, such as: In some embodiments, interpreter 715 may construct a prompt (e.g., by padding a prompt template with a place holder for a recorded history of the current conversation or a series of interactions) by creating a recorded history of the current conversation or a series of interactions, such as: # This is the current entertainment between the user and the bot: user intent: user was still 8.0The content of the request input described herein is merely exemplary and other variations may be implemented within the scope of the present disclosure.Returning to FIG. 6, the sensor system server 620 may translate detected input events 610 (e.g., a detected user input such as a detected gesture, a voice command, or a touch or click input; a detected feature or event associated with a user input such as a detected presence or absence of voice activity, a detected presence or absence of taps, a detected transcribed speech, a detected change in volume or speed of tapping; etc.) into standardized input events 630. In some embodiments, different sensor servers may process the detected input events 610 for different interaction modalities (e.g., a sensor server for translating detected gestures, a sensor server for translating detected voice commands, a sensor server for translating detected touch inputs, etc.). Thus, any sensor system server 620 may function as an event transponder that functions as a mediator between a corresponding input source and one or more downstream components (e.g., event gateway 180 of FIG. 1 ) by, for example, converting input events to a standardized format.Taking a sensor server for GUI input events as an example, the sensor server can effectively translate GUI input events (e.g., "user has clicked on the 'Chain Staff' button, is scrolld down, and has clicked on the 'Confirm' button) into standardized interaction level events (e.g., "user has selected the 'Chain Staff' option"). One possible example of a standardized interaction level event is an update event for the confirmation status (e.g. an event indicating a detected state or a change in state of a displayed confirmation status, such as "confirmed", "aborted" or "unknown"). For example, the sensor system server may translate various types of GUI inputs into corresponding confirmation status update events, and the translation logic may differ depending on the type of interaction element being presented or being interacted with. For example, a key press may be translated into a status update event "confirmed", or if a visual form contains a single form field input, the sensory server may translate an "enter" keyboard event into a status update event "confirmed". Another possible standardized interaction level event is an update event for selection (e.g., to indicate a detected change in the current selection of the user). For example, if a user selects a "chain slat" element from a list of multiple selection elements, the sensor system server may translate a corresponding recognized GUI input event (e.g., clicking or typing on a button or icon) into a standardized selection update event that indicates a recognized change in the current selection of the user. Another example of a possible standardized interaction level event is a form input update event that indicates an update of a requested form input. These are only a few examples, and others are conceivable within the scope of the present disclosure. Other examples of standardized interaction-level GUI input events (e.g., for recognized GUI gestures such as swipe, pull-up, or turn on touch screen devices), standardized interaction-level video input events (e.g., (e.g., representation of recognized visual gestures such as face recognition, pose recognition, object recognition, presence recognition, or motion tracking), standardized interaction-level audio input events (e.g., representation of recognized speech, recognized speech commands, recognition of keywords, other audio events, etc.), and / or others are contemplated within the scope of the present disclosure.FIG. 9 shows an example of an action server 930 according to some embodiments of the present disclosure. The action server 930 may correspond to the action server(s) 170 of FIG. 1 and / or the action server 670 of FIG. 6. At a high level, the action server 930 may be subscribed to or otherwise configured to capture and execute the events that the action server 930 is responsible for executing (e.g., one interaction modality per action server) from an event bus 910 (which may correspond to, e.g., the event gateway 180 of FIG. 1 ). In the embodiment depicted in FIG. 9, action server 900 includes one or more event operators 960 that advance incoming events to corresponding modality services, and an event interface manager 940 that manages event operator / s 960.For example, event interface manager 940 may subscribe to a global event channel of event bus 910 that transmits (e.g., standardized) events that indicate when an interaction channel connecting an interaction manager to an end user device has been detected (e.g., PipelineAc) or enabled (e.g., PipelineRelease). As such, event interface manager 940 may create a new event handler (e.g., event handler 960) in response to an event indicating that a new interaction channel has been acquired and / or delete an event handler in response to an event indicating that a corresponding interaction channel has been released. In some embodiments, event interface manager 940 performs periodic state checks (e.g., using any known technique such as interprocess communication) to ensure that event worker / workers 960 are healthy and are running. If the event interface manager 940 determines that one of the event workers 960 is not responding, the event interface manager 940 may restart the event worker.Event handler / s 960 may log in to one or more stream-related event channels of event bus 910 (e.g., a stream-related event channel dedicated to a particular interaction modality for which action server 930 is responsible), and may / may forward incoming events to the various modality services that registered for corresponding events. In some embodiments, event operators may run in separate (e.g., multi-process) processes (e.g., process 950) and manage incoming and outgoing events (e.g., using an Asycnio event loop).Modality services (e.g., modality services A and B in FIG. 9 ) may implement action-specific logic for each standardized action category and / or action event supported by an interaction modeling language and / or defined by an interaction categorization scheme for a particular interaction modality. Thus, a particular modality service may be used to map actions of a corresponding interaction modality to a particular implementation within an interactive system. In an example implementation, all supported actions are processed in a single interaction modality by a single modality service. In some embodiments, a modality service may support multiple interaction modalities, but different actions for the same interaction modality are not processed by different modality services. For example, in some embodiments that include an interactive visual content modality, different actions in that interaction modality (e.g., VisualFormSzenenAction, VisualSelectSzenenAction, VisualInformationAction) are processed by the same GUI modality service.FIG. 10 illustrates an example event flow by an example action server 1000 in accordance with some embodiments of the present disclosure. In this example, stream XY (e.g., which may correspond to one of the stream-related event channels of event bus 910 of FIG. 9 ) may include different events (e.g., events 1- 7), and an event handler 1010 (e.g., which may correspond to event handler / s 960 of FIG. 9 ) may subscribe to stream XY and provide a subscribed modality service (e.g., modality service A) with a corresponding event view (e.g., event view A, event view) filled with a subset of the events from stream XY for which the modality is subscribed to to receive. Thus, the modality service may perform instructed actions represented by its subscribed events, apply a corresponding modality policy to manage a corresponding action stack, and invoke corresponding action processors to perform the actions. As such, the action processors may execute the actions, generate internal events (e.g., to indicate timeouts), and place them in a corresponding event view (such that the modality service may take appropriate action and maintain the action stack and life cycle), and / or generate and place interaction modality events (e.g., to indicate that certain actions have been begun, ended, or updated) into the data stream XY.In some embodiments, each modality service registers with an event handler (e.g., event handler 1010) with a list of (e.g., types of) events of interest (e.g., processed by that modality service). Thus, event handler 1010 may provide an event view (e.g., event view A) that represents a subset of all events in that service's stream. The modality service may process the events within a corresponding event view in sequence. In some embodiments where action server 1000 includes multiple modality services, different modality services may process events in parallel (e.g., using an asynchronous event loop).In some embodiments, each modality service implements a particular modality policy (e.g., the modality policies illustrated in FIG. 4 ). In an example modality policy that allows parallel actions in a particular modality, a corresponding modality service may trigger, track, and / or otherwise manage parallel actions, and any number of actions may be performed simultaneously. The modality service may assign a common action identifier (e.g., Action_uid) that uniquely identifies a particular instance of an action, and may track the lifecycle of that instance of the action in response to action events generated by corresponding action processors and referencing the same action identifier.In an example of a modality policy that overrides overlapping actions in a particular modality, a corresponding modality service may manage a stack of actions, and the modality service may interrupt or skip a currently-performed action in response to a subsequently-instructed action. Once an action is completed and a corresponding (e.g., internal) action event representing that event is associated with the modality service, the modality service may trigger the top of the remaining actions in the stack to continue or render them visible again. For example, a modality service for animations may first trigger a Start GestureBotAction (gesture="Red") event, and if it subsequently receives a StartGesteBotAktion(Geste="nach point down") event before the speaking gesture (animation) is completed, the modality service may interrupt the speaking gesture, trigger the gesture "point down", and resume the speaking gesture when the gesture "point down" is completed.In some embodiments, a modality service may synchronize changes in action state with certain conditions (e.g., wait for a start of an action until a previous action of the same modality is completed, match the completion of two different actions in different modalities, match the start of an action with the end of another action, etc.). For purposes of illustration, an example of an action life cycle 1100 is shown in FIG. 11. For example, a modality service may receive a start action event 1110 indicating that an action should begin on a corresponding interaction modality operated on by that modality service. In the embodiment depicted in FIG. 1, the modality service may determine whether the modality is available at decision block 1120. For example, the modality service may implement a modality policy that waits for ongoing actions on that modality to be completed before a new action on that modality is started, the modality service may track the life cycle of initiated actions on that modality, and thus determine that there is another pending action that has been started and not yet completed. Accordingly, the modality service may wait until it receives an event (e.g., the modality event illustrated in FIG. 11 ) indicating that the action is complete, whereupon the modality service may proceed to decision block 1130. At decision block 1130, the modality service may determine whether a particular start condition is met (e.g., an instruction to synchronize the start of the new action with the start or completion of another action). As such, the modality service may wait until the designated start condition occurs (e.g., indicated by the synchronization event in FIG. 11 ) and the interaction modality remains free before initiating the action, and generate an event indicating that the action has begun in block 1140.At block 1150, prior to completion of the action, the modality service may determine whether a particular abort condition is met (e.g., abort of a blinking gesture if the bot has completed the disputed). Thus, the modality service may end the action when the designated abort condition occurs, or the action may last a certain duration and reach its natural completion. Once the action is completed or otherwise stopped, the modality service may generate an event indicating that the action has been stopped, at block 1160. Thus, the modality service may manage and track the lifecycle of the action and generate events representing changes in the state of the action during its lifecycle.Returning to FIG. 10, an action processor (e.g., Action1, Action2) may be responsible for executing a single category of supported (e.g., standardized) actions and implement a corresponding action state machine. For example, an action processor may receive events representing instructions for changing action status (e.g., start, stop, change) and may receive internal events from the modality service or from itself (e.g., API callback calls, timeouts, etc.). In some embodiments, action processors may directly inject events (e.g., standardized interaction modelling) into the stream XY (e.g., a change in action status such as "started", "ended", or "updated").In the following section, some example implementations of modality services are described, namely a GUI service that processes interactive visual content and an animation service that processes bot gesture actions.Example GUI service. In some embodiments, a GUI service (which may correspond to modality service B in FIG. 9, for example) handles interactive visual content actions (e.g., Visual InformationSzeneAction, VisualSelectSzeneAction, VisualFormSzeneAction), and corresponding events. In an example implementation, the GUI service translates standardized events representing commanded interactive visual content actions (e.g., commanded GUI updates) into calls to an API of a user interface server, applies a modality policy that overrides active actions by subsequently commanded, and manages a corresponding stack of visual information scene actions (e.g., in response to receiving an event that instructs a new interactive visual content action when there is at least one ongoing interactive visual content action). Thus, the GUI service may implement GUI updates that synchronize interactive visual content (e.g., visual information, a selection to which the user is prompted, or a field or form that the user is to fill in) with the current state of interaction with a call-carrying AI.In some embodiments, the GUI service may operate in coordination with a user interface server (e.g., on the same physical device, on a connected or networked physical device, etc.), such as user interface server 130 of FIG. 1. In general, the user interface server may be responsible for managing and providing user interfaces to a client device (e.g., client device 101 of FIG. 1 ) and providing front-end components (e.g., HTML files for content structuring, styling style sheets, JavaScript files for interactivity) that form the user interface of a web application. The user interface server may provide static elements such as images, fonts, and other resources to the user interface and / or use any known technique for providing user interfaces. The user interface server may function as a switch between the client device and the GUI service by converting GUI inputs to standardized GUI input events and standardized GUI output events to corresponding GUI outputs.The GUI service may manage an action state machine and / or an action stack for all interactive visual content actions. In an example implementation, the GUI service includes an action processor for each supported event for each supported interactive visual content action. FIGS. 12A-12C illustrate some example action processors for some example interactive visual content events, in accordance with some embodiments of the present disclosure. More specifically, FIG. 12A illustrates some example action processors for some example action events of a visual information scene, FIG. 12B illustrates some example action processors for some example action events of a visual selection, and FIG. 12C illustrates some example action processors for some example action events of a visual form.For example, an interactive visual content event (e.g., generated by an interaction manager such as the interaction manager 190 of FIG. 1 or the interaction manager 700 of FIG. 7 ) may instruct visualization of various types of visual information (e.g., in a 2D or 3D interface). In some embodiments, an interactive visual content event (e.g., payload data) includes fields indicating or encoding values representing a supported action type that categorizes a commanded action (e.g., Visual InformationSzenenAction, VisualSelectSzenenAction, VisualFormSzenenAction), an action status (e.g., "initiated", "scheduled", "starting", "running", "suspended", "continuing", "stopping", or "ending"), a representation of the commanded visual content, and / or other characteristics or information. The type of visual content indicated by an event may depend on the type of action.For example, (e.g., the payload) of an event (e.g., a launch event) for a visual information scene action may include fields indicating corresponding values such as a particular title, a particular summary of the information to be presented, particular content to be presented (e.g., a list of blocks of information to be presented to the user, where each block may include a particular text, a particular image (e.g., a description or identifier such as a uniform resource locator), or both), one or more particular support prompts that support or direct the user in selection, and / or others. Thus, for a corresponding event (e.g., a start event) for a visual information scene action, the action processor may translate the event into a (e.g., JSON) representation of a modular GUI configuration that specifies content blocks such as a pointer carousel block for one or more specified support blocks, a header block for a specified title, image and / or text blocks for the specified content, a (e.g., continue, abort) header(s), and / or other items. Accordingly, the action processor may use these content blocks to generate a user-defined page by filling a visual layout (e.g., a particular template or a visual frame layout with corresponding place holders) for a GUI overlay layout (e.g., an HTML layout), and may call the endpoint of the user interface server with the user-defined page to cause the user interface server to display the user-defined page.In some embodiments, (e.g., the payload) of an event (e.g., a launch event) for a visual selection action may include fields indicating corresponding values, such as a specified prompt (e.g., to describe a selection to be presented to the user), a specified image (e.g., a description or identifier such as a uniform source placer of an image to be presented with the selection), one or more specified assistance prompts that assist or direct the user in the selection, one or more specified options from which the user may choose (e.g., text, image, and / or other content for each option), a specified selection type (e.g., configuring the type of selection, The user may make such as selection, search bar, etc.), an indication of whether multiple selections are permitted, and / or others. Thus, for a corresponding event (e.g., a start event) for a visual selection action, the action processor may translate the event into a (e.g., JSON) representation of a modular GUI configuration that specifies content blocks, such as a pointer carousel block for one or more specified support blocks, a header block for a specified title, an image block for a specified image, a selectable option grid block for specified options, a (e.g., abort) destination(s), and / or other items. Accordingly, the action processor may use these content blocks to generate a user-defined page by filling a visual layout (e.g., a particular template or frame visual layout with corresponding place holders) for a GUI overlay layout (e.g., an HTML layout), and may call the endpoint of the user interface server with the user-defined page to cause the user interface server to display the user-defined page.FIGS. 13A-13F illustrate some example interactions with visual selection, in accordance with some embodiments of the present disclosure. For example, Figures 13A and 13D illustrate the presentation of a visual selection between four labeled images with an interactive avatar asking which image best fell to the user. FIG. 13B shows a scenario in which the user indicates the third image with a touch input, and FIG. 13E shows the same selection with a verbal input. In these scenarios, the verbal input may be detected, forwarded to a corresponding sensor server, and translated into a corresponding standardized event, forwarded to an interaction manager, and used to generate an event instructing a corresponding GUI update, an event instructing a verbal bot response, and / or an event instructing a corresponding gesture of the agent. As such, the events may be forwarded to corresponding action servers and executed. FIGS. 13C and 13F show an example of a bot response (e.g., visual highlighting of the selected selection and verbal confirmation).In some embodiments, (e.g., the payload) of an event (e.g., a launch event) for a visual form action may include fields indicating corresponding values, such as a particular prompt (e.g., a description of the user's desired input), a particular image (e.g., a description or identifier such as a uniform resource locator of an image to be presented with the selection), one or more specified assistance prompts that assist or direct the user in the selection, one or more specified user inputs (e.g., wherein each specified user input includes a specified input type such as numerical or date, a specified description such as "personal email address" or "birth place", etc.) and / or others. Thus, for a corresponding event (e.g., a start event) for a visual form action, the action processor may translate the event into a (e.g., JSON) representation of a modular GUI configuration defining content blocks defined by (e.g., (e.g., corresponding fields of the event), such as a hint carousel block for one or more specified support blocks, a header block for a specified prompt, an image block for a specified image, a list of input blocks representing corresponding form fields for specified inputs, a button (e.g., abort), and / or other items. Accordingly, the action processor may use these content blocks to generate a user-defined layout or page by filling a visual layout (e.g., a particular template or a visual shell layout with place holders for corresponding content blocks) for a GUI overlay page (e.g., an HTML page), and may call the endpoint of the user interface server with the user-defined layout or page to cause the user interface server to present the user-defined layout or page.In some embodiments where (e.g., the payload) of an event (e.g., a launch event) for an interactive visual content action specifies an image with a natural language description (e.g., "Summerbergs image"), a corresponding action processor for the event may trigger or perform an image search for a corresponding image. For example, an action processor may extract the natural language description of the desired image, interface to a suitable image search tool (e.g., via a corresponding API), and send the natural language description of the desired image to the search tool. In some embodiments, the search tool returns an identifier, e.g., a uniform resource locator for a matching image, and the action processor may insert the identifier into a corresponding block on a user-defined page. The action controller may provide the user-defined page to the user interface server (which may retrieve the specified image using the inserted identifier) for presentation.FIGS. 14A-14L show example layouts of visual elements for interactive visual content in accordance with some embodiments of the present disclosure. For example, FIG. 14A shows an example of a GUI overlay 1420 displayed over a scene with an interactive avatar. Figures 14B-14L show some example layouts of blocks of visual elements that could be used as corresponding GUI overlays. These are intended merely as examples, and other layouts may be implemented within the scope of the present disclosure.Example of an animation service. In some embodiments, an animation service (e.g., which may correspond to modality service A in FIG. 9 ) may process bot gesture actions (e.g., GestureBotAction) and corresponding events. In an example implementation, the animation service applies a modality policy that overrides active actions by subsequently commanded actions and creates a corresponding stack of actions in response to incoming Start GestureBotAction events when there are one or more current GestureBotAction actions. The animation service may manage an action state machine and an action stack for all GestureBotAktions, connect to an animation graph that implements a state machine with animation states and transitions between animations, and instruct the animation graph to set a corresponding state variable.In an example implementation, the animation service includes an action processor for each supported event for a bot gesture action. FIG. 12D illustrates some example action processors for some example GestureBotAction events, in accordance with some embodiments of the present disclosure.For example, a bot gesture action event (e.g., generated by an interaction manager such as the interaction manager 190 of FIG. 1 or the interaction manager 700 of FIG. 7) may instruct a particular animation (e.g., in a 2D or 3D interface). In some embodiments, a bot gesture action event (e.g., payload data) may include fields indicating or encoding values representing a supported action type that categorizes a commanded action (e.g., GestureBotAction), an action status (e.g., "initiated", "scheduled", "starting", "running", "pause", "continue", "stop", or "completed"), a representation of a commanded bot gesture, and / or other characteristics or information. For example, (e.g., the payload) of an event (e.g., a launch event) for a bot gesture action may include a field(s) that indicate (e.g., a natural language description or other identifier) a bot gesture. Depending on the implementation, one or more categories or types of actions (e.g., bot expression, pose, gesture, or other interaction or movement) may be standardized for corresponding bot functions, and a corresponding action event may indicate the desired action. Taking a bot gesture indicated for a bot gesture action category as a natural language description as an example, for a corresponding (e.g., start) event for the bot gesture action category, the action processor may extract the natural language description from the event, generate or access a sentence embedding for the natural language description of the bot gesture, use it to perform a similarity search on sentence embeddings for descriptions of the available animations, and select an animation using a similarity measure (e.g., nearest neighbor, within a threshold). If the best match is above a certain threshold, the action processor may trigger the animation graph to play a corresponding animation for the user. In some embodiments, for a corresponding event (e.g., launch) for the bot gesture action category, the action processor may extract the natural language description from the event and use any known generative technique to generate an animation from the natural language description (e.g., text-to-motion models, text-to-animation techniques, any other suitable animation technique).Example of Event Sequences. Some possible event sequences in example implementations are explained below. The following table illustrates, for example, a series of events that can be generated and distributed in an implementation in which a bot interacts with a user:UtteranceNutzerActionBeendet final_transkript="Hallo!"StartUßBotAktionScript="Hello together!"UtteranceBotActionGestartet between_script="Hello together!"UtteranceBotActionBeendet final_script="Hello together!"In this example, the event in the first row represents a detected completion of a user utterance ("Hello!"), which triggers an event instructing the bot to begin a response utterance ("Hello!"). The event in the second row indicates that the bot has started the utterance, and the event in the third row indicates that the bot has finished the utterance.The following table shows a series of events that can be generated and distributed in an implementation where a bot interacts with a user via gestures, emotions, and a display:...UtteranceBotActionBeendet final_script="Which option?".StartVisuelleSelectSzenenAktion=[... ]StartTimeTimeAction Duration=2secVisualAuswahlAktionTimerTimeActionTime Dauer-2secStartGesteAktionAktion="point to the display"GestureBotActionGestartetGestureBotActionBeendetTimerTimeActionTimeTime Duration=2secStartUßBotAktion="Take more time?".UtteranceBotActionGestartet script= "Take more time?"GestureNutzerAktion Geste="nick"StartGestBotActionWegbg="Back forward"GestureBotActionGestartetUtteranceBotActionBeendet final_script="Take more time?".GseBotActionBeendetExpression UserActionGestartet expression="hapy"In this example, the event in the first line indicates that the bot has completed the query "Which option?", triggering an event instructing the graphical user interface to present a visual selection. The event in the second row instructs a two second timer to start. The event in the third row indicates that the visual selection was presented and the event in the fourth row indicates that the timer was started, triggering an event instructing the bot to point to the display of the visual selection. The event in the fifth row indicates that the pointing gesture has begun and the event in the sixth row indicates that the pointing gesture has ended. The event in the seventh row indicates that the two-second timer has expired, triggering a bot utterance ("Take more time?"). The event in the eighth row indicates that the bot utterance has started. The event in the ninth row indicates that the recognition of a recognized user gesture (pitch) is complete, triggering a corresponding gesture of the agent (leaning forward). The event in the tenth row indicates that the bot gesture has begun. The event in the eleventh line indicates that the bot utterance ("Need Time?") has been ended, the event in the twelfth line indicates that the bot gesture (leaning forward) has been ended, and the event in the last line indicates that the start of a recognized user expression (malicious) has been recognized.In various scenarios, it may be advantageous to instruct an interactive system or one of its components (e.g., a sensor system server that controls input processing, an action server that implements bot actions), to perform a particular action or otherwise signal the expectation in anticipation of an event that an interaction manager (e.g., the interpreter) next awaits by the user or system. Some possible expectation actions in example implementations as well as other example functions will be explained below.For example, FIG. 15 shows an example event flow 1500 for a user utterance action in an implementation where a user 1518 is speaking with an interactive avatar 1504. In this example, the interactive avatar 1504 is implemented with a user interface 1516 (e.g., a microphone and audio interface), a voice activity detector 1514, an automatic speech recognition system 1512, and an action server 1510 responsible for processing events for user utterance actions (e.g., UtteranceUserAction 1508). In this example, the action server 1510 acts as both a sensor server and an action server that converts sensory inputs to standardized events and executes standardized events that command particular actions. An interaction manager 1506 may take over the decision making for the interactive avatar 1504. Although the interaction manager 1506 and the interactive avatar 1504 are shown as separate components, the interaction manager 1506 could be considered part of the interactive avatar 1504.In an example flow, the user 1518 begins speaking in step 1520. In step 1522, voice activity detector 1514 receives the voice and sends a voice stream to automatic speech recognition system 1512. In step 1524, voice activity detector 1514 notifies action server 1510 that voice activity has been detected, and in step 1526, automatic speech recognition system 1512 transmits the transcribed speech to action server 1510. In this sense, in step 1528, the action server 1510 generates a standardized event indicating that a recognized user utterance has begun (e.g., including the transcribed language) and sends the event (e.g., UtteranceUserActionGestartet) to the event gateway 1502 that the interaction manager 1506 accesses in step 1530.The following steps 1532-1546 may be executed in a loop. In step 1532, the user has spoken a few words, and in step 1534, the automatic speech recognition system 1512 sends a partial signature to the action server 1510. In step 1536, the action server 1510 generates a standardized event indicative of a detected update to a detected user utterance (e.g., including the transcribed language) and sends the event (e.g., UtteranceUserActionTranskriptAktualisiert) to the event gateway 1502 that the interaction manager 1506 accesses in step 1538. In step 1540, the user is speaking louder and in step 1542, the voice activity detector 1514 detects the increase in volume and notifies the action server 1510 of the detected change in volume. In step 1544, the action server 1510 generates a standardized event indicative of a detected update of a detected intensity of the user utterance (e.g., including a detected intensity or volume), and sends the event (e.g., UtteranceUserActionIntensitätAktualisiert) to the event gateway 1502 that the interaction manager 1506 accesses in step 1546.In some embodiments, the interaction manager 1506 generates a standardized event indicating expectation that the user is about to end speaking and / or instructs the interactive avatar 1504 to take a preparatory action at step 1548, and the interaction manager 1506 sends the event (e.g., StopUßUserAction) to the event gateway 1502 that the action server 1510 takes at step 1550. In response, in step 1552, the action server 1510 instructs the voice activity detector 1514 to decrease the audio dwell time (e.g., the amount of time a recognized voice signal is maintained before it is deemed inactive or muted).In step 1554, the user ceases speaking. In step 1556, voice activity detector 1514 detects voice inactivity and stops voice flow to automatic speech recognition system 1512, and in step 1558, automatic speech recognition system 1512 stops streaming a transcription to action server 1510. In step 1560, the hold time expires and in step 1562, voice activity detector 1514 notifies action server 1510 that voice inactivity has been detected. Thus, in step 1564, the action server 1510 generates a standardized event indicating a detected completion of a detected user utterance and sends the event (e.g., ÄußerungNutzerAktionBeended) to the event gateway 1502 that the interaction manager 1506 receives in step 1566.FIG. 16 shows an example event flow 1600 for a user utterance action in an implementation where a user 1618 is speaking with a chatbot 1604. In this example, chatbot 1604 is implemented with user interface 1616 (e.g., a hardware or software keyboard and driver), timer 1612, and action server 1610 responsible for processing events for user utterance actions (e.g., utteranceUserAction 1608). In this example, the action server 1610 functions as both a sensor server and an action server that converts sensory inputs (e.g., recognized text, tap speed) into standardized events and executes standardized events that command particular actions. An interaction manager 1606 may take the decision making for the chatbot 1604. Although the interaction manager 1606 and chatbot 1604 are shown as separate components, the interaction manager 1606 could be considered part of the chatbot 1604.In an example flow, the user 1618 begins typing in step 1620. In step 1622, the user interface 1616 notifies the action server 1610 that typing has begun, and in step 1624, the action server 1610 generates a standardized event indicating that a recognized user utterance has begun, and sends the event (e.g., UtteranceNutzerAktion) to the event gateway 1602 that the interaction manager 1606 accesses in step 1626.The following steps 1628-1640 may be executed in a loop. In step 1628, the user interface 1616 sends the typed text to the action server 1610, and in step 1630, the action server 1610 generates a standardized event that indicates a recognized update to a recognized user utterance (e.g., including the typed text), and sends the event (e.g., UtteranceUserActionTranskriptAktualisiert) to the event gateway 1602 that the interaction manager 1606 accesses in step 1634. In step 1632, the user begins to tap faster, and in step 1636, the user interface 1616 detects the increase in tap speed and notifies the action server 1610 of the detected speed change. In step 1638, the action server 1610 generates a standardized event that indicates a detected update of a detected intensity of the user utterance (e.g., including a detected intensity or tap speed), and sends the event (e.g., UtteranceUserActionIntensitätAktualisiert) to the event gateway 1602, which the interaction manager 1606 receives in step 1640.In some embodiments, the interaction manager 1606 generates a standardized event at step 1642 indicating that the user is about to cease typing and / or instruct the chatbot 1604 to take a preparatory action, and the interaction manager 1606 sends the event (e.g., StopUßUserAction) to the event gateway 1602 that the action server 1610 grasps at step 1644. In response, at step 1646, the action server 1610 reduces the timeout after the key press (e.g., the amount of time that a detected inactivity or lag in typing is interpreted as completion of the utterance).In step 1648, the user ceases to tap. In step 1650, the user interface 1610 sends a notification to the action server 1610 that the input has been ended, and in step 1652, the action server 1610 instructs the timer 1612 to start. In step 1654, timer 1612 notifies action server 1610 that the timer has expired, and action server 1610 informs user interface 1616 to disable the input field for further inputs. In step 1658, the user interface 1616 sends the completed text input to the action server 1610. In this sense, in step 1660, the action server 1610 generates a standardized event that indicates the detected completion of a detected user utterance (e.g., including the completed text input), and sends the event (e.g., ÄußerungNutzerAktionBeended) to the event gateway 1602 that the interaction manager 1606 accesses in step 1662.FIG. 17 shows an example event flow 1700 for a bot expectation action in an implementation where a user 1718 is speaking with an interactive avatar 1704. In this example, the interactive avatar 1704 is implemented with a client device 1716 (e.g., with a microphone and an audio interface), an automatic speech recognition system 1714, and an action server 1712 responsible for processing events for user utterance actions (e.g., UtteranceUserAction 1710) and bot expectation actions for user utterance actions (e.g., BotExAction 1708). In this example, the action server 1712 acts as both a sensor system server and an action server that converts sensory inputs to standardized events and executes standardized events that command particular actions. An interaction manager 1706 may take over the decision making for the interactive avatar 1704. Although the interaction manager 1706 and the interactive avatar 1704 are shown as separate components, the interaction manager 1706 could be considered part of the interactive avatar 1704.In step 1720, the interaction manager 1706 generates a standardized event that indicates that the start of a user utterance is being expected shortly and that is an instruction to perform a preparatory action in anticipation of the user utterance, and sends the event (e.g., StartBotErwartungAktion(ÄußerungNutzerAktionBeended)) to the event gateway 1702 that the action server 1712 is accessing in step 1722. Note that in this example, the argument of the keyword used to identify the expectation is an expected destination event (e.g., the completion of the user utterance) that can trigger a corresponding stop action indicating that the expectation of the interaction manager 1706 has been fulfilled or is no longer relevant, which in turn can trigger the reversal of the preparatory action, but this syntax is intended merely as an example and need not be used. In response, action server 1712 informs client device 1716 to deactivate its audio output in step 1724, informs client device 1716 to activate its microphone in step 1726, and informs automatic speech recognition system 1714 to activate automatic speech recognition in step 1728. In step 1730, the action server 1712 generates a standardized event confirming that the bot expectation action has begun and / or indicating that the preliminary action has been initiated, and sends the event (e.g., BotErwartungAktionGestartet(ÄußerungNutzerAktionBeended)) to the event gateway 1702 that the interaction manager 1706 is accessing in step 1732.In some embodiments, when the user 1718 begins speaking, the language is recognized (not shown), and in step 1734, the action server 1712 generates a standardized event indicating that a recognized user utterance has begun and sends the event (e.g., UtteranceUserActionGestartet) to the event gateway 1702 that the interaction manager 1706 accesses in step 1736. Once the user 1718 ceases speaking, the end of the utterance is detected (not shown), the action server 1712 generates a standardized event indicating a detected completion of a detected user utterance, and sends the event (e.g., ÄußerungNutzerAktionBeended) to the event gateway 1702 (not shown), which the interaction manager 1706 receives at step 1738. In this example, the interaction manager 1706 is programmed to stop the bot expectation action in response to receiving an event indicating the detected completion of a detected user utterance. Thus, in step 1740, the interaction manager 1706 generates a standardized event indicating that the expected user utterance is completed and instructs the reversal of the preparatory action, and sends the event (e.g., StopBotErwartungAktion(ÄußerungNutzerAktionBeended)) to the event gateway 1702 that the action server 1712 receives in step 1742. Then, in step 1744, the action server 1712 instructs the automatic speech recognition system 1714 to end the automatic speech recognition, and in step 1746, instructs the client device 1716 to deactivate its microphone. In step 1748, the action server 1712 generates a standardized event confirming that the bot expectation action is completed and / or indicating that the preliminary action has been reversed, and sends the event (e.g., BotErwartungAktionBeended(ÄußerungNutzerAktionBeended)) to the event gateway 1702 that the action server 1712 receives in step 1750.Flow Diagrams. Referring to FIGS. 18-27, each block of methods 1800-2700 described herein includes a computing process that may be performed with any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. The method 1800- 2700 may also be embodied in the form of computer usable instructions stored on computer storage media. The method 1800- 2700 may be provided by a stand-alone application, service or hosted service (stand-alone or in combination with another hosted service), or plug-in for another product, to name just a few. Moreover, methods 1800- 2700 are described with reference to an example system, e.g., interactive system 100 of FIG. 1, however, these methods may additionally or alternatively be performed by any system or combination of systems, including, but not limited to, the systems described herein.FIG. 18 is a flow diagram illustrating a method 1800 of generating a representation of a responsive agent action categorized using an interaction categorization scheme, in accordance with some embodiments of the present disclosure. The method 1800 includes, at block B 1802, receiving, by an interpreter of an interactive agent platform connected to an interactive agent, one or more representations of one or more recognized user actions categorized using an interaction categorization scheme. For example, with respect to the interactive system 100 of FIG. 1, a representation of user input, such as a gesture detected by the vision microservice 110, a voice command detected by the voice recognition microservice 120, or a touch or click input detected by the UI server 130, may be forwarded to a corresponding one of the sensor servers 160 responsible for a corresponding interaction channel. The sensor server / s 160 may translate this user input into a standardized representation of a corresponding event defined by an interaction categorization scheme and forward the event to an event gateway 180. The interaction manager 190 may implement an interpreter that subscribes to or otherwise is configured to receive these events from the event gateway 180.The method 1800 includes, at block B 1804, generating one or more representations of one or more responsive agent actions categorized using the interaction categorization scheme based at least on the interpreter executing one or more command lines of one or more interaction flows composed in an interaction modelling language and instructing one or more agent actions to generate responsive to the one or more detected user actions. For example, with respect to the interactive system 100 of FIG. 1, an interpreter implemented by the interaction manager 190 may support an interaction modelling language, and code implementing the decision logic for an interactive agent may be written in the interaction modelling language, loaded into or otherwise retrieved from the interaction manager 190 and executed by the interaction manager 190. As such, the interaction manager 190 may process events from the event gateway 180 (e.g., using an event driven state machine), determine which interactions are to be performed, and generate commands as corresponding events in a standardized representation, and forward to the event gateway 180.FIG. 19 is a flow diagram illustrating a method 1900 of generating a responsive agent action representation based at least on execution of one or more interaction flows, in accordance with some embodiments of the present disclosure. The method 1900 includes, at block B 1902, receiving, by an interpreter of an interactive agent platform that supports concurrent execution of agent actions in different interaction modalities, one or more representations of one or more recognized user actions. For example, with respect to the interactive system 100 of FIG. 1, the interaction manager 190 may implement an interpreter that is subscribed to or otherwise configured to receive events representing captured user actions from the event gateway 180. The interaction manager 190 may implement decision logic for an interactive agent written in an interaction modeling language, and the interaction modeling API and / or language used by the interaction manager 190 may define mutually exclusive interaction modalities such that events instructing actions in different interaction modalities may be performed independently (e.g., simultaneously) by corresponding action servers 170 dedicated to corresponding interaction modalities.The method 1900 includes, at block B 1904, generating one or more representations of one or more responsive agent actions, at least based on the interpreter executing one or more command lines of one or more interaction flows in response to the one or more detected user actions. For example, with respect to the interactive system 100 of FIG. 1, code implementing the decision logic for an interactive agent and defining one or more interaction flows in which interaction modelling language is written may be loaded into an interpreter implemented by the interaction manager 190 or otherwise accessed and executed by the interpreter. Thus, the interaction manager 190 may process events (e.g., detected user actions) from the event gateway 180 (e.g., using an event driven state machine), determine which interactions are to be performed, and generate commands as corresponding events in a standardized representation, and forward them to the event gateway 180.FIG. 20 is a flow diagram illustrating a method 2000 of triggering an interactive avatar to provide back channel feedback, in accordance with some embodiments of the present disclosure. The method 2000 includes, at block B 2002, receiving, by an interpreter associated with an interactive avatar supporting non-sequential human-machine interactions, one or more representations of one or more captured introductions of one or more user actions, and at block B 2004, triggering the interactive avatar based at least on the interpreter executing one or more command lines of one or more interaction actions in response to the one or more captured introductions to provide back channel feedback during the one or more user actions. For example, the interactive system 100 of FIG. 1 may support non-sequential interactions using various features described herein, such as an event driven architecture, an interpreter that supports various keywords, and / or decoupling sensory processing, interaction decision, and action execution. For example, to assist in executing a responsive agent action prior to completion of a triggering user action (e.g., an utterance), the sensor system server(s) 160 may generate an event representing a detected initiation of the user action and provide the event (via the event gateway 180) to the interaction manager 190, and the interaction manager 190 may check the event for an appropriate active (e.g., interrupted) flow waiting for such an event. Because the sensor server / s 160, interaction manager 190 and action server / s 170 may operate independently of each other, the sensor server / s 160 may continue to process the user input while the interaction manager 190 generates an event representing a response action and causes the action server / s 170 to execute the response action (e.g., back channelling feedback).FIG. 21 is a flow diagram illustrating a method 2100 of generating an interaction modeling event that instructs an interactive agent to execute a responsive agent or scene action, in accordance with some embodiments of the present disclosure. The method 2100 includes, at block B102, receiving one or more first interaction modelling events via one or more event gateways and by an interaction manager connected to an interactive agent, these events representing at least one of the following actions: one or more detected user actions, one or more commanded agent actions, or one or more commanded scene actions. For example, with respect to the interactive system 100 of FIG. 1, the sensor system server / servers 160 may translate detected user inputs into a standardized representation of a corresponding event and place the event on an event gateway 180. Additionally, referring to FIG. 6, the interaction manager 640 may generate internal events 660 representing internal state changes (e.g., expiration state changes) or commanded bot actions, and / or the action server 670 may generate events 665 representing the confirmation of action state changes. As such, the interaction manager 190 of FIG. 1 and / or the interaction manager 640 of FIG. 6 may be subscribed to or otherwise configured to receive the events from the event gateway 180.The method 2100 includes, at block B 2104, generating one or more second interaction modeling events that command the interactive agent to perform at least one of the following actions: one or more responsive agent actions or one or more responsive scene actions based at least on the interaction manager processing of the one or more first interaction modeling events using an event driven state machine. For example, with respect to the event driven interactive system 600 of FIG. 6, the interaction manager 640 (which may correspond to the interaction manager 190 of FIGS. 1 and / or 2 ) may be responsible for deciding which actions the interactive system 600 should perform in response to user actions or other events (e.g., the standardized input events 630, the internal events 660, the events 665 representing an acknowledgement of action state changes). The interaction manager 640 may interact with the rest of the interactive system 600 via an event driven mechanism. As such, the interaction manager 640 may evaluate various types of events (e.g., the standardized input events 630, the internal events 660, the events 665 representing the confirmation of action state changes), determine which actions to perform, and generate corresponding commanded bot action events 650 or event instruction updates for another aspect of the scene (e.g., interactive visual content actions).FIG. 22 is a flow diagram illustrating a method 2200 of triggering one or more responsive agent or scene actions specified by one or more matching interaction flows, in accordance with some embodiments of the present disclosure. The method 2200 includes, at block B2202, tracking one or more intermittent interaction flows representing one or more human-machine interactions. For example, with respect to the event driven interactive system 600 of FIG. 6, the interaction manager 640 may support and track multiple active flows (e.g., interrupted at corresponding event balancers).The method 2200 comprises, in block B 2204, checking one or more incoming interaction events for one or more matching interaction sequences of the one or more interrupted interaction sequences. For example, with respect to the event driven interactive system 600 of FIG. 6, the interaction manager 640 may use an event driven state machine to wait for events that match an event balancer of an active flow. For example, the flow alignment device 740 of FIG. 7 may evaluate incoming events to determine whether they match an event alignment device of an active flow, where incoming events are processed sequentially (e.g., from the internal event queue 790, from another queue or event gateway, such as the event gateway 180 of FIG. 1 ), and for each event, the event alignment device indicated by each active flow is tested for a match with the event.The method 2200 includes, at block B 2206, triggering one or more responsive agent or scene actions specified by the one or more matching interaction flows in response to identifying the one or more matching interaction flows. For example, with respect to the event driven interactive system 600 of FIG. 6, the interaction manager 640 may trigger corresponding events and actions specified in sequences corresponding to the event tested. For example, the flow balancer 740 of FIG. 7 may instruct the flow execution component 750 to forward matching (e.g., non-conflicting) flows, and the forwarding flows may instruct the flow execution component 750 to generate outgoing events that instruct an action.FIG. 23 is a flow diagram illustrating a method 2300 of generating a responsive agent or scene action based at least on prompting one or more big language models, in accordance with some embodiments of the present disclosure. The method 2300 includes, at block B 2302, receiving, by an interpreter of an interactive agent platform, one or more representations of one or more recognized user actions. For example, with respect to the interactive system 100 of FIG. 1, the sensor system server or servers 160 may translate detected user inputs into a standardized representation of a corresponding event and place the event on an event gateway 180, and the interaction manager 190 may implement an interpreter subscribed to or otherwise configured to retrieve or receive the events from the event gateway 180.The method 2300 includes, at block B 2304, generating one or more representations of one or more actions of the detected user or users for one or more matches with one or more interrupted interaction flows, at least based on the interpreter's request to create one or more big language models (LLMs) and evaluate the one or more representations of one or more actions of the agent or scene. For example, referring to FIG. 7, the interpreter 710 may support the use of natural language descriptions and the use of one or more LLMs. For example, interpreter 710 may request an LLM to generate a natural language description of one or more instruction lines defining a flow, generate one or more instruction lines for a particular flow, determine whether an event matches a flow description of an active flow, determine whether a non-matching event matches the name and / or instruction(s) of an active flow, generate a flow in response to a non-matching event, and / or others.FIG. 24 is a flow diagram illustrating a method 2400 of generating one or more outbound interaction modelling events that direct one or more action servers to perform one or more responsive agent or scene actions, in accordance with some embodiments of the present disclosure. The method 2400 includes, at block B 2402, generating, by one or more sensor servers in one or more input interaction channels, one or more incoming interaction modelling events representing one or more detected user actions. For example, with respect to the interactive system 100 of FIG. 1, the sensor system server or servers 160 may translate detected user inputs into a standardized representation of a corresponding event and place the event on an event gateway 180.The method 2400 includes, at block B2404, generating, by an interaction manager, one or more outbound interaction modelling events based on at least one or more inbound interaction modelling events that instruct one or more action servers in one or more outbound interaction channels to perform one or more responsive agent actions or scene actions associated with an interactive agent. For example, with respect to the interactive system 100 of FIG. 1, the interaction manager 190 may implement an interpreter that subscribes to or otherwise is configured to receive the events from the event gateway 180, process the events (e.g., using an event driven state machine), determine which interactions are to be performed, and generate commands as corresponding events in a standardized representation, and forward them to the event gateway 180. The action server / s 170 responsible for the corresponding interaction channel / channels may / may be subscribed to or otherwise configured to receive / receive the events for which he / she is responsible for executing from the event gateway 180. The action server / s 170 may execute, plan, and / or otherwise handle events for the corresponding interaction modality(s) by / are interfacing with a corresponding service that controls a corresponding output interface.FIG. 25 is a flow diagram illustrating a method 2500 of generating a visual layout representing an update specified by an event, in accordance with some embodiments of the present disclosure. The method 2500 includes, at block B 2502, receiving, from one or more action servers that handle one or more visual content overlays complementing one or more conversations with an interactive agent, one or more events that represent one or more visual content actions categorized using an interaction categorization scheme, and instructing one or more updates of the one or more overlays in one or more GUIs. For example, referring to FIG. 9, the action server 930 may include a GUI service (e.g., modality service B) that processes interactive visual content and corresponding events. An interactive visual content event (e.g., generated by an interaction manager such as the interaction manager 190 of FIG. 1 or the interaction manager 700 of FIG. 7 ) may instruct visualization of various types of visual information (e.g., in a 2D or 3D interface). In some embodiments, an interactive visual content event (e.g., payload data) includes fields indicating or encoding values representing a supported action type that categorizes an instructed action (e.g., Visual InformationSzenenAction, VisualSelectSzenenAction, VisualFormSzenenAction), an action status (e.g., "initiated", "scheduled", "starting", "running", "suspended", "continuing", "stopping", or "ending"), a representation of the instructed visual content, and / or other characteristics or information.The method 2500 includes, at block B 2504, generating, by the one or more action servers, one or more visual layouts representing the one or more updates specified by the one or more events. For example, referring to FIG. 9, the action server 930 may include a GUI service (e.g., modality service B) that includes an action processor for each supported event for each supported interactive visual content action, and the action processor for a corresponding (e.g., a start) event for a visual information scene action may translate the event into a (e.g., JSON) representation of a modular GUI configuration that includes content blocks such as a pointer carousel block for one or more specified support blocks, a header block for a specified title, image and / or text blocks for the specified content, a (e.g., continuation, image and / or text blocks for the specified content, Abort)Schaltfläche(s) and / or other elements. Accordingly, the action processor may use these content blocks to generate a user-defined page by filling a visual layout (e.g., a particular template or a visual shell layout with corresponding place holders) for a GUI overlay layout (e.g., an HTML layout), and may call the endpoint of the user interface server with the user-defined page to cause the user interface server to display the user-defined page.FIG. 26 is a flow diagram illustrating a method 2600 of triggering an animation state of an interactive agent in accordance with some embodiments of the present disclosure. The method 2600 includes, at block B 2602, receiving, from one or more action servers that handle animation of gestures of an interactive agent, one or more first interaction modeling events that command one or more target states of one or more agent gestures presented using an interaction categorization scheme. For example, referring to FIG. 9, the action server 930 may include an animation service (e.g., modality service A) that processes bot movements and / or gesture actions (e.g., gesture-gesture action) and corresponding events. For example, a bot gesture action event (e.g., generated by an interaction manager such as the interaction manager 190 of FIG. 1 or the interaction manager 700 of FIG. 7 ) may represent a particular animation (e.g., in a 2D or 3D interface) using fields indicating or encoding values that represent a supported action type that categorizes a commanded action (e.g., gesture gesture action), an action status (e.g., start, started, updated, stopped, ended), a representation of a commanded bot gesture, and / or other characteristics or information.The method 2600 includes, at block B 2604, triggering, by the one or more action servers, one or more animation states of the interactive agent corresponding to the one or more target states of the one or more agent gestures commanded by the one or more first interaction modelling events. For example, referring to FIG. 9, the action server 930 may include an animation service (e.g., modality service A) that includes an action processor for each supported event for each supported bot gesture action. FIG. 12D illustrates some example action processors for some example GestureBotAction events, in accordance with some embodiments of the present disclosure. Taking a bot gesture specified for a bot gesture action as a natural language description as an example, the action processor may extract the natural language description from the event for a corresponding event (e.g., launch) for the bot gesture action, generate or access a sentence embedding for the natural language description of the bot gesture, use it to perform a similarity search on sentence embeddings for descriptions of the available animations, and select an animation using a similarity measure (e.g., nearest neighbor, within a threshold).FIG. 27 is a flow diagram illustrating a method 2700 for performing one or more preparatory actions in accordance with some embodiments of the present disclosure. The method 2700 includes, at block B 2702, receiving, by one or more servers connected to an interactive agent, one or more first interaction modeling events that instruct one or more preparatory actions associated with expectation that one or more specified events will occur and that are presented using an interaction categorization scheme. For example, referring to FIG. 17, in step 1720, the interaction manager 1706 generates a standardized event indicating that a user utterance is likely to begin soon and instruct a preparatory action, and sends the event (e.g., StartBotErwartungAktion(ÄußerungNutzerAktionBeended)) to the event gateway 1702 that the action server 1712 is picking in step 1722.The method 2700 includes, at block B 2704, performing the one or more preparatory actions by the first server. For example, referring to FIG. 17, in step 1724, the action server 1712 informs the client device 1716 to deactivate its audio output, in step 1726, informs the client device 1716 to activate its microphone, and in step 1728, informs the automatic speech recognition system 1714 to activate automatic speech recognition. In step 1730, the action server 1712 generates a standardized event confirming that the bot expectation action has begun and / or indicating that the preliminary action has been initiated, and sends the event (e.g., BotErwartungAktionGestartet(ÄußerungNutzerAktionBeended)) to the event gateway 1702 that the interaction manager 1706 is accessing in step 1732.The systems and methods described herein may be used for a variety of purposes, including, but not limited to, control of machines (e.g., robots, vehicles, construction machines, storage vehicles / machines, autonomous, semi-autonomous, and / or other types of machines), machine travel, machine driving, synthetic data generation, model training (e.g., using real, augmented, and / or synthetic data such as synthetic data generated using a simulation platform or system, synthetic data generation techniques such as, but not limited to, those described herein, etc.), perception, augmented reality (AR), virtual reality (VR), mixed reality (MR), Robotics, security, and monitoring (e.g., in a smart ciies implementation), autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation, and / or digital twinning, data center processing, call-guiding AI, light transport simulation (e.g., ray tracing, path tracing, etc.), distributed or collaborative content creation for 3D assets (e.g., using universal scene descriptor (USD) data such as OpenUSD and / or other types of data), cloud computing, generative artificial intelligence (e.g., using one or more diffusion models, Transformer models, etc.) and / or other suitable applications.Included embodiments may be included in a variety of different systems, such as automotive systems (e.g., an autonomous or semi-autonomous machine control system, an autonomous or semi-autonomous machine perception system), systems implemented using a robot or robot platform, aeronautical systems, medial systems, boat systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations (e.g., in a driving or vehicle simulation, in a robot simulation, in a smart city or monitoring simulation, etc.), Systems for performing twin digital operations (e.g., in conjunction with a platform or system for collaborative content creation, such as OMNIVERSSE from NVIDIA and / or another platform, system, or service that uses USD or OpenUSD data types), systems implemented using an edge device, systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations (e.g., (e.g., using one or more neural rendering fields (NERFs), Gaussian Plate techniques, diffusion models, transformer models, etc.), systems, which are implemented at least partially in a data center, systems for performing conversation AI operations, systems implementing one or more language models - such as one or more large language models (LLMs), one or more visual language models (VLMs), one or more multi-modal language models, etc., systems for performing light transport simulations, systems for collaboratively creating content for 3D goods (e.g., using universal scene description data (USD) such as OpenUSD, CAD data (computer aided design), 2D and / or 3D graphics or design data, and / or other data types), systems, the at least some of which are implemented using cloud computing resources, and / or other types of systems.In some embodiments, the systems and methods described herein may be performed within a 3D content collaboration platform (e.g., OMNIVERSSE from NVIDIA) for 3D playback, industrial digitalization, generative physical AI, and / or other use cases, applications, or services. The content collaboration platform may, for example, house a framework for the development and / or deployment of interactive agents (e.g., an interactive avatar), and a system for the use or development of universal scene descriptors (USDs) (e.g., OpenUSDs) for the management of objects, features, scenes, etc., in a digital environment, a simulated environment, etc. The platform may include a real physics simulation, e.g., using the NVIDIA PhysX SDK to simulate real physics and physical interactions with virtual objects, characters, simulations, or other types of 3D content hosted by the platform. The platform may integrate OpenUSD into software tools and rendering workflows along with ray tracing / path tracing / light transport simulation (e.g., NVIDIAs RTX rendering technologies). In some embodiments, the development and / or deployment of an interactive agent (e.g., an interactive bot or robot) may utilize one or more cloud services and / or machine learning models (e.g., neural networks, large language models). The avatar cloud engine (ACE) of NVIDIA is, for example, a suite of cloud-based AI models and services developed for creating and managing interactive, life-fast avatars using hosted natural language processing, speech recognition, computer vision, and / or conversational AI services. In some embodiments, an interactive agent may be developed and / or deployed as part of an application hosted by a (e.g., streaming) platform, such as a cloud-based gaming platform (e.g., NVIDIA Ge FORCE NOW). Thus, interactive agents such as digital avatars may be developed and / or deployed for various applications, e.g., service, virtual assistants, interactive entertainment or games, digital twins (e.g., for video conferencing participants), training or training, health care, virtual or augmented reality experiences, social media interactions, marketing and advertising, and / or other applications.EXAMPLE SPEECH MODELSAt least in some embodiments, speech models such as large language models (LLMs), vision language models (VLMs), multimodal speech models (MMLMs), and / or other types of generative artificial intelligence (AI) may be implemented. These models may be capable of understanding, composing, translating, and / or otherwise generating text (e.g., natural language text, code, etc.), images, videos, computer aided design (CAD) goods, OMNIVERSSE, and / or METAVERSE file information (e.g., in USD format such as OpenUSD), and / or the like, based on the context provided in prompts or queries. These language models may be considered "large" in embodiments because the models are trained on large datasets and have architectures with a large number of trainable network parameters (weights and distortions), such as millions or billions of parameters. The LLMs / VLMs / MMLMs / etc. can be used to summarize text data, analyze and obtain findings from data (e.g., text, image, video, etc.), and generate new texts / images / videos / etc. in user-specific styles, tones, and / or formats. The LLMs / VLMs / MMLMs / etc. of the present disclosure may be used exclusively for word processing in certain embodiments, while in other embodiments multi-modal LLMs may be implemented to accept, understand, and / or generate text and / or other types of content such as images, audio, 2D and / or 3D data (e.g., in USD formats) and / or video. For example, vision speech models (VLMs) or more generally multi-modal speech models (MMLMs) may be implemented to accept image, video, audio, text, 3D design (e.g., CAD), and / or other input data types, and / or generate or output image, video, audio, text, 3D design, and / or other output data types.Various types of LLMs / VLMs / MMLMs / etc. architectures may be implemented in various embodiments. For example, various architectures may be implemented using different techniques for understanding and generating outputs, such as text, audio, video, image, 2D and / or 3D design or asset data, etc. In some embodiments, LLMs / VLMs / MMLMs / etc. architectures such as recurrent neural networks (RNNs) or long short-term memory (LSTMs) networks may be used, while in other embodiments, transformer architectures such as those based on self-attention and / or cross-attention (e.g., between context data and text data) may be used to understand and recognize relationships between words or tokens and / or context data (e.g., other text, video, image, design data, USD, etc.). One or more generative processing pipelines including LLMs / VLMs / MMLMs / etc. may also include one or more diffusion blocks (e.g., denozer). The LLMs / VLMs / MMLMs / etc. of the present disclosure may include one or more encoder and / or decoder blocks. For example, discriminative or pure coder models such as BERT (Bidirectional Coder Representations from Transformers) can be implemented for tasks involving speech understanding such as classification, mood analysis, question answering and named entity recognition. Another example is generative or pure decoder models such as GPT (Generative Biased Transformer) that can be used for tasks involving the generation of speech and content, such as the completion of texts, generation of layers, and dialog generation. LLMs / VLMs / MMLMs / etc., which include both encoder and decoder components, such as T5 (text-to-text transformers), can be used to understand and generate content, e.g., for translation and summary. These examples are not intended to be limiting, and any type of architecture, including but not limited to those described herein, may be implemented depending on the particular embodiment and task(s) performed with the LLMs / VLMs / MMLMs / etc.In various embodiments, the LLMs / VLMs / MMLMs / etc. may be trained by unsupervised learning in which an LLMs / VLMs / MMLMs / etc. pattern of large sets of unlabeled text / audio / video / image / design USD-etc. Data learns. Due to extensive training, the models may not require task-specific or domain-specific training. LLMs / VLMs / MMLMs / etc., which have undergone extensive pretraining on huge sets of unsigned data, may be referred to as basic models, and are suitable for a variety of tasks, e.g., question answering, summary, supplementing missing information, translation, generating images / videos / design / USD data. Some LLMs / VLMs / MMLMs / etc. may be tailored to a particular application by using techniques such as prompt tuning, fine tuning, Retrieval Augmented Generation (RAG), adding adapters (e.g., customized neural networks and / or neural network layers that tune or adjust prompts or tokens to align the language model to a particular task or domain), and / or other fine tuning or custom cutting techniques that optimize the models for use with particular tasks and / or within particular domains.In some embodiments, the LLMs / VLMs / MMLMs / etc. of the present disclosure may be implemented using various model matching techniques. For example, in some embodiments, guardrails may be implemented to identify impermissible or undesired inputs (e.g., prompts) and / or outputs of the models. In doing so, the system may use the guardrails and / or other model adaptation techniques to either prevent a particular undesirable input from being processed with the LLMs / VLMs / MMLMs / etc., and / or to prevent the output or presentation (e.g., display, audio output, etc.) of information generated with the LLMs / VLMs / MMLMs / etc. In some embodiments, one or more additional models, or layers thereof, may be implemented to detect problems with inputs and / or outputs of the models. For example, these "protection" models may be trained to identify inputs and / or outputs that are "safe" or otherwise okay or desired and / or that are "unsafe" or otherwise undesirable for the particular application / implementation. As a result, the LLMs / VLMs / MMLMs / etc. of the present disclosure are less likely to output voice / text / audio / video / design data / LJSD data / etc., which could be malicious, vulgy, improper, unsafe, out-of-range, and / or otherwise undesirable for the particular application / implementation.In some embodiments, the LLMs / VLMs / etc. may be configured or capable of accessing or using one or more plug-ins, application programming interfaces (APIs), databases, data stores, repositories etc. For example, for certain tasks or operations for which it is not ideally suited, the model may have instructions (e.g., as a result of training and / or based on instructions in a given request for input) to access one or more plug-ins (e.g., third party plug-ins) to assist in processing the current input. In such an example, where at least a portion of a request for input relates to restaurant or weather, the model may access one or more restaurant or weather plug-ins (e.g., via one or more APIs) to retrieve the relevant information. As another example, if at least a portion of a response requires mathematical computation, the model may access one or more math plug-ins or APIs to help solve the problem / s, and then may use the response of the plug-in and / or API in the output of the model. This process may be repeated, e.g., recursively, for any number of iterations and using any number of plug-ins and / or APIs until a response to the request input to each question / problem / process / operation / etc. can be generated. Thus, the model or models can be based not only on their own knowledge from training with a large dataset or large datasets, but also on the technical knowledge or the optimized nature of one or more external resources, such as APIs, plug-ins and / or the like.In some embodiments, multiple language models (e.g., LLMs / VLMs / MMLMs / etc.), multiple instances of the same language model, and / or multiple prompts conveyed to the same language model or instance of the same language model may be implemented, executed, or retrieved (e.g., using one or more plug-ins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide output responsive to the same query or to separate portions of a query. In at least one embodiment, multiple language models, e.g., language models with different architectures, language models trained on different (e.g., updated) data sets, may be provided with the same input query and request (e.g., a set of constraints, conditions, etc.). In one or more embodiments, the language models may be different versions of the same basic model. In one or more embodiments, at least one language model may be instantiated as multiple agents - e.g., more than one prompt may be provided to restrict, control, or otherwise influence a style, content, or character, etc., of the provided output. In one or more non-limiting examples, the same language model may be prompted to provide an output corresponding to a different role, perspective, character, or knowledge base, etc., as defined by a predetermined prompt.In each of these embodiments, the output of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiated agents of at least one language model, and / or two further prompts provided to at least one language model may be further processed, e.g., aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more embodiments, the output of one language model-or version, instance, or agent-may be provided as input to another language model for further processing and / or validation. In one or more embodiments, a language model may be prompted to generate or otherwise obtain an output related to an input source material, where the output is associated with the input source material. Such a link may include, for example, generating a label or text portion embedded (e.g., as metadata) in an input source text or image. In one or more embodiments, the output of a language model may be used to determine the validity of an input source material for further processing or inclusion in a dataset. For example, a language model may be used to assess the presence (or absence) of a target word in a text portion or object in an image, the text or image being annotated to note the presence (or absence). Alternatively, the determination of the language model may be used to determine whether the source material should be included in a kuratted dataset, for example and without limitation.FIG. 28A is a block diagram of an example generative language model system 2800 suitable for use in implementing at least some embodiments of the present disclosure. In the example illustrated in FIG. 28A, generative language model system 2800 includes a RAG component 2892, an input processor 2805, a tokenizer 2810, an embedding component 2820, plug-ins / APIs 2895, and a generative language model (LM) 2830 (which may include an LLM, a VLM, a multi-modal LM, etc.).At a high level, the input processor 2805 may receive an input 2801 comprising text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D construction data, CAD data, universal scene descriptor data (USD) - such as OpenUSD, etc.), depending on the architecture of the generative LM 2830 (e.g., LLM / VLM / MMLM, etc.). In some embodiments, input 2801 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, the input 2801 may include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., in table formats, JSON, or XML). In some implementations where generative LM 2830 is capable of processing multimodal input, input 2801 may combine (or omit) text with image data, audio data, video data, design data, USD data, and / or other types of input data, such as, but not limited to, those described herein. Taking raw input text as an example, the input processor 2805 may render raw input text in various ways. For example, the input processor 2805 may perform various types of text filtering to remove noise (e.g., special characters, interpolation, HTML tags, stop words, portions of an image or portions of audio, etc.) from the relevant text content. In an example with stop words (frequent words that typically have little semantic meaning), the input processor 2805 may remove stop words to reduce the noise and focus the generative LM 2830 on more meaningful content. The input processor 2805 may perform text normalization by, for example, converting all characters to lower case letters, removing accents, and / or processing special cases such as truncations or abbreviations to ensure consistency. These are just some examples, and other types of input processing may be applied.In some embodiments, a RAG component 2892 (which may include one or more RAG models and / or be executed using the generative LM 2830 itself) may be used to retrieve additional information used as part of the input 2801 or prompt. RAG can be used to extend the input to the LLM / VLM / MMLM / etc. with external knowledge so that the answers to specific questions or queries or requests are more relevant - for example, in a case where specific knowledge is required. The RAG component 2892 may retrieve this additional information (e.g., base information such as base text / image / video / audio / USD / CAD / etc.) from one or more external sources, which may then be fed to the LLM / VLM / MMLM / etc. along with the request to improve the accuracy of the model's responses or outputs.For example, in some embodiments, the input 2801 may be generated using the query or input to the model (e.g., a query, a query, etc.) in addition to the data retrieved with the RAG component 2892. In some embodiments, input processor 2805 may analyze input 2801 and communicate with RAG component 2892 (or, in embodiments, RAG component 2892 may be part of input processor 2805) to identify relevant text and / or other data provided to generative LM 2830 as additional context or information sources from which response, response, or output 2890 is generally identified. For example, if the input indicates that the user is interested in a desired tire pressure for a particular vehicle make and model, the RAG component 2892, using, for example, an RAG model that performs a vector search in an embedding space, may retrieve the tire pressure information or corresponding text from a digital (embedded) version of the user's manual for that particular vehicle make and model. Similarly, upon re-visit of a chatbot related to a particular product offer or service, the RAG component 2892 may retrieve a previous stored call history--or at least a summary thereof--and include the previous call history along with the current question / request as part of the input 2801 to the generative LM 2830.The RAG component 2892 may use various RAG techniques. For example, naive RAG may be used, where documents are indexed, divided into chunks, and applied to an embedding model to generate embeds corresponding to the chunks. A user query may also be applied to the embedding model and / or another embedding model of the RAG component 2892, and the embeddings of the chunks along with the embeddings of the query may be compared to identify the most similar / related embeddings of the query that may be provided to the generative LM 2830 to generate an output.In some embodiments, more advanced RAG techniques may be used. For example, the chunks may pass through pre-retrieval processes (e.g., routing, re-writing, metadata analysis, expansion, etc.) prior to delivery to the embedding model. Moreover, prior to generating the final embeds, post-retrieval processes (e.g., re-ranking, prompt compression, etc.) may be performed on the outputs of the embedding model before the final embeds are used for comparison with an input query.As another example, modular RAG techniques similar to naive and / or advanced RAG may be used, but also include features such as hybrid search, recursive retrieval and query engines, stepback approaches, sub-queries, and hypothetical document embedding.Another example: Graph RAG may use knowledge graphs as a source of context or subject information. Graph RAG may be implemented using a graph database as a source of contextual information sent to the LLM / VLM / MMLM / etc. Instead of (or in addition to) providing the model with pieces of data extracted from larger documents - which may result in a lack of context, properness of matter, accuracy of speech, etc. - Graph RAG may also provide structured entity information to the LLM / VLM / MMLM / etc. by combining the structured text description of the entity with its many properties and relationships, allowing the model to see deeper. In the implementation of Graph RAG, the systems and methods described herein use a graph as content storage and extract relevant portions of documents and request the LLM / VLM / MMLM / etc. to respond therefrom. The knowledge graph, in such embodiments, may include relevant text content and metadata about the knowledge graph and be integrated with a vector database. In some embodiments, the graph RAG may use a graph as a Binxperte, wherein descriptions of concepts and entities relevant to a request / prompt may be extracted and passed to the model as a semantic context. These descriptions may include relationships between the concepts. In other examples, the graph may be used as a database, where a portion of a query / prompt may be mapped to a graph query, the graph query may be performed, and the LLM / VLM / MMLM / etc. may summarize the results. In such an example, the graph may include relevant subject matter information, and a query (natural language query) may be used in a graph query tool (NL-to-graph query tool) and an entity link. In some embodiments, graph RAG (e.g., using a graph database) may be combined with standard RAG (e.g., vector database) and / or other RAG types to benefit from multiple approaches.In all embodiments, the RAG component 2892 may implement a plugin, an API, a user interface, and / or other functions to perform RAG. For example, a graph RAG plugin may be used by the LLM / VLM / MMLM / etc. to perform queries against the knowledge graph to extract relevant information for injection into the model, and a standard or vector RAG plugin may be used to perform queries against a vector database. For example, the graph database may interact with the REST interface of a plug-in such that the graph database is decoupled from the vector database and / or the embedding models.The tokenizer 2810 may segment the (e.g., processed) text data into smaller units (tokens) for subsequent analysis and processing. The tokens may represent individual words, subwords, characters, portions of audio / video / image, etc., depending on the implementation. Word-based tokenization divides the text into individual words, with each word being processed as a separate token. In tokenizing subwords, words are decomposed into smaller meaningful units (e.g., prefixes, suffixes, phrases) so that generative LM 2830 can understand morphological variations and better treat words not included in vocabulary. Character-based tokenization represents each character as a separate token and allows generative LM 2830 to process text at a fine-grained level. The choice of tokenization strategy may depend on factors such as the language to be processed, the task to be accomplished, and / or the features of the training dataset. Thus, the tokenizer 2810 may convert the (e.g., processed) text to a structured format according to the tokenization scheme implemented in the respective embodiment.The embedding component 2820 may use any known embedding technique to convert discrete tokens into (e.g., dense, continuous vector) representations of the semantic meaning. For example, embedding component 2820 may use pre-trained word embeds (e.g., Word2Ve, GloVe, or FastText), one-hot coding, term frequency-inverse document frequency (TF-IDF) coding, one or more neural network embedding layers, and / or other methods.In some implementations where input 2801 includes image data / video data, etc., input processor 2801 may scale the data to a default size compatible with the format of a corresponding input channel and / or normalize the pixel values to a common region (e.g., 0 to 1) to ensure consistent representation, and embedding component 2820 may encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations where the input 2801 includes audio data, the input processor 2801 may resample an audio file to a consistent sampling rate for consistent processing, and the embedding component 2820 may use any known technique to extract and encode audio features - for example, in the form of a spectrogram (e.g., a mel spectrogram). In some implementations where input 2801 includes video data, input processor 2801 may extract frames or apply a resize to extracted frames, and embedding component 2820 may extract features such as optical workflow embeddings or video embeddings and / or encode temporal information or sequences of frames. In some implementations where the input 2801 includes multi-modal data, the embedding component 2820 may fuse representations of the various types of data (e.g., text, image, audio, USD, video, design, etc.) using techniques such as early fusion (concatenation), late fusion (sequential processing), attention-based fusion (e.g., self-attention, cross-attention), etc.Generative LM 2830 and / or other components of generative LM system 2800 may use various types of neural network architectures depending on implementation. For example, transformer-based architectures as used in models such as GPT may be implemented and may include self-attention mechanisms that balance the importance of various words or tokens in the input sequence and / or feedforward networks that process the output of the self-attention layers by applying nonlinear transformations to the input representations and extracting higher level features. Some example non-limiting architectures include transformers (e.g., encoder-decoders only, multi-mode), RNNs, LSTMs, fusion models, diffusion models, cross modal embedding models learning common embedding spaces, graphical neural networks (GNNs), hybrid architectures combining different types of architectures, adversary networks such as generative adversary networks or GANs or adversary auto-encoders (AAEs) for common distribution learning, and others. Depending on the implementation and architecture, the embedding component 2820 may apply an encoded representation of the input 2801 to the generative LM 2830, and the generative LM 2830 may process the encoded representation of the input 2801 to generate an output 2890, which may include responsive text and / or other types of data.As described herein, in some embodiments, generative LM 2830 may be configured to access or use plug-ins / APIs 2895 - or to be capable of accessing or using them (which may include one or more plug-ins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations for which the generative LM 2830 is not ideally suited, the model may have instructions (e.g., as a result of training and / or based on instructions in a given request, such as those retrieved with the RAG component 2892) to access one or more plug-ins / APIs 2895 (e.g., third party plug-ins) to obtain assistance in processing the current input. In such an example, where at least a portion of a request relates to restaurant or weather, the model may access (e.g., via one or more APIs) one or more restaurant or weather plug-ins, send at least a portion of the request relating to the particular plug-in / API 2895 to the plug-in / API 2895, the plug-in / API 2895 may process the information and return a response to the generative LM 2830, and the generative LM 2830 may use the response to generate the output 2890. This process may be repeated, e.g., recursively, for any number of iterations and using any number of plug-ins / APIs 2895 until an output 2890 can be generated that addresses any question / query / process / operation / etc. from the input 2801. As such, the model(s) may rely not only on its own knowledge from training with a large dataset(s) and / or from data retrieved with the RAG component 2892, but also on the skill or optimized nature of one or more external resources, such as the plug-ins / APIs 2895.FIG. 28B is a block diagram of an example implementation where the generative LM 2830 includes a transformer encoder decoder. Suppose an input text such as "Who detected gravity?" is broken down into tokens (e.g., by the tokenizer 2810 of FIG. 28A ), and each token is encoded (e.g., by the embedding component 2820 of FIG. 928A ) into a corresponding embedding (e.g., size 512). Because these token embeds typically do not represent the position of the token in the input sequence, any known technique can be used to add position encoding to each token embed to encode the sequential relationships and context of the tokens in the input sequence. As such, the (e.g., resulting) embeddings may be applied to one or more encoders 2835 of the generative LM 2830.In an example implementation, the encoder or encoders 2835 form an encoder stack, where each encoder includes a self-observation layer and a feedforward network. In an example transformer architecture, each token (e.g., a word) traverses a separate path. Thus, each coder can accept a sequence of vectors, each vector being routed through the self-attention layer, then the feedforward network, and then up to the next coder in the stack. Any known technique of self-attenuation may be used. To calculate a self-attention score for each token (word), for example, a query vector, a key vector, and a value vector may be created for each token. A self-attention score can be calculated for token pairs by forming the point product of the query vector with the corresponding key vectors, normalizing the resulting scores, multiplying by the corresponding value vectors, and summing the weighted value vectors. The encoder may apply multi-head attention in which the attention mechanism is applied multiple times in parallel with different learned weight matrices. Any number of encoders may be cascaded to generate a context vector that encodes the input. An attention projection layer 2840 may convert the context vector to attention vectors (keys and values) for the decoder / s 2845.In an example implementation, the decoders 2845 form a decoder stack, each decoder comprising a self-attention layer, an encoder-decoder self-attention layer that uses the attention vectors (keys and values) of the encoder to focus on relevant portions of the input sequence, and a feedforward network. As with the encoder(s) 2835, in an example transformer architecture, each token (e.g., a word) traverses a separate path in the decoder(s) 2845. During a first pass, the decoder or decoders 2845, classifier 2850, and generation mechanism 2855 may generate a first token, and the generation mechanism 2855 may use the generated token as input during a second pass. The process may repeat in a loop, successively generating tokens (e.g., words) and adding the output from the previous pass, and using the token embeds of the composite sequence with position encodings as input to the / the decoders 2845 during a subsequent pass, sequentially generating one token at a time (known as autoregressive) until a symbol or token representative of the end of the response is predicted. Within each decoder, the self-observation layer is typically forced to only note previous positions in the output sequence by applying a masking technique (e.g., by setting future positions to negative infinity) prior to the softmax operation. In an example implementation, the coder-decoder attention layer operates similar to (e.g., multi-head) self-attention in coder(s) 2835, except that it generates its queries from the underlying layer and accepts the keys and values (e.g., the matrix) from the output of coder(s) 2835.As such, the decoder or decoders 2845 may output a decoded (e.g., vectorial) representation of the input applied during a particular pass. Classifier 2850 may include a multi-class classifier that includes one or more neural network layers that project the decoded (e.g., vectorial) representation into a corresponding dimensionality (e.g., a dimension for each word or token supported in the output vocabulary) and a softmax operation that converts logs into probabilities. As such, the generation mechanism 2855 may select or query a word or token based on a corresponding predicted probability (e.g., select the word with the highest predicted probability) and append it to the output from a previous pass, with each word or token generated sequentially. The generation mechanism 2855 may repeat the process by triggering successive decoder inputs and corresponding predictions until it selects or scans a symbol or token representing the end of the response, whereupon the generation mechanism 2855 may output the generated response.FIG. 28C is a block diagram of an example implementation where generative LM 2830 includes a decoder-transformer-only architecture. For example, the decoder / decoders 2860 of FIG. 28C may operate similar to the decoder / decoders 2845 of FIG. 28B, except that each of the decoders 2860 of FIG. 28C omits the encoder-decoder self-observation layer (since there is no encoder in this implementation). As such, the decoders 2860 may form a decoder stack, where each decoder includes a self-observation layer and a feedforward network. Moreover, instead of encoding the input sequence, a symbol or token representing the end of the input sequence (or the beginning of the output sequence) may be appended to the input sequence and the resulting sequence (e.g., corresponding embeddings with position encodings) may be applied to the decoder / s 2800. As with the decoder(s) 2845 of FIG. 28B, each token (e.g., word) may traverse a separate path in the decoder(s) 2860 and the decoder / s 2860, classifier 2865 and generation mechanism 2870 may use autoregressive to sequentially generate one token at a time until a symbol or token representative of the end of the response is predicted. Classifier 2865 and generation mechanism 2870 may operate similarly to classifier 2850 and generation mechanism 2855 of FIG. 28B, where generation mechanism 2870 selects or scans each successive output token based on a corresponding predicted probability and appends it to the output from a previous pass, where each token is sequentially generated until a symbol or token representing the end of the response is selected or scanned. These and other architectures described herein are merely exemplary and other suitable architectures may be implemented within the scope of the present disclosure.EXAMPLE OF CONTENT STREAMING SYSTEMFIG. 29 is an example system diagram for a content streaming system 2900 in accordance with some embodiments of the present disclosure. FIG. 29 includes one or more application servers 2902 (which may include similar components, features, and / or functions to example computing device 3000 of FIG. 30 ), one or more client devices 2904 (which may include similar components, features, and / or functions to example computing device 3000 of FIG. 30 ), and one or more networks 2906 (which may be similar to the networks described herein). In some embodiments of the present disclosure, system 2900 may support an application session corresponding to a gaming streaming application (e.g., NVIDIA Ge FORCE NOW), a remote desktop application, a simulation application (e.g., autonomous or semi-autonomous vehicle simulation), computer aided design (CAD) applications, virtual reality (VR) and / or augmented reality (AR) streaming applications, deep learning applications, and / or other application types.In system 2900, for an application session, client device(s) 2904 may receive only input data in response to input to the input device(s), transmit the input data to the application server(s) 2902, receive encoded display data from the application server(s) 2902, and display the display data on display 2924. Thus, the more computationally intensive computations and processing may be moved to the application server / s 2902 (e.g., rendering-in particular ray or path tracing-may be performed for the graphical output of the application session from the GPU(s) of the application server(s) 2902, such as a gaming server). In other words, the application session may be streamed from the application server(s) 2902 to the client device(s) 2904, thereby reducing the requirements of the client device(s) 2904 for graphics processing and playback.For example, with respect to instantiating an application session, a client device 2904 may display a frame of the application session on the display 2924, after receiving the display data from the application server(s) 2902. The client device 2904 may receive input at one of the input devices and generate input data in response. The client device 2904 may transmit the input data to the application server / servers 2902 via the communication interface 2920 and via the network(s) 2906 (e.g., the Internet), and the application server / servers 2902 may receive the input data via the communication interface 2918. The CPU(s) may receive the input data, process the input data, and transmit data to the GPU(s) that cause the GPU(s) to generate a playback of the application session. For example, the input data may represent a movement of a character of the user in a game session of a game application, firing a weapon, reloading, passing a ball, turning a vehicle, etc. The rendering component 2912 may render the application session (e.g., representative of the result of the input data), and the rendering capture component 2914 may capture the rendering of the application session as display data (e.g., as image data capturing the rendered image of the application session). The rendering of the application session may include ray or path tracked illumination and / or shadow effects calculated using one or more parallel processing units - such as GPUs, which may further utilize the use of one or more dedicated hardware accelerators or processor cores to perform ray or path tracking techniques - of the application server(s) 2902. In some embodiments, one or more virtual machines (VMs) - e.g., including one or more virtual components such as vPUs, vCPUs, etc.- may be used by the application server(s) 2902 to support the application sessions. The encoder 2916 may then encode the display data to generate encoded display data, and the encoded display data may be transmitted to the client device 2904 via the network(s) 2906 via the communication interface 2918. The client device 2904 may receive the encoded display data via the communication interface 2920, and the decoder 2922 may decode the encoded display data to generate the display data. The client device 2904 may then display the display data via the display 2924.EXAMPLE OF A DATA PROCESSING SYSTEMFIG. 30 is a block diagram of an example computing device(s) 3000 suitable for use in implementing some embodiments of the present disclosure. Computing device 3000 may include an interconnect system 3002 that directly or indirectly couples the following devices: memory 3004, one or more central processing units (CPUs) 3006, one or more graphics processing units (GPUs) 3008, a communication interface 3010, input / output ports (I / O) 3012, input / output components 3014, a power supply 3016, one or more presentation components 3018 (e.g., display(s)), and one or more logic units 3020. In at least one embodiment, computing device(s) 3000 may include one or more virtual machines (VMs) and / or each of its components may include virtual components (e.g., virtual hardware components). As non-limiting examples, one or more of the GPUs 3008 may include one or more vPUs, one or more of the CPUs 3006 may include one or more vCPUs, and / or one or more of the logic units 3020 may include one or more virtual logic units. Thus, a computing unit 3000 may include discrete components (e.g., a complete GPU associated with the computing unit 3000), virtual components (e.g., a portion of a GPU associated with the computing unit 3000), or a combination thereof.Although the various blocks are shown in FIG. 30 as being connected to lines via the connection system 3002, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 3018 such as a display device may be considered an I / O component 3014 (e.g., when the display is a touch screen). As another example, CPUs 3006 and / or GPUs 3008 may include memory (e.g., memory 3004 may represent a storage device in addition to the memory of GPUs 3008, CPUs 3006, and / or other components). In other words, the computing device of FIG. 30 is merely illustrative. Categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types are not distinguished since they all fall within the scope of the computing device of FIG. 30.The interconnect system 3002 may represent one or more interconnects or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 3002 may include one or more types of bus or interconnects, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standards association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect (PCIe) bus, and / or another type of bus or interconnect. In some embodiments, there are direct connections between the components. For example, the CPU 3006 may be directly connected to the memory 3004. Further, the CPU 3006 may be directly connected to the GPU 3008. In direct or point-to-point connections between components, the interconnect system 3002 may include a PCIe connection to establish the connection. In these examples, a PCI bus need not be integrated with computing device 3000.The memory 3004 may be comprised of a variety of computer readable media. The computer readable media may be any available media that can be accessed by computing device 3000. The computer readable media may include both volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer readable media may include computer storage media and communication media.The computer storage media may include both volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules, and / or other types of data. For example, the memory 3004 may store computer readable instructions (e.g., those representing a program or programs and / or a program element or elements, such as an operating system. Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device 3000. The term "computer storage medium" as used herein does not include signals per se.The computer storage media may embody computer readable instructions, data structures, program modules, and / or other types of data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information transmission media. The term "modulated data signal" may refer to a signal in which one or more of its characteristics are adjusted or altered to encode information in the signal. The computer storage media may include, for example, wired media such as a wired network or a direct cable connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the above media should also fall within the scope of computer readable media.The CPU(s) 3006 may / may be configured to execute at least some of the computer readable instructions to control one or more components of the computing device 3000 and perform one or more of the methods and / or processes described herein. The CPU(s) 3006 may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a plurality of software threads simultaneously. The CPU(s) 3006 may include any type of processor and may include different types of processors (e.g., processors with fewer cores for mobile devices and processors with more cores for servers), depending on the type of computing device 3000 implemented. Depending on the type of computing device 3000, the processor may be, for example, an Advanced RISC Machines (ARM) processor operating with Reduced Instruction Set Computing (RISC), or an x86 processor operating with Complex Instruction Set Computing (CISC). Computing device 3000 may include one or more CPUs 3006 in addition to one or more microprocessors or additional coprocessors, such as mathematical coprocessors.In addition to or as an alternative to the CPU(s) 3006, the GPU(s) 3008 may be configured to execute at least some of the computer readable instructions to control one or more components of the computing device 3000 to perform one or more of the methods and / or processes described herein. One or more of the GPU(s) 3008 may be an integrated GPU (e.g., with one or more of the CPU(s) 3006, and / or one or more of the GPU(s) 3008 may be a discrete GPU. In embodiments, one or more of the GPU(s) 3008 may be a coprocessor of one or more of the CPU(s) 3006. Graphics processor(s) 3008 may be used by computing device 3000 to render graphics (e.g., 3D graphics) or to perform general purpose computations. The GPU(s) 3008 may be used for general purpose computing on GPUs (GPGPU), for example. The GPU(s) 3008 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU(s) 3008 may generate pixel data for output images in response to rendering commands (e.g., rendering commands from the / the CPU(s) 3006 received via a host interface). The GPU(s) 3008 may include a graphics memory, e.g., a display memory, for storing pixel data or other suitable data, e.g., GPGPU data. The display memory may be included as part of the memory 3004. The GPU(s) 3008 may include two or more GPUs operating in parallel (e.g., via a link). The connection may connect the GPUs directly (e.g., to NVLINK) or via a switch (e.g., to NVSwit). When combined together, each GPU 3008 may generate pixel data or GPGPU data for different portions of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may have its own memory or share memory with other GPUs.In addition to or as an alternative to the CPU(s) 3006 and / or the GPU(s) 3008, the logic unit(s) 3020 may be configured to execute (execute) at least some of the computer readable instructions to control one or more components of the computing device 3000 to perform one or more of the methods and / or processes described herein. In embodiments, the CPU(s) 3006, the GPU(s) 3008, and / or the logic unit(s) 3020 may d...

Claims

A processor or multiple processors, comprising processing circuitry to: receive, by an interpreter of an interactive agent platform that supports concurrent execution of agent actions in different interaction modalities, one or more representations of one or more responsive agent actions based on at least the interpreter executing one or more instruction lines of one or more interaction actions in response to the one or more detected user actions; and cause a representation to render the interactive agent executing the one or more responsive agent actions based on at least the one or more representations of the one or more responsive agent actions.The one or more processors of claim 1, wherein the interactive agent platform supports processing of captured user actions independent of execution of the agent actions.The one or more processors of claim 1 or claim 2, wherein the interactive agent platform supports non-sequential human-machine interactions in the different interaction modalities.The one or more processors of any preceding claim, wherein the interpreter supports one or more keywords that direct the interpreter to interrupt the one or more interaction flows and wait for one or more specified events before continuing the one or more interaction flows.The one or more processors of any preceding claim, wherein the interpreter supports one or more keywords that direct the interpreter to trigger and wait one or more specified agent or scene actions until the one or more specified agent or scene actions are completed before proceeding.The one or more processors of any preceding claim, wherein the interpreter supports one or more keywords that direct the interpreter to trigger and proceed to one or more specified agent or scene actions without waiting until the one or more specified agent or scene actions are completed.The one or more processors of any preceding claim, wherein the interpreter supports one or more keywords that instruct the interpreter to start or stop one or more groups of supported agent actions in the different interaction modalities.The one or more processors of any preceding claim, wherein the interpreter supports tracking one or more active agent or scene actions initiated by the one or more interaction flows and stopping the one or more active agent or scene actions in response to termination of the one or more interaction flows.The one or more processors of any preceding claim, wherein the one or more processors are included in at least one of: an autonomous or semi-autonomous machine control system; an autonomous or semi-autonomous machine perception system; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing joint content creation for 3D goods; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more augmented reality, virtual reality, or mixed reality content; a system implemented with an edge device; a system implemented using a robot; a system for performing call-carrying AI operations; a system for implementing one or more speech models; a system for implementing one or more big speech models (LLMs); a system for implementing one or more picture speech models (VLMs); a system for implementing one or more multi-modal speech models; a system for generating synthetic data; a system for generating synthetic data using AI; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.A system comprising one or more processors to execute one or more instruction lines of one or more interaction flows by an interpreter of an interactive agent platform supporting concurrent execution of agent actions in different interaction modalities.The system of claim 10, wherein the interactive agent platform supports processing of captured user actions independent of execution of the agent actions.The system of claim 10 or claim 11, wherein the interactive agent platform supports non-sequential human-machine interactions in the different interaction modalities.The system of any of claims 10 to 12, wherein the interpreter supports one or more keywords that direct the interpreter to interrupt the one or more interaction flows and wait for one or more specified events before continuing the one or more interaction flows.The system of any of claims 10 to 13, wherein the interpreter supports one or more keywords that direct the interpreter to trigger one or more specified agent or scene actions and wait until the one or more specified agent or scene actions are completed before proceeding.The system of any of claims 10 to 14, wherein the interpreter supports one or more keywords that direct the interpreter to trigger and proceed to one or more specified agent or scene actions without waiting until the one or more specified agent or scene actions are completed.The system of any of claims 10 to 15, wherein the interpreter supports one or more keywords that instruct the interpreter to start or stop one or more groups of supported agent actions in the different interaction modalities.The system of any of claims 10 to 16, wherein the interpreter supports tracking one or more active agent or scene actions initiated by the one or more interaction flows and stopping the one or more active agent or scene actions in response to termination of the one or more interaction flows.The system of any of claims 10 to 17, wherein the system consists of at least one of: an autonomous or semi-autonomous machine control system; an autonomous or semi-autonomous machine perception system; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing joint content creation for 3D goods; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more augmented reality, virtual reality, or mixed reality content; a system implemented with an edge device; a system implemented using a robot; a system for performing call AI operations; a system for implementing one or more speech models; a system for implementing one or more big speech models (LLMs); a system for implementing one or more image speech models (VLMs); a system for implementing one or more multi-modal speech models; a system for generating synthetic data; a system for generating synthetic data using AI; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.A method comprising: receiving, by an interactive agent platform supporting simultaneous execution of agent actions in different interaction modalities, one or more representations of one or more captured user actions; and generating, based on at least the interactive agent platform executing one or more instruction lines of one or more interaction flows, one or more representations of one or more responsive agent actions in response to the one or more captured user actions.The method of claim 19, wherein the method is performed by at least one of: an autonomous or semi-autonomous machine control system; an autonomous or semi-autonomous machine perception system; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing joint content creation for 3D goods; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality, virtual reality, or mixed reality content; a system implemented with an edge device; a system implemented using a robot; a system for performing call AI operations; a system for implementing one or more speech models; a system for implementing one or more big speech models (LLMs); a system for implementing one or more image speech models (VLMs); a system for implementing one or more multi-modal speech models; a system for generating synthetic data; a system for generating synthetic data using AI; a system including one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Images