User-defined alert monitoring for application virtual assistant systems and applications

By converting natural language requests into structured logical expressions or executable code functions, the virtual assistant system dynamically monitors the application status, solving the shortcomings of conventional systems in terms of adaptability and scalability, and enabling flexible and efficient responses to user-defined alerts.

CN123114601APending Publication Date: 2026-11-17NVIDIA CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610666660.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2025-05-16
Filing Date
2026-05-14
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Conventional virtual assistant systems are limited by fixed predefined conditions and hard-coded triggers, making it difficult to adapt to changes in user needs in dynamic or highly interactive environments, resulting in insufficient responsiveness and limited scalability.

Method used

By using a language model to convert natural language requests into structured logical expressions or executable code functions, the system dynamically monitors application status and adapts user-defined alert conditions at runtime, supporting both lightweight logical evaluation and the triggering of complex conditions.

Benefits of technology

It enables the virtual assistant system to flexibly adapt to and efficiently respond to user-defined conditions in dynamic environments, improving the context relevance and personalization support of applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114601A_ABST
    Figure CN123114601A_ABST
Patent Text Reader

Abstract

This invention relates to user-defined alert monitoring for application virtual assistant systems and applications. In various examples, an artificial intelligence (AI)-based assistant can create, monitor, and trigger user-defined alerts associated with an interactive application. For instance, in response to an application user submitting a natural language request (such as "Notify me when my inventory is full"), the systems and methods of this disclosure can translate the request into a structured format that allows the AI ​​assistant to dynamically monitor the application state and alert the user when conditions are met. In some examples, this translation may include using a language model to generate interpretable logical expressions (e.g., key-value comparisons), executable code functions, and / or additional language model calls associated with real-time evaluation of the application state. When the application state indicates that a condition has been met, the corresponding alert can be triggered through one or more user interface mechanisms (such as visual highlighting, pop-up messages, and / or auditory signals).
Need to check novelty before this filing date? Find Prior Art

Description

Background Technology

[0001] Today, virtual assistants can be included in a wide variety of interactive applications to provide assistance and enhance user experience. For example, gaming applications, productivity tools, creative environments, or any other type of interactive application can rely on virtual assistants to monitor application state or user interactions and deliver contextual support such as task recommendations, real-time tips, or status updates. To interpret user context, evaluate conditions, or respond appropriately to changes within the application, these virtual assistants may rely on access to dynamic application state. This state information can include elements such as object presence, user activity, system events, or resource levels, and can be obtained through exposed application programming interfaces (APIs) or inferred using techniques such as computer vision, object detection, and optical character recognition. By processing this data in real time, virtual assistants can enhance their ability to operate in a context-appropriate and situation-aware manner.

[0002] However, despite these capabilities, the scope of a typical virtual assistant can be limited, often confined to responding to fixed, predefined conditions, inputs, or query sets. For example, a typical virtual assistant system may rely on static event mappings or hard-coded triggers defined by developers during application design. These constraints can inhibit the virtual assistant's ability to adapt to new scenarios, respond to evolving application states, or anticipate user needs beyond predefined boundaries. Therefore, in some instances, the level of support provided by the virtual assistant may be insufficient and / or it may not scale well across dynamic or highly interactive environments. Summary of the Invention

[0003] Various embodiments of this disclosure relate to user-defined alert monitoring for application virtual assistant systems and applications. Systems and methods for creating, monitoring, and triggering user-defined alerts associated with interactive applications, such as gaming applications, productivity tools, creative environments, and / or any other type of interactive application, are disclosed.

[0004] For example, in response to a user-submitted natural language request to create an alert (such as "Notify me when my inventory is full" or "Notify me when there are five minutes left"), the systems and methods disclosed herein can convert the natural language request into a structured format that allows an AI assistant to dynamically monitor the application's state and alert the user when conditions are met. In some examples, this conversion may include using a language model to generate interpretable logical expressions (e.g., including key-value comparisons), executable code functions, and / or additional language model calls associated with real-time evaluation of the application's state. When the application state indication conditions are met, the corresponding alert can be triggered through one or more user interface mechanisms (such as visual highlighting, pop-up messages, and / or auditory signals).

[0005] Compared to conventional systems, the systems and methods of this disclosure, in some embodiments, enable application users to define custom alerts using natural language without prior knowledge of the application's internal mechanisms or programming expertise. For example, by using a language model to transform natural language alert specifications into structured logical expressions and / or executable functions for real-time monitoring of dynamic application states, the systems and methods of this disclosure provide a flexible and scalable alert framework capable of adapting to a wide range of user-defined conditions. Additionally, by supporting lightweight logic-based evaluation and more complex compiled procedures, the disclosed systems and methods can adapt to alert conditions of varying complexity while maintaining high runtime performance. Furthermore, compared to conventional systems that rely on fixed developer-defined alert triggers or hard-coded event mappings, the systems of this disclosure support alert specifications and dynamically adapt to evolving user needs and contextual scenarios during application runtime. Therefore, the systems and methods of this disclosure offer significant technical advantages in adaptability, personalization, and scalability, thereby improving the responsiveness and contextual relevance of application virtual assistant systems across various application environments. Attached Figure Description

[0006] The following describes in detail, with reference to the accompanying drawings, the system and method for monitoring user-defined alerts for application virtual assistant systems and applications, wherein: Figure 1 This is a data flow diagram illustrating an example of a process for implementing a user-defined alert for an interactive application, according to some embodiments of the present disclosure; Figure 2 The illustration shows examples of key-value pairs indicating the state associated with an application according to some embodiments of this disclosure; Figure 3 The illustration shows an example of converting a natural language request into logic for triggering a user-defined alert, according to some embodiments of the present disclosure; Figure 4 It is a block diagram illustrating an example of a system that can perform one or more processes, methods or techniques of the present disclosure; Figure 5 This is a flowchart illustrating an example of a method for implementing a user-defined alert for an interactive application, according to some embodiments of the present disclosure; Figure 6 This is a flowchart illustrating an example of a method for converting natural language alert descriptions into logical conditions for triggering user-defined alerts associated with an application, according to some embodiments of the present disclosure; Figure 7 This is a block diagram of an example content streaming system applicable to implementing some embodiments of this disclosure; Figure 8A This is a block diagram of an example generative language model system applicable to implementing at least some embodiments of the present disclosure; Figure 8B This is a block diagram of an example generative language model including a converter encoder-decoder, applicable to implementing at least some embodiments of this disclosure; Figure 8C This is a block diagram of an example generative language model including a decoder-only converter architecture suitable for implementing at least some embodiments of this disclosure; Figure 9 This is a block diagram of an example computing device applicable to implementing at least some embodiments of the present disclosure; and Figure 10 This is a block diagram of an example data center applicable to implementing at least some embodiments of the present disclosure. Detailed Implementation

[0007] Systems and methods relating to user-defined alert monitoring for application virtual assistant systems and applications are disclosed. More specifically, the disclosed systems and methods can be used to create, monitor, and trigger user-defined alerts associated with interactive applications, such as game applications, productivity tools, creative environments, and / or any other type of interactive application. For example, in response to an application user submitting a natural language request to create an alert (such as “Notify me when my inventory is full” or “Notify me when there are five minutes left”), the disclosed systems and methods can convert the natural language request into a structured format that allows the AI ​​assistant to dynamically monitor the application state and alert the user when alert conditions are met. In some examples, this conversion may include using a language model to generate interpretable logical expressions (e.g., key-value comparisons), executable code functions, and / or additional language model calls associated with real-time evaluation of the application state. When application state indication conditions are met, the corresponding alert can be triggered through one or more user interface mechanisms (such as visual highlighting, pop-up messages, and / or auditory signals).

[0008] In some examples, one or more systems may receive input data representing alert instructions. That is, the alert instructions may instruct a natural language request to issue an alert in response to a condition or event associated with the interactive application. For example, a user could provide natural language input such as “Alert me when I have enough material to build a boat” or “Notify me when the blue dinosaur appears.” In some instances, the input data may include text data, audio data, image data, and / or combinations thereof. For example, the input data may include text data representing typed or spoken text corresponding to a natural language expression, or audio data representing a natural language expression. Additionally or alternatively, the input data may also include audio or image data representing visual, audio, or video content to further interpret the request in context (e.g., a screenshot or image indicating a specific object to be tracked). In various instances, alert instructions may be received and processed asynchronously during application runtime so as not to interrupt ongoing user interaction with the application or virtual assistant.

[0009] As described herein, one or more systems can apply input data (e.g., text data, audio data, image data, etc.) as input to one or more language models or other artificial intelligence (AI) models. One or more language models can process the input data and generate output data (e.g., text data) representing one or more logical conditions used to evaluate a requested alert. One or more logical conditions can be derived semantically based on the user's natural language expression and can correspond to event triggers associated with the application's state. For example, if a user of a game application requests an alert when they can build a specific object in the game, one or more language models can infer the required set of game state conditions (e.g., sufficient resources or game progress milestones) and encode them as executable conditions to be monitored.

[0010] In some instances, one or more language models can be used to retrieve reinforcement generation (RAG) configurations. In such instances, one or more language models and / or one or more systems can retrieve contextual information (e.g., game documentation, creation requirements, or system configurations) from a vector database associated with the application to help accurately interpret user requests / warning instructions. For example, if a video game user says, "Please let me know when I can create a guild," one or more systems can first retrieve the prerequisite game rules or dependencies required to create that structure. The retrieved content can then be incorporated into prompts (e.g., input data) applied to one or more language models or processing pipelines, enabling one or more language models to generate an accurate set of warning conditions based on up-to-date application knowledge.

[0011] While examples using generative AI models (such as language models) may be described in this paper, this is not intended to be limiting. For example, but not limited to, any AI model described in this paper can include any type of machine learning model, such as those using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoder neural networks, artificial neural networks (ANN), convolutional neural networks (CNN), recurrent neural networks (RNN), perceptrons, long short-term memory (LSTM) networks, multilayer perceptron (MLP) networks, deep stacked networks (DSN), generative pre-trained (GPT) models or networks, feedforward networks, radial basis function ANNs, self-organizing maps (SOM), Kohonen maps, Hopfield networks, Boltzmann machines, deep belief neural networks, deconvolutional neural networks, generative adversarial networks (GAN), liquid machines, modular neural networks, sequence-to-order... Column models, networks using transformer architectures, state-space models (SSM) (e.g., networks using Mamba architectures (e.g., Mamba-1, Mamba2, etc.), networks using selective state-space models, networks using structured state-space sequence models, etc.), diffusion models (e.g., diffusion probability models, fraction-based generative models, etc.), neural radiation field (NeRF) models, Gaussian sputtering models, Kolmogorov-Arnold networks (KAN), models with encoder-only architectures, models with decoder-only architectures, models with encoder-decoder architectures, generative machine learning models, language models, large language models (LLM), small language models (SLM), visual language models (VLM), multimodal language models (MMLM), large action models (LAM), visual-language-action (VLA) models, and one or more other types of machine learning models.

[0012] As described herein, one or more language models can generate alert monitoring logic in one or more formats based on the complexity and / or specificity of the user's alert request. In some instances, one or more language models can output structured logic expressions, such as triples including a status key, a threshold, and a comparison operator (e.g., ('timber_count', 25, 'greater than or equal to')). These structured logic expressions can then be evaluated during application runtime using one or more predefined comparison functions, as discussed in more detail herein. For example, the logic expression ('enemy_count', 3, 'greater than') could correspond to a user-defined alert, such as "Alert me when there are more than three enemies nearby." During runtime, the predefined comparison function can retrieve the current value of enemy_count from the application state and apply the 'greater than' operator to determine whether the condition is met. Furthermore, while some examples can describe alert logic as individual triple expressions (e.g., status key, threshold, operator), in some instances, these expressions can be combined into arbitrary Boolean logic. For example, one or more systems can evaluate multiple comparison expressions and use logical operators (e.g., AND, OR, NOT) to combine their Boolean results to form more complex alert conditions.

[0013] Additionally or alternatively, one or more language models can generate executable code functions (e.g., in C++, Python, or any other programming language) that analyze one or more elements of the application state and determine whether specified conditions are met. This format can be useful in some instances when alert conditions require complex logic or involve multi-step evaluation. For example, in response to a request such as “Notify me when I can begin shipbuilding,” one or more systems can use one or more RAG-enhanced language models to retrieve the preconditions for shipbuilding (e.g., 150 workers and a steel mill) from a knowledge base. One or more language models can then generate code functions that evaluate whether the current state meets these preconditions. In some examples, the generated functions can be compiled into shared objects and / or dynamically loadable libraries and invoked during runtime for more advanced alert processing.

[0014] In some examples, in addition to generating alert monitoring logic (or as an alternative), one or more systems may continuously apply the original input prompt (e.g., “Alert me when there are 5 minutes left”) to one or more language models and / or one or more second language models to determine whether the alert condition has been met. Alternatively or additionally, one or more language models may generate an improved version of the original input prompt to be continuously re-evaluated at runtime (e.g., using one or more language models or one or more second language models) to determine whether the alert condition has been met. In some instances, these techniques can be useful in contexts where the alert is ambiguous, context-dependent, or depends on visual or temporal information that cannot be easily reduced to static conditions. For example, a system could rewrite a request such as “Allow me when I look like I’m going to lose” into a structured prompt such as “Given the current game state, has the player lost a significant advantage?” and submit it to one or more language models every few seconds along with live game state data.

[0015] One or more systems can use the generated alert logic (whether in the form of data structures, executable code, or language model hints) to monitor the state of an interactive application to determine when a specified alert condition has occurred. In some embodiments, alert monitoring may include periodically evaluating the application state, such as querying key-value pairs representing runtime variables (e.g., inventory levels, active entities on the screen, remaining time, etc.). Additionally or alternatively, the application state may be accessed directly via an application programming interface (API) associated with the application (e.g., in an integrated deployment), or it may be inferred via computer vision and object detection pipelines (e.g., in a third-party virtual assistant system).

[0016] In some examples, the frequency of alert logic evaluation can be configurable (e.g., once per second or once per frame), and one or more systems can support concurrent monitoring of multiple active alerts. To optimize performance, one or more systems can prioritize or throttle alert evaluations based on user-defined alert priorities, system load, and / or trigger frequency. Additionally, concurrent alert evaluations can be handled in parallel threads or within a dedicated monitoring thread pool to prevent contention with user-facing processes. In various examples, one or more systems can determine that alert conditions are met and trigger one or more user interface elements to notify the user. Depending on the alert type, user preferences, and / or application context, this notification can include, but is not limited to, pop-up messages, flashing indicators, color changes in the user interface, graphical overlays (such as bounding boxes drawn around objects or characters), audio messages or sounds, or any other visual or auditory indicators. In some examples, users can specify how they prefer to be alerted, such as by selecting visual, text, or auditory indicators. The alert display system can also adapt its behavior based on the priority of the alert or associated game state variables—for example, flashing red for urgent alerts or displaying non-intrusive log messages for less urgent alerts.

[0017] In some instances, one or more systems can be configured to support alert snoozing or disabling, allowing users to temporarily mute or disable alerts. For example, when an alert is triggered, one or more systems can automatically suppress repeated notifications for a predefined duration (e.g., 5 seconds), or allow users to manually snooze alerts for configurable time intervals (e.g., 5 minutes or 1 hour). Snoozing / disabling mechanisms also help ensure alerts remain informative and context-sensitive, rather than becoming distracting. One or more systems can also allow previously snoozed alerts to be reactivated upon user request. Additionally, when registering a new alert, one or more systems can check its semantic equivalence to existing alerts. If a match is found, one or more systems can display the previous alert and request confirmation before continuing, helping to reduce repetition and streamline alert management.

[0018] In some embodiments, one or more systems may maintain a list of active alerts and expose interfaces that allow users to review, pause, or disable alerts as needed. This helps prevent redundant alerts from overburdening users, especially if the triggering conditions remain true for extended periods. In some examples, one or more systems may also support alert management operations beyond registration. For instance, users may issue natural language requests such as “List my alerts” or “Which alerts are active,” prompting one or more systems to return a structured summary of currently registered alerts. The list may include numeric identifiers, one or more logical conditions (e.g., in structured or natural language), and / or activity statuses (such as “enabled,” “disabled,” or “paused”). In some instances, one or more alerts (e.g., each) may also include a descriptive name, which may be automatically generated by a language model or provided by the user.

[0019] In some examples, one or more generative AI models (e.g., language models, LLM, SLM, VLM, MMLM, vision-language-action (VLA) models, etc.) described herein can be packaged as microservices (such as inference microservices, e.g., NVIDIA NIM)). These microservices can include containers (e.g., operating system (OS) level virtualization packages) that can include application programming interface (API) layers, server layers, runtime layers, and / or model "engines." For example, an inference microservice can include the container itself and one or more models (e.g., weights and biases). In some instances (such as when one or more AI models are small enough (e.g., have a sufficiently few parameters)), one or more models can be included within the container itself. In other instances (such as when one or more models are large), one or more models can be hosted / stored in the cloud (e.g., in a data center) and / or hosted on-premises and / or at the edge (e.g., on a local server or computing device, but outside the container). In these embodiments, one or more models can be accessed via one or more APIs (such as REST APIs). Thus, and in some embodiments, one or more AI models described herein can be deployed as inference microservices to accelerate the deployment of one or more models on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., execution software built using standardized AI model deployments, such as NVIDIA's Triton inference server), and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations for delivering low latency and high throughput for production applications (such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). The generative AI models described herein may be included as part of a microservice along with an acceleration infrastructure capable of deployment using a single command and / or orchestration and autoscaling using a container orchestration system on the acceleration infrastructure (e.g., reaching data center scale on a single device). Thus, an inference microservice may include one or more AI models (e.g., optimized for high-performance inference), inference runtime software that executes one or more AI models and provides responses / outputs to inputs (e.g., user queries, prompts, etc.), and enterprise management software that provides health checks, identity verification, and / or other monitoring. In some embodiments, the inference microservice may include software that performs in-situ replacement and / or updates to one or more models.When a replacement or update is performed, the software that performs the replacement / update can maintain the user configurations of the inference runtime software and the enterprise management software.

[0020] In one or more embodiments, the systems and methods described herein can be deployed in gaming applications that include or are enhanced by virtual AI assistants. For example, a game console, PC, tablet, or other interactive gaming device may include one or more onboard and / or remote processors (e.g., CPU, GPU, AI accelerator, or SoC), and local or remote memory / storage devices to support gameplay and assistant functionality. These devices may run a core game engine and operate in parallel with an AI assistant capable of processing natural language input from the user, including alert requests. In some embodiments, one or more language models (e.g., large language model (LLM), small language model (SLM), visual-language model (VLM), or multimodal model) may be deployed to translate user-provided natural language expressions into structured alert logic that is continuously evaluated during gameplay. This processing may occur locally on the user device or remotely in a cloud-hosted environment.

[0021] For example, in a cloud gaming context (e.g., NVIDIA's GeForce Now), an AI assistant can operate as a cloud service, receiving input prompts from the player's client device (e.g., controller, headset, or keyboard input), generating structured logic or executable alert checking functions, and returning alert notifications or UI overlays to the client in real time. The assistant can monitor user-defined conditions by integrating with the game engine or accessing game state data via a visual inference pipeline (e.g., OCR and object detection). In this cloud-based deployment, remote rendering, alert handling, and LLM evaluation can be performed on data center infrastructure, while lightweight client devices can handle input capture and display.

[0022] The systems and methods described herein can be used for a variety of purposes, as examples but not limited to machine (e.g., robots, vehicles, construction machinery, warehouse vehicles / machines, autonomous, semi-autonomous and / or other machine types) control, machine motion, machine driving, synthetic data generation, model training (e.g., using real, augmented and / or synthetic data, such as synthetic data generated using simulation platforms or systems, synthetic data generation techniques (such as, but not limited to, those described herein), perception, augmented reality (AR), virtual reality (VR), hybrid implementation (MR), robotics, security and supervision (e.g., in smart city implementations), autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), distributed or collaborative content creation for 3D assets (e.g., using generic scene descriptor (USD) data, such as OpenUSD and / or other data types), cloud computing, generative artificial intelligence (e.g., using one or more diffusion models, converter models, etc.) and / or any other suitable application.

[0023] The disclosed embodiments can comprise a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots or robotic platforms, aviation systems, medical systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations (e.g., in driving or vehicle simulations, in robot simulations, in smart city or supervised simulations, etc.), systems for performing digital twin operations (e.g., in conjunction with collaborative content creation platforms or systems, such as, but not limited to, NVIDIA's OMNIVERSE and / or another platform, system, or service using USD or OpenUSD data types), systems implemented using edge devices, systems containing one or more virtual machines (VMs), and systems for (e.g., using one or more) Systems that perform synthetic data generation operations (such as multiple neural rendering fields (NERF), Gaussian sputtering techniques, diffusion models, converter models, etc.), systems that are at least partially implemented in a data center, systems for performing conversational AI operations, systems that implement one or more language models (such as one or more large language models (LLM), one or more small language models (SLM), one or more visual language models (VLM), one or more multimodal language models, etc.), systems for performing optical transport simulations, systems for performing collaborative content creation for 3D assets (e.g., using generic scene descriptor (USD) data, such as OpenUSD, computer-aided design (CAD) data, 2D and / or 3D graphics or design data and / or other data types), systems that are at least partially implemented using cloud computing resources, and / or other types of systems.

[0024] refer to Figure 1 , Figure 1 This is an example data flow diagram illustrating a process 100 for implementing a user-defined alert for an interactive application according to some embodiments of the present disclosure. It should be understood that the arrangements and other arrangements described herein are illustrative by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or as alternatives to the arrangements and elements shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed using one or more processors that execute instructions stored in one or more memories. For example, in some embodiments, the systems and methods described herein may use one or more generative language models (e.g., such as...). Figures 8A-8C (as described), one or more computing devices or components thereof (e.g., such as) Figure 9 (as described) and / or one or more data centers or components thereof (e.g., such as) Figure 10 To achieve this, as described above.

[0025] As shown, process 100 can be implemented using one or more client devices 102, one or more language models 104, RAG component 106, monitoring component 108, and alert component 110, as well as additional or alternative components. In some examples, one or more of these components may correspond to an application assistant system (e.g., an application virtual assistant system). As a brief overview of process 100, input data 112 representing natural language alert descriptions (which may include text data 114, audio data 116, and / or image data 118) can be received from one or more client devices 102 and applied as input to one or more language models 104. Additionally, in some examples, RAG component 106 may apply RAG data 120 representing contextual information and / or other application-related information as input to one or more language models 104. One or more language models 104 can process the input data 112 and RAG data 120 and generate evaluation data 122, which may represent one or more logical conditions used to trigger an alert. Monitoring component 108 can evaluate status data 124, which represents the application's state with respect to evaluation data 122, to determine whether to return to 126 and re-evaluate evaluation data 122 with respect to new status data 124 (e.g., for the next state of the application), or to trigger an alert. Based on an alert trigger signal 128 issued by monitoring component 108, alerting component 110 can generate alert data 130 to cause one or more client devices 102 to output an alert.

[0026] In some examples, one or more client devices 102 may include any computing device capable of executing interactive applications (e.g., games, productivity tools, or creative environments) and interacting with a virtual assistant. For example, one or more example client devices 102 may include, but are not limited to, game consoles, personal computers (PCs), tablets, smart displays, mobile phones, augmented reality or virtual reality headsets, or any combination thereof. In some embodiments, one or more client devices 102 may include local processing and memory resources to support alert notification input and alert output rendering. Additionally or alternatively, in a cloud-based architecture, one or more client devices 102 may operate as thin clients, sending input and receiving alert feedback from a remote server or data center. One or more client devices 102 may also display alert notifications, visual overlays, or other user interface elements based on output received from alert component 110.

[0027] Input data 112 may include one or more forms of data representing alert instructions provided by users of one or more client devices 102. Input data 112 may include, but is not limited to, text data 114 (e.g., typed or transcribed speech), audio data 116 (e.g., spoken phrases or sound prompts), and / or image data 118 (e.g., screenshots or in-app visual symbols). Input data 112 may be generated by one or more client devices 102 and transmitted to an application assistant system for processing. In some examples, input data 112 may include metadata (such as timestamps, game context, or user preferences) to provide additional contextual information for interpreting the alert instructions. As shown, input data 112 may be applied as part of an input prompt to one or more language models 104.

[0028] In some examples, one or more language models 104 may include one or more artificial intelligence (AI) models configured to process natural language, multimodal input, or contextual data to generate alert monitoring logic, which can be represented using evaluation data 122. In various instances, one or more language models 104 may include one or more of large language models (LLMs), multimodal language models (MMLMs), visual language models (VLMs), small language models (SLMs), and / or any other kind of generative AI model. In some examples, one or more language models 104 may be fine-tuned on or supplemented with application-specific data to process input data 112 and generate evaluation data 122.

[0029] For example, in at least some examples, the application assistant system may apply RAG data 120 to one or more language models 104 before or concurrently with input data 112 (e.g., as part of an input prompt). In these examples, RAG component 106 may facilitate retrieval-enhanced generation by using prompts provided to one or more language models 104 with relevant contextual enrichment. For example, RAG component 106 may query one or more vector databases to retrieve RAG data 120 representing application documentation, application metadata, and / or user-specific knowledge graphs. This information can help one or more language models 104 interpret user input and ensure that the logical conditions generated by one or more language models 104 are accurate and aligned with current application parameters. For example, in a strategy game, if a user requests a warning such as "Please let me know when I can build a ship," RAG component 106 can retrieve the prerequisites for shipbuilding from in-game documentation to inform the logic generation.

[0030] While examples using generative AI models (such as language models) may be described in this paper, this is not intended to be limiting. For example, but not limited to, any AI model described in this paper can include any type of machine learning model, such as those using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoder neural networks, artificial neural networks (ANN), convolutional neural networks (CNN), recurrent neural networks (RNN), perceptrons, long short-term memory (LSTM) networks, multilayer perceptron (MLP) networks, deep stacked networks (DSN), generative pre-trained (GPT) models or networks, feedforward networks, radial basis function ANNs, self-organizing maps (SOM), Kohonen maps, Hopfield networks, Boltzmann machines, deep belief neural networks, deconvolutional neural networks, generative adversarial networks (GAN), liquid machines, modular neural networks, sequence-to-order... Column models, networks using transformer architectures, state-space models (SSM) (e.g., networks using Mamba architectures (e.g., Mamba-1, Mamba2, etc.), networks using selective state-space models, networks using structured state-space sequence models, etc.), diffusion models (e.g., diffusion probability models, fraction-based generative models, etc.), neural radiation field (NeRF) models, Gaussian sputtering models, Kolmogorov-Arnold networks (KAN), models with encoder-only architectures, models with decoder-only architectures, models with encoder-decoder architectures, generative machine learning models, language models, large language models (LLM), small language models (SLM), visual language models (VLM), multimodal language models (MMLM), large action models (LAM), visual-language-action (VLA) models, and one or more other types of machine learning models.

[0031] As described herein, one or more language models 104 can generate evaluation data 122 in one or more formats (e.g., text data), where the selected format may depend on the complexity and / or specificity of the user's alert request. That is, one or more language models 104 can determine whether a complexity score associated with the alert request or monitoring of the alert meets or exceeds one or more thresholds, and then generate evaluation data 122 in a given complexity score. For example, one or more language models 104 can generate evaluation data 122 to represent one or more logical expressions for simpler alerts (such as alerts that may depend on a single (or multiple) status key or value). In these examples, one or more logical expressions can be represented as a triple data structure including a status key, a threshold, and a comparison operator. As an example, for the logical expression ('timber_count', 25, 'greater than or equal to'), an alert can be triggered when the value of the 'timber_count' status variable is greater than or equal to 25.

[0032] Additionally or alternatively, one or more language models 104 may generate evaluation data 122 for representing (e.g., using C++, Python, or any other programming language) one or more executable code functions that can analyze one or more elements of the application state and determine whether specified conditions are met. In these examples, one or more language models 104 may generate these code functions for more complex alerts (such as alerts that may depend on multiple different state key-value pairs). As an example, a natural language alert description referencing multiple conditions (such as "Notify me when my health drops below 30% and I have fewer than two healing potions") may need to evaluate multiple state keys and apply conditional logic across those multiple state keys. In such cases, one or more language models 104 may be prompted to generate a complete function that accepts a structured application state (e.g., a mapping of key-value pairs or a dictionary) and returns a Boolean value indicating whether the compound alert conditions have been met. These generated code functions may include nested conditions, logical operators, or arithmetic expressions, and are stored as part of evaluation data 122 for use during alert monitoring.

[0033] In even more examples, evaluation data 122 may represent one or more language model input prompts to be applied to one or more language models 104 or one or more second language models. In such examples, one or more language model input prompts may simply include the same input prompts applied to one or more language models 104 based on input data 112, or one or more input prompts may include updated or refined input prompts generated by one or more language models 104. In some instances, these techniques can be useful where the alert is ambiguous, context-dependent, or depends on visual or temporal information that cannot be easily reduced to static conditions. For example, a system could rewrite a request such as “Please let me know when I look like I’m going to lose” into a structured prompt such as “Given the current game state, has the player lost a significant advantage?” and submit it to one or more language models every few seconds along with live game state data.

[0034] While these are just a few examples of various formats of logical conditions that evaluation data 122 can represent and that are generated by one or more language models 104, in additional or alternative examples, one or more language models 104 can generate any kind of conditions, rules, or procedures to be used by (monitoring component 108) to determine whether a user-defined alert is triggered. Additionally, it should be understood that any of the different formats of logical conditions, rules, procedures, etc., described herein (e.g., triplet logical expressions, code functions, language model calls / hints, etc., and combinations thereof) can be used to evaluate alert requests of any level of complexity. For example, triplet logical expressions can also be used to monitor complex alert requests (e.g., not just code functions), code functions can be used to evaluate simple alert requests, and so on.

[0035] The process 100 may further include: a monitoring component 108 receiving evaluation data 122 generated using one or more language models 104. The monitoring component 108 may be configured to evaluate the application state with respect to the logical conditions represented by the evaluation data 122. For example, the monitoring component 108 may obtain or otherwise evaluate state data 124, which may represent or include key-value pairs or other representations of the application state during runtime. In some instances, the values ​​of these key-value pairs may be obtained directly through (e.g., via an API) integration with the application, or inferred from perceptual data (e.g., from computer vision or object recognition) by analyzing image frames associated with the application using one or more AI models (such as one or more VLMs).

[0036] for example, Figure 2An example of key-value pairs 200 indicating the state associated with an application, according to some embodiments of the present disclosure, is illustrated. As shown, key-value pairs 200 may include a plurality of keys 202 and a plurality of values ​​204 corresponding to the keys 202. Keys 202 may correspond to various specific attributes, variables, or parameters associated with the current runtime state of the application. Values ​​204 may represent real-time values ​​or recently updated values ​​associated with these keys 202. In some examples, key-value pairs 200 may be stored in a data structure (e.g., a map, dictionary, or associative array) that monitoring component 108 can access to evaluate whether any alert conditions are met. These key-value pairs 200 may be frequently updated based on internal application logic, API polling, or inference pipelines derived from computer vision.

[0037] like Figure 2 As illustrated in the example, key-value pairs 200 may include application state keys 202, which correspond to application state-related information such as location (e.g., the player's location in the game environment), heading (e.g., the player's heading in the game environment), weather (e.g., the current weather in the game environment), health (e.g., the player's health points), stamina (e.g., the player's stamina), time (e.g., local time or time in the game environment world), equipped weapons (e.g., the weapon the player is wielding), weapon durability (e.g., ammo count, time before defeat, etc.), inventory items, and equipped armor, etc. Each of these keys 202 may correspond to a value 204. For example, the value of the location key may be the geographic coordinates indicating the player's current in-game location (e.g., 45.1188 degrees North latitude, 115.3259 degrees West longitude). The heading key may represent the azimuth angle that can be used to determine the orientation of an entity controlled by the user or player, such as 210 degrees. Other examples shown include weather set to 'cloudy', health at '85%', and stamina at '93%', which can indicate a player's environmental and physical condition. Similarly, a time set to '15:30' can reflect in-game time or real-world system time, depending on the implementation.

[0038] Now return to the reference Figure 1For example, process 100 may include: monitoring component 108 continuously and / or periodically (e.g., per frame or per time interval) evaluating state data 124 using evaluation data 122 (e.g., logical conditions represented in evaluation data 122). Based on this evaluation, monitoring component 108 may determine whether an alert condition has been met. If not met, process 100 may return 126 evaluation data 122 for subsequent evaluation as state data 124 continues to be updated. Otherwise, if monitoring component 108 determines that one or more logical conditions are true, and therefore the alert condition is met, monitoring component 108 may issue an alert trigger signal 128. In some examples, monitoring component 108 may evaluate whether an alert condition is triggered in several different ways based on the format of evaluation data 122. For example, if evaluation data 122 includes structured triples (e.g., status keys, comparison operators, and thresholds), monitoring component 108 may apply the triples to a predefined comparison function for evaluation. Additionally or alternatively, if the evaluation data 122 includes an executable code function, the monitoring component 108 may directly invoke or execute that function to determine whether the alert condition has been met. In yet another case, the evaluation data 122 may include a prompt that is reapplied to one or more language models (e.g., one or more language models 104 and / or one or more other models) to generate a runtime response indicating whether the condition has been met.

[0039] Upon receiving an alert trigger signal 128, the alert component 110 may generate alert data 130 representing one or more outputs to notify users of one or more client devices 102 that an alert condition has been met. Depending on the alert type and user settings, these outputs may include visual, auditory, or tactile feedback. For example, alert data 130 may represent visual and / or auditory content to be output by one or more client devices 102 in response to the alert. This may include static UI elements (e.g., pop-up text), dynamic overlays (e.g., bounding boxes), visual effects (e.g., making elements blink), information indicators (e.g., resource bars), audio messages (e.g., natural language speech), sounds or tones, or any other output.

[0040] In some embodiments, prior to rendering an application frame on one or more client devices 102, the alert component 110 may use alert data 130 to modify or composite the rendered frame to visually reflect the triggered alert. For example, the alert component 110 may draw dashed boxes around a specific object or character in the frame (e.g., an NPC such as 'Dodo'), or apply temporary color filters or animations to indicate that alert conditions have been met. In some examples, these modifications may occur on the server side or in an intermediate rendering pipeline and be streamed to one or more client devices 102 as part of the final video or image frame. In some examples, these changes may be frame-limited or time-bound (e.g., displayed for 3-5 seconds) to avoid persistent visual clutter. The alert component 110 may determine the appropriate form of visual enhancement based on the type of alert and associated status key.

[0041] Similarly, for auditory alerts, the alert component 110 can inject synthesized speech (e.g., using a TTS system), pre-recorded audio messages, or tone indicators into the outgoing audio stream before it reaches one or more client devices 102. This allows alerts to be delivered in real time without requiring additional decoding or playback logic to be performed by one or more client devices 102. Visual and auditory alert outputs can be selected based on default preferences, context-specific relevance (e.g., alert severity), or user-configured display modes.

[0042] In some examples, the alert component 110 can also manage alert lifecycle events such as pausing, repeating, disabling, or updating alert preferences. For example, when an alert condition is detected to have been met and the alert has been displayed, the alert component 110 can initiate a pause timer that temporarily suppresses repeated notifications of that particular alert, such as for a fixed duration (e.g., 5 seconds, 1 minute, or a user-configured interval). In some embodiments, the alert component 110 can support user-driven alert pausing, allowing users to manually pause or defer individual alerts via interactive menus or toggle controls. Additionally, the alert component 110 can manage repeated alerts by determining whether to re-display the alert if the trigger condition remains true across multiple application states. In some examples, alerts can be configured to trigger only once per session, or repeat until explicitly acknowledged or closed by the user. The alert component 110 can also expose controls that allow users to update conditions, thresholds, or display styles associated with existing alerts, thereby supporting a dynamic, user-customizable alert system that adapts in real-time to evolving gameplay or application scenarios.

[0043] Now for reference Figure 3 , Figure 3The illustration shows an example of converting a natural language request into logic for triggering a user-defined alert, according to some embodiments of the present disclosure. As shown, input data 302 may represent a natural language instruction provided by a user of an interactive application. For example, the natural language instruction includes a request to “notify me when my health drops below 20%”. Input data 302 may be provided via a client device through text, voice, or other modalities and submitted to one or more language models 304 for processing.

[0044] One or more language models 304 (which may correspond to one or more language models 104) can interpret the semantics of the natural language input and generate corresponding evaluation data 306, which includes one or more logical expressions or structured conditions based on application status keys. In the illustrated example, evaluation data 306 includes a triplet expression representing a warning condition that should be satisfied when the value of the “HEALTH” status key is less than (e.g., “LT”) 20%: ( <health> 、<20%>、 <lt>The evaluation data 306 can be forwarded to the alert triggering system 308, which may include the monitoring component 108 and the alert component 110, as previously mentioned. Figure 1 As described.

[0045] Monitoring component 108 can receive status data 314 reflecting the current or periodically updated application state—such as HEALTH=19% and TIME=5. Status data 314 can originate from the application's runtime engine, API, or visual analytics pipeline. For example, a frame 310 associated with the application can be applied to one or more language models 312 (e.g., a visual language model, a multimodal language model, etc.). In some instances, one or more language models 312 can be the same as or different from one or more language models 304. One or more language models 312 can process frame 310 to generate or update status data 314, which can include key-value pairs 200 representing the application state. For example, one or more language models 312 can process frame 310 and assign the values ​​"19%" and "5" to the status keys HEALTH and TIME, respectively.

[0046] The monitoring component 108 can compare the status data 314 with the evaluation data 306 and determine that the alert condition is met (e.g., 19% < 20% is true). Based on this determination, the monitoring component 108 can send a signal to the alert component 110 to generate a user-facing alert.

[0047] In response to a triggered alert, alert component 110 can generate alert data 316 to be rendered to the user. As shown, alert data 316 includes visual notifications, such as a pop-up or overlay indicating "Health BELOW 20%". The alert can be rendered within an application activity frame 310, which includes gameplay elements such as player characters, user interface controls, a health bar, a minimap, and inventory elements. The alert can be composited onto the game frame before being rendered or streamed to the user's display. Additionally, in some examples, alert data 316 can be auditory output. For example, a machine-generated voice expressing "Health BELOW 20%" can be generated and included in alert data 316, and the client device can output audio data as a supplement to or alternative to the visual notification.

[0048] Now for reference Figure 4 , Figure 4 This is a block diagram illustrating an example of a system 402 capable of performing one or more processes, methods, or techniques of this disclosure. As shown, the system 402 may include one or more processors 404 (which may correspond to one or more CPUs 906 and / or one or more GPUs 908 of the example computing device 900) and memory 406 (which may correspond to memory 904 of the example computing device 900). In various examples, one or more processors 404 may execute one or more software components stored in memory 406 to perform one or more methods or operations described herein. For example, one or more software components stored in memory 406 may include one or more of the following: RAG component 106, monitoring component 108, alerting component 110, one or more models 408 (which correspond to one or more language models 104) and / or status component 410, and one or more processors 404 may execute one or more of these components to translate natural language alert descriptions into user-defined alert logic, and then monitor this user-defined alert logic to issue alerts to the user based on application status updates or events.

[0049] For example, the system 402 may receive input data 112 from one or more client devices 102 via a content streaming system 412 (which may correspond to one or more application servers 702). This input data 112 represents a natural language request to alert or notify the user of one or more client devices 102 upon the occurrence of one or more specific conditions. The system 402 may use one or more models 408 to process the input data 112 to generate alert monitoring logic (e.g., evaluation data 122), and the monitoring component 108 may evaluate this logic based on application status information received from the status component 410. Upon determining that one or more alert logic conditions are met, the monitoring component 108 may trigger the alert component 110 to generate alert data 130 to output an alert to the user. The alert data 130 may then be used to modify application data 414 (which may represent image frames or audio output associated with the application) and sent to one or more client devices 102.

[0050] Now for reference Figure 5 and Figure 6 Each block of methods 500 and 600 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed using one or more processors that execute instructions stored in one or more memories. These methods can also be embodied as computer-usable instructions stored on a computer storage medium. These methods can be provided by a standalone application, by a service or managed service (standalone or in conjunction with another managed service), as a microservice via an application programming interface (API), or as a plug-in to other products, to name a few. Additionally, by way of example, regarding... Figure 1 The application assistant system describes methods 500 and 600. However, the method may be additionally or alternatively performed by any system or any combination of systems, including but not limited to the systems described herein.

[0051] First refer to Figure 5 , Figure 5 This is a flowchart illustrating an example of a method 500 for implementing user-defined alerts for an interactive application according to some embodiments of the present disclosure. At B502, the method 500 may include: obtaining input data representing a natural language expression of a user associated with the interactive application, the natural language expression indicating one or more requests to issue one or more alerts in response to one or more events associated with the interactive application. The input data may be received during the execution of the interactive application, such as during a session of the interactive application. For example, an application assistant system may obtain input data 112 representing a natural language expression of one or more users of one or more client devices 102, which may be executing or streaming an interactive application.

[0052] At box B504, the method may include: generating text data corresponding to one or more logical conditions related to issuing one or more alerts, based at least on processing the input data using one or more language models. For example, an application assistant system may apply input data 112 as at least part of an input prompt to one or more language models 104, and one or more language models 104 may generate evaluation data 122 (e.g., text data) representing one or more logical conditions related to issuing one or more alerts.

[0053] At box B506, method 500 may include: evaluating one or more logical conditions based at least on state data representing one or more states of an interactive application to determine whether one or more events have occurred. For example, monitoring component 108 may determine whether one or more events have occurred based at least on evaluating one or more logical conditions represented by evaluation data 122 with regard to state data 124.

[0054] At box B508, the method 500 may include: outputting one or more alerts based on at least one or more events having occurred. One or more alerts may be output during a session of an interactive application. For example, based on determining that one or more events have occurred, monitoring component 108 may send an alert trigger signal 128 to alerting component 110. Alerting component 110 may then generate alert data 130 and forward that alert data 130 to one or more client devices 102 (e.g., within application data).

[0055] Figure 6 This is a flowchart illustrating an example of a method 600 for converting natural language alert descriptions into logical conditions for triggering user-defined alerts associated with an application, according to some embodiments of the present disclosure. At block B602, the method 600 may include: generating text data corresponding to logical conditions for evaluating alert conditions indicated by the natural language expression, using one or more language models and based at least on input data representing a user's natural language expression associated with an interactive application. For example, one or more language models 104 may process input data 112 and RAG data 120 to generate evaluation data 122 (e.g., text data) corresponding to the logical conditions for evaluating the alert conditions.

[0056] At box B604, the method 600 may include: determining that an alert condition is met based at least on evaluating the state of the interactive application using logical conditions. For example, monitoring component 108 may determine that an alert condition is met based at least on evaluating state data 124 representing the state of the interactive application using logical conditions represented by evaluation data 122.

[0057] At box B606, method 600 may include an indication that an output alert condition is met during the runtime of an interactive application. For example, monitoring component 108 may send alert trigger signal 128 to alert component 110. Alert component 110 may then generate alert data 130 and forward the alert data 130 to one or more client devices 102 (e.g., within application data).

[0058] The systems and methods described herein can be used for a variety of purposes, as examples but not limited to, machine (e.g., robots, vehicles, construction machinery, warehouse vehicles / machines, autonomous, semi-autonomous and / or other machine types) control, machine motion, machine driving, synthetic data generation, model training (e.g., using real, augmented and / or synthetic data, such as synthetic data generated using simulation platforms or systems, synthetic data generation techniques (such as, but not limited to, those described herein), perception, augmented reality (AR), virtual reality (VR), hybrid implementation (MR), robotics, safety and supervision (e.g., in smart city implementations), autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or participant simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), distributed or collaborative content creation of 3D assets (e.g., using generic scene descriptor (USD) data, such as OpenUSD and / or other data types), cloud computing, generative artificial intelligence (e.g., using one or more diffusion models, transformer models, etc.) and / or any other suitable application.

[0059] The disclosed embodiments can be included in a variety of different systems, such as motor vehicle systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots or robotic platforms, aviation systems, medical systems, marine systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations (e.g., in driving or vehicle simulations, in robotics simulations, in smart city or supervisory simulations, etc.), systems for performing digital twin operations (e.g., in conjunction with collaborative content creation platforms or systems (such as, but not limited to, NVIDIA's OMNIVERSE and / or another platform, system, or service using USD or OpenUSD data types)), systems implemented using edge devices, systems containing one or more virtual machines (VMs), and systems for (e.g., enabling...) Systems that perform synthetic data generation operations using one or more neural rendering fields (NERF), Gaussian sputtering techniques, diffusion models, transformer models, etc.; systems implemented at least partially in data centers; systems for performing conversational AI operations; systems that implement one or more language models (such as one or more large language models (LLM), small language models (SLM), one or more visual language models (VLM), one or more multimodal language models, etc.); systems for performing optical transport simulations; systems for performing collaborative content creation of 3D assets using (e.g., using generic scene descriptor (USD) data (such as OpenUSD, computer-aided design (CAD) data, 2D and / or 3D graphics or design data) and / or other data types); systems implemented at least partially using cloud computing resources; and / or other types of systems.

[0060] Example content streaming system Now for reference Figure 7 , Figure 7 This is an example system diagram of a content streaming system 700 according to some embodiments of the present disclosure. Figure 7 Includes one or more application servers 702 (which may include...) Figure 9 Example computing device 900 (similar components, features, and / or functions), one or more client devices 704 (which may include similar components, features, and / or functions to the example computing device 900), and one or more client devices 704. Figure 9 The example computing device 900 may have similar components, features, and / or functions, and one or more networks 706 (which may be similar to one or more networks described herein). In some embodiments of this disclosure, system 700 may be implemented. Application sessions may correspond to game streaming applications (e.g., NVIDIA GeForce NOW), remote desktop applications, simulation applications (e.g., autonomous or semi-autonomous vehicle simulations), computer-aided design (CAD) applications, virtual reality (VR), augmented reality (AR) and / or hybrid implementation (MR) streaming applications, deep learning applications, and / or other application types.

[0061] In system 700, for an application session, one or more client devices 704 may receive input data in response to input from one or more input devices, transmit the input data to one or more application servers 702, receive encoded display data from one or more application servers 702, and display the display data on a display 724. Thus, computationally intensive computation and processing are offloaded to one or more application servers 702 (e.g., rendering of the application session's graphics output (specifically, ray or path tracing) is performed by one or more GPUs of one or more game servers 702). In other words, the application session is streamed from one or more application servers 702 to one or more client devices 704, thereby reducing the graphics processing and rendering requirements of one or more client devices 704.

[0062] For example, regarding the instantiation of an application session, client device 704 may be displaying frames of the application session on display 724 based on display data received from one or more application servers 702. Client device 704 may receive input from one of one or more input devices and generate input data in response. Client device 704 may transmit the input data to one or more application servers 702 via communication interface 720 and through one or more networks 706 (e.g., the Internet), and one or more application servers 702 may receive the input data via communication interface 718. One or more CPUs may receive the input data, process the input data, and transmit the data to one or more GPUs, which causes one or more GPUs to generate a rendering of the application session. For example, the input data may represent a user's character moving, firing weapons, reloading, passing, turning vehicles, etc., in a game session of a game application. Rendering component 712 may render the application session (e.g., the result representing the input data), and rendering capture component 714 may capture the rendering of the application session as display data (e.g., as image data of the rendered frames of the captured application session). Rendering of an application session may include ray or path tracing lighting and / or shadow effects computed using one or more parallel processing units (such as GPUs) of one or more application servers 702, which may also employ one or more dedicated hardware accelerators or processing cores to perform ray or path tracing techniques. In some embodiments, one or more virtual machines (VMs) (e.g., including one or more virtual components such as vGPUs, vCPUs, etc.) may be used by one or more application servers 702 to support the application session. Encoder 716 may then encode display data to generate encoded display data, and the encoded display data may be transmitted to client device 704 via communication interface 718 through one or more networks 706. Client device 704 may receive the encoded display data via communication interface 720, and decoder 722 may decode the encoded display data to generate display data. Client device 704 may then display the display data via display 724.

[0063] Example language model In at least some embodiments, language models such as Large Language Models (LLM), Visual Language Models (VLM), Small Language Models (SLM), Multimodal Language Models (MMLM), and / or other types of generative artificial intelligence (AI) can be implemented. These models may be able to understand, summarize, translate, and / or otherwise generate text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., USD formats such as OpenUSD), and / or the like based on context provided in input prompts or queries. In embodiments, these language models may be considered "large" because they are trained on massive datasets and have architectures with a large number of learnable network parameters (weights and biases)—e.g., millions or billions of parameters. This disclosure allows for the implementation of LLM / SLM / VLM / MMLM, etc., for summarizing textual data, analyzing data (e.g., text, images, videos, etc.), extracting insights from data (e.g., text, images, videos, etc.), and generating new text / images / videos / , etc., in a user-specified style, tone, and / or format. In some embodiments, the LLM / SLM / VLM / MMLM, etc., of this disclosure may be specifically designed for text processing, while in other embodiments, multimodal LLMs may be implemented to accept, understand, and / or generate text and / or other types of content, such as images, audio, 2D and / or 3D data (e.g., USD format) and / or video. For example, a Visual Language Model (VLM) or more specifically a Multimodal Language Model (MMLM) may be implemented to accept images, videos, audio, text, 3D designs (e.g., CAD), and / or other input data types and / or generate or output images, videos, audio, text, 3D designs, and / or other output data types.

[0064] Various types of LLM / SLM / VLM / MMLM / etc. architectures can be implemented in various embodiments. For example, different architectures can be implemented using different techniques to understand and generate outputs (e.g., text, audio, video, images, 2D and / or 3D design or asset data, etc.). In some embodiments, LLM / SLM / VLM / MMLM / etc. architectures (e.g., recurrent neural networks (RNNs) or long short-term memory networks (LSTMs)) can be used, while in other embodiments, converter architectures (e.g., architectures relying on self-attention and / or cross-attention (e.g., between contextual data and textual data) mechanisms) can be used to understand and recognize relationships between words or tokens and / or contextual data (e.g., other text, video, images, design data, USD, etc.). One or more generative processing pipelines including LLM / SLM / VLM / MMLM / etc. may also include one or more diffusion blocks (e.g., noise reduction blocks). The LLM / SLM / VLM / MMLM / etc. of this disclosure may include encoder and / or decoder blocks. For example, discriminative or encoder-only models (e.g., BERT (Bidirectional Encoder Representations from Transformers)) can be implemented for tasks involving language understanding (e.g., classification, sentiment analysis, question answering, and named entity recognition). As another example, generative or decoder-only models (e.g., GPT (Generative Pretrained Transformer)) can be implemented for tasks involving language and content generation (e.g., text completion, story generation, and dialogue generation). LLM / SLM / VLM / MMLM / etc., including encoder and decoder components (e.g., T5 (Text-to-Text Transformer)), can be implemented to understand and generate content, such as for translation and summarization. These examples are not intended to be limiting and any architecture type (including, but not limited to, those described herein) can be implemented depending on the specific implementation and the task performed using LLM / SLM / VLM / MMLM / etc.

[0065] In various embodiments, LLM / SLM / VLM / MMLM / etc. can be trained using unsupervised learning, whereby LLM / SLM / VLM / MMLM / etc. learns patterns from a large amount of unlabeled text / audio / video / image / design / USD / etc. data. Due to the extensive training, in these embodiments, the model may not require task-specific or domain-specific training. An LLM / SLM / VLM / MMLM / etc. extensively pre-trained on a large amount of unlabeled data can be referred to as a base model and can excel at various tasks, such as question answering, summarizing, filling in missing information, translation, and image / video / design / USD / data generation. Some LLM / SLM / VLM / MMLM / etc. can be customized for specific use cases using techniques such as cue tuning, fine-tuning, retrieval augmentation generation (RAG), adding adapters (e.g., custom neural networks and / or neural network layers to tune or adjust cues or labels to bias the language model towards a specific task or domain), and / or using optimization models for specific tasks and / or other fine-tuning or customization techniques within a specific domain.

[0066] In some embodiments, the LLM / SLM / VLM / MMLM / etc. disclosed herein can be implemented using various model alignment techniques. For example, in some embodiments, guardrails can be implemented to identify incorrect or unwanted inputs (e.g., prompts) and / or outputs of the model. In this process, the system can use guardrails and / or other model alignment techniques to prevent the processing of specific unwanted inputs using LLM / SLM / VLM / MMLM / etc., and / or to prevent the output or presentation of information generated by LLM / SLM / VLM / MMLM / etc. (e.g., displays, audio outputs, etc.). In some embodiments, one or more additional models (or layers thereof) can be implemented to identify problems with the model's inputs and / or outputs. For example, these "protective" models can be trained to identify "safe" or otherwise okay or desired inputs and / or outputs and / or "unsafe" or otherwise unwanted inputs and / or outputs for a particular application / implementation. Therefore, the LLM / SLM / VLM / MMLM / etc. disclosed herein are unlikely to output language / text / audio / video / design data / USD data / etc. that may be offensive, vulgar, inappropriate, insecure, out of scope, and / or unwanted for a particular application / implementation.

[0067] In some embodiments, LLM / SLM / VLM / etc. can be configured or able to access or use one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc. For example, for certain tasks or operations where the model is not ideally suited, the model may have instructions for accessing one or more plugins (e.g., third-party plugins) to help process the current input (e.g., as a result of training, and / or based on instructions in a given prompt). In such an example, when at least part of the prompt relates to restaurants or weather, the model can access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. Another example is that if at least part of the response requires mathematical computation, the model can access one or more mathematical plugins or APIs to help solve the problem, and then the response from the plugins and / or APIs can be used in the model's output. This process can be repeated (e.g., recursively) an arbitrary number of iterations, using any number of plugins and / or APIs, until a response to each query / question / request / process / operation / etc. can be generated in response to the input prompt. Therefore, models can rely not only on their own knowledge gained from training on large datasets, but also on the expertise or optimized properties of one or more external resources (such as APIs, plugins, etc.).

[0068] In some embodiments, multiple language models (e.g., LLM / SLM / VLM / MMLM / etc., multiple instances of the same language model, and / or multiple hints provided to the same language model or instances of the same language model) can be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data stores, repositories, etc.) to provide output in response to the same query or in response to separate parts of a query. In at least one embodiment, the same input query and hints (e.g., a set of constraints, condition generators, etc.) can be provided to multiple language models (e.g., language models with different architectures, language models trained on different (e.g., updated) data corpora). In one or more embodiments, the language models can be different versions of the same base model. In one or more embodiments, at least one language model can be instantiated as multiple agents, for example, providing more than one hint to constrain, guide, or otherwise influence the style, content, or character of the provided output. In one or more exemplary non-limiting embodiments, the same language model can be required to provide output corresponding to different roles, perspectives, characters, or different knowledge bases, as defined by the provided hints.

[0069] In any such embodiment, the outputs of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiated proxies of at least one language model, and / or provided to two or more prompts for at least one language model can be further processed, such as aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more embodiments, the output from one language model (or version, instance, or proxy) can be provided as input to another language model for further processing and / or validation. In one or more embodiments, the language model can be required to generate or otherwise obtain output about the input source material, wherein the output is associated with the input source material. Such association can include, for example, generating captions or text portions embedded (e.g., as metadata) within the input source text or image. In one or more embodiments, the output of the language model can be used to determine the validity of the input source material for further processing or inclusion in a dataset. For example, the language model can be used to evaluate the presence (or absence) of a target word in a text portion or the presence (or absence) of an object in an image, wherein the text or image is annotated to indicate such presence (or absence). Alternatively, the determination from the language model can be used to determine whether the source material should be included in a curatorial dataset, for example, but not limited to this.

[0070] Figure 8A This is a block diagram of an example generative language model system 800 suitable for implementing at least some embodiments of the present disclosure. Figure 8A In the example shown, the generative language model system 800 includes a retrieval augmentation generation (RAG) component 892, an input processor 805, a tokenizer 810, an embedding component 820, a plugin / API 895, and a generative language model (LM) 830 (which may include LLM, VLM, multimodal LM, etc.).

[0071] At a high level, the input processor 805 can receive input 801, which includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasound, etc.), 3D design data, CAD data, generic scene descriptor (USD) data (e.g., OpenUSD, etc.), depending on the architecture of the generative LM 830 (e.g., LLM / SLM / VLM / MMLM, etc.). In some embodiments, input 801 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 801 may include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., tabular format, JSON, or XML). In generative LM In some implementations of 830 capable of handling multimodal input, input 801 can combine text (or text that may be omitted) with image data, audio data, video data, design data, USD data, and / or other types of input data (e.g., but not limited to the data described herein). Taking raw input text as an example, input processor 805 can prepare the raw input text in various ways. For example, input processor 805 can perform various types of text filtering to remove noise from relevant text content (e.g., special characters, punctuation marks, HTML tags, stop words, portions of images, portions of audio, etc.). In examples involving stop words (common words that often have little semantic meaning), input processor 805 can remove stop words to reduce noise and allow the generative LM 830 to focus on more meaningful content. Input processor 805 can apply text normalization, for example, by converting all characters to lowercase, removing accent marks, and / or handling special cases (such as abbreviations or abbreviations) to ensure consistency. These are just a few examples; other types of input processing can be applied.

[0072] In some embodiments, RAG component 892 (which may include one or more RAG models, and / or may be performed using generative LM 830 itself) may be used to retrieve additional information to be used as part of input 801 or a prompt. RAGs can be used to enhance input to LLM / SLM / VLM / MMLM / etc. with external knowledge to make the answer to a specific question or query or request more relevant, such as when specific knowledge is required. RAG component 892 may obtain this additional information from one or more external sources (e.g., basic information such as basic text / images / videos / audio / USD / CAD / etc.), which can then be fed along with the prompt to LLM / SLM / VLM / MMLM / etc. to improve the accuracy of the model's response or output.

[0073] For example, in some embodiments, in addition to the data retrieved using RAG component 892, input 801 may be generated using query or model inputs (e.g., questions, requests, etc.). In some embodiments, input processor 805 may analyze input 801 and communicate with RAG component 892 (or in embodiments, RAG component 892 may be part of input processor 805) to identify relevant text and / or other data to provide to generative LM 830 as additional context or information source, typically from which to identify responses, answers, or outputs 890. For example, when the input indicates that a user is interested in the required tire pressure for a particular brand and model of vehicle, RAG component 892 may use a RAG model, for example, to perform a vector search in the embedding space to retrieve tire pressure information or its corresponding text from a digital (embedded) version of the owner's manual for that particular vehicle brand and model. Similarly, when a user revisits a chatbot related to a particular product for sale or service, RAG component 892 may retrieve previously stored conversation history (or at least its summary) and provide the previous conversation history along with the current inquiry / request as part of input 801 to generative LM 830.

[0074] RAG component 892 can use various RAG techniques. For example, naïve RAG can be used, where documents are indexed, chunked, and applied to an embedding model to generate embeddings corresponding to chunks. User queries can also be applied to this embedding model and / or another embedding model of RAG component 892, and the embeddings of chunks can be compared with the embeddings of the query to identify the most similar / relevant embeddings to the query. These most similar / relevant embeddings can be provided to generative LM 830 to generate output.

[0075] In some embodiments, more advanced RAG techniques can be used. For example, chunks can undergo pre-retrieval processes (e.g., routing, rewriting, metadata analysis, expansion, etc.) before being passed to the embedding model. Furthermore, post-retrieval processes (e.g., re-ranking, hint compression, etc.) can be performed on the output of the embedding model before generating the final embedding, which is then used for comparison with the input query.

[0076] As a further example, modular RAG techniques can be used, such as those similar to Naive RAG and / or Advanced RAG, but also including features such as hybrid search, recursive retrieval and query engines, StepBack methods, subqueries and hypothetical document embeddings.

[0077] As another example, Graph RAG can use a knowledge graph as a source of context or factual information. Graph RAG can be implemented using a graph database as a source of contextual information sent to LLM / SLM / VLM / MMLM / etc. Instead of providing the model with chunks of data extracted from larger documents (which can lead to a lack of context, factual accuracy, linguistic accuracy, etc.) (or anything other than providing the model with chunks of data extracted from larger documents), Graph RAG can also provide structured entity information to LLM / SLM / VLM / MMLM / etc. by combining structured entity text descriptions with their many attributes and relationships, thus giving the model deeper insights. In implementing Graph RAG, the systems and methods described herein use graphs as content stores and extract relevant document chunks, requiring LLM / SLM / VLM / MMLM / etc. to use them to answer questions. In such embodiments, the knowledge graph may contain relevant textual content and metadata about the knowledge graph, or it may be integrated with a vector database. In some embodiments, Graph RAG can use the graph as a subject matter expert, where descriptions of concepts and entities relevant to the query / hint can be extracted and passed to the model as semantic context. These descriptions may include relationships between concepts. In other examples, the graph can be used as a database where a portion of a query / hint can be mapped to a graph query, the graph query can be executed, and LLM / SLM / VLM / MMLM / etc. can aggregate the results. In such examples, the graph can store relevant factual information and can be used for queries (natural language queries) and entity links to graph query tools (NL to graph query tools). In some embodiments, the graph RAG (e.g., using a graph database) can be combined with standard (e.g., vector database) RAGs and / or other RAG types to benefit from a variety of approaches.

[0078] In any embodiment, the RAG component 892 can implement plugins, APIs, user interfaces, and / or other functions to perform RAG. For example, LLM / SLM / VLM / MMLM / etc. can use graph RAG plugins to run queries on knowledge graphs to extract relevant information to feed into the model, and can use standard or vector RAG plugins to run queries on vector databases. For example, the graph database can interact with the plugin's REST interface, thus decoupling the graph database from the vector database and / or embedded models.

[0079] The tokenizer 810 can segment (e.g., processed) text data into smaller units (tags) for subsequent analysis and processing. Depending on the implementation, the tags can represent individual words, sub-words, characters, audio / video / images, etc. Word-based tokenization divides the text into individual words, treating each word as a separate tag. Sub-word tokenization breaks words down into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 830 to understand morphological changes and process words outside the vocabulary more effectively. Character-based tokenization represents each character as a separate tag, enabling the generative LM 830 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the task at hand, and / or the characteristics of the training dataset. Therefore, the tokenizer 810 can transform (e.g., processed) text into a structured format according to the tokenization scheme implemented in a particular embodiment.

[0080] Embedding component 820 can use any known embedding technique to transform discrete tokens into semantically meaningful (e.g., dense, continuous vector) representations. For example, embedding component 820 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or others.

[0081] In some implementations where input 801 includes image data / video data, etc., input processor 801 may resize the data to a standard size compatible with the format of the corresponding input channel and / or normalize pixel values ​​to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 820 may encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations where input 801 includes audio data, input processor 801 may resample the audio file to a consistent sampling rate for uniform processing, and embedding component 820 may use any known technique to extract and encode audio features, such as in the form of a spectrogram (e.g., a Mel spectrogram). In some implementations where input 801 includes video data, input processor 801 may extract frames or apply resizing to extracted frames, and embedding component 820 may extract features such as optical flow embedding or video embedding and / or encode temporal information or frame sequences. In some implementations where input 801 includes multimodal data, the embedded component 820 can use techniques such as early fusion (concatenation), late fusion (sequential processing), and attention-based fusion (e.g., self-attention, cross-attention) to fuse representations of different types of data (e.g., text, images, audio, USD, video, design, etc.).

[0082] Other components of the generative LM 830 and / or generative LM system 800 may use different types of neural network architectures depending on the implementation scheme. For example, a transducer-based architecture (such as the one used in models like GPT) may be implemented, and it may include a self-attention mechanism that weights the importance of different words or tokens in the input sequence and / or a feedforward network that processes the output of the self-attention layer, applying a nonlinear transformation to the input representation and extracting higher-level features. Some non-limiting example architectures include transducers (e.g., encoder-decoder, decoder-only, multimodal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of adversarial networks (such as generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning), etc. Therefore, depending on the implementation scheme and architecture, the embedded component 820 can apply the encoded representation of the input 801 to the generative LM 830, and the generative LM 830 can process the encoded representation of the input 801 to generate an output 890, which may include response text and / or other types of data.

[0083] As described herein, in some embodiments, the generative LM 830 may be configured to access or use (or be able to access or use) plugins / APIs 895 (which may include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations where the generative LM 830 is not ideally suited, the model may have instructions (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using RAG component 892) to access one or more plugins / APIs 895 (e.g., third-party plugins) to help process the current input. In such an example, when at least part of the prompt is related to a restaurant or weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs), sending at least part of the prompt related to a particular plugin / API 895 to the plugin / API 895, which may process the information and return an answer to the generative LM 830, which may use the response to generate output 890. This process can be repeated (e.g., recursively) an arbitrary number of iterations and repeated with any number of plugins / APIs 895 until an output 890 that resolves each query / question / request / process / action / etc. from input 801 is generated. Therefore, the model can rely not only on its own knowledge acquired from training on a large dataset and / or from data retrieved using the RAG component 892, but also on the expertise or optimized properties of one or more external resources (e.g., plugins / APIs 895).

[0084] Figure 8B This is a block diagram of an example implementation scheme, where the generative LM 830 includes a converter encoder-decoder. For example, suppose the input text (e.g., "Who discovered gravity") is tokenized (e.g., by...) Figure 8A The tokenizer 810) is used for tokens such as words, and each token is encoded (e.g., by...). Figure 8A The embedding component 820 is a corresponding embedding (e.g., of size 512). Since these token embeddings do not typically represent the position of the tokens in the input sequence, positional encoding can be added to each token embedding using any known technique to encode the order relation and context of the tokens in the input sequence. Thus, the (e.g., the resulting) embeddings can be applied to one or more encoders 835 of the generative LM 830.

[0085] In the example implementation, encoder 835 forms an encoder stack, where each encoder includes a self-attention layer and a feedforward network. In the example converter architecture, each token (e.g., a word) flows through a separate path. Therefore, each encoder can accept a sequence of vectors, pass each vector through the self-attention layer, then through the feedforward network, and then up to the next encoder in the stack. Any known self-attention technique can be used. For example, to compute a self-attention score for each token (word), a query vector, a key vector, and a value vector can be created for each token. The self-attention score for a token pair can be computed by taking the dot product of the query vector and the corresponding key vector, normalizing the resulting score, multiplying by the corresponding value vector, and summing the weighted value vectors. The encoder can apply multi-head attention, where the attention mechanism is applied multiple times in parallel with different learned weight matrices. Any number of encoders can be cascaded to generate context vectors that encode the input. Attention projection layer 840 can transform the context vectors into attention vectors (keys and values) for decoder 845.

[0086] In the example implementation, decoder 845 forms a decoder stack, where each decoder includes a self-attention layer, an encoder-decoder self-attention layer that uses attention vectors (keys and values) from the encoder to focus on relevant parts of the input sequence, and a feedforward network. Similar to encoder 835, in the example converter architecture, each token (e.g., a word) flows through a separate path in decoder 845. During the first pass, decoder 845, classifier 850, and generation mechanism 855 can generate a first token, and generation mechanism 855 can apply the generated token as input during a second pass. This process can be repeated cyclically, generating tokens (e.g., words) and adding them to the output of the previous pass, and in subsequent passes applying token embeddings of positionally encoded composite sequences as input to decoder 845, generating one token at a time (called autoregression) until a symbol or token representing the end of the response is predicted. In each decoder, the self-attention layer is typically restricted to focusing only on preceding positions in the output sequence by applying a masking technique (e.g., setting future positions to negative infinity) before the softmax operation. In the example implementation, the encoder-decoder attention layer operates similarly to the (e.g., multi-head) self-attention operation in the encoder 835, except that it creates its query from the layer below it and obtains keys and values ​​(e.g., matrices) from the output of the encoder 835.

[0087] Therefore, decoder 845 can output some decoded (e.g., vector) representation of the input applied during a particular pass. Classifier 850 can include a multi-class classifier comprising one or more neural network layers and a softmax operation that transforms logit probabilities into probabilities, the neural network layers projecting the decoded (e.g., vector) representation onto corresponding dimensions (e.g., one dimension for each supported word or token in the output vocabulary). Thus, generation mechanism 855 can select or sample words or tokens based on corresponding predicted probabilities (e.g., selecting the word with the highest predicted probability) and append it to the output of the previous pass, thereby generating each word or token sequentially. Generation mechanism 855 can repeat this process, triggering successive decoder inputs and corresponding predictions until a symbol or token representing the end of the response is selected or sampled, at which point generation mechanism 855 can output the generated response.

[0088] Figure 8C This is a block diagram of an example implementation where the generative LM 830 includes a decoder-only converter architecture. For example, Figure 8C The decoder 860 can be used with Figure 8B The decoder 845 operates similarly, except... Figure 8C Each decoder 860 omits the encoder-decoder self-attention layer (because there is no encoder in this implementation). Therefore, decoders 860 can form a decoder stack, where each decoder includes a self-attention layer and a feedforward network. Furthermore, instead of encoding the input sequence, a symbol or tag indicating the end of the input sequence (or the beginning of the output sequence) can be appended to the input sequence, and the resulting sequence (e.g., a corresponding embedding with positional encoding) can be applied to decoder 860. Figure 8B Similar to decoder 845, each tag (e.g., a word) can flow through a separate path in decoder 860, and decoder 860, classifier 865, and generation mechanism 870 can use autoregression to generate one tag at a time sequentially until a symbol or tag indicating the end of the response is predicted. Classifier 865 and generation mechanism 870 can be combined with... Figure 8B The classifier 850 and the generation mechanism 855 operate similarly, wherein the generation mechanism 870 selects or samples each consecutive output label based on the corresponding predicted probability and appends it to the output of the previous iteration, generating each label sequentially until a symbol or label representing the end of the response is selected or sampled. The architectures described herein, and others, are merely examples, and other suitable architectures may be implemented within the scope of this disclosure.

[0089] Example computing device Figure 9 This is a block diagram of an example computing device 900 suitable for implementing some embodiments of the present disclosure. The computing device 900 may include an interconnect system 902 directly or indirectly coupled to: a memory 904, one or more central processing units (CPUs) 906, one or more graphics processing units (GPUs) 908, a communication interface 910, input / output (I / O) ports 912, input / output components 914, a power supply 916, one or more presentation components 918 (e.g., one or more displays), and one or more logic units 920. In at least one embodiment, one or more computing devices 900 may include one or more virtual machines (VMs), and / or any of their components may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 908 may include one or more vGPUs, one or more CPUs 906 may include one or more vCPUs, and / or one or more logic units 920 may include one or more virtual logic units. Thus, one or more computing devices 900 may include discrete components (e.g., a full GPU dedicated to computing device 900), virtual components (e.g., a portion of the GPU dedicated to computing device 900), or a combination thereof.

[0090] although Figure 9 The various blocks are shown as connected via interconnect system 902 using lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 918 (such as a display device) may be considered I / O component 914 (e.g., if the display is a touchscreen). As another example, CPU 906 and / or GPU 908 may include memory (e.g., memory 904 may represent a storage device other than the memory of GPU 908, CPU 906, and / or other components). Therefore, Figure 9 The computing devices described are for illustrative purposes only. No distinction is made between such categories as "workstation," "server," "laptop computer," "desktop computer," "tablet computer," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all are considered within the scope of this category. Figure 9 Within the scope of computing devices.

[0091] Interconnect system 902 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 902 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 906 may be directly connected to memory 904. Further, CPU 906 may be directly connected to GPU 908. Where there is a direct or point-to-point connection between components, interconnect system 902 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required to be included in computing device 900.

[0092] The memory 904 may include any computer-readable medium from a variety of computer-readable media. A computer-readable medium may be any available medium accessible by the computing device 900. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0093] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 904 may store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 900. As used herein, computer storage media does not include the signal itself.

[0094] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information transmission medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0095] CPU 906 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. Each CPU 906 may contain one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPU 906 may contain any type of processor and may contain different types of processors depending on the type of computing device 900 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 900, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as math coprocessors), computing device 900 may also include one or more CPUs 906.

[0096] In addition to or in lieu of one or more CPUs 906, one or more GPUs 908 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. One or more GPUs 908 may be integrated GPUs (e.g., having one or more CPUs 906) and / or one or more GPUs 908 may be discrete GPUs. In embodiments, one or more GPUs 908 may be coprocessors of one or more CPUs 906. GPUs 908 may be used by computing device 900 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 908 may be used for general-purpose computing on a GPU (GPGPU). GPUs 908 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPUs 908 may generate pixel data for an output image in response to rendering commands (e.g., rendering commands received from CPUs 906 via a host interface). GPU 908 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 904. GPU 908 may include two or more GPUs operating in parallel (e.g., via links). The links may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 908 may produce pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for an analog image). Each GPU may include its own memory or may share memory with other GPUs.

[0097] In addition to or in lieu of CPU 906 and / or GPU 908, logic unit 920 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 906, one or more GPUs 908, and / or one or more logic units 920 may execute any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 920 may be a portion of one or more CPUs 906 and / or GPUs 908 and / or integrated into one or more CPUs 906 and / or GPUs 908, and / or one or more logic units 920 may be discrete components or otherwise external to CPUs 906 and / or GPUs 908. In embodiments, one or more logic units 920 may be coprocessors of one or more CPUs 906 and / or GPUs 908.

[0098] Examples of logic unit 920 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree lateral unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), programmable vision accelerator (PVA) (which may include one or more direct memory access (DMA) systems), one or more vision or vector processing units (VPU), and one or more pixel processing engines (PPE) (e.g.) Examples include 2D arrays of processing elements, each of which communicates north, south, east, and west with one or more other processing elements in the array; one or more decoupled accelerators or units (e.g., decoupled lookup table (DLUT) accelerators or units); vision processing units (VPUs); optical flow accelerators (OFAs); field-programmable gate arrays (FPGAs); neuromorphic chips; quantum processing units (QPUs); associative processing units (APUs); arithmetic logic units (ALUs); application-specific integrated circuits (ASICs); floating-point units (FPUs); input / output (I / O) elements; peripheral component interconnects (PCIs) or fast peripheral component interconnects (PCIe) elements; and so on.

[0099] The communication interface 910 may include one or more receivers, transmitters, and / or transceivers enabling the computing device 900 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). The communication interface 910 may include components and functions for enabling communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wirelessband), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 920 and / or the communication interface 910 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 902 to one or more GPUs 908 (e.g., memory of one or more GPUs 908).

[0100] I / O port 912 enables computing device 900 to be logically coupled to other devices including I / O component 914, one or more presentation components 918, and / or other components, some of which may be built into (e.g., integrated into) computing device 900. Illustrative I / O component 914 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, scanners, printers, wireless devices, etc. I / O component 914 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 900. Computing device 900 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, the computing device 900 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, the computing device 900 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0101] Power supply 916 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 916 may provide power to computing device 900 so that the components of computing device 900 can operate.

[0102] The presentation component 918 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 918 may receive data from other components (e.g., GPU 908, CPU 906, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0103] Example Data Center Figure 10 An example data center 1000 that may be used in at least one embodiment of this disclosure is shown. The data center 1000 may include a data center infrastructure layer 1010, a framework layer 1020, a software layer 1030, and / or an application layer 1040.

[0104] like Figure 10 As shown, the data center infrastructure layer 1010 may include a resource coordinator 1012, grouped computing resources 1014, and node computing resources ("nodes CRs") 1016(1)-1016(N), where "N" represents any complete positive integer. In at least one embodiment, nodes CRs 1016(1)-1016(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules and / or cooling modules, etc. In some embodiments, one or more node CRs from nodes CRs 1016(1)-1016(N) may correspond to servers having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CRs 1016(1)-1016(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more of nodes CRs 1016(1)-1016(N) may correspond to virtual machines (VMs).

[0105] In at least one embodiment, the grouped computing resources 1014 may include individual groups of node CRs 1016 housed within one or more racks (not shown), or multiple racks housed within a data center at different geographical locations (also not shown). Individual groups of node CRs 1016 within the grouped computing resources 1014 may include grouped computing, networking, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several node CRs 1016, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0106] Resource coordinator 1022 may be configured or otherwise control one or more nodes CRs 1016(1)-1016(N) and / or groups of computing resources 1014. In at least one embodiment, resource coordinator 1022 may include a Software Design Infrastructure ("SDI") management entity for data center 1000. Resource coordinator 1022 may include hardware, software, or some combination thereof.

[0107] In at least one embodiment, such as Figure 10 As shown, framework layer 1020 may include job scheduler 1028, configuration manager 1034, resource manager 1036, and / or distributed file system 1138. Framework layer 1020 may include a framework for software 1032 supporting software layer 1030 and / or one or more applications 1042 supporting application layer 1040. Software 1032 or application 1042 may respectively contain web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1020 may be, but is not limited to, free and open-source software web application frameworks (such as Apache Spark) that can utilize distributed file system 1138 for large-scale data processing (e.g., "big data"). TM (Hereinafter referred to as "Spark") is a type of resource manager. In at least one embodiment, job scheduler 1028 may include Spark drivers to facilitate the scheduling of workloads supported by different layers of data center 1000. Configuration manager 1034 may be able to configure different layers, such as software layer 1030 and framework layer 1020 (which includes Spark and distributed file system 1138 for supporting large-scale data processing). Resource manager 1036 may be able to manage computing resources mapped to or allocated to clusters of distributed file system 1138 and job scheduler 1028 or to support clusters of distributed file system 1138 and job scheduler 1028. In at least one embodiment, clusters or groups of computing resources may include grouped computing resources 1014 in data center infrastructure layer 1010. Resource manager 1036 may coordinate with resource coordinator 1012 to manage these mapped or allocated computing resources.

[0108] In at least one embodiment, the software 1032 included in software layer 1030 may include software used in at least a portion of the nodes CRs 1016(1)-1016(N), the grouped computing resources 1014, and / or the distributed file system 1138 of framework layer 1020. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0109] In at least one embodiment, the application 1042 included in the application layer 1040 may include one or more types of applications used at least in part by nodes CRs 1016(1)-1016(N), grouped computing resources 1014, and / or the distributed file system 1138 of the framework layer 1020. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in combination with one or more embodiments.

[0110] In at least one embodiment, any of the configuration manager 1034, resource manager 1036, and resource coordinator 1012 can implement any number and type of self-modification actions based on any amount and type of data obtained in any technically feasible manner. Self-modification actions can free the data center operator of data center 1000 from making potentially poor configuration decisions and may prevent underutilization and / or poor performance of the data center.

[0111] According to one or more embodiments described herein, data center 1000 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information. For example, one or more machine learning models can be trained by using the software and / or computing resources described above with respect to data center 1000 to compute weight parameters according to a neural network architecture. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks can be used to infer or predict information using the resources described above with respect to data center 1000 by using weight parameters computed through one or more training techniques, such as, but not limited to, those described herein.

[0112] In at least one embodiment, the data center 1000 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform services that infer information, such as image recognition, speech recognition, or other artificial intelligence services.

[0113] Example network environment A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 9 This is implemented on one or more instances of computing devices 900—for example, each device may include similar components, features, and / or functions of one or more computing devices 900. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of a data center 1000, examples of which are described in this document. Figure 10 To describe in more detail.

[0114] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In cases where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0115] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0116] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0117] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). If the connection to a user (e.g., a client device) is relatively close to an edge server, the core server can assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0118] One or more client devices may include the information described in this article. Figure 9 At least some of the components, features, and functions of one or more example computing devices 900 described. By way of example and not limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0119] Example Terms A. A method comprising: obtaining input data representing at least one natural language expression of a user associated with an interactive application, the at least one natural language expression indicating one or more requests to issue one or more alerts in response to one or more events associated with the interactive application; generating text data corresponding to one or more logical conditions associated with and issuing the one or more alerts, based at least on processing the input data using one or more language models; evaluating the one or more logical conditions based at least on state data representing one or more states of the interactive application to determine that the one or more events have occurred; and causing the one or more alerts to be output, based at least on the occurrence of the one or more events.

[0120] B. The method as described in paragraph A, wherein the text data represents one or more data structures corresponding to the one or more logical conditions, the one or more data structures including at least one or more identifiers of one or more status keys, one or more thresholds, and one or more comparison operators.

[0121] C. The method as described in any one of paragraphs A and B, wherein the text data represents code corresponding to the one or more logical conditions, the code comprising one or more functions for determining whether the one or more events have occurred, at least based on evaluating one or more state key values ​​indicating the one or more states of the interactive application.

[0122] D. The method as described in any one of paragraphs A, C, and D, wherein the code comprises one or more functions, the method further comprising: compiling the code into one or more dynamically loadable libraries; and, during the runtime of the interactive application, executing at least one of the one or more functions from the one or more dynamically loadable libraries to evaluate whether the one or more events have occurred.

[0123] E. The method as described in any one of paragraphs A, D, and E, wherein the text data represents a natural language prompt corresponding to the one or more logical conditions, the method further comprising: applying the text data as an input prompt to one or more second language models; and processing the text data and the state data based at least on the one or more second language models to obtain output data indicating whether the one or more events have occurred, wherein determining that the one or more events have occurred is based at least on the output data.

[0124] F. The method as described in any one of paragraphs A, E, and F, further comprising: obtaining data representing contextual information corresponding to the interactive application from one or more vector databases associated with the interactive application; wherein generating the text data using the one or more language models further processes the data representing the contextual information based at least on the one or more language models.

[0125] G. A system comprising one or more processors, the processors being configured to: generate text data corresponding to logical conditions for evaluating alert conditions indicated by the at least one natural language expression of a user associated with an interactive application, using one or more language models and based at least on input data representing at least one natural language expression of a user associated with an interactive application; determine that the alert conditions are satisfied, based at least on evaluating the state of the interactive application using the logical conditions; and, during runtime of the interactive application, cause an indication that the alert conditions are satisfied to be output.

[0126] H. The system as described in paragraph G, wherein the text data represents a data structure corresponding to the logical condition, the data structure including at least an identifier for a status key, a threshold, and a comparison operator.

[0127] I. The system as described in any one of paragraphs GH, wherein evaluating the state of the interactive application comprises: obtaining state data representing key-value pairs indicating the state of the interactive application, the key-value pairs including the state key; and evaluating the value of the state key with respect to the threshold using the comparison operator.

[0128] J. The system as described in any one of paragraphs GI, wherein the text data represents code corresponding to the logical condition, the code including a function for determining whether the alert condition is satisfied based at least on evaluating one or more state key values ​​associated with the state of the interactive application.

[0129] K. As described in any one of paragraphs GJ, the one or more processors are further configured to: compile the code into a dynamically loadable library; and execute the functions from the dynamically loadable library during the runtime of the interactive application.

[0130] L. The system as described in any one of paragraphs GK, wherein the one or more processors are further configured to: obtain data representing contextual information corresponding to the interactive application from a vector database associated with the interactive application; wherein generating the text data using the one or more language models further processes the data representing the contextual information based at least on the one or more language models.

[0131] M. A system as described in any one of paragraphs GL, wherein outputting the indication comprises: rendering one or more visual indications in a graphical user interface associated with the interactive application that the warning condition has been met.

[0132] N. The system as described in any one of paragraphs GM, wherein evaluating the state of the interactive application using the logical conditions comprises: at a first time, evaluating one or more first values ​​of one or more state variables representing the state of the interactive application; and at a second time after the first time, evaluating one or more second values ​​of the one or more state variables.

[0133] O. The system as described in any one of paragraphs GN, wherein: the first time corresponds to a first session of the interactive application, and the second time corresponds to a second session of the interactive application.

[0134] P. The system as described in any one of paragraphs GO, wherein the text data represents a natural language prompt corresponding to the logical condition, the one or more processors are further configured to: apply the text data as an input prompt to one or more second language models; and process the text data and state data representing the state of the interactive application based at least on the one or more second language models to obtain output data indicating that the warning condition is satisfied, wherein determining that the warning condition is satisfied is based at least on the output data.

[0135] Q. A system as described in any one of paragraphs GP, wherein the system comprises at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing analog operations; a system for performing digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system implementing one or more multimodal language models; a system implementing one or more large language models (LLMs); a system implementing one or more small language models (SLMs); a system implementing one or more visual language models (VLMs); a system for generating synthetic data; a system for generating synthetic data using AI; a system containing one or more virtual machines (VMs); a system using or deploying one or more inference microservices; a system containing one or more machine learning models deployed in services or microservices and OS-level virtualization packages; a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0136] R. One or more processors, including a processing circuitry system, the processing circuitry system being configured to: process input data representing a natural language request to issue an alert in response to the occurrence of an event associated with an interactive application using one or more language models; generate text data representing one or more logical conditions for evaluating one or more states of the interactive application during runtime; and determine whether to cause the alert to be output in a graphical user interface associated with the interactive application.

[0137] S. One or more processors as described in paragraph R, wherein the input data further represents context information corresponding to the interactive application, the context information being obtained from one or more vector databases associated with the interactive application.

[0138] T. One or more processors as described in any one of paragraphs R and S, wherein the one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing analog operations; a system for performing digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation of 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; or a system implemented using a robot. Systems that include: systems for performing conversational AI operations; systems that implement one or more multimodal language models; systems that implement one or more large language models (LLMs); systems that implement one or more small language models (SLMs); systems that implement one or more visual language models (VLMs); systems for generating synthetic data; systems for generating synthetic data using AI; systems that contain one or more virtual machines (VMs); systems that use or deploy one or more inference microservices; systems that contain one or more machine learning models deployed in services or microservices and OS-level virtualization packages; systems that are at least partially implemented in a data center; or systems that are at least partially implemented using cloud computing resources.

[0139] This disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions, such as program modules, that are executed by a computer or other machine, such as a personal data assistant or other handheld device. Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be practiced in distributed computing environments, where tasks are performed by remote processing devices linked via a communication network.

[0140] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" could include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" could include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" could include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0141] This document describes in detail the subject matter of this disclosure to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the discloser has envisioned that the claimed subject matter may be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be construed as implying any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.< / lt> < / health>

Claims

1. A method comprising: Obtain input data representing at least one natural language expression of a user associated with an interactive application, the at least one natural language expression indicating one or more requests to issue one or more alerts in response to one or more events associated with the interactive application; At least based on processing the input data using one or more language models, text data corresponding to one or more logical conditions associated with issuing the one or more alerts is generated; Evaluate the one or more logical conditions based at least on state data representing one or more states of the interactive application to determine that the one or more events have occurred; and At least one or more of the events have occurred, causing one or more alerts to be output.

2. The method of claim 1, wherein the text data represents one or more data structures corresponding to the one or more logical conditions, the one or more data structures including at least one or more identifiers of one or more status keys, one or more thresholds, and one or more comparison operators.

3. The method of claim 1, wherein the text data represents code corresponding to the one or more logical conditions, the code comprising one or more functions for determining whether the one or more events have occurred, at least based on evaluating one or more state key values ​​indicating the one or more states of the interactive application.

4. The method of claim 3, wherein the code comprises one or more functions, and the method further comprises: Compile the code into one or more dynamically loadable libraries; as well as During the runtime of the interactive application, at least one function from one or more dynamically loadable libraries is executed to evaluate whether one or more events have occurred.

5. The method of claim 1, wherein the text data represents a natural language prompt corresponding to the one or more logical conditions, the method further comprising: The text data is used as an input prompt in one or more second language models; as well as The text data and the state data are processed based on at least one or more second language models to obtain output data indicating whether one or more events have occurred. The determination that one or more events have occurred is based at least on the output data.

6. The method of claim 1, further comprising: Obtain data representing context information corresponding to the interactive application from one or more vector databases associated with the interactive application; The text data generated using one or more language models also includes processing the data representing the context information based at least on one or more language models.

7. A system comprising one or more processors, said one or more processors being used to: Using one or more language models and based at least on input data representing at least one natural language expression of a user associated with an interactive application, text data corresponding to logical conditions used to evaluate alert conditions indicated by the at least one natural language expression is generated. The alert condition is determined to be met at least based on evaluating the state of the interactive application using the aforementioned logical conditions. as well as During the runtime of the interactive application, an indication is given that the warning conditions are met.

8. The system of claim 7, wherein the text data represents a data structure corresponding to the logical condition, the data structure including at least an identifier for a status key, a threshold, and a comparison operator.

9. The system of claim 8, wherein evaluating the state of the interactive application comprises: Obtain state data representing key-value pairs indicating the state of the interactive application, the key-value pairs including the state key; as well as The comparison operator is used to evaluate the value of the state key with respect to the threshold.

10. The system of claim 7, wherein the text data represents code corresponding to the logical condition, the code comprising a function for determining whether the alert condition is satisfied based at least on evaluating one or more state key values ​​associated with the state of the interactive application.

11. The system of claim 10, wherein the one or more processors are further configured to: Compile the code into a dynamically loadable library; and During the runtime of the interactive application, the functions from the dynamically loadable library are executed.

12. The system of claim 7, wherein the one or more processors are further configured to: Obtain data representing context information corresponding to the interactive application from a vector database associated with the interactive application; The text data generated using one or more language models also includes processing the data representing the context information based at least on one or more language models.

13. The system of claim 7, wherein causing the indication to be output comprises: Render one or more visual indications that the alert condition has been met in the graphical user interface associated with the interactive application.

14. The system of claim 7, wherein evaluating the state of the interactive application using the logical conditions comprises: At the first moment, one or more first values ​​of one or more state variables representing the state of the interactive application are evaluated; as well as At a second time point following the first time point, one or more second values ​​of the one or more state variables are evaluated.

15. The system of claim 14, wherein: The first time corresponds to the first session of the interactive application, and The second time corresponds to the second session of the interactive application.

16. The system of claim 7, wherein the text data represents a natural language prompt corresponding to the logical condition, and the one or more processors are further configured to: The text data is used as input prompts and applied to one or more second language models; and The text data and state data representing the state of the interactive application are processed based on at least one or more second language models to obtain output data indicating that the warning condition has been met. The determination that the warning condition is met is based at least on the output data.

17. The system of claim 7, wherein the system comprises at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system used to perform deep learning operations; A system used to perform remote operations; Systems used for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; Systems implemented using edge devices; Systems implemented using robots; Systems used to perform conversational AI operations; A system that implements one or more multimodal language models; A system that implements one or more large language model LLMs; A system that implements one or more small language models (SLMs); A system that implements one or more Visual Language Models (VLMs); A system for generating synthetic data; Systems for generating synthetic data using AI; A system containing one or more virtual machines (VMs); A system that uses or deploys one or more inference microservices; A system that includes one or more machine learning models deployed in a service or microservice, as well as an OS-level virtualization package; A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

18. One or more processors, including a processing circuitry system, said processing circuitry system being used for: Input data representing a natural language request to issue an alert in response to an event associated with the interactive application is processed using one or more language models to generate text data representing one or more logical conditions for evaluating one or more states of the interactive application during runtime, in order to determine whether to output the alert in the graphical user interface associated with the interactive application.

19. The processor of claim 18, wherein the input data further represents context information corresponding to the interactive application, the context information being obtained from one or more vector databases associated with the interactive application.

20. The processor of claim 18, wherein the processor is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system for collaborative content creation of 3D assets; A system used to perform deep learning operations; A system used to perform remote operations; Systems used for performing real-time streaming; A system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; Systems implemented using edge devices; Systems implemented using robots; Systems used to perform conversational AI operations; A system that implements one or more multimodal language models; A system that implements one or more large language model LLMs; A system that implements one or more small language models (SLMs); A system that implements one or more Visual Language Models (VLMs); A system for generating synthetic data; Systems for generating synthetic data using AI; A system containing one or more virtual machines (VMs); A system that uses or deploys one or more inference microservices; A system that includes one or more machine learning models deployed in a service or microservice, as well as an OS-level virtualization package; A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.