Scene graph for video scene understanding and information retrieval

Abstract

The invention relates to a scene graph for video scene understanding and information retrieval. Generation and use interaction graphs for video information retrieval systems and applications are described herein in various examples. Systems and methods of processing a video generated using one or more image sensors to generate a graph representing at least interactions between entities depicted by the video are disclosed. For example, nodes of a graph may be associated with entities (e.g., persons and / or other objects) and attributes associated with the entities. Further, edges of the graph may be associated with interactions between entities, times at which interactions occur, and / or indications as to which videos describe those interactions. Systems and methods for performing retrieval of information associated with a video using the graph are then further disclosed. For example, the graph may be used to identify relevant information associated with a query, where these information may then be used to generate a response.

Description

Background Technology

[0001] The system uses cameras for various applications, such as monitoring warehouse, factory, and / or retail environments to optimize routes and / or product or object locations, determining information about the surrounding environment for autonomous driving, or providing security for individuals and / or businesses. In some cases, determining information about the video captured by such cameras can be important. For example, if a camera captures an event (such as an accident caused by a forklift driven by a worker in a warehouse), determining information about that event and / or retrieving portions of the video describing that event can be important. However, accurately identifying the important information and / or portions of the video can be difficult. For example, if the warehouse in the example above contains multiple cameras capturing the interior of the warehouse, with each camera continuously generating video, identifying which videos describe the event can be challenging.

[0002] Therefore, techniques have been developed for retrieving information about videos based on user requests. For example, a system might process a video to generate a description containing information about each frame. For instance, if a frame represents a worker driving a forklift, the description for that frame might indicate the identifier of the worker driving the forklift, the time the worker is driving the forklift, and the location of the forklift within the warehouse. When a query is received to retrieve information related to an event, the system might then compare the query (e.g., the embedding associated with the query) with the generated descriptions (e.g., the embedding associated with the description) to identify which descriptions contain information related to the event. Furthermore, the system might then return these identified descriptions to the user who provided the query.

[0003] However, several problems may arise with techniques that generate descriptions for frames only for subsequent retrieval. For example, because these systems generate descriptions for individual frames of a video, these descriptions may lack context and / or causal relationships between frames, potentially reducing the quality of the retrieved information. For instance, if multiple videos capture an event that occurred over a period of time and / or at different time intervals, the system may not associate the frames describing that event, making it unlikely that the most relevant information will be retrieved for the query. Furthermore, since multiple cameras may be used to generate the video, these systems may require significant computational resources to generate and / or store these descriptions for the video. For example, storing data for each frame (e.g., embeddings) may require substantial memory, and data for a single video over a period of time may contain tens of thousands of frames. Moreover, for similar reasons, these systems may involve significant latency when traversing the stored data to search for information that may be relevant to the query. Summary of the Invention

[0004] Embodiments of this disclosure relate to the generation and use of interaction graphs for video information retrieval systems and applications. A system and method are disclosed that process video generated using one or more image sensors to generate a graph containing representations of interactions between entities depicted in the video. For example, nodes in the graph may be associated with entities (e.g., people and / or other objects) and attributes associated with those entities. Furthermore, edges in the graph may be associated with interactions between entities, the timing of those interactions, and / or indications of which videos depict those interactions. Systems and methods for using the graph to perform video-related information retrieval are then further disclosed. For example, when a query is received, the graph and / or one or more other sources (e.g., a database storing descriptions of frames) may be used to determine information relevant to the query. One or more language models may then process input data associated with that information to generate a response to the query.

[0005] Compared to conventional systems, the systems of this disclosure, in some embodiments, may generate and then use an interaction graph that represents the context and / or causal relationships between frames (e.g., interactions) of a video. Therefore, the systems of this disclosure may identify information more relevant to a query, such as by identifying information relevant across multiple frames based on entities and / or interactions. Furthermore, unlike conventional systems, in some embodiments, this graph may allow the systems of this disclosure to answer time-related and / or entity-related queries more efficiently. For example, one or more portions of the graph may be filtered based on text in the query, such that relevant portions of the graph are processed when performing information retrieval. Additionally, as described in more detail herein, information identified from the graph may be used to filter descriptions related to the video and / or video frames, which may further reduce latency when performing information retrieval. Attached Figure Description

[0006] The following will describe in detail, with reference to the accompanying drawings, the system and method for generating and using interactive graphs in video information retrieval systems and applications, wherein:

[0007] Figure 1A Examples of processes for generating interactive graphs associated with video, according to some embodiments of the present disclosure, are shown;

[0008] Figure 1B Examples of processes for performing information retrieval using interactive graphs according to some embodiments of the present disclosure are shown;

[0009] Figure 2 Examples of videos showing interactions between entities according to some embodiments of this disclosure are shown;

[0010] Figure 3Examples of generating computer vision information associated with frames of a video according to some embodiments of the present disclosure are shown;

[0011] Figure 4 Examples of generating frame-related descriptions of a video according to some embodiments of this disclosure are shown;

[0012] Figure 5 Examples of generating summaries associated with video descriptions and / or frames are shown according to some embodiments of this disclosure;

[0013] Figure 6 Examples of generating interactive graphs associated with video according to some embodiments of the present disclosure are shown;

[0014] Figure 7 Examples of generating and performing queries associated with information retrieval according to some embodiments of this disclosure are shown;

[0015] Figure 8 Examples of generating a query-associated summary according to some embodiments of the present disclosure are shown, wherein the summary is used to perform information retrieval;

[0016] Figure 9 Examples of using a summary to retrieve information from a figure according to some embodiments of this disclosure are shown;

[0017] Figure 10 Examples of using retrieved information to generate a response associated with a query, according to some embodiments of this disclosure, are shown;

[0018] Figure 11 Examples of one or more systems that can be configured to perform one or more processes described herein are shown according to some embodiments of the present disclosure;

[0019] Figure 12 A flowchart illustrating a method for generating an interactive graph associated with one or more videos, according to some embodiments of the present disclosure, is shown;

[0020] Figure 13 A flowchart illustrating a method for performing information retrieval using an interactive graph, according to some embodiments of the present disclosure, is shown;

[0021] Figure 14A This is a block diagram of an example generative language model system suitable for implementing at least some embodiments of the present disclosure;

[0022] Figure 14B It is a block diagram of an example generative language model that includes a converter encoder-decoder suitable for implementing at least some embodiments of the present disclosure;

[0023] Figure 14CIt is a block diagram of an example generative language model that includes a decoder-only converter architecture suitable for implementing at least some embodiments of the present disclosure;

[0024] Figure 15 This is a block diagram of an example computing device suitable for implementing at least some embodiments of the present disclosure; and

[0025] Figure 16 This is a block diagram of an example data center applicable to implementing at least some embodiments of the present disclosure. Detailed Implementation

[0026] Systems and methods related to generating and using interaction graphs for video information retrieval systems and applications are disclosed. For example, one or more systems may receive, retrieve, acquire, access, and / or store video data generated using one or more image sensors (e.g., one or more cameras). As described herein, the video data may represent one or more videos that at least depict entities and / or interactions between entities. In some examples, entities may include, but are not limited to, people, vehicles, machines, animals, equipment, shelves, boxes, and / or any other type of object. Furthermore, interactions may include, but are not limited to, approaching an entity, walking away from an entity, talking to an entity, instructing an entity, pushing an entity, placing an entity, lifting an entity, carrying an entity, driving an entity, causing a collision with an entity, and / or any other type of interaction that may occur between two or more entities.

[0027] Subsequently, one or more systems may process the video data (e.g., by using one or more computer vision (CV) models, algorithms, and / or any other type of processing components) to determine CV information associated with frames of one or more videos. As described herein, CV information may include, but is not limited to, identifiers of entities, the location of entities within a frame, the boundary shape (e.g., a bounding box) representing the portion of the frame depicting the entity, attributes associated with the entity, actions being performed by the entity, and / or any other information. One or more systems may also process the video data and / or CV information (e.g., by using one or more language models (e.g., one or more visual language models) and / or any other type of processing components) to generate a description representing the information associated with the frame. For example, in some examples, the description associated with a frame may include identifiers of the entities depicted by the frame, the location depicted by the frame, the time the frame was generated, one or more interactions between entities depicted by the frame, attributes associated with the entities, and / or any other information associated with the frame.

[0028] In some examples, one or more systems generate a corresponding description for each frame. Alternatively, in some examples, one or more systems generate corresponding descriptions for groups of frames. In any of these examples, one or more systems then store the CV information and / or descriptions in one or more databases for later retrieval. For example, one or more systems use one or more encoders to generate embeddings associated with the CV information and / or descriptions. The one or more systems then store the embeddings in one or more vector databases.

[0029] One or more systems may then process these descriptions (e.g., by using one or more language models (e.g., one or more text2cypher language models) and / or any other type of processing component) to generate text associated with these descriptions (referred to in some examples as a “summary”). As described herein, the summary associated with a description may indicate at least one or more entities described in the description, one or more interactions between one or more entities described in the description, and / or timestamps associated with one or more interactions. For example, if the description associated with a frame describes a first entity providing instructions to a second entity located in a warehouse at 5:00, the summary might include “Entity 1 → Instruction (5:00) → Entity 2”. For example, in some examples, the summary may be associated with a specific type of language, such as the Cypher Query Language (and / or any other type of query language).

[0030] Subsequently, one or more systems may use these summaries to generate a graph associated with one or more videos, also known as an "interaction graph." For example, one or more systems may generate a graph such that the nodes of the graph are associated with entities in one or more videos and / or additional information associated with those entities (e.g., attributes corresponding to the entities (e.g., CV information)). One or more systems may also generate a graph such that the edges of the graph are associated with interactions between entities and / or additional information associated with those interactions (e.g., timestamps indicating when the interaction occurred). Furthermore, as one or more image sensors continue to generate one or more videos and / or receive new descriptions for one or more videos, one or more systems may continuously update the graph.

[0031] For example, if a first summary associated with a first frame indicates a first person, one or more systems will generate a graph to include a first node associated with the first person. Next, if a second summary associated with a second frame indicates a second person is instructing the first person, one or more systems may update the graph to include a second node associated with the second person, as well as an edge between the first and second nodes indicating the interaction. Next, if a third summary associated with a third frame indicates the first person is performing an action associated with the instruction (e.g., driving a forklift), one or more systems may update the graph to include a third node associated with the forklift, as well as a second edge between the first and third nodes indicating the driving interaction. This process may then continue to repeat as one or more systems continue to generate additional summaries associated with one or more videos.

[0032] As described herein, one or more systems may then use graphs to perform information retrieval, such as when receiving a query from a user associated with one or more videos. For example, one or more systems may receive a query to retrieve information related to an event that has occurred and is depicted by one or more videos. In some examples, one or more systems may then process the query (e.g., by using one or more Automatic Speech Recognition (ASR) models, one or more Natural Language Understanding (NLU) models, one or more language models, and / or any other type of processing component) to generate text associated with the query. For example, the text may represent transcribed text of the user's speech associated with the query. Furthermore, in some examples, one or more systems may process the text (e.g., by using one or more language models (e.g., one or more text2cypher models)) to generate a summary associated with the query. For example, the summary may indicate one or more entities, one or more interactions, one or more timestamps, one or more actions to be performed, and / or any other information associated with the query.

[0033] Then, one or more systems may use the query summary to traverse the graph for searching and identify information relevant to the query. For example, if the query is requesting the name of a person performing a specific interaction, the retrieved information may at least indicate an identifier associated with that person. In some examples, such as to improve the search, one or more systems may use one or more terms from the query summary to filter at least a portion of the graph. For the first example, if the query summary indicates a time period, one or more systems may filter the graph to traverse the portion of the graph associated with that time period for searching (e.g., interactions occurring within a threshold time interval near that time period). For the second example, if the query summary indicates an entity's identifier, one or more systems may filter the graph to traverse the initial node associated with that entity and / or one or more additional nodes connected to that initial node for searching. While these are just a few examples of using restrictive terms to filter portions of a graph, in other examples, additional and / or alternative terms may be used to filter the graph during information retrieval.

[0034] In some examples, one or more systems may then use information retrieved from the graph to generate a response to the query. For example, one or more systems may process input data associated with the retrieved information (e.g., by using one or more language models and / or any other type of processing component) to generate a response. However, in other examples, one or more systems may process other data in generating the response, such as text associated with the query, prompts indicating instructions to generate a response, and / or one or more descriptions associated with one or more videos.

[0035] For example, one or more systems might process the text associated with a query (e.g., by using one or more encoders and / or any other type of processing component) to generate one or more embeddings corresponding to the query. One or more systems might then use the query embeddings to traverse one or more vector databases to perform a search, thereby identifying one or more descriptions associated with the query. In some examples, to improve this search (e.g., reduce search-related latency), one or more systems might filter the descriptions using information retrieved from a graph. For example, one or more systems might filter the descriptions using one or more timestamps from the retrieved information, such that the search returns descriptions of frames generated within a threshold time interval to the timestamps. In this example, one or more systems might further process the input data representing the retrieved one or more descriptions when generating a response.

[0036] In some examples, one or more systems may also provide additional information related to the query. For example, one or more systems may retrieve one or more portions of one or more videos (e.g., portions depicting the event indicated by the query) and responses associated with the query. In some examples, one or more systems may use information retrieved from a graph to identify one or more portions of one or more videos. For example, one or more systems may use timestamps associated with the retrieved information to identify one or more portions of one or more videos generated near the time of the event (e.g., within a threshold time interval). Thus, by performing one or more of the processes described herein, one or more systems may use graphs to improve the information retrieval process, such as by identifying relevant information from the graph, identifying relevant descriptions, identifying relevant portions of the video, and / or generating more relevant responses.

[0037] As described herein, these processes may be used for a variety of technologies. For the first example, if the machine (e.g., an autonomous vehicle and / or a semi-autonomous vehicle) includes cameras for navigation, one or more systems described herein may process the video to generate an interaction graph and / or vector database associated with the video. One or more systems may then use the interaction graph and / or vector database to perform information retrieval for events related to autonomous driving (e.g., events crucial for analyzing how the machine navigates). For the second example, if the warehouse includes cameras that generate video representing the interior of the warehouse, one or more systems described herein may again process the video to generate an interaction graph and / or vector database associated with the video. Furthermore, one or more systems may use the interaction graph and / or vector database to perform information retrieval for events related to the warehouse (e.g., specific interactions occurring within the warehouse).

[0038] In some examples, one or more models described herein may be packaged as microservices—such as inference microservices (e.g., NVIDIA NIM)—which may contain a container (e.g., an operating system (OS)-level virtualization package) that may contain an application programming interface (API) layer, a server layer, a runtime layer, and / or a model “engine.” For example, an inference microservice may include the container itself and one or more models (e.g., weights and biases). In some cases, such as when one or more models are small enough (e.g., have a sufficiently small number of parameters), one or more models may be contained within the container itself. In some embodiments, one or more models described herein may be deployed as inference microservices to accelerate model deployment on any cloud, data center, or edge computing system while ensuring data security. For example, an inference microservice may include one or more APIs, pre-configured containers for simplified deployment, an optimized inference engine (e.g., deployment execution software built using standardized AI models, such as NVIDIA's Triton Inference Server, and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations that provide low latency and high throughput for production applications, such as NVIDIA's TensorRT), and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring). One or more models described herein may be included as part of a microservice along with acceleration infrastructure that has the capability for deployment using a single command and / or orchestration and auto-scaling using a container orchestration system on the acceleration infrastructure (e.g., from a single device up to data center scale). Therefore, an inference microservice may include one or more models (e.g., models optimized for high-performance inference), inference runtime software for executing one or more models and providing output / response to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing health checks, identity verification, and other monitoring capabilities. In some embodiments, the inference microservice may include software for in-situ replacement and / or updating of one or more machine learning models. During replacement or updating, the software performing the replacement / update may maintain user configurations for the inference runtime software and enterprise management software.

[0039] The systems and methods described herein can be used by, but are not limited to, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones, and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, such as, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or participant simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, and / or any other suitable application.

[0040] The disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, marine systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems implementing large language models (LLMs), systems implementing one or more multimodal language models, systems using or deploying one or more inference microservices, systems combining one or more machine learning models deployed in services or microservices with OS-level virtualization packages (e.g., containers), systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing optical transmission simulations, systems for performing collaborative content creation for 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0041] refer to Figure 1A , Figure 1AExamples of a process 100 for generating an interactive graph associated with video, according to some embodiments of this disclosure, are shown. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groups of functions, etc.) may be used in addition to, or in place of, the arrangements and elements shown, and certain elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components, or in combination with other components, and may be implemented in any suitable combination and in any suitable location. The various functions performed by the entities described herein can be implemented by hardware, firmware, and / or software. For example, various functions may be implemented by a processor executing instructions stored in memory.

[0042] Process 100 may include generating video data 104 representing one or more videos using one or more image sensors 102 (e.g., one or more cameras). As described herein, one or more image sensors 102 may be associated with one or more objects and / or environments, such as being located on and / or inside a building (e.g., a warehouse), on a machine (e.g., a vehicle), etc. Furthermore, one or more videos may at least depict entities and / or interactions between entities. In some examples, entities may include, but are not limited to, people, vehicles, machines, animals, equipment, shelves, boxes, and / or any other type of object. Furthermore, interactions may include, but are not limited to: approaching an entity, walking away from an entity, pointing an entity, pushing an entity, placing an entity, lifting an entity, carrying an entity, driving an entity, causing a collision with an entity, and / or any other type of interaction that may occur between two or more entities. In some examples, video data 104 may represent additional information, such as timestamps indicating the time when frames of one or more videos were generated using one or more image sensors 102, and / or identifiers indicating which image sensor generated the corresponding video.

[0043] For example, Figure 2An example of video 202 representing interactions between entities according to some embodiments of the present disclosure is shown. As shown, video 202 includes frames 204(1)-(4) (also referred to in the singular as “frame 204” or in the plural as “multiple frames 204”), which depict entities 206(1)-(3) (also referred to in the singular as “entity 206” or in the plural as “multiple entities 206”). Furthermore, frames 204 depict different interactions that may occur between entities 206. For example, the first frame 204(1) indicates that the second entity 206(2) manages the third entity 206(3). Furthermore, the second frame 204(2) indicates that the first entity 206(1) approaches the second entity 206(2) and / or the second entity 206(2) still carries the third entity 206(3). Furthermore, the third frame 204(3) indicates that the first entity 206(1) provides instructions to the second entity 206(2) and / or the second entity 206(2) still carries the third entity 206(3). In addition, the fourth frame 204(4) indicates that the first entity 206(1) walks away from the second entity 206(2) and / or the second entity 206(2) places the third entity 206(3) on the ground.

[0044] review Figure 1A For example, process 100 may include one or more video processors 106 processing video data 104 to generate CV data 108 associated with one or more videos. As described herein, one or more video processors 106 may use and / or include one or more models (e.g., one or more CV models) and / or any other type of processing component to perform one or more processes described herein. Furthermore, in some examples, the information represented by the CV data 108 may include, but is not limited to, entity identifiers (e.g., names, usernames, generic identifiers, etc.), the entity's position within a frame, the boundary shape representing the portion of the frame depicting the entity (e.g., a bounding box), entity-related attributes (e.g., color, texture, pattern, etc.), the action the entity is performing, and / or any other information. In some examples, one or more video processors 106 may generate corresponding CV data 108 associated with each frame of one or more videos. Furthermore, or alternatively, in some examples, one or more video processors 106 may generate corresponding CV data 108 for multiple frames of one or more videos.

[0045] For example, Figure 3Examples of generating computer vision information associated with a third frame 204(3) of video 202 according to some embodiments of the present disclosure are shown. As shown, one or more video processors 106 may determine that the CV information includes at least a first boundary shape 302(1) associated with a first entity 206(1), a second boundary shape 302(2) associated with a second entity 206(2), and a third boundary shape 302(3) associated with a third entity 206(3). However, in other examples, one or more video processors 106 may determine additional CV information associated with the third frame 204(3), such as identifiers of entity 206 (e.g., name, username, generic identifier, etc.) and / or attributes associated with entity 206. Furthermore, in some examples, one or more video processors 106 may determine CV information associated with one or more of the other frames 204 of video 202.

[0046] review Figure 1A For example, process 100 may include one or more language models 110 processing video data 104 and / or CV data 108 to generate description data 112 representing descriptions associated with frames of one or more videos. As described herein, one or more language models 110 may include any type of language model, such as one or more visual language models, configured to perform at least a portion of the processing described herein. Furthermore, in some examples, the description associated with a frame may describe the identifier of the entity depicted by the frame, the location depicted by the frame, the time the frame was generated, one or more interactions between the entities depicted by the frame, attributes associated with the entity, and / or any other information associated with the frame. In some examples, one or more language models 110 may generate a corresponding description for each frame of one or more videos. Additionally, or alternatively, in some examples, one or more language models 110 may generate corresponding descriptions for multiple frames of one or more videos.

[0047] For example, Figure 4Examples of generating a description 402 associated with a third frame 204(3) of video 202 according to some embodiments of the present disclosure are shown. As shown, one or more language models 110 may process input data representing the third frame 204(3) and / or CV information associated with the third frame 204(3). Based at least on this processing, one or more language models 110 may generate data representing the description 402 of the third frame 204(3). As shown, the description 402 includes information associated with the third frame 204(3), such as the identity of entity 206 (e.g., person one and person two), the type of environment depicted (e.g., warehouse), the interaction occurring between entities 206(1)-(2) (e.g., first entity 206(1) is instructing second entity 206(2)), and an indication of the time of occurrence (e.g., 5:00). In some examples, one or more language models 110 may perform a similar process to generate one or more additional descriptions associated with one or more other frames 204 of video 202.

[0048] review Figure 1A For example, process 100 may include storing CV data 108 and / or description data 112 in one or more databases, such as one or more vector databases 114. As shown, in some examples, to store the data, process 100 may include one or more embedding components 116 processing the description represented by description data 112 (and / or, in some examples, CV information represented by CV data 108) to generate an embedding 118 associated with the description (and / or CV information). For example, one or more embedding components 116 may include one or more encoders (and / or any other type of processing component) configured to generate embedding 118. Process 100 may then include storing embedding 118 in one or more vector databases 114.

[0049] Process 100 may also include one or more language models 120 processing at least the description data 112 to generate extracted text data 122 representing a summary associated with the frame and / or description. As described herein, one or more language models 120 may include any type of language model, such as one or more text2cypher language models, configured to perform at least a portion of the process described herein. Furthermore, the summary may use one or more specific types of languages ​​associated with the search and / or retrieval information, such as the Cypher query language (and / or any other type of query language). For example, the summary associated with the description and / or frame may include specific information such as one or more entities depicted by the frame, one or more interactions represented by the frame, and / or the time step associated with the frame.

[0050] For example, Figure 5 An example process for generating a summary 502 associated with a third frame 204(3) of description 402 and / or video 202 according to some embodiments of the present disclosure is illustrated. As shown, one or more language models 120 may determine that the summary 502 includes at least an identifier associated with a first entity 206(1) (e.g., person one), an identifier associated with a second entity 206(2) (e.g., person two), an interaction occurring between the first entity 206(1) and the second entity 206(2) (e.g., an instruction), and the time when the third frame 204(3) was generated (e.g., 5:00). However, in other examples, the summary 502 may include additional information associated with the third frame 204(3), such as at least a portion of the CV information represented by the CV data associated with the third frame 204(3) (e.g., attributes associated with entity 206). Additionally, in some examples, one or more language models 120 may determine a summary associated with one or more of the other frames 204 of video 202.

[0051] review Figure 1A For example, process 100 uses one or more graph models 124 to generate and / or update a graph associated with video data 104, which may be stored in one or more graph databases 126. As described herein, one or more graph models 124 may include any type of processing, such as one or more language models configured to obtain summaries represented by extracted text data 122 and generate the graph. In some examples, the graph may at least represent interactions between entities represented by one or more videos. For example, the graph may include nodes associated with entities, such as nodes representing entity identifiers and / or attributes associated with entities (e.g., CV information). Furthermore, the graph may include edges associated with interactions occurring between entities and / or timestamps indicating when the interactions occurred. In some examples, the graph may initially be generated using one or more initial summaries associated with one or more initial frames (e.g., with one or more initial nodes and / or edges). In these examples, the graph may subsequently be updated as a new summary associated with a new frame (e.g., with one or more new nodes and / or edges) is generated and / or received. Furthermore, in some examples, at least a portion of the graph may be generated using CV data 108.

[0052] For example, Figure 6An example process for generating an interactive graph 602 associated with video 202 according to some embodiments of the present disclosure is shown. In some examples, graph 602 may be generated using a summary associated with a first frame 204(1) to include at least a first node 604(1) associated with a first entity 206(1) and a second node 604(2) associated with a second entity 206(2). Next, graph 602 may be updated using a summary associated with the second frame 204(2) to include a first edge 606(1) associated with a first interaction between the first entity 206(1) and the second entity 206(1), wherein the first interaction includes the first entity 206(1) approaching the second entity 206(2).

[0053] Next, Figure 602 may be updated using the summary 503 associated with the third frame 204(3) to include a second edge 606(2) associated with a second interaction between the first entity 206(1) and the second entity 206(2), wherein the second interaction includes the first entity 206(1) indicating the second entity 206(2). Finally, Figure 602 may be updated using the summary associated with the fourth frame 204(4) to include a third node 604(3) associated with the third entity 206(3) and a third edge 606(3) associated with a third interaction between the second entity 206(2) and the third entity 206(3), wherein the third interaction includes the second entity 206(2) placing the third entity 206(3).

[0054] As shown, these processes may then be repeated as additional frames associated with video 202 are processed using one or more of the processes described herein. For example, the additional frames may depict the interaction of a first entity 206(1) with a fourth entity (e.g., a machine), such as by driving the fourth entity. Thus, Figure 602 may be updated to include a fourth node 604(4) associated with the fourth entity and a fourth edge 606(4) associated with the interaction between the first entity 206(1) and the fourth entity, where the interaction includes the first entity 206(1) driving the fourth entity. Next, the additional frames may depict the interaction of the fourth entity with a fifth entity (e.g., a shelf), such as by colliding with the fifth entity. Thus, Figure 602 may be updated to include a fifth node 604(5) associated with the fifth entity and a fifth edge 606(5) associated with the interaction between the fourth entity and the fifth entity, where the interaction includes the fourth entity colliding with the fifth entity.

[0055] In some examples, nodes 604(1)-(5) (also referred to as “node 604” in the singular or “multiple nodes 604” in the plural) and / or edges 606(1)-(5) (also referred to as “edge 606” in the singular or “multiple edges 606” in the plural) may include additional information associated with video 202. For example, node 604 may include information describing entity 206, such as attributes associated with entity 206. Additionally, edge 606 may include information describing interactions, such as timestamps indicating when an interaction occurs and / or identifiers of frames 204 associated with the interaction. Thus, Figure 602 may represent one or more (e.g., all) interactions occurring between one or more (e.g., all) entities 206 as depicted in video 202.

[0056] In some examples, such as via Figure 6 As further illustrated in the examples, Figure 602 may contain a user-viewable format, such as a document, spreadsheet, memo, and / or the like. For example, a user might view Figure 602 to identify entity 206 associated with video 202, interactions occurring between entities 206, and / or additional information associated with the interactions (e.g., the time when the interaction occurred). Furthermore, in some examples, as described in more detail herein, Figure 602 may contain a format that can be searched by one or more systems to identify information associated with video 202.

[0057] review Figure 1A The example, although Figure 1A The examples illustrate one or more graph databases 126 as separate from one or more vector databases 114, but in other examples, at least a portion of the data from the vector databases 114 may be stored in the graph databases 126, at least a portion of the data from one or more graph databases 126 may be stored in one or more vector databases 114, and / or one or more graph databases 126 and one or more vector databases 114 may be combined into one or more databases. Process 100 may continue to repeat in order to add more data to one or more vector databases 114, update graphs stored in one or more graph databases 126, and / or generate new graphs associated with new videos. Furthermore, as described herein, after the graphs are generated, these graphs can be used to perform various tasks, such as information retrieval. For example, Figure 1B Examples of a process 128 for performing information retrieval using an interactive graph, according to certain embodiments of the present disclosure, are shown.

[0058] Process 128 may include one or more language models 130 processing input data 132 associated with at least one query. As described herein, input data 132 may include, but is not limited to: audio data representing user speech associated with the query, text data representing text describing the query, selection data representing selection of interactive elements (e.g., buttons, etc.) describing the query, and / or any other type of input data. Therefore, in some examples, input data 132 may be preprocessed before being received by one or more language models 130, and / or processed using one or more language models 130. For example, if input data 132 represents audio data, the audio data may be processed using one or more ASR models and / or one or more NLU models to generate transcribed text associated with user speech, where one or more ASR models and / or one or more NLU models may be represented by one or more language models 130.

[0059] Then, process 100 may include generating and / or outputting query data 134 representing the query, based at least on the processed input data 132. For example, in some examples, query data 134 may represent text corresponding to the query, such as one or more letters, numbers, words, sentences, symbols, and / or similar elements associated with the query. In some examples, one or more language models 130 may generate enhanced queries for performing information retrieval. For example, query data 134 may not only represent the query but may also represent additional information for performing information retrieval, such as information associated with one or more nodes and / or edges of a graph.

[0060] For example, Figure 7 An example process for generating and performing queries associated with information retrieval according to some embodiments of this disclosure is illustrated. As shown, user 702 may provide input in the form of user voice 704, where this input corresponds to a question about video 202. For example, this question is associated with a request for information about the person who gives the instruction to put down the box around 5:00. Therefore, one or more language models 130 may process audio data representing user voice 704, and at least based on this processing, generate data representing a query 706 corresponding to user voice 704. Figure 7 In the example, query 706 might include the transcribed text of user speech 704. However, in other examples, query 706 could include any other type of representation of user speech 704.

[0061] review Figure 1BFor example, process 128 may include one or more language models 136 processing at least the query data 134 to generate extracted text data 138 representing a summary associated with the query. As described herein, one or more language models 136 may include any type of language model (e.g., one or more text2cypher language models) configured to perform at least a portion of the process described herein. For example, in some examples, one or more language models 136 may include Figure 1A One or more language models 120 in the example. Furthermore, the summary may use one or more specific types of languages ​​associated with the search and / or retrieval of information (e.g., Cypher query language (and / or any other type of query language)). For example, a summary associated with a query may contain specific information from the query, such as one or more entities, one or more interactions, one or more time steps, and / or one or more actions that should be performed for information retrieval.

[0062] For example, Figure 8 An example process for generating a summary 802 associated with query 706 according to some embodiments of the present disclosure is illustrated, wherein the summary 802 is used to perform information retrieval. As shown, the summary 802 may at least include information associated with entity 206 in query 706 (including people and boxes), interactions from query 706 (including providing instructions and placing boxes), and the time from query 706 (including 5:00). Furthermore, the summary 802 includes actions to be performed for the information retrieval, including retrieving a person's name. However, in other examples, the summary 802 may include additional and / or alternative information for performing the information retrieval. For example, the summary 802 may include any information that facilitates traversing a graph to perform a search in order to perform the information retrieval.

[0063] review Figure 1BFor example, process 100 might include using a summary to traverse the graph for searching and retrieving information 140 associated with a query. As described herein, any type of search might be performed to identify information 140 from the graph, such as matching text (e.g., words) in the summary with text (e.g., words) in the graph. For example, if the summary contains identifiers for one or more entities and / or interactions, the graph might be searched to identify one or more nodes and / or edges associated with those one or more identifiers. These matches can then be used to identify information 140, for example, by retrieving information 140 from one or more identified nodes and / or one or more edges, and / or from one or more connected nodes and / or edges. For example, if an identifier in the summary matches a node, information 140 associated with that node, information 140 associated with one or more connected edges, and / or information 140 associated with one or more connected nodes might be retrieved.

[0064] For example, Figure 9 An example process for retrieving information from FIG. 602 using a summary 802 according to some embodiments of the present disclosure is illustrated. As shown, a third entity 206(3) in summary 802 (which in these examples contains “box”) may initially match a third node 604(3) associated with the third entity 206(3), as indicated by 902. Next, an interaction in summary 802 (which contains “placement”) may match a third edge 606(3) of placement associated with the third entity 206(3), as indicated by 904. Thus, it may be determined that the second entity 206(2) associated with the second node 604(2) is the person who placed the box. However, query 706 requests information indicating another person who placed the box.

[0065] Therefore, another instruction in summary 802 (which contains the word "instruction") may match the second side 606(2) associated with instructing the second entity 206(2) to place the third entity 206(3), as indicated by 906. Ultimately, it may be determined that the first entity 206(1) associated with the first node 604(1) instructs the second entity 206(2) to place the third entity 206(3). Therefore, information 908 may be retrieved from figure 602 associated with the first entity 206(1). For example, an identifier associated with the first entity 206(1) may be retrieved from figure 602, in which, in these examples, figure 602 again contains person 1. However, in other examples, additional information may be retrieved from figure 602, such as one or more attributes associated with the first entity 206(1) and / or the identifier of the second entity 206(2), which may also be helpful in generating a response to query 706.

[0066] In some examples, as described herein, one or more filtering techniques may be used to improve the performance of retrieving information 908 from Figure 602. For the first example, since the summary 802 contains restrictive terms associated with a time period (i.e., 5:00), it may be possible to search only the portion of Figure 602 within a threshold time interval (e.g., 20 minutes, 1 hour, 2 hours, etc.) around that time period. For example, if Figure 602 indicates that the proximity interaction associated with the first side 606(1) occurs at 4:55, the instruction interaction associated with the second side 606(2) occurs at 4:58, the placement interaction associated with the third side 606(3) occurs at 5:00, the driving interaction associated with the fourth side 606(4) occurs up to 8:00, and the collision interaction associated with the fifth side 606(5) occurs at 8:05, then when retrieving information 908 from query 706, the portion of Figure 602 containing nodes 604(4)-(5) and edges 606(4)-(5) may not be searched, because edges 606(4)-(5) are associated with interactions occurring outside the threshold time interval up to 5:00.

[0067] For the second example, since summary 802 includes restrictive terms associated with entity types (which include people and boxes), it is possible to search only a portion of Figure 602 associated with entities of those types. For example, if Figure 602 indicates that entities 206(1)-(3) associated with nodes 604(1)-(3) include people and boxes, but entities 206(4)-(5) associated with nodes 604(4)-(5) include machines and shelves respectively, then when retrieving information 908 for query 706, it is possible to search only the portion of Figure 602 containing nodes 604(1)-(3). While these are just a few examples of using restrictive terms from summary 802 to filter Figure 602 when performing information retrieval, in other examples, additional and / or alternative restrictive terms may be used to filter Figure 602 during information retrieval.

[0068] review Figure 1BIn some examples, process 128 may include retrieving additional information relevant to the query. For example, process 128 may include one or more language models 130 further generating and / or outputting query data 142 representing the query. As described herein, in some examples, query data 142 may represent text corresponding to the query, such as one or more letters, numbers, words, sentences, symbols, and / or similar elements related to the query. In some examples, query data 142 used to retrieve additional information may be similar to query data 134 used to retrieve information 140. However, in other examples, query data 142 used to retrieve additional information may differ from query data 134 used to retrieve information 140. For example, as described herein, query data 134 may have been enhanced with additional information relevant to the query (e.g., information from a graph), where query data 142 has not been enhanced with the same additional information.

[0069] Then, process 128 may include using query data 142 to retrieve one or more descriptions stored in one or more vector databases 114. For example, as shown, process 128 may include using one or more embedding components 144 to process query data 142 to generate one or more embeddings 146 associated with the query. In some examples, one or more embedding components 144 may include one or more embedding components 116 for generating embeddings 118 associated with the descriptions. However, in other examples, one or more embedding components 144 may differ from one or more embedding components 116.

[0070] Then, process 128 may include using one or more embeddings 146 to perform a search across one or more vector databases 114 to identify additional information. As described herein, in some examples, any type of search may be used to identify additional information. For example, based on the search, one or more embeddings 148 similar to one or more embeddings 146 stored in one or more vector databases 114 may be identified and / or retrieved. In some examples, information 140 from the graph may be used to perform the search for additional information, such as by filtering the embedded 118 being searched. For the first example, if information 140 contains a time period (e.g., a timestamp) during which the interaction associated with the query occurred, this time period may be used to filter the embeddings 118 in order to search for a portion of the embeddings 118 associated with that time period (e.g., associated with frames generated within a threshold time period of that time period). For the second example, if information 140 indicates an identifier of an entity associated with the query, that identifier may be used to filter the embeddings 118 in order to search for a portion of the embeddings 118 associated with that identifier (e.g., containing text describing the identifier).

[0071] As described herein, at least a portion of one or more embeddings 148 may be associated with one or more descriptions associated with one or more videos. Additionally, or alternatively, in some examples, at least a portion of one or more embeddings 148 may be associated with additional information (e.g., CV information) related to one or more videos. In any example, process 128 may include providing one or more embeddings 148 as additional information 150, and / or decoding one or more embeddings 148 to generate text associated with the additional information 150.

[0072] Then, process 128 may include: one or more language models 130 processing input data associated with information 140 and / or additional information 150 to generate and / or output response data 152 representing a response to a query. As described herein, in some examples, response data 152 may include any type of data, such as text data representing text corresponding to the response, audio data representing speech corresponding to the response, image data representing graphics associated with the response, and / or any other type of data. In some examples, additional data may be input into one or more language models 130 to generate response data 152, such as input data 132, query data 134, query data 142, and / or prompt data representing prompts associated with generating the response.

[0073] For example, Figure 10 An example process is illustrated for generating a response 1002 associated with query 706 using retrieved information, according to some embodiments of this disclosure. As shown, input data for one or more language models 130 may at least represent information 908 retrieved from FIG. 602 and description 402 retrieved from one or more vector databases. Based at least on the processing of the input data, one or more language models 130 may generate response 1002. Figure 10 In the examples, response 1002 includes the identifier of the person who provided the instructions to place the box, and in these examples again, it includes person one.

[0074] review Figure 1B In some examples, process 128 may include retrieving additional data relevant to the query. For example, one or more portions of one or more videos may be retrieved and / or provided with the response to depict information relevant to the query. In these examples, information 140 retrieved from the graph may be used to identify one or more portions of one or more videos. For example, if information 140 indicates a time period associated with the query, that time period can be used to identify one or more portions of one or more videos generated within a threshold time interval to that time period.

[0075] Figure 11Examples of one or more systems 1102 according to some embodiments of the present disclosure are shown, which may be configured to perform one or more processes described herein. As shown, one or more systems 1102 may include one or more processors 1104 (which may include and / or be similar to one or more CPUs 1526 and / or one or more GPUs 1528), one or more communication interfaces 1106 (which may include and / or be similar to one or more communication interfaces 1510), and memory 1108 (which may include and / or be similar to memory 1524). However, in other examples, one or more systems 1102 may include additional and / or alternative components, such as one or more image sensors 102.

[0076] As shown, memory 1108 may store one or more video processors 106, one or more language models 110, one or more vector databases 114, one or more embedding components 116, one or more language models 120, one or more graph databases 126, one or more language models 130, one or more language models 136, and / or one or more embedding components 144. Furthermore, one or more processors 1104 may execute one or more video processors 106, one or more language models 110, one or more vector databases 114, one or more embedding components 116, one or more language models 120, one or more graph databases 126, one or more language models 130, one or more language models 136, and / or one or more embedding components 144 to perform one or more processes described herein, such as... Figure 1A In the example, process 100 and / or Figure 1B The example is process 128.

[0077] For example, one or more client devices 1110 may send input data 132 to one or more systems 1102. As described herein, input data 132 may represent one or more queries for information related to one or more videos. One or more systems 1102 may then execute one or more processes described herein to generate one or more responses associated with one or more queries. Furthermore, one or more systems 1102 may send response data 152 representing one or more responses to one or more client devices 1110.

[0078] Now for reference Figure 12-13Each block of methods 1200 and 1320 described herein contains a computational process that may be executed using any combination of hardware, firmware, and / or software. For example, various functions may be implemented by a processor executing instructions stored in memory. Methods 1200 and 1320 may also be embodied as computer-usable instructions stored on a computer storage medium. Methods 1200 and 1320 may be provided by a standalone application, service, or managed service (standalone or in combination with other managed services), or a plug-in to another product. Furthermore, methods 1200 and 1320 are combined with... Figure 1A-1B These methods 1200 and 1320 are described by way of example. However, these methods 1200 and 1320 may be performed additionally or alternatively by any system or combination of systems, including but not limited to those described herein.

[0079] Figure 12 A flowchart of a method 1200 for generating an interaction graph associated with one or more videos, according to some embodiments of the present disclosure, is shown. Method 1200 may include, at block B1202, processing video data representing one or more frames based at least on one or more language models to determine one or more descriptions associated with the one or more frames. For example, one or more language models 110 may process video data 104 to generate description data 112 representing one or more descriptions associated with the one or more frames. As described herein, in some examples, one or more language models 110 may process additional data to generate one or more descriptions, such as CV data 108 generated using one or more video processors 106.

[0080] Method 1200 at box B1204 may include: processing input data representing one or more descriptions based on at least one or more language models to determine one or more entities and one or more interactions associated with one or more frames. For example, one or more language models 120 may process description data 112 to generate extracted text data 122 representing one or more summaries associated with one or more descriptions. As described herein, one or more summaries may include at least one or more identifiers associated with one or more entities and one or more interactions occurring between one or more entities. Furthermore, one or more summaries may be associated with a specific type of language, such as the Cypher query language (and / or any other type of query language).

[0081] At box B1206, method 1200 may include: generating a graph containing one or more nodes associated with one or more entities and one or more edges associated with one or more interactions. For example, the graph may be generated to contain one or more nodes associated with one or more entities and one or more edges associated with one or more interactions. As described herein, in some examples, the graph may be generated to contain additional information associated with one or more videos. For the first example, the graph may be generated such that one or more nodes are further associated with one or more attributes associated with one or more entities. Furthermore, the graph may be generated such that one or more edges are further associated with one or more timestamps indicating the time when one or more interactions occurred.

[0082] At box B1208, method 1200 may include performing one or more operations using the graph. For example, the graph may be stored in one or more graph databases 126 and made available to one or more users to view, analyze for information retrieval, and / or to perform any other type of operation.

[0083] Figure 13 A flowchart illustrating a method 1320 for performing information retrieval using an interaction graph according to some embodiments of the present disclosure is shown. Method 1320 may include, at block B1322, obtaining a graph containing one or more nodes associated with one or more entities represented by one or more videos, and one or more edges associated with one or more interactions of one or more entities. For example, it is possible to obtain a graph containing one or more nodes associated with one or more entities and one or more edges associated with one or more interactions from one or more graph databases 126. As described herein, the graph may represent additional information associated with one or more videos, such as attributes associated with one or more entities and / or timestamps associated with one or more interactions.

[0084] Method 1320 at box B1324 may include receiving a query associated with one or more videos. For example, one or more language models 130 may receive input data 132 associated with the query. As described herein, input data 132 may include audio data, text data, selection data, and / or any other type of input data. One or more language models 130 may then process the input data 132 to generate query data 134 (and / or query data 142) representing the query.

[0085] Method 1320 at box B1326 may include: determining a response associated with the query based at least on at least a portion of a graph. For example, query data 134 may be used to retrieve information 140 from the graph. Furthermore, in some examples, query data 142 may also be used to retrieve additional information 150 from one or more vector databases 114. One or more language models 130 may then process input data representing information 140 and / or additional information 150. Based at least on this processing, one or more language models 130 may generate response data 152 representing the response to the query. In some examples, additional data for the query may be retrieved, such as one or more portions of one or more videos associated with the query.

[0086] Method 1320 at box B1328 may include: outputting a response associated with the query. For example, this may be done by outputting speech representing the response, displaying text representing the response, displaying content representing the response, and / or using any other techniques to output the response.

[0087] Example language model

[0088] In at least some embodiments, language models may be implemented, such as large language models (LLMs), visual language models (VLMs), multimodal language models (MMLMs), and / or other types of generative artificial intelligence (AI). These models may be able to understand, summarize, translate, and / or otherwise generate text (e.g., natural language text, code, etc.), images, videos, computer-aided design (CAD) assets, OMNIVERSE and / or METAVERSE file information (e.g., in USD format, such as OpenUSD), and / or the like, based on context provided in input prompts or queries. In embodiments, these language models can be considered “large” because they are trained on massive datasets and have architectures containing a large number of learnable network parameters (weights and biases), such as millions or billions of parameters. LLMs / VLMs / MMLMs / etc. can be used to summarize textual data, analyze and extract insights from data (e.g., text, images, videos, etc.), and generate new text / images / videos / etc. in user-specified styles, tones, and / or formats. In some embodiments, the LLM / VLM / MMLM / etc. disclosed herein may be specifically designed for text processing, while in others, a multimodal LLM may be implemented to accept, understand, and / or generate text and / or other types of content, such as images, audio, 2D and / or 3D data (e.g., in USD format) and / or video. For example, a visual language model (VLM), or more generally a multimodal language model (MMLM), may be implemented to accept images, video, audio, text, 3D designs (e.g., CAD), and / or other input data types, and / or generate or output images, video, audio, text, 3D designs, and / or other output data types.

[0089] Various types of LLM / VLM / MMLM / etc. architectures may be implemented in various embodiments. For example, different architectures may be implemented that use different techniques to understand and generate outputs such as text, audio, video, images, 2D and / or 3D design or asset data, etc. In some embodiments, LLM / VLM / MMLM / etc. architectures such as recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) may be used, while in other embodiments, transformer architectures—such as those relying on self-attention and / or cross-attention (e.g., between contextual data and textual data) mechanisms—may be used to understand and recognize relationships between words or tags and / or contextual data (e.g., other text, video, images, design data, USD, etc.). A generative processing pipeline containing one or more LLM / VLM / MMLM / etc. may also include one or more diffusion blocks (e.g., noise reduction blocks). The LLM / VLM / MMLM / etc. disclosed herein may include one or more blocks of encoders and / or decoders. For example, discriminative or encoder-only models, such as BERT (Bidirectional Encoder Representation from Transformer), might be implemented for tasks involving language understanding (e.g., classification, sentiment analysis, question answering, and named entity recognition). As another example, generative or decoder-only models, such as GPT (Generative Pre-trained Transformer), might be implemented for tasks involving language and content generation (e.g., text completion, story generation, and dialogue generation). LLM / VLM / MMLM models, such as T5 (Text-to-Text Transformer), which include encoder and decoder components, might be implemented to understand and generate content, for example, for translation and summarization. These examples are not intended to be limiting and may be implemented depending on the specific implementation and the specific tasks being performed using LLM / VLM / MMLM models, etc., to achieve any architecture type (including, but not limited to, the architecture types described herein).

[0090] In various embodiments, LLM / VLM / MMLM / etc. may be trained using unsupervised learning, where LLM / VLM / MMLM / etc. learns patterns from large amounts of unlabeled text / audio / video / image / design / USD / etc. data. Due to the extensive training, in these embodiments, the model may not require task-specific or domain-specific training. An LLM / VLM / MMLM / etc. that has already undergone extensive pre-training on large amounts of unlabeled data may be referred to as a base model and may excel at various tasks, such as question answering, summarizing, filling in missing information, translation, and image / video / design / USD / data generation. Some LLM / VLM / MMLM / etc. may be customized for specific use cases, using techniques such as cue tuning, fine-tuning, retrieval augmentation generation (RAG), adding adapters (e.g., custom neural networks and / or neural network layers that tune or adjust cues or labels to bias the language model towards a specific task or domain), and / or using other fine-tuning or customization techniques to optimize the model for a specific task and / or domain.

[0091] In some embodiments, the LLM / VLM / MMLM / etc. disclosed herein may be implemented using various model alignment techniques. For example, in some embodiments, protection mechanisms may be implemented to identify inappropriate or undesirable inputs (e.g., prompts) and / or outputs of the model. In doing so, the system may use protection mechanisms and / or other model alignment techniques to prevent the processing of specific unintended inputs using LLM / VLM / MMLM, etc., and / or to prevent the output or presentation of information generated using LLM / VLM / MMLM, etc. (e.g., displays, audio outputs, etc.). In some embodiments, one or more additional models (or layers thereof) may be implemented to identify problems with model inputs and / or outputs. For example, these “safety-protected” models may be trained to identify “safe,” or otherwise acceptable or desired, inputs and / or outputs, and / or identify “unsafe,” or otherwise undesirable, inputs and / or outputs, for a particular application / implementation. Therefore, the LLM / VLM / MMLM etc disclosed herein may be less likely to produce offensive, vulgar, inappropriate, insecure, out-of-scope and / or otherwise undesirable language / text / audio / video / design data / USD data for a particular application / implementation.

[0092] In some embodiments, an LLM / VLM / etc. may be configured to or able to access or use one or more plugins, application programming interfaces (APIs), databases, data repositories, stores, etc. For example, for certain tasks or operations where the model is not ideally suited, the model may have instructions (e.g., as a result of training, and / or based on instructions in a given prompt) to access one or more plugins (e.g., third-party plugins) to help process the current input. In such an example, if at least part of the prompt is related to a restaurant or weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs) to retrieve relevant information. As another example, if at least part of the response requires mathematical computation, the model may access one or more mathematical plugins or APIs to help solve one or more problems, and then may use the response from the plugins and / or APIs in the model's output. This process can be repeated (e.g., recursively) an arbitrary number of iterations, using any number of plugins and / or APIs, until a response to the input prompt is generated to address each problem / question / request / process / operation, etc. Therefore, one or more models may rely not only on the knowledge they gain from training on large datasets, but also on the expertise or optimized features of one or more external resources (such as APIs, plugins, and / or similar objects).

[0093] In some embodiments, multiple language models (e.g., LLM / VLM / MMLM / etc., multiple instances of the same language model, and / or multiple hints provided to the same language model or instances of the same language model) may be implemented, executed, or accessed (e.g., using one or more plugins, user interfaces, APIs, databases, data repositories, storage repositories, etc.) to provide output in response to the same query or in response to separate portions of the query. In at least one embodiment, the same input query and hints (e.g., a set of constraints, conditioners, etc.) may be provided for multiple language models (e.g., language models with different architectures, language models trained on different (e.g., updated) data corpora). In one or more embodiments, the language models may be different versions of the same underlying model. In one or more embodiments, at least one language model may be instantiated as multiple agents, for example, providing more than one hint to constrain, guide, or otherwise influence the style, content, or characteristics of the provided output. In one or more examples (non-limiting embodiments), the same language model may be required to provide output corresponding to different roles, perspectives, features, or different knowledge bases, as defined by the provided hints.

[0094] In any such embodiment, the outputs of two or more (e.g., each) language models, two or more versions of at least one language model, two or more instantiated proxies of at least one language model, and / or provided to two or more prompts for at least one language model may be further processed, such as aggregated, compared, or filtered, or used to determine (and provide) a consensus response. In one or more embodiments, the output of one language model (or version, instance, or proxy) may be provided as input to another language model for further processing and / or validation. In one or more embodiments, the language model may be required to generate or otherwise acquire output about the input source material, which is associated with the input source material. Such association may include, for example, generating a title or portion of text embedded (e.g., as metadata) in the input source text or image. In one or more embodiments, the output of the language model may be used to determine the validity of the input source material for further processing or inclusion in a dataset. For example, the language model may be used to evaluate the presence (or absence) of a target word in a portion of text or the presence (or absence) of an object in an image annotated to indicate such presence (or absence). Alternatively, the determination results from the language model can be used to determine whether the source material should be included in the screened dataset, for example, but not limited to this.

[0095] Figure 14A This is a block diagram of an exemplary generative language model system 1400 suitable for implementing at least some embodiments of this disclosure. Figure 14A In the example shown, the generative language model system 1400 includes a retrieval augmentation generation (RAG) component 1492, an input processor 1405, a tokenizer 1410, an embedding component 1420, a plugin / API 1495, and a generative language model (LM) 1430 (which may include LLM, VLM, multimodal LM, etc.).

[0096] At a higher level, the input processor 1405 may receive input 1401, which includes text and / or other types of input data (e.g., audio data, video data, image data, sensor data (e.g., LiDAR, RADAR, ultrasonic, etc.), 3D design data, CAD data, generic scene descriptor (USD) data (e.g., OpenUSD, etc.), depending on the architecture of the generative LM1430 (e.g., LLM / VLM / MMLM, etc.). In some embodiments, input 1401 includes plain text in the form of one or more sentences, paragraphs, and / or documents. Additionally or alternatively, input 1401 may include numerical sequences, pre-computed embeddings (e.g., word or sentence embeddings), and / or structured data (e.g., tabular format, JSON, or XML). In some implementations where the generative LM1430 can handle multimodal input, input 1401 may combine text (or possibly omit text) with image data, audio data, video data, design data, USD data, and / or other types of input data (e.g., but not limited to the data described herein). Taking raw input text as an example, input processor 1405 may prepare the raw input text in various ways. For example, input processor 1405 may perform various types of text filtering to remove noise (e.g., special characters, punctuation marks, HTML tags, stop words, portions of one or more images, portions of audio, etc.) from the relevant text content. In examples involving stop words (common words that often carry little semantic meaning), input processor 1405 may remove stop words to reduce noise and allow the generative LM 1430 to focus on more meaningful content. Input processor 1405 may apply text normalization, for example, by converting all characters to lowercase, removing accents, and / or handling special cases such as abbreviations or acronyms to ensure consistency. These are just a few examples, and other types of input processing may be applied.

[0097] In some embodiments, RAG component 1492 (which may contain one or more RAG models and / or may be performed using the generative LM1430 itself) may be used to retrieve additional information to be used as part of input 1401 or a prompt. RAGs can be used to enhance inputs to LLM / VLM / MMLM, etc., by leveraging external knowledge to make the answer to a specific question, query, or request more relevant, for example, where specific knowledge is required. RAG component 1492 may obtain this additional information (e.g., basic information, such as basic text / images / videos / audio / USD / CAD / etc.) from one or more external sources and then feed it along with the prompt to LLM / VLM / MMLM / etc. to improve the accuracy of the model's response or output.

[0098] For example, in some embodiments, in addition to the data retrieved using RAG component 1492, input 1401 may also be generated using queries or inputs to the model (e.g., questions, requests, etc.). In some embodiments, input processor 1405 may analyze input 1401 and communicate with RAG component 1492 (or in embodiments, RAG component 1492 may be part of input processor 1405) to identify relevant text and / or other data to provide as additional context or information sources to generative LM 1430, typically identifying responses, answers, or outputs 1490 from the additional context or information sources. For example, if the input indicates that the user is interested in the expected tire pressure of a particular brand and model of vehicle, RAG component 1492 may, for example, perform a vector search in the embedding space using the RAG model to retrieve tire pressure information or its corresponding text from a digital (embedded) version of the owner's manual for that particular vehicle brand and model. Similarly, when a user revisits a chatbot related to a specific product offering or service, the RAG component 1492 may retrieve previously stored conversation history or at least its summary, and incorporate the previous conversation history along with the current inquiry / request as part of the input 1401 of the generative LM 1430.

[0099] RAG component 1492 may use various RAG techniques. For example, it may use naive RAG, in which documents are indexed, chunked, and applied to an embedding model to generate embeddings corresponding to those chunks. User queries may also be applied to the embedding model of RAG component 1492 and / or another embedding model, and the embeddings of the chunks may be compared with the embeddings of the query to identify the embeddings most similar to / most relevant to the query, which may be provided to generative LM1430 to generate output.

[0100] In some embodiments, more advanced RAG techniques may be used. For example, the blocks may undergo pre-retrieval processing (e.g., routing, rewriting, metadata analysis, expansion, etc.) before being passed to the embedding model. Furthermore, post-retrieval processes (e.g., reordering, on-the-fly compression, etc.) may be performed on the output of the embedding model before the final embedding is generated, and then the final embedding is used for comparison with the input query.

[0101] As another example, modular RAG techniques may be used, such as those similar to Naive RAG and / or Advanced RAG, but also include features such as hybrid search, recursive retrieval and query engines, StepBack methods, subqueries and hypothetical document embedding.

[0102] As another example, Graph RAG might use a knowledge graph as a source of contextual or factual information. Graph RAG might utilize a graph database as a source of context-sensitive information sent to LLM / VLM / MMLM / etc. Instead of (or in addition to) providing the model with data chunks extracted from large documents (which could lead to a lack of context, factual accuracy, linguistic accuracy, etc.), Graph RAG might also provide structured entity information to LLM / VLM / MMLM / etc. by combining structured entity text descriptions with their numerous attributes and relationships, allowing the model to achieve deeper insights. In implementing Graph RAG, the systems and methods described herein use a graph as a content repository, extract relevant document chunks, and ask LLM / VLM / MMLM / etc. to use them to answer questions. In such embodiments, the knowledge graph might contain relevant textual content and metadata about the knowledge graph, as well as integration with a vector database. In some embodiments, Graph RAG might use a graph as a subject matter expert, where descriptions of concepts and entities relevant to the query / hint might be extracted and passed to the model as semantic context. These descriptions might include relationships between concepts. In other examples, the graph can be used as a database, where a portion of a query / hint may be mapped to a graph query, that graph query may be executed, and results may be aggregated using LLM / VLM / MMLM, etc. In such examples, the graph may store relevant factual information and may use queries (natural language queries) to graph query tools (NL-to-Graph-query tools) and entity links. In some embodiments, the graph RAG (e.g., using a graph database) may be combined with standard (e.g., vector database) RAGs and / or other RAG types to benefit from a variety of approaches.

[0103] In any embodiment, the RAG component 1492 may implement plugins, APIs, user interfaces, and / or other functionalities to perform RAG. For example, LLM / VLM / MMLM may use graph RAG plugins to run queries on the knowledge graph to extract relevant information to feed into the model, and may use standard or vector RAG plugins to run queries on a vector database. For example, the graph database may interact with the plugin's REST interface, thereby decoupling the graph database from the vector database and / or the embedded model.

[0104] The tokenizer 1410 can segment (e.g., processed) text data into smaller units (tags) for subsequent analysis and processing. Depending on the implementation, the tags can represent individual words, sub-words, characters, parts of audio / video / images, etc. Word-based tokenization divides the text into individual words, treating each word as a separate tag. Sub-word tokenization breaks words down into smaller meaningful units (e.g., prefixes, suffixes, stems), enabling the generative LM 1430 to understand morphological changes and process words outside the vocabulary more effectively. Character-based tokenization represents each character as a separate tag, enabling the generative LM 1430 to process text at a fine-grained level. The choice of tokenization strategy can depend on factors such as the language being processed, the task at hand, and / or the characteristics of the training dataset. Therefore, the tokenizer 1410 can transform (e.g., processed) text into a structured format according to the tokenization scheme implemented in a particular embodiment.

[0105] Embedding component 1420 can use any known embedding technique to transform discrete tokens into semantically meaningful (e.g., dense, continuous vector) representations. For example, embedding component 1420 can use pre-trained word embeddings (e.g., Word2Vec, GloVe, or FastText), one-hot encoding, Term Frequency-Inverse Document Frequency (TF-IDF) encoding, one or more embedding layers of a neural network, and / or others.

[0106] In some implementations where input 1401 includes image data / video data, etc., input processor 1401 may resize the data to a standard size compatible with the format of the corresponding input channel and / or normalize pixel values ​​to a common range (e.g., 0 to 1) to ensure consistent representation, and embedding component 1420 may encode the image data using any known technique (e.g., using one or more convolutional neural networks (CNNs) to extract visual features). In some implementations where input 1401 includes audio data, input processor 1401 may resample the audio file to a consistent sampling rate for uniform processing, and embedding component 1420 may use any known technique to extract and encode audio features, such as in the form of a spectrogram (e.g., a Mel spectrogram). In some implementations where input 1401 includes video data, input processor 1401 may extract frames or apply resizing to extracted frames, and embedding component 1420 may extract features such as optical flow embedding or video embedding and / or encode temporal information or frame sequences. In some implementations where input 1401 includes multimodal data, the embedded component 1420 may use techniques such as early fusion (concatenation), late fusion (sequential processing), and attention-based fusion (e.g., self-attention mechanism, cross-attention mechanism) to fuse representations of different types of data (e.g., text, images, audio, data, video, design, etc.).

[0107] Other components of the generative LM 1430 and / or generative LM system 1400 may use different types of neural network architectures depending on the implementation scheme. For example, a transducer-based architecture (e.g., the architecture used in models such as GPT) may be implemented, and it may include a self-attention mechanism that weights the importance of different words or tokens in the input sequence and / or a feedforward network that processes the output of the self-attention layer, applying a nonlinear transformation to the input representation and extracting higher-level features. Some non-limiting example architectures include transducers (e.g., encoder-decoder, decoder-only, multimodal), RNNs, LSTMs, fusion models, diffusion models, cross-modal embedding models that learn a joint embedding space, graph neural networks (GNNs), hybrid architectures that combine different types of adversarial networks (such as generative adversarial networks or GANs or adversarial autoencoders (AAEs) for joint distribution learning), etc. Therefore, depending on the implementation scheme and architecture, the embedded component 1420 can apply the encoded representation of the input 1401 to the generative LM 1430, and the generative LM 1430 can process the encoded representation of the input 1401 to generate an output 1490, which may include response text and / or other types of data.

[0108] As described herein, in some embodiments, the generative LM 1430 may be configured to access or use (or be able to access or use) plugins / APIs 1495 (which may include one or more plugins, application programming interfaces (APIs), databases, data stores, repositories, etc.). For example, for certain tasks or operations where the generative LM 1430 is not ideally suited, the model may have instructions (e.g., as a result of training, and / or based on instructions in a given prompt, such as instructions retrieved using RAG component 1492) to access one or more plugins / APIs 1495 (e.g., third-party plugins) to help process the current input. In such an example, when at least part of the prompt is related to a restaurant or weather, the model may access one or more restaurant or weather plugins (e.g., via one or more APIs), sending at least part of the prompt related to a particular plugin / API 1495 to the plugin / API 1495, which may process the information and return an answer to the generative LM 1430, which may then use the response to generate output 1490. This process can be repeated (e.g., recursively) an arbitrary number of iterations and repeated using any number of plugins / APIs 1495 until an output 1490 that resolves each query / question / request / process / action / etc. from input 1401 is generated. Therefore, the model can rely not only on its own knowledge acquired from training on a large dataset and / or from data retrieved using the RAG component 1492, but also on the expertise or optimized properties of one or more external resources (e.g., plugins / APIs 1495).

[0109] Figure 14B This is a block diagram of an example implementation scheme, where the generative LM 1430 includes a converter encoder-decoder. For example, suppose the input text (e.g., “Who discovered gravity”) is tokenized (e.g., by...) Figure 14A The tokenizer 1410) is used for tokens such as words, and each token is encoded (e.g., by...). Figure 14A The embedding component 1420 is a corresponding embedding (e.g., of size 512). Since these token embeddings do not typically represent the position of the tokens in the input sequence, positional encoding can be added to each token embedding using any known technique to encode the order relation and context of the tokens in the input sequence. Thus, (e.g., the resulting) embeddings can be applied to one or more encoders 1435 of the generative LM1430.

[0110] In the example implementation, encoder 1435 forms an encoder stack, where each encoder includes a self-attention layer and a feedforward network. In the example converter architecture, each token (e.g., a word) flows through a separate path. Therefore, each encoder can accept a sequence of vectors, pass each vector through the self-attention layer, then through the feedforward network, and then up to the next encoder in the stack. Any known self-attention technique can be used. For example, to compute a self-attention score for each token (word), a query vector, a key vector, and a value vector can be created for each token. The self-attention score for a token pair can be computed by taking the dot product of the query vector and the corresponding key vector, normalizing the resulting score, multiplying by the corresponding value vector, and summing the weighted value vectors. The encoder can apply multi-head attention, where the attention mechanism is applied multiple times in parallel with different learned weight matrices. Any number of encoders can be cascaded to generate context vectors that encode the input. Attention projection layer 1440 can transform the context vectors into attention vectors (keys and values) for decoder 1445.

[0111] In the example implementation, decoder 1445 forms a decoder stack, where each decoder includes a self-attention layer, an encoder-decoder self-attention layer that uses attention vectors (keys and values) from the encoder to focus on relevant parts of the input sequence, and a feedforward network. Similar to encoder 1435, in the example converter architecture, each token (e.g., a word) flows through a separate path in decoder 1445. During the first pass, decoder 1445, classifier 1450, and generation mechanism 1455 can generate a first token, and generation mechanism 1455 can apply the generated token as input during a second pass. This process can be repeated cyclically, generating tokens (e.g., words) and adding them to the output of the previous pass, and in subsequent passes applying token embeddings of positionally encoded composite sequences as input to decoder 1445, generating one token at a time (called autoregression) until a symbol or token indicating the end of the response is predicted. In each decoder, the self-attention layer is typically restricted to focusing only on earlier positions in the output sequence by applying a masking technique (e.g., setting future positions to negative infinity) before the softmax operation. In the example implementation, the encoder-decoder attention layer operates similarly to the (e.g., multi-head) self-attention operation in encoder 1435, except that it creates its query from the layer below it and obtains keys and values ​​(e.g., matrices) from the output of encoder 1435.

[0112] Therefore, decoder 1445 can output some decoded (e.g., vector) representation of the input applied during a particular pass. Classifier 1450 can include a multi-class classifier comprising one or more neural network layers and a softmax operation that converts logit probabilities into probabilities, the neural network layers projecting the decoded (e.g., vector) representation onto corresponding dimensions (e.g., one dimension for each supported word or token in the output vocabulary). Thus, generation mechanism 1455 can select or sample words or tokens based on corresponding predicted probabilities (e.g., selecting the word with the highest predicted probability) and append it to the output of the previous pass, thereby generating each word or token sequentially. Generation mechanism 1455 can repeat this process, triggering successive decoder inputs and corresponding predictions until a symbol or token representing the end of the response is selected or sampled, at which point generation mechanism 1455 can output the generated response.

[0113] Figure 14C This is a block diagram of an example implementation where the generative LM 1430 includes a decoder-only converter architecture. For example, Figure 14C The decoder 1460 can be used with Figure 14B The decoder 1445 operates similarly, except... Figure 14C Each decoder 1460 omits the encoder-decoder self-attention layer (because there is no encoder in this implementation). Therefore, decoders 1460 can form a decoder stack, where each decoder includes a self-attention layer and a feedforward network. Furthermore, instead of encoding the input sequence, a symbol or tag indicating the end of the input sequence (or the beginning of the output sequence) can be appended to the input sequence, and the resulting sequence (e.g., a corresponding embedding with positional encoding) can be applied to decoder 1460. Figure 14B Similar to decoder 1445, each tag (e.g., a word) can flow through a separate path in decoder 1460, and decoder 1460, classifier 1465, and generation mechanism 1470 can use autoregression to generate one tag at a time sequentially until a symbol or tag indicating the end of the response is predicted. Classifier 1465 and generation mechanism 1470 can be combined with... Figure 14B The classifier 1450 and generation mechanism 1455 operate similarly, wherein the generation mechanism 1470 selects or samples each consecutive output label based on the corresponding predicted probability and appends it to the output of the previous iteration, generating each label sequentially until a symbol or label representing the end of the response is selected or sampled. The architectures described herein, and others, are merely examples, and other suitable architectures may be implemented within the scope of this disclosure.

[0114] Example computing device

[0115] Figure 15This is a block diagram of an example computing device 1400 suitable for implementing some embodiments of the present disclosure. The computing device 1500 may include an interconnect system 1502 directly or indirectly coupled to the following devices: a memory 1504, one or more central processing units (CPUs) 1506, one or more graphics processing units (GPUs) 1508, a communication interface 1510, input / output (I / O) ports 1512, input / output components 1514, a power supply 1516, one or more presentation components 1518 (e.g., one or more displays), and one or more logic units 1520. In at least one embodiment, one or more computing devices 1500 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 1508 may include one or more vGPUs, one or more CPUs 1506 may include one or more vCPUs, and / or one or more logic units 1520 may include one or more virtual logic units. Thus, one or more computing devices 1500 may include discrete components (e.g., a full GPU dedicated to computing device 1500), virtual components (e.g., a portion of the GPU dedicated to computing device 1500), or a combination thereof.

[0116] although Figure 15 The various blocks are shown as connected via interconnect system 1502 using lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 1518 (such as a display device) may be considered I / O component 1514 (e.g., if the display is a touchscreen). As another example, CPU 1506 and / or GPU 1508 may include memory (e.g., memory 1504 may represent a storage device other than the memory of GPU 1508, CPU 1506, and / or other components). Therefore, Figure 15 The computing devices described are for illustrative purposes only. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of… Figure 15 Within the scope of computing devices.

[0117] Interconnect system 1502 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 1502 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 1506 may be directly connected to memory 1504. Further, CPU 1506 may be directly connected to GPU 1508. In cases where there is a direct or point-to-point connection between components, interconnect system 1502 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required to be included in computing device 1500.

[0118] The memory 1504 may include any computer-readable medium from a variety of computer-readable media. The computer-readable medium may be any available medium accessible by the computing device 1500. The computer-readable medium may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer-readable medium may include computer storage media and communication media.

[0119] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1504 may store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 1500. As used herein, computer storage media does not include the signal itself.

[0120] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information transmission medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0121] CPU 1506 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1500 to perform one or more of the methods and / or processes described herein. Each CPU 1506 may contain one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPU 1506 may contain any type of processor and may contain different types of processors depending on the type of computing device 1500 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 1500, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as math coprocessors), computing device 1500 may also include one or more CPUs 1506.

[0122] In addition to or in lieu of one or more CPUs 1506, one or more GPUs 1508 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1500 to perform one or more of the methods and / or processes described herein. One or more GPUs 1508 may be integrated GPUs (e.g., having one or more CPUs 1506) and / or one or more GPUs 1508 may be discrete GPUs. In embodiments, one or more GPUs 1508 may be coprocessors of one or more CPUs 1506. GPUs 1508 may be used by computing device 1500 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 1508 may be used for general-purpose computing on a GPU (GPGPU). GPUs 1508 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPUs 1508 may produce pixel data of an output image in response to rendering commands (e.g., rendering commands received from CPUs 1506 via a host interface). GPU 1508 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 1504. GPU 1508 may include two or more GPUs operating in parallel (e.g., via links). The links may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 1508 may produce pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for an analog image). Each GPU may include its own memory or may share memory with other GPUs.

[0123] In addition to or in lieu of CPU 1506 and / or GPU 1508, logic unit 1520 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1500 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 1506, one or more GPUs 1508, and / or one or more logic units 1520 may execute any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 1520 may be a portion of one or more CPUs 1506 and / or GPUs 1508 and / or integrated into one or more CPUs 1506 and / or GPUs 1508, and / or one or more logic units 1520 may be discrete components or otherwise external to CPUs 1506 and / or GPUs 1508. In an embodiment, one or more of the logic units 1520 may be coprocessors of one or more of the CPU 1506 and / or one or more of the GPU 1508.

[0124] Examples of logic unit 1520 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree lateral unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), programmable vision accelerator (PVA) (which may include one or more direct memory access (DMA) systems), one or more vision or vector processing units (VPU), and one or more pixel processing engines (PPE) (e.g.) Examples include a 2D array containing processing elements, each of which communicates with one or more other processing elements in the array in the north, south, east, and west directions; one or more decoupled accelerators or units (e.g., decoupled lookup table (DLUT) accelerators or decoupled units); a vision processing unit (VPU); an optical flow accelerator (OFA); a field-programmable gate array (FPGA); a neuromorphic chip; a quantum processing unit (QPU); an associative processing unit (APU); an arithmetic logic unit (ALU); an application-specific integrated circuit (ASIC); a floating-point unit (FPU); input / output (I / O) elements; and peripheral component interconnect (PCI) or fast peripheral component interconnect (PCIe) elements.

[0125] Communication interface 1510 may include one or more receivers, transmitters, and / or transceivers enabling computing device 1500 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). Communication interface 1510 may include components and functions for implementing communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wirelessband), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, logic unit 1520 and / or communication interface 1510 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 1502 to one or more GPUs 1508 (e.g., memory of one or more GPUs 1508).

[0126] I / O port 1512 enables computing device 1500 to be logically coupled to other devices including I / O component 1514, one or more presentation components 1518, and / or other components, some of which may be built into (e.g., integrated into) computing device 1500. Illustrative I / O component 1514 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, scanners, printers, wireless devices, etc. I / O component 1514 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 1500. Computing device 1500 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, the computing device 1500 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, the computing device 1500 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0127] The power supply 1516 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1516 may provide power to the computing device 1500 so that the components of the computing device 1500 can operate.

[0128] The presentation component 1518 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 1518 may receive data from other components (e.g., GPU 1508, CPU 1506, etc.) and output said data (e.g., as images, videos, sounds, etc.).

[0129] Example Data Center

[0130] Figure 16 An example data center 1600 that may be used in at least one embodiment of this disclosure is shown. The data center 1600 may include a data center infrastructure layer 1610, a framework layer 1620, a software layer 1630, and / or an application layer 1640.

[0131] like Figure 16 As shown, the data center infrastructure layer 1610 may include a resource coordinator 1612, grouped computing resources 1614, and node computing resources (“nodes CRs”) 1616(1)-1616(N), where “N” represents any complete positive integer. In at least one embodiment, nodes CRs 1616(1)-1616(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules and / or cooling modules, etc. In some embodiments, one or more node CRs from nodes CRs 1616(1)-1616(N) may correspond to servers having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CRs 1616(1)-16161(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more nodes CRs 1616(1)-1616(N) may correspond to virtual machines (VMs).

[0132] In at least one embodiment, the grouped computing resources 1614 may include individual groups of node CRs 1616 housed within one or more racks (not shown), or multiple racks housed within a data center at different geographical locations (also not shown). Individual groups of node CRs 1616 within the grouped computing resources 1614 may include grouped computing, networking, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several node CRs 1616, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0133] Resource coordinator 1612 may be configured or otherwise control one or more nodes CRs 1616(1)-1616(N) and / or grouped computing resources 1614. In at least one embodiment, resource coordinator 1612 may include a Software Design Infrastructure (“SDI”) management entity for data center 1600. Resource coordinator 1612 may include hardware, software, or some combination thereof.

[0134] In at least one embodiment, such as Figure 16 As shown, framework layer 1620 may include job scheduler 1628, configuration manager 1634, resource manager 1636, and / or distributed file system 1638. Framework layer 1620 may include a framework for software 1632 supporting software layer 1630 and / or one or more applications 1642 supporting application layer 1640. Software 1632 or application 1642 may respectively contain web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1620 may be, but is not limited to, free and open-source software web application frameworks (such as Apache Spark) that can utilize distributed file system 1638 for large-scale data processing (e.g., "big data"). TM(Hereinafter referred to as "Spark") is a type of resource. In at least one embodiment, the job scheduler 1628 may include Spark drivers to facilitate the scheduling of workloads supported by different layers of data center 1600. The configuration manager 1634 may be able to configure different layers, such as the software layer 1630 and the framework layer 1620 (which includes Spark and a distributed file system 1638 for supporting large-scale data processing). The resource manager 1636 may be able to manage compute resources mapped to or allocated to clusters of distributed file system 1638 and job scheduler 1628 to support the distributed file system 1638 and job scheduler 1628. In at least one embodiment, the clustered or grouped compute resources may include grouped compute resources 1614 in the data center infrastructure layer 1610. The resource manager 1636 may coordinate with the resource coordinator 1612 to manage these mapped or allocated compute resources.

[0135] In at least one embodiment, the software 1632 included in software layer 1630 may include software used in at least a portion of the nodes CRs 1616(1)-1616(N), the grouped computing resources 1614, and / or the distributed file system 1638 of framework layer 1620. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0136] In at least one embodiment, the application 1642 included in the application layer 1640 may include one or more types of applications used at least in part by nodes CRs 1616(1)-1616(N), grouped computing resources 1614, and / or the distributed file system 1638 of the framework layer 1620. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in combination with one or more embodiments.

[0137] In at least one embodiment, any of the configuration manager 1634, resource manager 1636, and resource coordinator 1612 can implement any number and type of self-modification actions based on any amount and type of data obtained in any technically feasible manner. Self-modification actions can free the data center operator of data center 1600 from making potentially poor configuration decisions and may prevent underutilization and / or poor performance of the data center.

[0138] According to one or more embodiments described herein, data center 1600 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information. For example, one or more machine learning models may be trained by using the software and / or computing resources described above with respect to data center 1600 to compute weight parameters according to a neural network architecture. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1600 by using weight parameters computed through one or more training techniques, such as, but not limited to, those described herein.

[0139] In at least one embodiment, the data center 1600 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform services that infer information, such as image recognition, speech recognition, or other artificial intelligence services.

[0140] Example network environment

[0141] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 15 This is implemented on one or more instances of computing devices 1500—for example, each device may include similar components, features, and / or functions of one or more computing devices 1500. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of a data center 1600, examples of which are described in this document. Figure 16 To describe in more detail.

[0142] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0143] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0144] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0145] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). The core server may assign at least a portion of the functionality to the edge server if the connection to the user (e.g., a client device) is relatively close to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0146] One or more client devices may include the information described in this article. Figure 15At least some of the components, features, and functions of one or more example computing devices 1500 described. By way of example and not limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0147] This disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, which are executed by a computer or other machine such as a personal digital assistant or other handheld device. Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a wide variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. This disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices linked via a communication network.

[0148] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0149] This document describes in detail the subject matter of this disclosure to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the discloser has envisioned that the claimed subject matter may be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be construed as suggesting any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

[0150] Example paragraph

[0151] A: A method comprising: processing video data representing one or more frames based on at least one or more language models to determine one or more descriptions associated with content depicted in the one or more frames; processing input data representing the one or more descriptions based on at least one or more of the language models to determine one or more entities associated with the one or more frames and one or more interactions associated with the one or more entities; generating a graph containing one or more nodes associated with the one or more entities and one or more edges associated with the one or more interactions; and performing one or more operations using the graph.

[0152] B: The method described in paragraph A further includes: determining one or more timestamps associated with the one or more frames; and associating the one or more edges of the graph with the one or more timestamps.

[0153] C: The method according to paragraph A or paragraph B further includes: processing the video data based on at least one or more computer vision models to determine one or more attributes associated with the one or more entities; and associating the one or more nodes of the graph with the one or more attributes.

[0154] D: The method according to any one of paragraphs A, C, and D, wherein: the one or more entities include at least a first entity and a second entity; the one or more interactions include at least an interaction between the first entity and the second entity; the one or more nodes of the graph include at least a first node associated with the first entity and a second node associated with the second entity; and the one or more edges of the graph include at least an edge between the first node and the second node associated with the interaction.

[0155] E: The method according to any one of paragraphs A and D further includes: processing second video data representing one or more second frames based at least on the one or more language models to determine one or more second descriptions associated with the one or more second frames; processing second input data representing the one or more second descriptions based at least on the one or more language models to determine one or more second entities associated with the one or more second frames and one or more second interactions associated with the one or more second entities; and updating the graph to include one or more second nodes associated with the one or more second entities and one or more second edges associated with the one or more second interactions.

[0156] F: The method according to any one of paragraphs AE further includes: generating one or more embeddings associated with the one or more descriptions; and storing the one or more embeddings associated with the graph in one or more databases.

[0157] G: The method according to any one of paragraphs AF, wherein the one or more language models comprise at least: one or more visual language models that process the video data to determine the one or more descriptions; and one or more large language models that process the input data to determine the one or more entities and the one or more interactions.

[0158] H: The method according to any one of paragraphs AG, wherein performing the one or more operations comprises: receiving a query corresponding to information associated with the one or more frames; determining a response associated with the query based at least on the graph; and causing an output associated with the response.

[0159] I: According to the method described in paragraph H, determining the response associated with the query includes: processing second input data representing the query based at least on one or more language models to determine text associated with the query; retrieving information associated with the query based at least on searching the graph using the text; and processing third input data representing the information based at least on one or more language models to calculate the response associated with the query.

[0160] J: A system comprising: one or more processors for: acquiring a graph containing: one or more nodes associated with one or more entities and one or more edges associated with one or more interactions between the one or more entities, the one or more interactions being depicted by one or more videos; receiving a query associated with the one or more videos; determining a response associated with the query based at least on at least a portion of the graph; and generating an output associated with the response.

[0161] K: According to the system described in paragraph J, the determination of the response associated with the query includes: processing first input data representing the query based on at least one or more language models to determine text associated with the query; determining information from the graph associated with the query based on at least a portion of the text; and processing second input data representing the information based on at least one or more language models to determine the response associated with the query.

[0162] L: According to the system described in paragraph K, determining the information from the graph associated with the query includes: determining that one or more first words from at least the portion of the text correspond to one or more second words associated with at least one of the one or more nodes or the one or more edges; and using the one or more nodes or the one or more edges to determine the information.

[0163] M: The system according to any one of paragraphs JL, wherein the one or more processors are further configured to: determine one or more restrictive terms associated with the query; and identify the portion of the graph based at least on the one or more restrictive terms, wherein the response is further determined based at least on the portion of the graph.

[0164] N: The system according to any one of paragraphs JM, wherein the one or more processors are further configured to: access one or more databases containing data representing one or more descriptions associated with the one or more videos; and determine at least one description associated with the query from the one or more descriptions, at least based on the query, wherein the response is further determined at least based on the description.

[0165] O: According to the system described in paragraph N, the determination of the response associated with the query includes: determining information associated with the query based at least on the graph; applying input data representing the information and the description to one or more language models; and processing the input data based at least on the one or more language models to generate output data representing the response associated with the query.

[0166] P: According to the system described in paragraph O, the one or more processors are further configured to: determine one or more timestamps associated with the information, wherein the description associated with the query is further determined based on at least one or more timestamps.

[0167] Q: In the system according to any one of paragraphs JP, wherein the one or more processors are further configured to: process video data representing the one or more videos based on at least one or more language models to determine one or more descriptions associated with the one or more videos; process input data representing the one or more descriptions based on at least one or more of the language models to determine the one or more entities and the one or more interactions associated with the one or more videos; and generate the graph containing the one or more nodes associated with the one or more entities and the one or more edges associated with the one or more interactions.

[0168] R: A system according to any one of paragraphs JQ, wherein the system comprises at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more analog operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation for 3D assets; a system for providing one or more cloud gaming applications; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language model LLMs; Systems for performing operations using one or more visual language models (VLMs); systems for performing operations using one or more multimodal language models; systems for performing one or more conversational AI operations; systems for generating synthetic data; systems for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; systems for implementing one or more multimodal language models; systems for using or deploying one or more inference microservices; systems that combine the deployment of one or more machine learning models in services or microservices with OS-level virtualization packages, such as containers; systems containing one or more virtual machines (VMs); systems implemented at least partially in a data center; or systems implemented at least partially using cloud computing resources.

[0169] S: One or more processors, including: processing circuitry for: processing text represented as a graph using one or more language models to generate a response to a query associated with one or more videos, wherein the graph includes one or more graph nodes associated with one or more entities represented by the one or more videos, and one or more graph edges associated with one or more interactions between the one or more entities; and generating output associated with the response.

[0170] T: According to paragraph S, one or more processors, wherein the one or more processors are included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation for 3D assets; a system for providing one or more cloud gaming applications; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language model LLMs. Systems that include: systems for performing operations using one or more visual language models (VLMs); systems for performing operations using one or more multimodal language models; systems for performing one or more conversational AI operations; systems for generating synthetic data; systems for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; systems that implement one or more multimodal language models; systems that use or deploy one or more inference microservices; systems that combine the deployment of one or more machine learning models in services or microservices with OS-level virtualization packages, such as containers; systems that contain one or more virtual machines (VMs); systems that are at least partially implemented in a data center; or systems that are at least partially implemented using cloud computing resources.

Claims

1. A method comprising: Process video data representing one or more frames based on at least one or more language models to determine one or more descriptions associated with the content depicted in the one or more frames; Process input data representing the one or more descriptions based at least on the one or more language models to determine one or more entities associated with the one or more frames and one or more interactions associated with the one or more entities; Generate a graph that includes one or more nodes associated with the one or more entities and one or more edges associated with the one or more interactions; as well as Use the graph to perform one or more operations.

2. The method according to claim 1, further comprising: Determine one or more timestamps associated with the one or more frames; as well as Associate one or more edges of the graph with one or more timestamps.

3. The method according to claim 1, further comprising: The video data is processed based on at least one or more computer vision models to determine one or more attributes associated with the one or more entities; as well as Associating one or more nodes of the graph with one or more attributes.

4. The method according to claim 1, wherein: The one or more entities include at least a first entity and a second entity; The one or more interactions include at least the interaction between the first entity and the second entity; The one or more nodes in the graph include at least a first node associated with the first entity and a second node associated with the second entity; as well as The one or more edges of the graph include at least the edge between the first node and the second node associated with the interaction.

5. The method according to claim 1, further comprising: Process second video data representing one or more second frames based at least on the one or more language models to determine one or more second descriptions associated with the one or more second frames; Process second input data representing the one or more second descriptions based at least on the one or more language models to determine one or more second entities associated with the one or more second frames and one or more second interactions associated with the one or more second entities; as well as The graph is updated to include one or more second nodes associated with the one or more second entities and one or more second edges associated with the one or more second interactions.

6. The method according to claim 1, further comprising: Generate one or more embeddings associated with the one or more descriptions; as well as The one or more embeddings associated with the graph are stored in one or more databases.

7. The method according to claim 1, wherein, The one or more language models include at least: One or more visual language models, which process the video data to determine the one or more descriptions; and One or more large language models process the input data to determine one or more entities and one or more interactions.

8. The method according to claim 1, wherein, Performing one or more of the operations includes: Receive queries corresponding to information associated with the one or more frames; Based at least on the graph, determine the response associated with the query; and Generate an output associated with the response.

9. The method according to claim 8, wherein, The determination of the response associated with the query includes: Process the second input data representing the query based on at least one or more language models to determine the text associated with the query; At least based on searching the graph using the text, information associated with the query is retrieved; and The third input data representing the information is processed based on at least one or more language models to calculate the response associated with the query.

10. A system comprising: One or more processors are used for: Obtain a graph comprising: one or more nodes associated with one or more entities and one or more edges associated with one or more interactions between the one or more entities, the one or more interactions being depicted by one or more videos; Receive queries associated with the one or more videos; The response associated with the query is determined based on at least a portion of the graph; as well as Generate an output associated with the response.

11. The system according to claim 10, wherein, The determination of the response associated with the query includes: Process the first input data representing the query based on at least one or more language models to determine the text associated with the query; Based on at least a portion of the text, determine information from the graph associated with the query; and The response associated with the query is determined by processing second input data representing the information based on at least one or more language models.

12. The system according to claim 11, wherein, The information derived from the graph and associated with the query includes: Determine that one or more first words from at least said portions of the text correspond to one or more second words associated with at least one of said one or more nodes or said one or more edges; and The information is determined using at least one of the one or more nodes or the one or more edges.

13. The system according to claim 10, wherein, The one or more processors are also used for: Identify one or more restrictive terms associated with the query; and The portion of the figure is identified based on at least one or more of the restrictive terms. The response is further determined based at least on the portion of the graph.

14. The system according to claim 10, wherein, The one or more processors are also used for: Access one or more databases, said one or more databases containing data representing one or more descriptions associated with said one or more videos; and Based at least on the query, determine at least one description associated with the query from the one or more descriptions. The response is further determined at least based on the description.

15. The system according to claim 14, wherein, The determination of the response associated with the query includes: Based at least on the graph, determine the information associated with the query; The input data representing the information and the description are applied to one or more language models; and The input data is processed based on at least one or more language models to generate output data representing the response associated with the query.

16. The system according to claim 15, wherein, The one or more processors are also used for: Identify one or more timestamps associated with the information. The description associated with the query is further determined based on at least one or more timestamps.

17. The system according to claim 10, wherein, The one or more processors are also used for: Process video data representing the one or more videos based on at least one or more language models to determine one or more descriptions associated with the one or more videos; Process input data representing the one or more descriptions based on at least one or more language models to determine the one or more entities and the one or more interactions associated with the one or more videos; as well as Generate the graph containing the one or more nodes associated with the one or more entities and the one or more edges associated with the one or more interactions.

18. The system according to claim 10, wherein, The system is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for performing collaborative content creation for 3D assets; A system that provides one or more cloud gaming applications; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models; A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system that implements one or more multimodal language models; A system that uses or deploys one or more inference microservices; A system that combines the deployment of one or more machine learning models within a service or microservice with OS-level virtualization packages, such as containers; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

19. One or more processors, comprising: Processing circuitry, used for: By processing text represented as a graph using one or more language models, a response to a query associated with one or more videos is generated, wherein the graph includes one or more graph nodes associated with one or more entities represented by the one or more videos, and one or more graph edges associated with one or more interactions between the one or more entities; and Generate an output associated with the response.

20. One or more processors according to claim 19, wherein, The one or more processors are included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system for performing one or more simulation operations; A system for performing one or more digital twin operations; A system for performing optical transmission simulation; A system for performing collaborative content creation for 3D assets; A system that provides one or more cloud gaming applications; A system for performing one or more deep learning operations; Systems implemented using edge devices; Systems implemented using robots; A system for performing one or more generative AI operations; A system for performing operations using one or more large language model LLMs; A system for performing operations using one or more visual language models (VLMs); A system for performing operations using one or more multimodal language models; A system for performing one or more conversational AI operations; A system for generating synthetic data; A system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; A system that implements one or more multimodal language models; A system that uses or deploys one or more inference microservices; A system that combines the deployment of one or more machine learning models within a service or microservice with OS-level virtualization packages, such as containers; A system containing one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

Images