Computer implementation methods, systems, and computer program products (converters for real-world video question and answer).

Abstract

To provide a computer-implemented method, system and computer program product for transformers for real world video question answering.SOLUTION: The method comprises: receiving a video including a plurality of video frames in a sequence; receiving a question about the video; for a video frame in the plurality of video frames, parsing the video frame into objects and relationships between the objects; and creating a subgraph of nodes representing the objects and edges representing the relationships, where the parsing and creating are performed for each video frame in the plurality of video frames, where a plurality of subgraphs is created; creating a hypergraph connecting the subgraphs by learning relationships between the nodes of the subgraphs, where a hyper-edge is created to represent a relationship between at least one node of one subgraph and at least one node of another subgraph in the plurality of subgraphs; and generating an answer to the question based on the hypergraph.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] This application generally relates to computers and computer applications, and more particularly to machine learning, online chat, human-robot communication, robotics understanding, and autonomous visual inference and visual question answering by machines.

Background Art

[0002] Humans may unconsciously make quick and feasible judgments about how to act according to the situation. Such an ability may be logic-centered, but in the real world, cognition is usually not separated from the context, so it is not isolated or detached from the surrounding situation. Such situation inference is not easy with current state-of-the-art methods. The ability of a machine to learn from and infer real-world situations can help the machine intelligently perform automated operations.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Provide a computer-implemented method, system, and computer program product for a converter for real-world video question answering.

Means for Solving the Problems

[0004] The summary of the present disclosure is provided to assist in understanding the computer system and method of visual question answering, and is not intended to limit the present disclosure or the invention. It should be understood that various aspects and features of the present disclosure can advantageously be used separately in some examples or in combination with other aspects and features of the present disclosure in other examples. Therefore, modifications and changes may be made to the computer system or its operating method or both to achieve different effects.

[0005] A computer implementation method may, in one embodiment, include receiving a video containing multiple video frames in a sequence. The method may also include receiving a question about the video. The method may also include analyzing and creating subgraphs of nodes and edges, where nodes represent objects and edges represent relationships, for each video frame in the multiple video frames. This analysis and creation can be performed for each video frame in the multiple video frames, and multiple subgraphs may be created. The method may also include creating a hypergraph connecting at least some of the subgraphs by learning the relationships between the nodes of the subgraphs, and hyperedges representing relationships between at least one node in one subgraph and at least one node in another subgraph in the multiple subgraphs. The method may also include generating answers to questions based on the hypergraphs.

[0006] In one embodiment, the system may include a processor and a memory device coupled to the processor. The processor may be configured to receive a video containing multiple video frames in a sequence. The processor may also be configured to receive questions about the video. The processor may also be configured to analyze the video frames in multiple video frames into at least object-to-object relationships and create a subgraph of nodes representing objects and edges representing relationships. The processor may be configured to perform the analysis and creation for each video frame in multiple video frames, so that multiple subgraphs can be created. The processor may also be configured to create a hypergraph connecting at least some of the subgraphs by learning the relationships between the nodes of the subgraphs, so that hyperedges are created representing relationships between at least one node in one subgraph and at least one node in another subgraph in multiple subgraphs. The processor may also be configured to generate answers to questions based on the hypergraphs.

[0007] Computer-readable storage media for storing a program of machine-executable instructions for performing one or more of the methods described herein may also be provided.

[0008] Further features, as well as the structure and operation of various embodiments, are described in detail below with reference to the accompanying drawings. In the drawings, similar reference numerals indicate identical or functionally similar elements. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows the system architecture in the embodiment. [Figure 2] This figure shows further details of the converter in the embodiment. [Figure 3] This figure shows an example of a hypergraph or a set of hypergraphs that the converter can build or learn in the embodiment. [Figure 4] This figure shows an example of the structure of a representational hypergraph token sequence in an embodiment. [Figure 5] This figure shows the context abstraction and graph initialization in the embodiment. [Figure 6] This is a flowchart illustrating the method in the embodiment. [Figure 7] This figure shows the components of a system in one embodiment that can automate video question answering. [Figure 8] This is a schematic diagram of an exemplary computer or processing system that can implement a system according to one embodiment. [Figure 9] This diagram shows a cloud computing environment in one embodiment. [Figure 10] This figure shows a set of functional abstraction layers provided by a cloud computing environment in one embodiment of the present disclosure. [Modes for carrying out the invention]

[0010] In one or more embodiments, systems, methods, and techniques can be provided to enable a machine to grasp current knowledge from its surroundings or circumstances and to reason accordingly by machine learning methods. The machine can capture a visual surrounding situation, which may be expressed as, for example, a video or similar visual data, using a camera or similar device, and to reason from it. In embodiments, a machine such as a robot can be trained to answer questions about a given visual surrounding or a given video, for example, so that it can reason from a given visual surrounding, such as a given video. For example, a machine can be trained to dynamically understand a situation and to reason in the current situation according to that understanding. In this way, for example, a machine or robot can autonomously learn to act in accordance with a situation, or effectively understand human actions in the real world, or both. For example, a machine or robot can learn to reason about actions such as moving or lifting an object based on its surroundings. Simply put, a situation represents entities, events, moments, environments, etc. A situation can be provided in the form of a video, which may include multiple consecutive or overlapping actions and interactions.

[0011] In one embodiment, real-world situational reasoning can be provided via video, in which a system or machine captures current knowledge from a dynamic situation as a structured representation and answers questions. The system or machine learns and reasons in the dynamic real-world situation in order to answer questions. In an embodiment, the situation provided via video can be represented by an annotation-based hierarchical graph. In an embodiment, situational reasoning can be performed iteratively on such a graph. For example, in an embodiment, a given situation in a real-world video can be represented by a hypergraph connecting extracted atomic entities and relationships (e.g., actions, people, objects, and relationships). In an embodiment, relationships in the situation may include person-object contractual and spatial relationships, object-object spatial relationships, or action-action temporal relationships, or a combination thereof.

[0012] In embodiments, a transformation-based question answering architecture or system, or both, can be provided. Such an architecture or system, or both, recognizes humans, objects, or human-object relationships or combinations thereof in the input video as a set of basic entities by a set of video parsers (e.g., object detector, relationship detector, human posture detector, motion detector); performs bottom-up abstraction and inference on an abstract situational graph from the video as context by a situational token sequence generator and a stacked situational motion converter deep network; represents questions / choices together as a whole question-answer embedding by combining language encoders for questions and choices; and can predict possible answers by a question answering head based on question-answer embeddings and pooled situational embeddings. In embodiments, the types of questions may include interaction, sequence, prediction, and feasibility.

[0013] In embodiments, a visual abstraction and inference system, method, model, or engine, or a combination thereof, may also be called a situational behavior converter and may include a system that separates visual perception, situational abstraction, and provides question-answering capabilities. In embodiments, a situational graph encoder connects entities to a structured situational representation (situational graph). The system or method, or both, can convert the connected graph into a token sequence by hierarchical stacking. In embodiments, a stacked situational graph converter or graph sequence decoder, or both, learns to predict masked (unseen) entity tokens in the input situational graph and behavior tokens by simulating implicit state transition processing between entities. A hypergraph generator can construct an output behavior hypergraph by connecting both predicted behaviors and existing and predicted missing entities.

[0014] In embodiments, an architecture for a situational action transformer for real-world video question answering may include multiple layers of transformers capable of recognizing basic entities and relationships from the video context, performing bottom-up inference on an abstract situational graph, and predicting possible answers. For example, a system or transformer-based question answering architecture may recognize humans, objects, or human-object relationships in the input video as a set of basic entities by a set of video parsers (e.g., object detector, relationship detector, human posture detector, action detector). The system may perform bottom-up abstraction and inference on an abstract situational graph from the video as context by a situational token sequence generator and a stacked deep network (also known as a situational action transformer (SAT)). In embodiments, the system may represent a question or a choice, or both, together as a whole question-answer embedding by combining language encoders for the question and choice. In embodiments, the system may predict possible answers by a question-answering head based on the question-answer embedding and pooled situational embedding. The system is applicable to online chat, human-robot communication, or other combinations.

[0015] In embodiments, one or more processors can automatically or autonomously perform video question answering, for example, by being given a video scene (which may include, for example, one or more video frames) and a question, and automatically or autonomously answering the question. For example, a bot or robot may have a model trained to answer questions about a video scene. In embodiments, the model receives a video as input, for example, from a user, which includes a video frame and one or more questions. Such a video may be a scene or image of the surrounding environment from the real world, for example, from a robot communicating with a user and capturing images of the surroundings through its camera or similar vision-capturing device. Depending on the video content (e.g., video from the real world), the model can answer one or more questions or output answers to one or more questions about the video. In embodiments, a transducer including a multi-layer architecture can be trained to perform such video question answering. For example, a human and a robot can communicate communicatively. For example, when communicating with a robot, a user may use natural language, for example, human spoken or written language, to say sentences, phrases, words, or ask questions, or both. The robot can see the scene or surrounding environment through its camera or similar device, and can see objects and actions, such as actions being performed, within the scene or surrounding environment. This model helps the robot to understand the process of action and the surrounding environment in a video, so that it can communicate with a user and answer questions such as what the person is doing, what they plan to do, what they have already done, or a combination of these. This model can help the robot intelligently evaluate its surrounding environment and assist in answering questions, communicating with a user, or a combination of these.

[0016] FIG. 1 is a diagram showing a system architecture in an embodiment. The components shown include computer-implemented components that are implemented or executed or both on one or more hardware processors, or are coupled to one or more hardware processors. The one or more hardware processors may include components such as, for example, programmable logic devices, microcontrollers, memory devices, or other hardware components or combinations thereof, which may be configured to perform each task described in the present disclosure. The coupled memory device may be configured to selectively store instructions executable by one or more hardware processors.

[0017] The processor may be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), other suitable processing components or devices, or a combination of one or more of them. The processor may be coupled to a memory device. The memory device may include random access memory (RAM), read only memory (ROM), or another memory device, and may store data or processor instructions or both for implementing various functionalities related to the methods or systems or both described herein. The processor may execute computer instructions stored in the memory or received from another computer device or medium.

[0018] In embodiments, the system may include a video parser 104 or similar component or function that can analyze a given video 102a and extract objects and relationships. For example, the video parser 104 may receive a video 102a that may contain multiple video frames. The video 102a may contain a time series of video frames or segments. The system may also include a converter 106 (also called a converter-based model or situational behavior converter) that learns or builds a hypergraph, which is a structure that connects the interactions between objects. For example, model 106 may convert the video footage of video 102a into a hypergraph. In embodiments, the hypergraph may include connected subgraphs, each subgraph representing a video frame. In embodiments, the initial graph or situational graph associated with each video frame may be built based on objects, people, or relationships, or combinations thereof, identified or extracted by the video parser 104. The graphs associated with video frames may be called subgraphs. Initially, the subgraphs are isolated, for example, with no connections between them. The transducer 106 can further learn more unique connections between nodes in a subgraph and between subgraphs to generate a hypergraph. In embodiments, the transducer 106 may perform bottom-up abstraction and inference on an abstract situation graph from a video as context by a situation token sequence generator and a stacked situational behavior transducer deep network. A situation graph, which may include node instances, subgraph edges, and hypergraph edges, can be thought of as an abstraction of the visual content of a video. A single situation graph may contain multiple subgraphs, each subgraph corresponding to a video frame. For example, each subgraph corresponds to a video frame in a video. In embodiments, a function (which may be called, for example, a situation token sequence generator) may construct an initial situation graph with connected nodes and edges, which do not yet have hyperedges. The stacked situational behavior transducer deep network may be a deep neural network model having a behavioral transition model and a transducer architecture.

[0019] The system can also include a pooling component 108 that can be a neural network that encodes a hypergraph into a vector, such as a fixed-length visual embedding vector. For example, the pooling component 108 can be a neural network layer that converts a hypergraph structure into a visual embedding vector. In an embodiment, this pooling layer can calculate the average of the inputs in order to convert the input hypergraph or the embedding vector for a unified representation.

[0020] The system can further include a language encoder 110 that encodes the received question or utterance 102b into a vector or an embedding vector, such as a language embedding vector. The question or utterance 102b can be, for example, illustratively, "After placing the bottle, which object can the person pick up?" In an embodiment, for example, optionally, candidate answers or options such as "table", "bottle", "book", "chair", etc. can be provided to the language encoder 110 as part of the input. In an embodiment, the language encoder 110 can represent the question and the options (e.g., when candidate options are provided) together as a whole question-and-answer embedding by combining the language encoders for the question and the options.

[0021] The system can also include, for example, a question-and-answer component 112 called a question-and-answer head, which receives a visual representation (e.g., a visual embedding vector) from the pooling component 108, such as a neural network, and a language representation (e.g., a language embedding vector) from the language encoder 110, and outputs an answer 114 based on those representations. For example, the question-and-answer component 112 can predict one or more possible answers based on the question-and-answer embedding and the pooled context embedding.

[0022] In an embodiment, the architecture shown in Figure 1 includes a neurosymbolic framework. In an embodiment, neurosymbolic inference can be employed, which can separate visual perception, perform contextual abstraction, understand language, and perform symbolic inference.

[0023] The video parser 104 can be a visual recognition module that includes a set of detectors capable of obtaining human-centered, object-centered, or both interactions from keyframe inputs of a video. In embodiments, the video parser 104 can use a set of video parsers (such as object detectors, relationship detectors, human pose detectors, and motion detectors) to recognize humans, objects, or human-object relationships in an input video as a set of basic entities. Given an input video or video segment, the video parser 104 can output a situation data type. In embodiments, the video parser 104 can use bounding box ground truth to localize objects / humans in the video. In embodiments, objects can be detected using an object detector (e.g., a region-based convolutional neural network (R-CNN), Faster R-CNN, or other or a combination thereof). In embodiments, visual representations for each object or person can be extracted using techniques such as deep neural networks. Relationships can be detected, relationship representations extracted, and further skeleton representations for motion can be extracted using existing techniques such as visual and pattern recognition, natural language processing, and pose parsers. In embodiments, for tasks involving query behavior in a question (e.g., feasibility or sequence), an existing pre-trained behavior recognizer can be employed to recognize behavior observed in a video as a prerequisite. In embodiments, the video parser 104 can be trained on video keyframes from a training set to obtain features of appearance, recognition, and bounding box position. The video parser 104 recognizes entities, relationships, and human or object interactions or combinations thereof in a visual scene or situation. In embodiments, the current scene or situation is sent to the transducer 106 to learn a situational abstraction and predict future situations in the form of a hypergraph (also called a situational hypergraph).

[0024] Figure 2 shows further details of the transducer (also called the transducer-based model) 106 in an embodiment. In the embodiment, the transducer can process and predict current and future situations in the form of a hypergraph. In the embodiment, the graph encoder 202 performs dynamic state transitions on the situation hypergraph. State transitions are attention update processes of the self-attention mechanism in the behavioral transition model. For example, initially, the situation hypergraph may be incomplete or lack complete parts such as hyperedges. The transducer generates and outputs a complete hypergraph. The graph encoder 202 constructs an "initial" graph by connecting detected entities or relationships and encodes the graph into a structured graph token sequence 206. In the embodiment, the graph encoder 202 connects entities to a structured situation representation (situation graph) and converts the connected graph into a token sequence 206 by hierarchical stacking. For example, the graph encoder 202 follows a self-monitoring learning paradigm. The graph encoder 202 takes a token sequence as input and predicts the masked tokens in that sequence. In this embodiment, sequence 206 describes the top-down structure of the situation graph and signifies situation segments, subgraph segments, and entity tokens. A situation segment may be a discrete segment token for a situation mark in the sequence. t situation segments 0 ,...,s T Given >, we assume that each situation segment at time t contains a set of multiple predicate tokens and triplet tokens. Each predicate is in situation s t Hyperedge relationships exist that connect the subgraphs of the connected situation, resulting in the emergence of atomic behavior a j Represents a triplet token. <h i ,o i ,r i ​> represents human-relation-object interaction. Action predicates can consist of a combination of an atomic action verb and an object noun (e.g., pick up a book, sit in a chair). In embodiments, each situation segment may be padded with zero tokens for a fixed length. To represent graph entities, hyperedges, segments, and situations, the graph encoder 202 may sum up multiple types of embedding vectors for each token, such as token embeddings, type or hyperedge embeddings, situation embeddings, location embeddings, and segment embeddings.

[0025] The graph transformer 208 can dynamically predict behavioral states or relationships by learning the relationships between input data types in a given situation (e.g., objects in a given video). In embodiments, the model architecture includes a multi-layer stacked transformer with downstream task predictors. With respect to the stacked transformer, in embodiments, the graph transformer 208 can use the stacked transformer block to compute a self-attention score for an input token sequence having multiple heads. In embodiments, the stacked transformer can include a Bidirectional Encoded Representation by Transformers (BERT) model. Attention describes the "connection" of each potential relationship between two nodes in a situational graph or subgraph (e.g., behavioral hyperedge or human-relationship-object triplet). In embodiments, the internal structure of the transformer's self-attention corresponds to token pairs, so all attention on input tokens performs dynamic relationship modeling. In embodiments, connections between adjacent nodes are aggregated into a single node. The aggregated effect is stored in the current state at time t and can be applied to predict missing information in the current step or the next state at time t+1. Such dynamic attention modeling can treat all possible relationships as implicit connections and can provide robust modeling even when relationships are unknown or some visual cues are unreliable. In embodiments, this model can also be employed to predict several unknown situations for predictive or feasibility questions. A graph transformer generates, for example, a predicted token sequence 210.

[0026] The graph sequence decoder 212 may perform the following self-supervised tasks: behavior type prediction, human-object relationship type prediction, and masked token modeling (for objects or people). For example, behavior type prediction and human-object relationship type prediction can use a classifier that predicts behavior hyperedges or relationships using a multilayer perceptron (MLP), artificial neural network, having a pooled global representation of all states in the corresponding situation. Masked token modeling can be made more robust to the representation by reconstructing their embedding vectors. The graph sequence decoder 212 outputs a hypergraph. In an embodiment, the stacked graph transducer 208 and the graph sequence decoder 212 learn to predict masked entity tokens in the input situation graph and behavior tokens by simulating implicit state transition processing between entities. In an embodiment, all output tokens of 210 may also be inputs to the graph sequence decoder 212, which includes a multilayer MLP in addition to pooling operations. The hypergraph generator can construct an output behavior hypergraph by connecting both predicted behaviors and existing and predicted missing entities. Missing entities can be invisible instances in a video.

[0027] Figure 3 shows an example of a hypergraph, for example, shown in Figure 1, 116, which a converter can construct or learn in an embodiment. For example, a given situation in a real-world video can be represented by a hypergraph connecting extracted atomic entities and relationships (e.g., actions, people, objects, and relationships). For example, in an embodiment, a hypergraph can be learned to represent or describe dynamic processing in a video, for example, in a real-world situation. A hypergraph can represent, for example, actions and internal relationships within a situation in a video frame or segment, as well as their hierarchical structure. For example, a video may include a set of subgraphs with nodes for people and objects, and edges representing relationships in a frame (person-object or object-object). Each action hyperedge can connect multiple subgraphs. In some embodiments, multiple actions can overlap and share nodes in a subgraph. In an embodiment, dynamic processing in a video can be extracted as a set of consecutive and overlapping hypergraphs. Formally, a hypergraph H is where X is a set of nodes for objects or people appearing in a video frame, and E is a non-empty hyperedge subgraph S for actions. i The set can be a pair H=(X,E). In embodiments, the hypergraph structure describes the behavior as a hyperedge. Such a structure can reflect a hierarchical abstraction from a real-world situation or symbolic representation.

[0028] Referring back to Figure 2, in an embodiment, the graph encoder 202 may convert a situation graph corresponding to a video, which is an initial hypergraph having several subgraphs, into a structured token sequence (referred to as a hypergraph token sequence). In an embodiment, to represent graph entities, hyperedges, segments, and situations, the graph encoder 202 may sum up several types of embedding vectors for each token, e.g., 1) token embedding: appearance of an object, person pose, relation class, or predicate class; 2) type or hyperedge embedding: indicating an action predicate, person, object, or relation; 3) situation embedding: recording the time order of situation frames; 4) position embedding: bounding boxes for objects and people; 5) segment embedding. Separating tokens can be used to indicate the boundaries of segments in a token sequence. Tokens represent entity nodes, edges, or hyperedges in the situation hypergraph. Separating tokens may also exist. In one embodiment, a situation corresponds to the entire given video, and a segment corresponds to a video frame within the video. In an embodiment, as an implementation example, the system may set parameters that can specify the maximum number of situation graphs in a video, and the maximum number of actions and relations within a single situation graph (e.g., a subgraph). A hypergraph token sequence can, in an embodiment, be represented as a row-column data structure (e.g., a multidimensional vector). Each situation column can be a structured situation subgraph, and related situation subgraphs can share the same behavior and be connected by behavior hyperedges. Each row can represent a specific embedding token type. A situation column can have multiple columns, each column can represent an embedding related to the subgraph (e.g., a person, object, behavior, or relationship embedding). Figure 4 shows an example structure of a hypergraph token sequence for representation in an embodiment.

[0029] Returning to Figure 1, the language encoder 110 can parse each question or utterance into a functional program in the form of a program statement. A functional program can consist of a series of nested operations. For example, different kinds of indivisible operations (e.g., query functions) may be defined to construct a stepwise inference program. In an embodiment, the system can use an attention-based Seq2Seq model to parse an input question into a corresponding program. In an embodiment, two models can be used to parse the question and the choices separately. In an embodiment, each model may include a bidirectional long short-term memory (LSTM) encoder and an LSTM decoder. In an embodiment, as an example, the system can use two hidden layers, 256 hidden units and an embedding layer, to obtain a 300-dimensional word vector for both the encoder and the decoder.

[0030] In an embodiment, the question answering component (also called the question answering head in the architecture of the embodiment) 112 can answer a given question 102b by executing a functional program on a discrete graph, such as a generated hypergraph, which explicitly performs symbolic inference for the resulting answer. For example, in an embodiment, the question answering component or head 112 acts as an inference engine. In an embodiment, the question answering component or head 112 receives the program or output of the language encoder 110 and the predicted situation hypergraph 116 as symbolic and discrete inputs and executes functional operations in the program on the hypergraph. In an embodiment, predefined operations can be performed based on entities and relationships in the structured situation hypergraph. Each operation takes a specific entity or relationship as input and outputs its prediction as input for the next inference step or as the final answer prediction. In an embodiment, with the hypergraph 116 as input, the inference starts with a queue in the question (object, action, or other basic data type) as an initial query, then iterates through all operations and finally outputs an answer 114. In an embodiment, the answer can be selected from a set of input candidates (e.g., multiple choices). In another embodiment, this is an answer provided to a free-form question (e.g., one with no candidate options or no multiple options).

[0031] Figure 5 shows the situation abstraction and graph initialization in an embodiment. Video 502 may have multiple frames 502a, 502b, 502c, 502d, 502e, 502f. The situation in video 502 may include preconditions, actions (atomic actions), and termination. Situation decomposition may include extracting or decomposing entities (e.g., people, objects, actions, relationships) from video frames. Relationships may be attention or contact (person-object relationships), temporal (action-action relationships), spatial (person-object, object-object relationships), etc. Graphs may be constructed with nodes (e.g., representing objects, people) and edges representing relationships related to video frames, as shown, for example, in 504, 506, and 508. For example, situational video question answering may include providing a video showing a person's activity in a sequence of video frames and providing a question such as "How does that person take the book to the door?". The answer may be "Puts the bottle down, picks up the book, and walks past the table." In one embodiment, possible responses may also be provided, such as "Put down the bottle, pick up the book, and walk past the table," "Walk straight, pick up the book, and put down the phone," "Pick up the table, sit on the door, and put down the can," or "Close the door, pick up the bottle, and open the book," from which the automated machine can select the correct response.

[0032] In embodiments, a machine such as a bot or robot can learn to reason about actions, for example, to act based on a specific event or a given situation, and can be trained to effectively understand human actions in a real-world setting. Examples of actions taken by a machine, bot, or robot based on a given or current situation include, but are not limited to, grasping an item or object from a table, shelf or something else, moving an object such as a box in a warehouse, and lifting an object. In embodiments, the machine may be given the task of evaluating or answering questions for action reasoning in a video representing a real-world setting.

[0033] Figure 6 is a flowchart illustrating the method in an embodiment. The method may be implemented or executed on one or more computer processors or hardware processors or both. In 602, video can be received. The video may be a sequence, such as a time series, and may contain multiple video frames. In 604, questions about the video may also be received. For example, questions may be received about what is happening in the video, what could happen in the video, or both, or other questions about the video or a combination thereof. For example, the types of questions may include, but are not limited to, interaction questions, sequence questions, prediction questions, and feasibility questions. Each question may be associated with an action-centered situation from a variety of scenes and locations, and the situation may contain multiple actions.

[0034] In 606, video frames within a video can be analyzed into objects and relationships between objects. In embodiments, objects may include people, items, or both that appear in the video frame. Subgraphs of nodes and edges can be created. Nodes represent objects analyzed from the video frame, and edges represent relationships between objects that appear in the video frame. Such analysis and subgraph creation can be performed for individual video frames in a video. For a video with multiple video frames, multiple corresponding subgraphs can be created.

[0035] In 608, a hypergraph is created that includes subgraphs representing video frames within a video. The hypergraph also includes one or more edges or connections that connect at least some of the subgraphs. Such edges or connections can be created by learning the relationships between nodes in the subgraphs. An edge (also called a hyperedge) represents a relationship between at least one node in one subgraph and at least one node in another subgraph across multiple subgraphs. In embodiments, the hypergraph provides a structured representation of the entities and relationships of current knowledge and dynamic changes in the video.

[0036] In 610, answers to questions about videos can be generated based on a hypergraph. In one embodiment, an automated machine, such as a bot or robot, can interact or communicate with a user who asks questions about videos that can represent the current visual surroundings. For example, the machine can grasp current knowledge from a dynamic situation as a structured representation. The machine can answer such questions in interaction with the user.

[0037] In an embodiment, the hypergraph can be encoded as a visual embedding vector. In an embodiment, a neural network can be trained to predict the relationships between nodes in a subgraph. In an embodiment, a question can be encoded as a language embedding vector. In an embodiment, a bidirectional long short-term memory (LSTM) encoder and decoder can analyze the question. In an embodiment, the subgraph is converted into a structured token sequence. Such a token sequence can be converted into a visual embedding vector.

[0038] Figure 7 shows the components of a system in one embodiment that can automate video question answering. One or more hardware processors 702, such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA) or a combination thereof, an application-specific integrated circuit (ASIC), or another processor or a combination thereof, are coupled with a memory device 704 and may generate predictive models to recommend communication opportunities. The memory device 704 may include random access memory (RAM), read-only memory (ROM), or another memory device and may store data or processor instructions or both for implementing various functionalities related to the methods or systems or both described herein. One or more processors 702 may execute computer instructions stored in the memory 704 or received from another computer device or medium. The memory device 704 may store instructions or data or both for operating one or more hardware processors 702, and may include other programs such as an operating system and instructions or data or both. One or more hardware processors 702 may receive input including video and video-related questions. One or more hardware processors 702 may analyze video frames into at least objects and relationships between objects, and create subgraphs of nodes representing objects and edges representing relationships, the analysis and creation being performed for each video frame in a plurality of video frames, and the plurality of subgraphs being created, for example, subgraphs corresponding to video frames. One or more hardware processors 702 may create a hypergraph connecting at least some of the subgraphs by learning the relationships between the nodes of the subgraphs, where hyperedges are created to represent the relationship between at least one node in one subgraph and at least one node in another subgraph in the plurality of subgraphs. One or more hardware processors 702 may generate answers to questions based on the hypergraphs.In one embodiment, video data and question data may be stored in a storage device 706, or received from a remote device via a network interface 708, and may be temporarily loaded into a memory device 704 for building or generating a model capable of predicting relationships and creating one or more hypergraphs. The learned model may be stored in the memory device 704 for execution by, for example, one or more hardware processors 702. One or more hardware processors 702 may be coupled with interface devices such as a network interface 708 for communicating with a remote system via a network, and an input / output interface 710 for communicating with input or output devices, or both, such as a keyboard, mouse, display, or other or a combination thereof.

[0039] Figure 8 is a schematic diagram of an exemplary computer or processing system that may implement the system in one embodiment. The computer system is merely an example of a suitable processing system and is not intended to imply any limitations on the scope or functionality of the embodiments of the methodology described herein. The shown processing system may operate in a number of other general-purpose or dedicated computing system environments or configurations. Examples of well-known computing systems, environments, or configurations or combinations that may be suitable for use with the processing system shown in Figure 8 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable home appliances, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0040] A computer system is sometimes described in the general context of a computer system executing computer system executable instructions, such as program modules. Generally, a program module may include routines, programs, objects, components, logic, data structures, etc., that perform a specific task or implement a specific abstract data type. A computer system may also be implemented in a distributed cloud computing environment where tasks are executed by remote processing units linked over a communication network. In a distributed cloud computing environment, program modules may reside on both local and remote computer system storage media, including memory storage devices.

[0041] The components of the computer system may include, but are not limited to, one or more processors or processing units 12, system memory 16, and a bus 14 that connects various system components, including the system memory 16, to the processor 12. The processor 12 may include a module 30 that performs the methods described herein. The module 30 may be programmed into the integrated circuit of the processor 12, or it may be loaded from memory 16, a storage system 18, or a network 24, or a combination thereof.

[0042] Bus 14 can represent one or more of several types of bus structures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of the various bus architectures. Examples, but not limited to, such architectures include the Industry Standard Architecture (ISA) bus, Microchannel Architecture (MCA) bus, Extended ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0043] A computer system may include various computer system-readable media. Such media may be any available media accessible by the computer system and may include both volatile and non-volatile media, and removable and non-removable media.

[0044] The system memory 16 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM), cache memory, or both. The computer system may further include other removable / non-removable, volatile / non-volatile computer system storage media. For example only, the storage system 18 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (e.g., a “hard drive”). Not shown, it is possible to provide a magnetic disk drive for reading from and writing to removable, non-volatile magnetic disks (e.g., a “floppy disk”), and an optical disk drive for reading from and writing to removable, non-volatile optical disks such as CD-ROMs, DVD-ROMs, or other optical media. In such cases, each may be connected to the bus 14 by one or more data medium interfaces.

[0045] The computer system may also communicate with one or more external devices 26 such as a keyboard, pointing device, or display 28, one or more devices that enable a user to interact with the computer system, or any devices (e.g., network cards, modems, etc.) or combinations thereof that enable the computer system to communicate with one or more other computer devices. Such communication may occur via an input / output (I / O) interface 20.

[0046] Furthermore, the computer system can communicate with one or more networks 24, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via the network adapter 22. As depicted, the network adapter 22 communicates with other components of the computer system via the bus 14. It should be understood that other hardware or software components, or combinations thereof, may be used with the computer system, although these are not shown. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0047] This disclosure includes a detailed description of cloud computing, but it should be understood that implementations of the teachings described herein are not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in combination with any other type of computing environment currently known or to be developed in the future. Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or minimal interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0048] The characteristics are as follows:

[0049] On-demand self-service: Cloud consumers can unilaterally prepare computing power, such as server time and network storage, automatically as needed, without requiring human interaction with service providers.

[0050] Broad network access: Computing power is available over the network and accessible through standard mechanisms. This facilitates utilization by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, PDAs).

[0051] Resource pooling: A provider's computing resources are pooled and delivered to multiple consumers using a multi-tenant model. Various physical and virtual resources are dynamically allocated and reallocated as needed. Generally, consumers have a sense of location independence because they do not manage or know the exact location of the resources provided. However, consumers may be able to identify the location at a higher level of abstraction (e.g., country, state, data center).

[0052] Rapid Elasticity: Computing power can be prepared quickly and flexibly, allowing it to scale out automatically and immediately, and to be quickly released and scale in immediately. To consumers, the computing power available for preparation often appears unlimited and can be purchased in any quantity at any time.

[0053] Measured Services: Cloud systems leverage metric capabilities at a certain level of abstraction, appropriate for the type of service (e.g., storage, processing, bandwidth, active user accounts), to automatically control and optimize resource usage. Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0054] The service model is as follows:

[0055] Software as a Service (SaaS): The functionality offered to consumers is the ability to use the provider's applications running on a cloud infrastructure. These applications can be accessed from various client devices via thin client interfaces such as web browsers (e.g., webmail). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functions, except for configuring a limited number of user-specific applications.

[0056] Platform as a Service (PaaS): The functionality offered to consumers is the ability to deploy applications they have created or acquired to cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, and storage, but they can control the deployed applications and, in some cases, the configuration of their hosting environment.

[0057] Infrastructure as a Service (IaaS): The functionality provided to consumers is the provision of processors, storage, networking, and other basic computing resources that enable consumers to deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they can control the operating system, storage, and deployed applications, and in some cases, partially control certain network components (e.g., host firewalls).

[0058] The deployment model is as follows:

[0059] Private Cloud: This cloud infrastructure is operated exclusively for a specific organization. This cloud infrastructure can be managed by that organization or a third party and can reside on-premises or off-premises.

[0060] Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with common interests (e.g., mission, security requirements, policies, and compliance). This cloud infrastructure can be managed by the organization or a third party and can reside on-premises or off-premises.

[0061] Public Cloud: This cloud infrastructure is provided to a large number of people or large industry groups and is owned by organizations that sell cloud services.

[0062] Hybrid Cloud: This cloud infrastructure combines two or more cloud models (private, community, or public). While maintaining the unique entities of each model, they are bound together by standards or individual technologies to achieve data and application portability (e.g., cloud bursting for load balancing across clouds).

[0063] Cloud computing environments are service-oriented environments that emphasize statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is the infrastructure, which includes a network of interconnected nodes.

[0064] Here, Figure 9 shows an exemplary cloud computing environment 50. As shown in the figure, the cloud computing environment 50 includes one or more cloud computing nodes 10. Local computer devices used by cloud consumers (e.g., PDAs or mobile phones 54A, desktop computers 54B, laptop computers 54C, or automotive computer systems 54N, or a combination thereof) can communicate with these nodes. The nodes 10 can communicate with each other. The nodes 10 can be grouped physically or virtually (not shown) in one or more networks, such as the private, community, public, or hybrid clouds or a combination thereof. This allows the cloud computing environment 50 to provide infrastructure, platforms, or software as a service, or a combination thereof, without requiring cloud consumers to maintain resources on their local computer devices. Note that the types of computer devices 54A-N shown in Figure 9 are merely examples, and it should be understood that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of electronic device via any type of network or network addressable connection (e.g., using a web browser) or both.

[0065] Here, Figure 10 shows a set of functional abstraction layers provided by the cloud computing environment 50 (Figure 9). It should be understood that the components, layers, and functions shown in Figure 10 are merely illustrative, and the embodiments of the present invention are not limited to these. As illustrated, the following layers and corresponding functions are provided.

[0066] The hardware and software layer 60 includes hardware components and software components. Examples of hardware components include a mainframe 61, a reduced instruction set computer (RISC) architecture-based server 62, server 63, blade server 64, storage 65, and a network and network components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0067] The virtualization layer 70 provides an abstraction layer. From this layer, for example, the following virtual entities can be provided: virtual servers 71, virtual storage 72, virtual networks 73 including virtual private networks, virtual applications and operating systems 74, and virtual clients 75.

[0068] As an example, the management layer 80 can provide the following functions: Resource preparation 81 enables the dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing 82 enables cost tracking as resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. As an example, these resources may include licenses for application software. Security enables not only protection of data and other resources, but also identification and verification of cloud consumers and tasks. The user portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 enables the allocation and management of cloud computing resources to ensure that requested service levels are met. Service Level Agreement (SLA) planning and execution 85 enables the pre-arrangement and procurement of cloud computing resources that are expected to be needed in the future in accordance with the SLA.

[0069] Workload layer 90 provides examples of the capabilities available in a cloud computing environment. Examples of workloads and capabilities that can be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analytics processing 94, transaction processing 95, and video question answering processing 96.

[0070] The present invention may be a system, method, or computer program product or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium storing computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0071] A computer-readable storage medium can be a tangible device capable of holding and storing instructions used by an instruction execution device. Examples of computer-readable storage media may be electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or appropriate combinations thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM (or flash memory), SRAM, CD-ROM, DVD, memory stick, floppy disk, punch cards, or grooved raised structures, and mechanically encoded devices on which instructions are recorded, and appropriate combinations thereof. Computer-readable storage devices as used herein should not be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0072] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computer device / processor. Alternatively, they can be downloaded to an external computer or external storage device via a network (e.g., the Internet, LAN, WAN, or wireless network, or a combination thereof). The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers or edge servers, or a combination thereof. A network adapter card or network interface within each computer device / processor receives computer-readable program instructions from the network and transfers them for storage in a computer-readable storage medium in the respective computer device / processor.

[0073] The computer-readable program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions can be executed as a standalone software package, either entirely on the user's computer or partially on the user's computer. Alternatively, they can be executed partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including LANs and WANs, or it may be connected to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), and programmable logic arrays (PLAs), can execute computer-readable program instructions by utilizing state information of computer-readable program instructions in order to customize the electronic circuits for the purpose of performing aspects of the present invention.

[0074] Embodiments of the present invention are described herein with reference to flowcharts or block diagrams, or both, of methods, apparatus (systems), and computer program products according to embodiments of the present invention. Each block in a flowchart or block diagram, or both, and combinations of blocks in a flowchart or block diagram, or both, are executable by computer-readable program instructions.

[0075] The above computer-readable program instructions may be provided to a computer or other programmable data processing device processor for the purpose of producing a machine. This creates means for these instructions, executed via such computer or other programmable data processing device processor, to perform functions / operations identified in one or more blocks in a flowchart or block diagram, or both. The above computer-readable program instructions may further be stored in a computer-readable storage medium that can be instructed to function in a particular manner to a computer, programmable data processing device, or other device, or a combination thereof. This constitutes a product in which the computer-readable storage medium storing the instructions includes instructions for performing functions / operations identified in one or more blocks in a flowchart or block diagram, or both.

[0076] Alternatively, a computer execution process may be generated by loading computer-readable program instructions into a computer, another programmable device, or other device, and having a series of operational steps executed on that computer, other programmable device, or other device. This ensures that the instructions executed on the computer, other programmable device, or other device perform functions / operations identified in one or more blocks in a flowchart, block diagram, or both.

[0077] The flowcharts and block diagrams in the drawings of this disclosure illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions for performing a particular logical function. In some other implementations, the functions shown within a block may be executed in an order different from the order shown in each figure. For example, two consecutively shown blocks may actually be executed substantially simultaneously, in a manner that partially or entirely overlaps in time, or in reverse order, depending on the functions involved. Each block in a block diagram or flowchart or both, and combinations of multiple blocks in a block diagram or flowchart or both, are executable by a dedicated hardware-based system that performs a particular function or operation, or executes a combination of dedicated hardware and computer instructions.

[0078] The terms used herein are intended solely to describe and not limit to specific embodiments. Where used herein, the singular forms “a,” “an,” and “the” are intended to include the plural unless the context explicitly indicates otherwise. Where used herein, the term “or” is an inclusion operator and may mean “and / or” unless the context explicitly or clearly indicates otherwise. Where used herein, the terms “comprise,” “comprises,” “comprising,” “include,” “includes,” “including,” or “having,” or any combination thereof, specify the presence of a described feature, integer, step, operation, element, or component or combination thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups or combinations thereof. Where used herein, the expression “in an embodiment” refers to possible, but not necessarily, the same embodiment. As used herein, the phrase "in one embodiment" does not necessarily refer to the same embodiment, although it may be possible. Similarly, as used herein, the phrase "in another embodiment" does not necessarily refer to a different embodiment, although it may be possible. Furthermore, embodiments, components of embodiments, or both may be freely combined with each other, provided they are not mutually exclusive.

[0079] All means-plus-function elements or step-plus-function elements in the following claims, along with their corresponding structures, materials, actions, and equivalents, are intended to include any structures, materials, or actions for performing a function in combination with elements of any other claims specifically claimed. The description of the present invention is presented for illustrative and explanatory purposes, but is not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments are selected and described to best illustrate the principles and practical applications of the invention, and to enable those skilled in the art to understand the invention in various embodiments with various modifications to suit specific intended uses.

Claims

1. Receiving a video containing multiple video frames within a sequence, Receiving questions regarding the aforementioned video, For the video frames in the aforementioned plurality of video frames, The aforementioned video frames are analyzed to at least the relationship between objects and the relationship between those objects, This involves creating a subgraph of nodes representing objects and edges representing the aforementioned relationships, The aforementioned analysis and creation is performed for each video frame in the aforementioned multiple video frames, and multiple subgraphs are created, and the analysis and creation are performed. Creating a hypergraph connecting at least some of the subgraphs by learning the relationships between the nodes of the subgraphs, wherein hyperedges are created representing the relationships between at least one node in one subgraph and at least one node in another subgraph, and the hypergraph includes future states for the plurality of video frames, which are the result of predicting missing information for the next time state using the current time state. A method performed by a computer, comprising generating an answer to the question based on the hypergraph.

2. The computer method according to claim 1, wherein the hypergraph is encoded as a visual embedding vector.

3. The method performed by the computer according to claim 1, wherein the aforementioned question is encoded as a language embedding vector.

4. The method performed by a computer according to claim 1, further comprising training a neural network to predict the relationships between the nodes of the subgraph.

5. A method performed by a computer according to claim 1, further comprising analyzing the question using a bidirectional long-short-term memory (LSTM) encoder and decoder.

6. The method performed by a computer according to claim 1, wherein the subgraph is converted into a structured token sequence.

7. Processor and A memory device coupled to the aforementioned processor, Includes, The aforementioned processor comprises at least, Receiving a video containing multiple video frames within a sequence, Receiving questions regarding the aforementioned video, For the video frames in the aforementioned plurality of video frames, The aforementioned video frames are analyzed to at least the relationship between objects and the relationship between those objects, This involves creating a subgraph of nodes representing objects and edges representing the aforementioned relationships, The aforementioned analysis and creation is performed for each video frame in the aforementioned multiple video frames, and multiple subgraphs are created, and the analysis and creation are performed. Creating a hypergraph connecting at least some of the subgraphs by learning the relationships between the nodes of the subgraphs, wherein hyperedges are created representing the relationships between at least one node in one subgraph and at least one node in another subgraph, and the hypergraph includes future states for the plurality of video frames, which are the result of predicting missing information for the next time state using the current time state. A system configured to generate answers to the aforementioned questions based on the aforementioned hypergraph.

8. The system according to claim 7, wherein the hypergraph is encoded as a visual embedding vector.

9. The system according to claim 7, wherein the aforementioned question is encoded as a language embedding vector.

10. The system according to claim 7, wherein the processor is further configured to train a neural network to predict the relationships between the nodes of the subgraph.

11. The system according to claim 10, wherein the aforementioned relationship includes an operational relationship.

12. The system according to claim 7, wherein the processor is further configured to analyze the question using a bidirectional long-short-term memory (LSTM) encoder and decoder.

13. The system according to claim 7, wherein the subgraph is converted into a structured token sequence.

14. A computer program, wherein the computer program is readable by a device, and the device Receiving a video containing multiple video frames within a sequence, Receiving questions regarding the aforementioned video, For the video frames in the aforementioned plurality of video frames, The aforementioned video frames are analyzed to at least the relationship between objects and the relationship between those objects, This involves creating a subgraph of nodes representing objects and edges representing the aforementioned relationships, The aforementioned analysis and creation is performed for each video frame in the aforementioned multiple video frames, and multiple subgraphs are created, and the analysis and creation are performed. Creating a hypergraph connecting at least some of the subgraphs by learning the relationships between the nodes of the subgraphs, wherein hyperedges are created representing the relationships between at least one node in one subgraph and at least one node in another subgraph, and the hypergraph includes future states for the plurality of video frames, which are the result of predicting missing information for the next time state using the current time state. A computer program that generates answers to the aforementioned questions based on the aforementioned hypergraph.

15. The computer program according to claim 14, wherein the hypergraph is encoded as a visual embedding vector.

16. The computer program according to claim 14, wherein the aforementioned question is encoded as a language embedding vector.

17. The computer program according to claim 14, further causing the device to train a neural network to predict the relationships between the nodes of the subgraph.

18. The computer program according to claim 17, wherein the aforementioned relationship includes an operational relationship.

19. The computer program according to claim 14, further causing the device to analyze the question using a bidirectional long-short-term memory (LSTM) encoder and decoder.

20. The computer program according to claim 14, wherein the subgraph is converted into a structured token sequence.

Images