Method, computer program, and apparatus for performing artificial intelligence-based video question answering in a data processing system (neural symbolic action transformer for video question answering)

Abstract

To provide mechanisms that implement neural-symbolic action transformers to perform artificial intelligence-based video question answering.SOLUTION: A video parser parses an input video data sequence to generate situation data structures, each situation data structure comprising data elements corresponding to entities, and first relationships between the entities, identified by the video parser as present in images of the input video data sequence. First machine learning computer models operate on the situation data structures to predict second relationships between the situation data structures. Second machine learning computer models execute on a received input question to predict an executable program to answer the received question. The program is executed on the situation data structures and the predicted second relationships. An answer to the question is output based on results of executing the program.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] This application generally relates to improved data processing apparatuses and methods, and more particularly, to mechanisms for implementing a neural symbolic action transformer for performing video question answering.

Background Art

[0002] As recognized in "Inventing AI: Tracing the Diffusion of Artificial Intelligence with U.S. Patents", IP Data Highlights, No. 5, published in October 2020 by the Chief Economist's Office of the United States Patent and Trademark Office, artificial intelligence (AI) is becoming increasingly important for the latest innovations and has the potential to fundamentally change the way people perceive the world around them and lead their daily lives. As described in this publication, such fundamental changes are essential for technological progress, and these changes are being realized through innovation.

[0003] One field in which AI is applied is the field of visual question answering (VQA). The field of VQA involves addressing the task of answering open-ended (i.e., unconstrained) natural language questions regarding a given image or set of images. Most humans can relatively easily reason about real-world situations represented within an image (due to their innate knowledge, understanding of the physical world through experience, and inferential capabilities based on human intelligence), but the same does not hold true for computer-based AI systems. Such AI systems do not have innate knowledge, understanding of the physical world, experience, or inferential capabilities. Therefore, having an AI system that can understand the physical world from visual input and make logical judgments is a significant advancement in advanced AI.

Prior Art Documents

[0004] [Non-Patent Document 1] "Inventing AI: Tracing the Diffusion of Artificial Intelligence with US Patents," by the Chief Economist's Office of the United States Patent and Trademark Office, IP Data Highlights, No. 5, U.S. Patent and Trademark Office, October 2020. [Overview of the project] [Problems that the invention aims to solve]

[0005] However, existing research primarily focuses on object recognition that does not involve implicit logical reasoning. As a result, existing AI systems lack models capable of accurately generating responses to complex questions such as those concerning interaction, sequencing, prediction, and feasibility. [Means for solving the problem]

[0006] This summary of the invention is provided in a simplified form to introduce a selection of concepts that will be further described in embodiments for carrying out the invention herein. This summary of the invention is not intended to identify any important factors or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0007] In one exemplary embodiment, a method is provided for performing artificial intelligence-based video question answering in a data processing system. The method comprises the step of having a video parser of the data processing system parse an input video data sequence to generate one or more contextual data structures, each contextual data structure comprising data elements corresponding to entities identified by the video parser as existing in the images of the input video data sequence, and first relationships between entities. The method further comprises the step of running at least one first machine learning computer model of the data processing system on the one or more contextual data structures to predict one or more second relationships between the one or more contextual data structures. The method also comprises the step of having at least one second machine learning computer model of the data processing system run on an received input natural language question to determine an executable program to run in response to the received input natural language question. In addition, the method comprises the step of having the data processing system run the determined executable program on the one or more contextual data structures and the predicted one or more second relationships between the one or more contextual data structures. Furthermore, the method comprises a step in which the data processing system outputs a response to the input natural language question based on the result of executing the determined executable program. Thus, the mechanism of the exemplary embodiment provides an artificial intelligence computing system that can perform logical reasoning on an input video data sequence to respond to a natural language question.

[0008] In some exemplary embodiments, the at least one first machine learning computer model comprises a situation encoder, a machine learning-trained dynamics transformer computer model, and a sequence decoder. In some exemplary embodiments, the steps of running the at least one first machine learning computer model include: the situation encoder encoding the situation data structure to generate token sequences corresponding to the entities and relationships represented in the one or more situation data structures; and the machine learning-trained dynamics transformer computer model processing the token sequences to predict missing or subsequent token sequences and generating a predicted token sequence including the predicted missing or subsequent token sequences. Thus, the mechanism of the exemplary embodiment provides a machine learning computer model mechanism that can be trained to predict token sequences based on recognized entities and relationships in an input video data sequence.

[0009] In some exemplary embodiments, the at least one first machine learning computer model further comprises a sequence decoder, which generates the predicted one or more second relationships based on the predicted token sequence as one or more predicted hypergraph data structures. Thus, according to at least one exemplary embodiment, the machine learning computer model mechanism can predict actions or relationships between image data sequences in the input video data that represent relationships between situations that may not be explicitly shown in the input video data itself.

[0010] In some exemplary embodiments, the at least one second machine learning computer model comprises a language / program parser and a program execution unit. In some exemplary embodiments, the step of determining the executable program includes: the language / program parser processing the input natural language question and predicting a plurality of program modules to be executed to respond to the input natural language question; the program execution unit combining the plurality of program modules into an executable program to be executed on the one or more predicted hypergraph data structures to generate a final response to the input natural language question; and the data processing system outputting the final response to the input natural language question. In this way, at least one exemplary embodiment is capable of translating natural language questions into a predicted computer executable model that is capable of responding to those questions based on predicted relationships between situations in input video data.

[0011] In some exemplary embodiments, each hypergraph data structure includes one or more hyperedges connecting a first situation data structure within the one or more situation data structures to at least one second situation data structure, the at least one situation data structure being either the first situation data structure or a different situation data structure within the one or more situation data structures. In some exemplary embodiments, each hyperedge within the one or more hyperedges includes a predicted action corresponding to at least one first entity in the first situation data structure, accompanied by at least one second entity in the at least one second entity data structure. In this way, the hypergraph data structure provides predicted actions that connect the situations represented by the situation data structures, which can then be processed by a predicted program to generate a response to an input natural language query.

[0012] In some exemplary embodiments, each situation, comprising one or more images in the input video data sequence, has a corresponding situation data structure within the one or more situation data structures. In some exemplary embodiments, the situation data structure is a situation graph data structure, where the data elements include nodes corresponding to entities and edges corresponding to the relationships between entities. In some exemplary embodiments, the input natural language question is a logical reasoning question of any type: interaction question, sequence question, prediction question, or feasibility question. Thus, the mechanism of the exemplary embodiments is capable of responding to logical reasoning questions that ask about relationships and actions that are not necessarily explicitly shown in the image data of the input video data itself.

[0013] In some exemplary embodiments, a computer program product is provided comprising a computer-readable storage medium storing a computer-readable program. When executed in a data processing system, the computer-readable program causes the data processing system, through its video parser, to parse an input video data sequence to generate one or more contextual data structures, each contextual data structure comprising data elements corresponding to entities identified by the video parser as existing within the images of the input video data sequence, and first relationships between entities. The computer-readable program further causes the data processing system to run at least one first machine learning computer model of the data processing system on the one or more contextual data structures to predict one or more second relationships between the one or more contextual data structures. Furthermore, the computer-readable program causes the data processing system to determine an executable program to be executed in response to the received input natural language question by at least one second machine learning computer model of the data processing system executed on the received input natural language question, and to execute the determined executable program on the one or more situational data structures and the one or more predicted second relationships between the one or more situational data structures. In addition, the computer-readable program causes the data processing system to output a response to the input natural language question based on the result of the execution of the determined executable program by the data processing system.

[0014] In some exemplary embodiments, the at least one first machine learning computer model comprises a situation encoder, a machine learning-trained dynamics transformer computer model, and a sequence decoder. In some exemplary embodiments, running the at least one first machine learning computer model comprises: the situation encoder encoding the situation data structures to generate token sequences corresponding to the entities and relationships represented in the one or more situation data structures; and the machine learning-trained dynamics transformer computer model processing the token sequences to predict missing or subsequent token sequences and to generate a predicted token sequence including the predicted missing or subsequent token sequences.

[0015] In some exemplary embodiments, the at least one first machine learning computer model further comprises a sequence decoder, which generates the predicted one or more second relationships based on the predicted token sequence as one or more predicted hypergraph data structures. In some exemplary embodiments, the at least one second machine learning computer model comprises a language / program parser and a program executor. In some exemplary embodiments, the computer-readable program causes the data processing system to determine the executable program by at least the language / program parser processing the input natural language query and predicting a plurality of program modules to be executed to respond to the input natural language query; the program executor combining the plurality of program modules into an executable program to be executed on the one or more predicted hypergraph data structures to generate a final response to the input natural language query; and the data processing system outputting the final response to the input natural language query.

[0016] In some exemplary embodiments, each hypergraph data structure includes one or more hyperedges connecting a first situation data structure within the one or more situation data structures to at least one second situation data structure, the at least one situation data structure being either the first situation data structure or a different situation data structure within the one or more situation data structures. In some exemplary embodiments, each hyperedge within the one or more hyperedges includes a predicted action corresponding to at least one first entity in the first situation data structure, accompanied by at least one second entity in the at least one second entity data structure.

[0017] In some exemplary embodiments, each situation, comprising one or more images in the input video data sequence, has a corresponding situation data structure within the one or more situation data structures. In some exemplary embodiments, the situation data structure is a situation graph data structure, where the data elements include nodes corresponding to entities and edges corresponding to the relationships between entities. In some exemplary embodiments, the input natural language question is a logical reasoning question of any type: interaction question, sequence question, prediction question, or feasibility question.

[0018] In some exemplary embodiments, a device is provided comprising at least one processor and at least one memory coupled to the at least one processor. The at least one memory has instructions, when executed by the at least one processor, that cause the at least one processor to have a video parser of the data processing system parse an input video data sequence to generate one or more context data structures, each context data structure comprising entities identified by the video parser as existing in the images of the input video data sequence, and data elements corresponding to a first relationship between entities. The instructions further cause the at least one processor to run at least one first machine learning computer model of the data processing system on the one or more context data structures to predict one or more second relationships between the one or more context data structures. The instructions also cause the at least one processor to have at least one second machine learning computer model of the data processing system run on an received input natural language question to determine an executable program to run in response to the received input natural language question. In addition, the instruction causes the at least one processor to execute the determined executable program on the one or more situation data structures and the predicted one or more second relationships between the one or more situation data structures, as performed by the data processing system. Furthermore, the instruction causes the at least one processor to output a response to the input natural language query based on the result of executing the determined executable program, as performed by the data processing system.

[0019] These and other features and advantages of the present invention will be described in the following detailed description of exemplary embodiments of the present invention, or will become apparent to those skilled in the art who have considered this detailed description. [Brief explanation of the drawing]

[0020] The present invention, as well as its preferred usage forms and further objectives and advantages, will be best understood when read in conjunction with the accompanying drawings by referring to the following detailed description of exemplary embodiments.

[0021] [Figure 1] FIG. is a block diagram showing the main operational components of an improved computing tool based on a situation inference transformer (SR-transformer) according to one exemplary embodiment.

[0022] [Figure 2] FIG. is an exemplary diagram showing an exemplary situation shown in input video data, together with a corresponding situation graph data structure and hyperedges for answering logical inference questions according to one exemplary embodiment.

[0023] [Figure 3] FIG. is an exemplary diagram of the structure of a hypergraph token sequence for representation in SRT encoding according to one exemplary embodiment.

[0024] [Figure 4] FIG. is an exemplary list of program modules for automatic program generation by a program parser according to one exemplary embodiment.

[0025] [Figure 5] FIG. is an example of an automatically generated program for four exemplary question types according to one exemplary embodiment.

[0026] [Figure 6] FIG. is a flowchart outlining the exemplary operation of an improved computing tool according to one exemplary embodiment.

[0027] [Figure 7] FIG. is an exemplary diagram of a distributed data processing system in which aspects of an exemplary embodiment can be implemented.

[0028] [Figure 8] This is an illustrative block diagram of a computing device in which an exemplary embodiment may be implemented. [Modes for carrying out the invention]

[0029] As mentioned above, visual question answering (VQA) is a recent area of ​​artificial intelligence (AI) work in which open-ended natural language questions about a given image or set of images are answered through AI computer systems such as neural networks, convolutional neural networks (CNNs), and deep learning neural networks (DNNs). In order to be able to answer such questions given images, the AI ​​system must be trained to have some knowledge base for visual and linguistic understanding and for generating responses. This is a rather complex problem, as it limits the AI ​​system's ability to answer questions about what is specifically represented in the input image and does not allow for any logical reasoning beyond what is explicitly shown in the input image.

[0030] For example, given images of a man wearing a hat and a woman not wearing a hat, current AI systems may be able to answer questions such as "Who is wearing a hat?" or "Who is not wearing a hat?" or "What color is the woman's clothing?" or "How many people are in the image?", but all of these questions are about what is already represented in the image itself. For example, some efforts have proposed AI challenges to design algorithms that can answer visual questions asked by blind people. In such VQA efforts, the AI ​​system is given an image and a question about that image, the question asking about what is shown in the image itself, for example, "Does this foundation contain sunscreen?" (image of an ingredient list on a bottle), "What is this?" (image of money), "What color is this?" (image of a shirt), "What is this item?" (image of a can of food), "Is it sunny outside?" (image of a house), "What is the temperature of the oven?" (image of an oven control panel), etc.

[0031] For example, current VQA mechanisms lack the ability to perform logical reasoning to answer complex questions related to interaction, sequencing, prediction, and feasibility, such as "How does the man get the woman's hat?", "How does the man sit on the sofa?", "What should the man do to get to the woman?", and "Will the woman's hat fall to the ground?". In other words, existing VQA artificial intelligence (AI) systems are limited to answering questions about what is explicitly shown in an image, i.e., in-frame content, and are unable to learn relationships from visual information that could later be applied to new situations to answer complex questions requiring logical reasoning rather than simply object recognition.

[0032] The ability to learn relationships from visual information that demonstrates human interaction with their environment in a computer-recognizable way, enabling them to perform complex question-answering, has many different applications. For example, such learning can be used to enable AI-based systems to learn how to reason for themselves about how to achieve a goal given a similar physical environment. For instance, by observing a human moving through a real-world environment, an AI system can determine how the human should navigate around obstacles or interact with objects in the environment to achieve a desired outcome, namely a sequence of actions and relationships between the human and objects in the environment, thereby enabling it to predict the sequence of actions. As one example, a series of images may be presented showing a human entering a room through a door, walking around a sofa, and then sitting on the sofa. When presented with the question, "What should the man do to sit on the sofa?", by observing the images, the AI ​​system can learn the relationships between the human and objects in the environment, as well as the timing and sequencing of such actions and relationships. Subsequently, from these relationships, the AI ​​system can determine that the sequence of entering the room through the door, walking around the sofa, and then sitting on the sofa achieves the desired outcome stated in the question. Thus, the AI ​​system learns how a human can achieve the desired outcome of sitting on a sofa and can apply similar logical reasoning to other situations, such as sitting in a chair or sitting at a desk, where similar patterns of objects and relationships may exist.

[0033] Subsequently, we can see how such learning can be translated into computer-initiated robot commands that cause a robot system to perform desired actions within a physical environment to achieve a desired outcome, for example, the robot system sitting on a sofa in this example. For example, the desired outcome can be translated into a natural language question, and the response can be translated into a command signal. Therefore, for example, if a robot system is required to traverse a physical environment, the goal may be imposed as "reach the exit," which can then be translated into a natural language question, "how do I reach the exit?", and the resulting sequence may be imposed as a command that the robot system should follow. The AI ​​system itself can generate these questions and commands and thereby autonomously learn how to operate in a given situation and a given physical environment. In other words, by combining the VQA mechanism and the logical reasoning AI mechanism, an improved AI system can be achieved that can use logical reasoning to autonomously respond to complex questions beyond simply recognizing content in an image.

[0034] Current VQA AI systems are not capable of learning relationships and applying them to complex questions to generate logical reasoning-based responses to these complex questions. These limitations of existing VQA AI systems are addressed by the mechanism of the exemplary embodiment, which provides an improved AI system and corresponding AI models that implement a neural symbolic situation action transformer for VQA. The mechanism of the exemplary embodiment utilizes computer execution models that implement multiple AI computer models, i.e., machine learning and computer execution models that implement artificial intelligence mechanisms such as transformer-based models, neural networks, convolutional neural networks (CNNs), deep learning neural networks (DNNs), long short-term memory (LSTM)-based neural networks, etc., which implement situation inference using object / relationship detection in input video image sequences. Situation inference refers to learning relationships between objects, sequences of relationships, etc., about a specific situation represented in image data, and then using the learned relationships and sequences to infer new situations in newly received image data, making predictions to deal with these new situations and achieving desired results or generating desired responses to provided queries.

[0035] In one exemplary embodiment, an AI computer model called Situated Reasoning Bidirectional Encoder Representations from Transformers (SR-BERT) is provided. BERT is a transformer-based machine learning computer model originally developed for natural language processing using word embeddings and bidirectional context. Transformer-based machine learning computer models are deep learning models that employ an attention mechanism to weight each part of the input data differently in importance. Similar to RNNs, transformers deal with sequential input data, but unlike RNNs, transformers do not necessarily process the data in order. Rather, the attention mechanism provides context for any position in the input sequence. For example, if the input data is a natural language sentence, the transformer does not need to process the beginning of the sentence before its end. Rather, the transformer identifies the context that gives meaning to each word in the sentence. This feature allows for further parallelization compared to RNNs and reduces training time. The transformer computer model uses an attention mechanism without RNNs to process all tokens simultaneously and compute attention weights between them in successive layers.

[0036] In transformer computer models, the model's attention layer can access all previous states and weight them according to a learned measure of relevance, providing relevant information about distant tokens. For example, in natural language processing techniques, the attention mechanism of transformer models can be more clearly illustrated with the example of language translation, where context is essential for assigning meaning to words within a sentence. For instance, in an English-to-French translation system, the first word of the French output likely heavily depends on the first few words of the English input. However, in typical RNN or LSTM computer models, to generate the first word of the French output, the model is given only the state vector of the last English word. Theoretically, this vector can encode information about the entire English sentence, giving the model all the necessary knowledge. In practice, this information is often not well maintained by RNN or LSTM computer models. To address this problem, an attention mechanism can be added, in which the decoder is given access to the state vectors of all English input words, not just the last word, and can learn attention weights that indicate how much attention should be given to each English input state vector. Adding an attention mechanism to an RNN improves its performance.

[0037] In one exemplary embodiment, the SR-BERT AI computer model utilizes a dynamics transformer model that implements an attention mechanism of the transformer model to extract states from any preceding point in time along a sequence, where this sequence may be a sequence of image frames of input video data. The video parser parses the input video data, which consists of image frames having corresponding timestamp information. Each image frame is parsed to identify people, objects, and relationships represented within the image data of the image frame. These people, objects, and relationships are transformed into a situation graph data structure having nodes representing people and objects and edges representing relationships, so that each relationship can be either a person-object relationship or an object-object relationship. Thus, each portion of the input video data corresponding to each timestamp has a corresponding situation graph data structure. The timestamp information is maintained for each situation graph data structure, thereby generating a sequence of saturated graph data structures.

[0038] It should be understood that the symbolic visual inference performed by the mechanism of the exemplary embodiment is designed to perform multiple stages of logic, not just a single stage. Therefore, this symbolic visual inference mechanism needs to acquire multiple stages of inference capability through multiple hops of connections in graph-structured data. The exemplary embodiment employs situational graph-structured data extracted to explicitly represent not only entities but also facts about an image, including both identified entities (people, objects, relationships, etc.) and the connections that existed (e.g., who interacts with which, which relationships reflect which actions, etc.), so that the inference operation can be based on an explicitly partial connection graph in subsequent inference operations. If only identified entities were used without partial connections, the mechanism would not be able to successively perform multiple stages of inference that proceed step by step.

[0039] The situation graph data structure is input to a transformer-based action transition computer model comprising a situation graph encoder, a dynamics transformer model, and a graph sequence decoder. The situation graph encoder encodes the sequence of the situation graph data structure into a token sequence according to an encoding schema. During training of the dynamics transformer model, this encoding of the sequence of the situation graph data structure may be masked according to one or more mask data structures that identify portions of the token sequence that the dynamics transformer model will learn through the machine learning process. For example, certain patterns of actions within the token sequence may be masked so that the dynamics transformer model acts to predict these actions and learns through machine learning appropriate attention weights to apply to generate correct predictions of such actions.

[0040] The dynamics transformer model operates on the token sequence output by the situation encoder to generate a predicted token sequence. That is, during the runtime operation of SR-BERT, the situation graph encoder may have missing tokens in the sequence, similar to masking during training of the dynamics transformer model. The dynamics transformer model predicts these tokens based on its machine learning of attention weights for state information about each token in the input token sequence from the situation encoder. Furthermore, the dynamics transformer model may predict the next set of tokens in the predicted token sequence for timestamps that are not present in the input video data. For example, during testing (unlike masking during training), selected token types may be masked, for instance, action token positions may be demasked to reserve space for possible occurring actions in each situation frame. Each masked token, for example, each unmasked action token, corresponds to a discrete class, for example, an action class, and a predefined class, for example, a range of values ​​from the values ​​of the predefined action class to a "MASK" value (which does not need to be filled in) or "NA" (no action). Therefore, in the prediction stage, tokens with "MASK", such as action tokens, need to be predicted by the dynamic transformer model.

[0041] Therefore, the dynamics transformer model predicts the token sequence based on its machine learning training of attention weights about state information about tokens in the input token sequence. The output of the dynamics transformer model is the predicted token sequence, which is then input to the graph sequence decoder. The graph sequence decoder generates a predicted hypergraph data structure based on the predicted token sequence, where the predicted portion of the predicted token sequence, i.e., the portion that was not present in the input token sequence from the situation encoder, may be represented as hyperedges linking one or more situation graphs of the situation graph data structure to other situation graphs of the same or different situation graph data structures. The hyperedges represent the predicted relationships between situation graphs of various situation graph data structures. Each hyperedge in the predicted hypergraph data structure connects multiple situation graph data structures. Multiple hyperedges may overlap, and nodes in a situation graph data structure (hereinafter referred to as a "situation graph") may be shared within the predicted hypergraph data structure (hereinafter referred to as a "hypergraph"). The entire dynamic process within a situation, including one or more situation graphs, may be represented as a set of one or more consecutive and overlapping hypergraphs. Formally, a hypergraph H is a pair H=(X,E), where X is a set of nodes about objects or people that appeared in the input frames of the input video data, and E is a set of non-empty hyperedges connecting one or more situation graphs, representing the relationships between nodes of the connected situation graphs. The hypergraph represents a type of symbolic inference performed by the SR-BERT AI computer model. In some exemplary embodiments, hyperedges can connect a particular type of class, e.g., an action class, to different situation graph data structures, and different hyperedges can, in some cases, connect to the same situation graph.

[0042] The predicted hypergraph is input to the program executor, which receives the predicted hypergraph along with a program automatically and dynamically generated by the program parser. The program parser includes multiple trained machine learning computer models that operate on the input natural language question to generate predictions of a set of program modules to apply to the predicted hypergraph data structure in order to generate a response to the input natural language question. That is, a predefined set of program modules having different program module types is provided. The trained machine learning computer models learn the relationships between the input natural language terms or phrases of the input natural language question and specific ones of the predefined program modules. Therefore, when presented with an input natural language question, each of the trained machine learning computer models predicts one or more program modules to apply to the input natural language question.

[0043] One or more predicted program modules are combined into an executable program for an input natural language query, and the executable program has a nested structure of one or more program modules. For example, after a query versus program parsing by a program parser, there exists a set of predicted program modules, e.g., a [Filter\_Actions\_with\_Obj] with program separators in the form of a sequence (e.g., (), [], {}). The program executor has predefined behaviors and parameter amounts for each program module / function. The program executor may execute a sequence of program modules using a predefined ordering algorithm, such as a first-in, first-out (FIFO) algorithm. Such behavior translates the sequence of program modules into a nested program that can be processed by the program executor.

[0044] Therefore, the combination of program modules predicted by the program parser's machine learning computer model is input to the program execution unit, which is configured to execute the program modules in a nested manner. Each program module predicts a set of outputs, which are then used as input to the next program module in nested order. In this way, the outputs are iteratively paired and reduced until they become the final response to the input query. The program modules operate on the predicted nodes, edges, and hyperedges of the hypergraph data structure to generate their outputs, which are then provided as input to the next program module in nested order until the final program module is executed, and the output of the final program module is then used as the final response to the input query.

[0045] It should be understood that, by generating a predicted hypergraph, the AI ​​computer model mechanism of the exemplary embodiment is capable of summarizing the situation represented in the input video data, generating predictions of the relationships between entities represented in the input video data, and providing symbolic inference of the situation represented by the hypergraph and hyperedges. In addition, logical inference questions such as questions about interaction, sequence, prediction, or feasibility may be evaluated by providing a program parser that takes natural language questions and represents them as a sequence of program modules to be executed on the hypergraph data structure. For example, given input video data showing a sequence of a person in an office situation with a desk, book, computer, clothing, food, etc., the response is not limited to questions about what is specifically shown in the input video data, such as "What color is the person's shirt?", but rather questions such as "Which object was eaten by the person?" (interaction question), "Which object did the person pick up after closing the book?" (sequence question), "What will the person do next?" (prediction question), and "Which other objects could be placed down by the person?" (feasibility question). Interaction-type and sequence-type questions refer to those shown in the images, but the difference between the exemplary embodiment and existing VQA mechanisms lies in "how questions are inferred" rather than "how questions are asked." That is, the exemplary embodiment performs hierarchical inference, using lower-level entities and interaction relationships to infer higher-level actions using relational inference, rather than inferring actions by using appearance alone. Therefore, all questions in the exemplary embodiment are not limited to using only explicitly represented entities and relationships in the video input, but are processed using a predicted nested program that operates on hypergraph data structures.

[0046] The question-answering process enables the execution of logical inference on input video data, which can later provide responses that may prompt further analysis or output for human use. However, it should be understood that the operation of the exemplary embodiments is automated and performed using complex machine learning computer models and AI mechanisms. Accordingly, exemplary embodiments may be a specifically configured computing system, a method implemented by such specifically configured computing system, or a computer program product comprising software logic loaded into the computing system to specifically configure the computing system to implement the mechanisms and functions described herein, or a combination thereof.

[0047] Whether described as a system, a method, or a computer program product, the exemplary embodiments described herein should be understood to specifically focus on improved computing tools and methodologies implemented by such improved computing tools. In particular, the improved computing tools of the exemplary embodiments specifically provide an AI computer system that implements multiple machine learning-based computer models, as well as a video parser and a program executor, to automatically extract a situation graph from input video data, convert the situation graph into an encoded token sequence, perform predictions on the encoded token sequence to generate a predicted token sequence, generate a predicted hypergraph based on the predicted token sequence, generate a program based on a natural language question, and execute the generated program on the generated hypergraph to identify a response to the natural language question. The improved computing tools implement mechanisms and functions such as a video parser, a transformer-based action transition model, a program parser, and a program executor, which cannot actually be executed by humans outside of a technical environment, or even with the assistance of a technical environment, such as mental processes. The improved computing tool provides a practical application of the methodology, at least in that it is capable of automatically responding to logical reasoning questions concerning concepts outside of the in-frame content of an input video sequence through computer artificial intelligence. Therefore, the improved computing tool is capable of predicting sequences of objects, people, and relationships between these objects / people, which responds to logical reasoning questions.

[0048] Furthermore, while humans can be the subjects of the input video data and can utilize the output generated by the mechanism of the exemplary embodiment, the exemplary embodiment itself does not concern itself with actions performed by these humans, but rather with specific configurations of computer models, logic, and functions that are specifically performed on the input video data by a particular artificial intelligence computer mechanism, such as an improved computing tool. While the exemplary embodiment may provide output that ultimately assists humans in responding to logical questions concerning interaction, sequence, prediction, or feasibility, the exemplary embodiment of the present invention does not concern itself with actions performed by humans utilizing the results of the improved computing tool, but rather with specific operations performed by a particular improved computing tool of the present invention, thereby facilitating the automated processing of input video data for responding to logical questions through artificial intelligence. Therefore, the exemplary embodiment does not organize any human activity, but rather concerns the automated logic and functions of the improved computing tool.

[0049] It should be further understood that the exemplary embodiments described herein implement and use artificial intelligence (AI) systems or cognitive systems, or both. The purpose of these AI systems or cognitive systems, or both, is not to replace human intelligence, but to extend it. These AI systems or cognitive systems, or both, are designed to enhance and expand human capabilities and potential through specific improved computer tools and improved computer tool operations. These improved computer tools perform operations at speeds, complexities, and volumes that are not actually possible for human intelligence to perform. Such AI systems or cognitive systems, or both, may mimic achieving results similar to those of human intelligence, but they do so by using different methodologies and mechanisms inherent in computer tools that are not the same as any human mental process or manual labor, due to the inherent differences between the modes in which computing devices operate and the modes in which the human mind operates.

[0050] AI systems or cognitive systems, or both, implemented by exemplary embodiments, may operate on various types of data, which may include input video data containing personal or private information of individuals, for example, persons shown in the input video data. While AI systems or cognitive systems, or both, may operate on such personal or private information, AI computing systems or cognitive computing systems, or both, may implement various mechanisms (not specifically shown in the drawings) for maintaining the privacy and security of individuals' personal or private information, and may implement principles of trustworthiness and transparency with respect to the security of such personal or private information. These principles of trustworthiness and transparency recognize that any person whose data is tracked and shared should always be given the option to opt in or opt out of such tracking and sharing of their personal or private data. These principles of trustworthiness and transparency recognize that any person whose data is tracked and shared should always have control over how their data is used, which entities have access to it, and their ability to delete it. Furthermore, this principle of trustworthiness and transparency recognizes that an individual's personal or private data should be kept secure from cyber threats, and that such data should not be used for purposes such as government tracking and surveillance without the specific authorization of the individual who is the ultimate owner of the personal, private, or both data.

[0051] Therefore, an AI system or a cognitive system, or both, may operate on any such personal or private information that may be represented in the input video data, and these AI system mechanisms or cognitive system mechanisms, or both, may implement features that allow individuals to opt in or opt out of the use of their personal / private data, authorize entities to access their personal / private data, and provide security mechanisms to ensure that an individual's personal / private data is secure from cyber threats. These mechanisms do not require individuals to relinquish ownership of their personal / private data or the insights derived from it in order to enjoy the benefits of the exemplary embodiments. The exemplary embodiments may facilitate and utilize the free movement of data across one or more data networks that may extend organizational and geopolitical boundaries, but such free movement of data is carried out using mechanisms that facilitate the security of personal / private data flows.

[0052] With the foregoing in mind, before continuing the description of exemplary embodiments and various specific aspects of the improved computer operations performed by such exemplary embodiments, it should first be understood that throughout this description, the term “mechanism” is used to refer to elements of the present invention that perform various operations, functions, etc. “Mechanism” may, as used herein, be an implementation of a function or aspect of an exemplary embodiment in the form of an apparatus, procedure, or computer program product. In the case of a procedure, the procedure is implemented by one or more devices, apparatus, computers, data processing systems, etc. In the case of a computer program product, logic represented by computer code or instructions embodied within or on the computer program product is executed by one or more hardware devices to implement a function associated with a particular “mechanism” or to perform an operation associated with a particular “mechanism.” Therefore, the mechanisms described herein may be implemented as special hardware, software executed on the hardware that configures the hardware to implement special functions of the present invention that the hardware cannot otherwise perform, software instructions stored on a medium that make the instructions easily executable by the hardware, thereby specifically configuring the hardware to perform the functions and specific computer operations described herein, or any combination thereof.

[0053] This specification and claims may use the terms “one,” “at least one of,” and “one or more of” with respect to certain features and elements of exemplary embodiments. It should be understood that these terms and phrases are intended to state that at least one of certain features or elements present in a particular exemplary embodiment is present, but more than one may be present. That is, these terms / phrases are not intended to limit this specification or claims to a single feature / element present, nor to require the presence of multiple such features / elements. On the contrary, these terms / phrases require only at least a single feature / element, and multiple such features / elements may be present within the scope of this specification and claims.

[0054] Furthermore, it should be understood that the use of the term "engine," when used herein in connection with describing embodiments and features of the present invention, is not intended to limit any particular implementation to realizing or performing any action, stage, process, etc., that is caused by, or performed by, or both, an engine. An engine may, but is not limited to, software, specialized computer hardware and / or firmware, or any combination thereof, running on computer hardware that performs a specified function, including, but is not limited to, the use of any general-purpose processor or specialized processor or both, loaded into or stored in machine-readable memory and executed by a processor, and combined with appropriate software that specifically configures the processor to perform a particular function of an exemplary embodiment. Moreover, any name associated with a particular engine is for convenience of reference unless otherwise specified and is not intended to limit it to a particular implementation. In addition, any function belonging to a certain engine may be performed equally by multiple engines, may be incorporated into or combined with the function of another engine of the same or different type, or may be distributed across one or more engines of various configurations.

[0055] In addition, it should be understood that the following description uses several different examples of various elements of the exemplary embodiment to further illustrate the exemplary implementation of the exemplary embodiment and to aid in understanding the mechanism of the exemplary embodiment. These examples are intended to be non-limiting and do not exhaust the various possibilities for implementing the mechanism of the exemplary embodiment. It will be apparent to those skilled in the art who have considered this specification that there are many other alternative implementations of these various elements that can be used in addition to or instead of the examples provided herein without departing from the spirit and scope of the invention.

[0056] The present invention may be a system, method, or computer program product, or a combination thereof. The computer program product may include one (or more) computer-readable storage media having computer-readable program instructions that cause a processor to execute an aspect of the present invention.

[0057] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of those described above. A non-exhaustive list of more specific examples of computer-readable storage media includes, namely, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved raised structures recording instructions, and any suitable combination of those described above. When used herein, computer-readable storage media should not be interpreted as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or transient signals such as electrical signals transmitted through wires.

[0058] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers such computer-readable program instructions for storage in a computer-readable storage medium within each computing / processing device.

[0059] The computer-readable program instructions that perform the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, where one or more programming languages ​​include object-oriented programming languages ​​such as Java®, Smalltalk®, C++, etc., and conventional procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially executed on the user's computer as a standalone software package, partially executed on the user's computer and partially executed on a remote computer, or fully executed on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), and the connection may be to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may be personalized by executing computer-readable program instructions using state information of computer-readable program instructions in order to perform an aspect of the present invention.

[0060] Aspects of the present invention are described herein with reference to flowcharts or block diagrams, or both, of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram, or both, and any combination of blocks in a flowchart or block diagram, or both, can be implemented by computer-readable program instructions.

[0061] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing device to generate a machine, so that instructions executed via the processor of the computer or other programmable data processing device create means for implementing functions / operations specified in one or more blocks of a flowchart or block diagram, or both. These computer-readable program instructions may also be stored in a computer-readable storage medium, which can instruct a computer, a programmable data processing device, or other device, or a combination thereof, to function in a particular manner, so that the computer-readable storage medium storing the instructions contains a product containing instructions that implement modes of functions / operations specified in one or more blocks of a flowchart or block diagram, or both.

[0062] Furthermore, computer-readable program instructions may be loaded into a computer, another programmable data processing device, or another device to execute a series of operational steps on the computer, another programmable device, or another device, thereby generating a computer implementation process in which the instructions executed on the computer, another programmable device, or another device implement the functions / operations specified in one or more blocks of a flowchart or block diagram, or both.

[0063] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions that implement a specified logical function. In some alternative implementations, the functions described in a block may be performed in an order different from the order shown in the drawings. For example, two blocks shown consecutively may actually be executed substantially simultaneously, and blocks may be executed in reverse order depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, or both, and any combination of blocks in a block diagram or flowchart, or both, may be implemented by a dedicated hardware-based system that performs a specified function or operation, or a combination of dedicated hardware and computer instructions.

[0064] As discussed above, the exemplary embodiment provides an improved computing tool and improved computing tool operation that provides an artificial intelligence mechanism for performing video question answering (VQA) using logical reasoning-based question answering. The exemplary embodiment implements an improved computing tool comprising a video parser, a transformer-based action transition model, a program parser, and a program execution unit. Furthermore, the exemplary embodiment operates on data structures generated by these elements, such as situation graph data structures, token sequences, and hypergraph data structures, and on a program module that executes a program generated on the generated predicted hypergraph data structure to generate responses to logical reasoning questions regarding input video data parsed by the video parser.

[0065] Figure 1 is a block diagram showing the main operating components of an improved situational reasoning transformer (SRT)-based computing tool according to one exemplary embodiment. As shown in Figure 1, an AI system 100 implementing an SRT-based architecture uses a video parser 120 to represent input video data 110 received from a video data source computing system 102 as a set of data representations 121, 122, and 123 about human entities, objects, or both, and the relationships between these entities. For example, the video parser 120 may be a perception module having a set of recognition models 124-127, which take a human-centered or object-centered representation of a given situation s at time t, for example, each timestamp t may have one or more image frames from the input video data 110 defining the situation s for that timestamp. To simplify object / person recognition, an object detection bounding box mechanism is used to locate objects and humans in the input video data. The recognition models 124-127 of the video parser 120 include an object detector 124 that extracts a visual representation for each object or person in the input video data 110, a relation detector 125, and a pose parser 126 that extracts a skeletal representation of human motion. The video parser 120 may include, for example, an object detection bounding box mechanism that detects objects and people present in the input video data. The relation detector may include a pipeline that predicts a predicate for each entity pair, for example, by using a softmax distribution. The pose parser 126 may utilize multi-domain, multi-person, or both pose estimation to identify key points of a person or object structure and determine the pose of the person / object based on these key points. Any type of object, person, and relation parsing mechanism may be used without departing from the spirit and scope of the invention.

[0066] An action recognizer 127 may be implemented to recognize actions visible within the input video data 110, in contrast to unvisible actions not represented in the input video data 110 itself. The action recognizer 127 may utilize a neural network architecture to analyze the input video data and classify the input data for specific actions present within the input video data. These visible actions are multi-action inference situations, such as feasibility and preconditions for sequence questions, which are addressed by mechanisms in exemplary embodiments as described herein. The video parser 120 may be trained using a machine learning process on a set of training video data. The library of training video data includes various input video data sequences representing different human-object interventions in different environments. The training video data may be randomly sampled from a library of training videos, which can be stored in a video source computing system 102, to obtain visual features, object categories, human poses, and bounding box positions for various types of video data sequences.

[0067] The video parser 120 identifies people, objects, and relationships between people, objects, or both within the input video data 110 for various situations corresponding to different timestamps t1 to tn. For example, for one or more image frames of the input video data 110 corresponding to timestamp t1, a set of tuples of people, objects, and relationships may be generated. These tuples identify either person-object relationships or object-object relationships. For example, if the situation represented at timestamp t1 is a man picking up a book, the tuple may be (man,book,pick up), where "man" is a person, "book" is an object, and "pick up" is the relationship between the person and the object. Furthermore, if the represented situation also includes a cup sitting on a desk, the tuple may be of type (cup,desk,on), etc.

[0068] The people, objects, and relationships identified through the operation of the video parser 120 are used to generate a situation graph data structure 131, one for each situation corresponding to a certain timestamp in the input video data, and these situation graph data structures 131 are input to the transformer-based action transition model 130. The situation graph data structure 131 may include a graph having nodes representing entities and edges representing relationships between entities, i.e., entity relationships. In some exemplary embodiments, these graph data structures may be directed graph data structures that specify the direction of the relationships represented by the edges. In addition, time information regarding the timing of relationships identified in the input video data is maintained for objects and relationships, and for example, the timestamp associated with an individual video segment, e.g., one or more image frames, is associated with the corresponding situation graph data structure representing that video segment, and a different situation graph data structure is generated for each video segment in the sequence of video segments of the input video data 110, e.g., for each timestamp of the input video data 110. Therefore, the video parser 120 detects entities and their relationships that define a situation present in the input video data, where the situation describes entities, events, moments, and environments, and corresponds to trimmed portions of a video having multiple consecutive or overlapping actions and interactions.

[0069] The transformer-based action transition model 130 implements a situation inference transformer (SRT) mechanism comprising a situation encoder 132, a dynamics transformer model 134, and a graph sequence decoder 136. The SRT mechanism learns an implicit transition process to predict missing information in the form of generalizations of entity relationships represented in the situation graph data structure 131 as one or more predicted hypergraph data structures 140, where the hypergraph is a generalization of one or more situation graph data structures, in which hyperedges can connect any number of nodes (or vertices), rather than edges connecting only two nodes (or vertices) as in the situation graph data structure. In other words, SRT includes logic 132-136, trained through a machine learning training process, to predict the relationships between nodes of individual situation graph data structures 131, thereby generating predicted hypergraph data structures 140, where the hyperedges 141, 142 of the predicted hypergraph data structure 140 connect one or more nodes of a situation graph data structure to one or more nodes of the same or different situation graph data structures. SRT uses a unified schema to describe dynamic processes in real-world situations in the form of these predicted hypergraph data structures. The predicted hypergraph data structures represent the actions, internal relationships, and their hierarchical structure within a situation represented in the input video data 110. Each sequence of video segments corresponding to a given situation is a set of situation graph data structures 131 representing in-frame relationships, e.g., person-object relationships or object-object relationships. On the other hand, each predicted hypergraph data structure 140 also shows predicted relationships within and between situations represented in the input video data 110, as well as in-frame relationships.

[0070] The situation encoder 132 of the transformer-based action transition model 130 takes in the situation graph data structure 131 and encodes them into a token sequence 133. The situation encoder 132 is designed for situation representation. The situation encoder 132 encodes the connected situation graph data structure 131 as a hierarchically structured token sequence 133. The time order of the situation, represented by the timestamps associated with the image frames in the input video data 110, is the token sequence. 0 ,...,s t > is organized, where s is the situation for the corresponding timestamps 0 to t. Each situation s at time t corresponds to two segments, namely the situation graph segment and the action hyperedge segment.

[0071] As described above, the situation graph data structure 131 segments are atomic tokens, i.e., people, objects, and spatial or contact relationships. <h i ,o i ,r i Includes a set of human-object or object-object interaction tuples having >. All connected human or object nodes are involved in a situation graph data structure segment encoded by the situation encoder 132, and each set of situation segment tokens in the token sequence 133 generated by the situation encoder 132 is padded with zero tokens for a unified number of situation segment tokens. Each token in the set of one or more action tokens for each situation graph data structure connecting the hyper-edge segments, e.g., situation graphs, is of a specific action type a j ​Each token represents a sum that captures information from the following types of embedding vectors: (1) token embedding: appearance representation for an object, posture representation for a person, or distributed embedding for an action or relation category; (2) type embedding: indicating the token type; (3) hyperedge embedding: indicating the hyperedge location within each situation; (4) situation embedding: recording the situation time order; and (5) segment embedding. Situation and segment embeddings are used to indicate the situation time order and segment type for each token. Separation embeddings mark the boundaries of situations and segments within the token sequence. The constructed token sequence 133 describes the structure of the set of situation graph data structures 131.

[0072] The dynamics transformer model 134 is designed to receive a constructed token sequence 133 from a situation encoder 132 and dynamically predict action states or relationships by learning the relationships between input data types in a given situation. The dynamics transformer model 134 includes a multi-layer stacked transformer architecture with downstream task predictors. The stacked transformer blocks are used with multiple heads to compute a self-attention score for the input token sequence 133. The attention describes the connection of each potential relationship between two nodes in the situation graph data structure 131, e.g., an action hyperedge or a human-relationship-object triplet. Since the self-attention internal structure of the transformer corresponds to token pairs, the entire attention across input tokens performs dynamic relationship modeling. The connections of neighboring nodes are summed up into a single node. The aggregated effect is stored in the current state at time t and applied for predictions about missing information in the current state or the state at the next time point t+1. Such dynamic attention modeling treats all possible relationships as implicit connections. While this is more robust, the relationships are unknown or some of the visual cues are unreliable. This dynamics transformer model 134 is also used to predict some unvisualized situations for predictive or feasibility questions.

[0073] The dynamics transformer model 134 generates a predicted token sequence 135 based on the constructed input token sequence 133, which includes predictions for any missing information in the input token sequence 133, such as tokens masked during training, and predictions for the token sequence in one or more subsequent timestamps after the last timestamp t of the input video data 110. The predicted token sequence 135 is input to the graph sequence decoder 136.

[0074] There are three self-supervision tasks implemented by graph sequence decoders: action type prediction, human-object relationship type prediction, and masked token modeling (for objects or humans). The first two tasks predict action hyperedges or relationships using multilayer perceptrons (MLPs) with a pooled global representation of all states in previous situations, e.g., a situation graph data structure with previous timestamps corresponding to earlier situations in a video sequence, using machine learning computer models such as RNNs and CNNs (classifiers).

[0075] While recognition models, such as the object detector 124 and human pose detector 126 in the video parser 120, are assumed to achieve high accuracy, some objects or human poses in the input video data may be blurred or invisible. Masking token modeling aims to improve representational robustness by reconstructing their embedding vectors. The graph sequence decoder 136 takes in the input predicted token sequences 135 from the dynamics transformer model 134 and applies these classifiers and masking token modeling to these predicted token sequences 135 to generate a predicted hypergraph 140.

[0076] Therefore, the graph sequence decoder 136 generates a predicted hypergraph 140 having hyperedges 141 and 142 that link one or more nodes of parts 143 to 146 of the situation graph data structure 131. For example, the hyperedges 141 to 142 of the predicted hypergraph 140 correspond to the predicted tokens in the predicted token sequence 135. The predicted hypergraph 140 is input to the program execution unit 170, which executes the generated program on the predicted hypergraph 140 to generate a response 180 to the input question 150.

[0077] The AI ​​system 100 also includes a language / program parser 160 that parses an input natural language question 150 and generates an executable program corresponding to the natural language question 150. The language / program parser 160 includes a natural language parser 166 that parses the natural language question 150 in a manner generally known in the art and provides the parsed natural language content to an encoder 161 to generate an encoding of the input natural language question 150 in a manner generally known in the art. The encoding of the natural language question 150 is then input to a group of machine learning models 162-164 that are trained to predict which program module should be used to deal with response generation to the natural language question based on a set of input features of the question encoding from the encoder 161. These machine learning models 162-164 may be implemented, for example, as long short-term memory (LSTM) decoders with an attention mechanism. Each of the machine learning models 162-164 outputs a prediction of a program module or program function, or both, from predefined program modules in the program module library 190, which should be applied to a predicted hypergraph to generate a response to the input question 150. These machine learning models 162-164 may be trained using a training set of natural language input questions that are processed to generate program module predictions, which are then corrected according to ground truth program module information, thereby iteratively training the machine learning models to generate improved predictions about which program modules correspond to which linguistic elements of the input natural language question.

[0078] The predicted program modules / functions are provided to the program execution unit 170, which uses the nested program modules / functions to perform symbolic inference across the predicted hypergraph data structures 140, actions, or other data types. Inference is a top-down process for a given input question 150. The program execution unit 170 takes all predicted hypergraphs as initial input and starts with the first data type mentioned in the input question 150. The program execution unit 170 then iteratively goes through all valid actions and outputs a response in the final stage.

[0079] Therefore, given input video data 110 containing multiple input images for different timestamps of a video sequence, the AI ​​system 100 of the exemplary embodiment generates a predicted hypergraph data structure that includes predictions for missing information in the context graph data structure of the input video data 110, and predictions for one or more future timestamps not represented in the input video data 110. These predicted hypergraphs 140 include hyperedges 141, 142 that represent this missing information, or the predicted relationships between nodes in the context graph data structure of the timestamped video data, or both. The AI ​​system 100 of the exemplary embodiment also generates a program having nested program modules / functions that can receive an input natural language question, respond to this input natural language question, and run on the predicted hypergraph to generate a response to this input natural language question. These input natural language questions may be of a type that requires logical reasoning, such as questions relating to interaction, sequence, prediction, or feasibility assessment.

[0080] It should be understood that the AI ​​system 100 of the exemplary embodiment utilizes a machine learning model that is trained through a machine learning process to learn the relationship between patterns of input features and specific output predictions. Thus, during training, the AI ​​system 100 may be provided with training input video data 110, where annotations for people, objects, and relationships may be known, for example, ground truth values. The model organizes the input graph data into a structured token sequence consisting of graph nodes and relationships. The situation encoder 132 may generate token sequences using various masks that mask specific tokens in the token sequence, corresponding to the type of tokens for which machine learning training is training a dynamics transformer model 134 to predict. For example, machine learning may iteratively apply different masks to a token sequence 133 to generate masked token sequences, which are then input to a dynamics transformer 134 that operates to predict a predicted token sequence containing tokens about the masked tokens. Based on ground truth comparison, the error in the predictions generated by the dynamics transformer model 134 is determined and used to adjust the attention weights, etc., of the dynamics transformer model 134, thereby reducing the error in the predictions of the dynamics transformer model 134. This process may be repeated for multiple different input video data sequences 110 and for multiple different masks.

[0081] Similar training can be applied to the machine learning model of the graph sequence decoder 136. It should be understood that the entire transformer model (encoder-decoder-classifier) ​​may be trained using an end-to-end strategy, rather than training each distinct element separately, e.g., separately training the encoder, separately training the decoder, and separately training the classifier.

[0082] During runtime operation, after the dynamics transformer model 134 and graph sequence decoder 136 have been trained, these elements operate on a new input video data sequence to predict any missing tokens or token sequences, or both, at future time points not represented in the input video data 110, thereby generating a predicted hypergraph data structure. That is, the new input video data may not be masked using a mask, while the new input video data may have missing information, such as blurred image data, which can be predicted by the dynamics transformer model 134. Furthermore, the dynamics transformer model 134 may operate to predict a sequence of tokens for the next timestamp t+1 that is not represented in the input video data, so that the corresponding predicted hypergraph can be used to answer prediction questions or feasibility questions.

[0083] In addition, during training, the language / program parser 160 may be presented with a set of training input questions corresponding to different logical reasoning questions that will be translated into executable programs to be executed on the predicted hypergraph. The machine learning process for training the model implemented by the language / program parser 160 may use a similar iterative process that adjusts attention weights, etc., to reduce errors in predictions using a ground truth error evaluation mechanism. That is, the language / program parser 160 may operate on input natural language questions to predict program modules / functions that will be combined to generate a program about the input questions, these predictions are compared to ground truth to determine an error, and then attention weights may be adjusted over iterations to reduce this error. Once trained, the language / program parser 160 may be applied to new input questions to generate a program that the program execution unit 170 will execute on the predicted hypergraph 140 to produce the resulting response 180.

[0084] Therefore, the exemplary embodiment provides an improved computing tool and improved computing tool operation that predicts relationships not explicitly shown in the input video sequence and responds to logical reasoning questions based on these predictions. The exemplary embodiment uses the predicted token sequence to generate a predicted hypergraph from a set of input situation graph data structures and uses a predicted program to run on the predicted hypergraph to generate responses to such logical reasoning questions. Such logical reasoning questions may take the form of interactions spanning multiple situations, sequences of interactions, questions asking about predictions of what will happen next, and questions about the feasibility of a particular action, all of which cannot be answered by existing video question answering (VQA) mechanisms that are limited to responding to questions about what is explicitly shown in the video images themselves.

[0085] As mentioned above, the AI ​​system of the exemplary embodiment implements a video parser 120 that identifies people, objects, and relationships actually seen in the input video data 110, which is then used as the basis for generating a situation graph data structure for each timestamp in the input video data 110. Furthermore, input questions for logical reasoning may be encoded by a language / program parser 160 and responded to by a program execution unit 170 based on the execution of a generated program against the predicted hypergraph data structure. Figure 2 is an illustrative diagram showing an exemplary situation shown in the input video data, along with the corresponding situation graph data structure and hyperedges for responding to the logical reasoning questions, according to one exemplary embodiment. Furthermore, Figure 2 shows examples of logical reasoning questions, their encoding, and the corresponding correct / incorrect answers to the logical reasoning questions.

[0086] As shown in Figure 2, the video sequence 210 includes a pre-adjustment section 211, a process section 212, an end section 213, and an unseen or future section 214. The pre-adjustment section 211 includes a depiction of the environment prior to human interaction with objects within the environment. The process section 212 includes one or more situations corresponding to timestamps representing multiple atomic actions in the input video data, such as interactions between humans and objects within the environment. The end section 213 represents the portion of the video sequence where no more interactions exist. The unseen or future section 214 includes actions not represented in the input video data sequence 210.

[0087] Each situation represented in the input video data sequence 210 may be represented as a situation graph data structure 220, 222, 224, 226. In some exemplary embodiments, these situations correspond to timestamps in the input video data sequence 210. Situation graph data structures 220-226 include nodes representing objects / people present in the corresponding images of the situation and edges representing the relationships between these objects / people. It should be understood that some objects / relationships may be masked or not represented in the situation and may be grayed out. For example, in situation graph data structures 220-226, the object "phone" and the relationship between the person node and the masked or invisible object "phone" may be grayed out, meaning that they are not visible or are masked in the input video data sequence 210.

[0088] Therefore, for example, the situation graph data structure 220 includes nodes for “person,” “bottle,” “book,” “table,” and “telephone.” The situation graph data structure 220 further includes edges indicating that the person is holding the bottle, that the person is facing the table, that the person is looking at the telephone, that the book is on the table, and that the bottle is near the book. The situation graph data structure 224 further includes a node for “trash,” and an edge indicating that the person is facing the trash. The situation graph data structure 226 further includes a node for the “oven” object, and an edge indicating that the trash is near the oven. These objects, people, and relationships are detected by the video parser using known video parsing mechanisms for object identification and relationship identification, as discussed above.

[0089] Through the mechanism of the exemplary embodiment, the AI ​​system of the exemplary embodiment generates a predicted hypergraph that provides predicted hyperedges between nodes of the same or different situation graph data structures. For example, the mechanism of the exemplary embodiment may predict that a missing edge in the situation graph data structure is a hyperedge 230, which represents a person being able to "drink from" the bottle. Furthermore, the exemplary embodiment may predict that another missing edge in the situation graph data structure 222 may be a person being able to "put down" the bottle, which is represented by a hyperedge 232. Furthermore, hyperedges 234 and 236 represent other relationships or actions that may be predicted by the mechanism of the exemplary embodiment, such as a person being able to "pick up" the book 234.

[0090] Figure 2 also shows an example of a logical reasoning question that may be processed by the mechanism of the exemplary embodiment for an input video data sequence 210. For example, an interaction logical reasoning question that can be input to a language / program parser and translated into a program that is executed on a predicted hypergraph to generate a response to the input question may be of the type "What did the person do while he was sitting across from the table?". The elements in the diagram, such as Q:[V,O], S:[P,T_R,S_R,O], etc., specify the data types in the question (Q) and situation (S). As shown in Figure 2, processing a question through the mechanism of the exemplary embodiment may generate a variety of responses, one of which is the correct answer and is represented in ground truth during training to train a model of the AI ​​system to generate that correct answer. For example, for an interaction question, the correct answer is c "Drink from the bottle".

[0091] An example of a sequence logic reasoning question might be of the type, "What happened before the person picked up the bottle?", with the correct answer being, "Put the bottle down." Similarly, an example of a predictive logic reasoning question might be of the type, "Which object will the person open next?", with the correct answer being d, "The door." An example of a feasibility logic reasoning question might be of the type, "Which object can the person pick up after putting the bottle down?", with the correct answer being, "The book." The correct answers may be specified as ground truth responses during training, and may be responses generated by the mechanism of the exemplary embodiment during runtime operation of a VQA using the logic reasoning AI system of the exemplary embodiment.

[0092] Figure 3 is an illustrative diagram of the structure of a hypergraph token sequence for representation in SRT encoding according to one exemplary embodiment. As shown in Figure 3, each token in sequences 1 and 2 is represented by a column having multiple rows, each row corresponding to a different embedding that embeds information about the element of the input image sequence represented by the token. For example, each token has a token embedding (vector), a type embedding, a triplet embedding, a hyperedge embedding, and a context embedding. The integer values ​​in Figure 3 represent either an order ID (for position tokens) or a class ID (for other tokens), and the value "MAX" means not applicable, i.e., "NA". The letter symbols indicate the type of token, e.g., action (A), person (P), relationship (R), object (O), or separator (M).

[0093] Figure 4 is an exemplary program module list for automatic program generation by a program parser, according to one exemplary embodiment. In the depiction of Figure 4, “Action,” “Verb,” “Object,” and “Relationship” are lists of corresponding entities from the predicted hypergraph data structure corresponding to these basic data types. The action data type may be a tuple containing the action class, start time, and end time, e.g., (c102,15.0,22.1). The verb, object, and relationship may be the corresponding verb, object, or relationship within a given vocabulary, e.g., “Pick up,” “Laptop,” and “Over.” “Situation” is a combination of basic data types. “Set” represents a set of basic verbs, objects, relationships, or actions.

[0094] Program modules and corresponding functions may correspond to a given program module in a program module library, which can be selected using the language / program parser's prediction mechanism based on the processing of an input natural language query. As shown in Figure 4, a program module / function has a program module type, e.g., input, element, filter, query, and logic, and a corresponding program module or function is defined by a function name and a description that describes what action the program module / function performs on a predicted hypergraph data structure. For example, an input type program module might include a "situation" program module or function whose action is to return all situations represented in the predicted hypergraph. Similarly, a filter-type program module / function may have multiple different program modules / functions, each performing a different type of operation as specified in its description. For example, the "Filter_Actions_with_Verb" program module / function selects an action from input actions having an input verb, where the input action is an action represented by an edge in a predicted hypergraph data structure, and the verb is the verb specified in the input natural language question, or a verb associated with the verb specified in the input natural language question, such as a synonym, or both.

[0095] The input and output columns in Figure 4 show the inputs to the corresponding program modules / functions and the outputs generated by those programs / functions. For example, the input to the program module "Filter_Actions_with_Verb" is (action, verb), where the action is obtained from the expected hypergraph data structure input to the program execution unit, and the verb is the verb (or synonym or both) specified in the input natural language query. The "Filter_Actions_with_Verb" program module / function outputs an action that includes the input verb.

[0096] These program modules / functions may be predefined in a program module / function library and may be selected to be combined into an executable program based on a specific input natural language question processed by a machine learning computer model of a language / program parser in an exemplary embodiment. Therefore, different combinations of program modules / functions may be combined to address a particular input question based on a specific input question received. The selection of a particular program module / function is based on machine learning training of a machine learning computer model that predicts the program module / function that is most appropriate to respond to the input question.

[0097] The predicted program modules / functions are input to the program executor, which combines the program modules / functions into an executable program containing nested execution sequences of these program modules / functions. Each level of the nested execution sequence produces output, and once this is done, the next level of the nested execution sequence operates. In this way, the output is iteratively reduced until a single final response to the input question is obtained.

[0098] Figure 5 shows an example of an automatically generated program for four exemplary question types according to one exemplary embodiment. Figure 5 shows examples of interaction questions, sequence questions, prediction questions, and feasibility questions. For each of these question types, an example of a corresponding program generated by the program executor based on the selection of a program module / function from the predefined list of program modules / functions in Figure 4 is shown. As can be seen from Figure 5, each of these programs includes a nested program module / function architecture in which the output of a program module / function is provided as input to the next program module / function in a nested architecture.

[0099] Figure 6 is a flowchart outlining the exemplary operation of an improved computing tool according to one exemplary embodiment. The operation shown in Figure 6 assumes that the machine learning computer model for video question answering (VQA) using the logical reasoning AI system of the exemplary embodiment is trained through a machine learning process to generate predictions about predicted token sequences, predicted hypergraph data structures, and predicted program modules / functions, as already described above. Therefore, the operation outlined in Figure 6 is runtime operation that processes logical reasoning questions, such as questions aimed at interactions, asking about sequences, generating predictions, or determining feasibility, for which the responses may be “not visible” or missing in the input video data sequence. Although Figure 6 shows operations executed sequentially, this is not necessary, and operations that can be executed in parallel or substantially concurrently with other operations may be executed in such a parallel manner. For example, it should be understood that the operation of the language / program parser may operate substantially concurrently with the input natural language question while the video parser and transformer-based action transition model are operating on the input video data sequence.

[0100] As shown in Figure 6, the operation begins by receiving an input video data sequence and an input natural language question (step 610). The input video data sequence is parsed by a video parser to generate data representations of the people, objects, and relationships identified within the input video data sequence, with the temporal features of these identified people, objects, and relationships preserved, for example, a timestamp associated with their identification within the input video data sequence (step 620). A situation graph data structure is generated from the identified people, objects, and relationships (step 630). The situation graph data structure is input to a situation encoder, which generates a token sequence for the situation graph data structure (step 640). The token sequence is input to a trained dynamics transformer model that generates a predicted token sequence based on the input token sequence, where the predicted token sequence includes predictions for missing information in the input token sequence, e.g., missing tokens, or predictions for the token sequence at a point in time not represented in the input video data sequence, e.g., time t+1, or both, where t is the final timestamp of the input video data sequence (stage 650).

[0101] The predicted token sequence is input to the graph sequence decoder, which generates one or more predicted hypergraph data structures based on the predicted token sequence (step 660). The predicted hypergraph data structures are input to the program execution unit (step 670).

[0102] The input natural language question is input to a language / program parser, which parses the input natural language question and encodes it for input to multiple trained machine learning computer models (step 680). The multiple trained machine learning computer models operate on the features represented by the question encoding to generate predictions of predefined program modules / functions that should be used to generate a program that evaluates the response to the input natural language question (step 690). The predicted program modules / functions are input to a program executor, which combines the predicted program modules / functions into a nested program (step 692). The program executor executes the nested program on the predicted hypergraph data structure to generate a response to the natural language input question (step 694). The final response is then output (step 696), and the operation ends.

[0103] It should be understood that the final response output in step 696 will be used differently depending on the specific implementation of the mechanism of the exemplary embodiment. For example, in some exemplary embodiments, the final response may be returned to the originator of the input natural language question, where the final response may be rendered on a computing device or the like to notify the user of the response to the question. In other exemplary embodiments, the final response may be used to generate a command causing a computing device or robotic device to implement an action or operation corresponding to the final response. Various other possible uses of the final response, such as intelligent chat systems, medical assistance systems, home robotics systems, etc., will also be apparent to those skilled in the art who have considered this specification. Any essentially any system that operates on a basis of logical reasoning with respect to an input image sequence may integrate and benefit from the mechanism of the exemplary embodiment to assist in responding to logical reasoning questions about an input image sequence (video) in order to prompt the execution of other computerized or robotic operations. Any potential implementations and uses of the final response are intended to be incorporated herein.

[0104] The exemplary embodiments may be used in many different types of data processing environments, which may include an environment comprising a single computing system (in which the mechanism of the exemplary embodiment operates, and in which a user can interact through interfaces and peripheral devices to input natural language questions about video data and receive responses), or a distributed data processing environment comprising multiple computing devices. Furthermore, as briefly mentioned above, in some exemplary embodiments, the mechanism of the exemplary embodiment may be implemented as an autonomous computing system, such as an autonomous robot system, which may operate on video received via sensors of the autonomous computing system, such as a video camera, and formulate questions about objects identified in the video data, and cause the mechanism of the exemplary embodiment to respond to those questions to generate commands for the robot or autonomous computing system to perform actions. Many other types of computing environments will become apparent to those skilled in the art who have considered this specification, and each of these other computing environments is intended to fall within the spirit and scope of the present invention and the exemplary embodiments.

[0105] To provide the illustrative context of this specification with respect to exemplary computer environments, Figures 7 and 8 are provided hereafter as examples of such environments in which aspects of exemplary embodiments may be implemented. It should be understood that Figures 7 and 8 are merely examples and are not intended to claim or imply any limitation on any aspects of the present invention or any environments in which embodiments may be implemented. Many modifications may be made to the environments shown without departing from the spirit and scope of the present invention.

[0106] Figure 7 shows a pictorial representation of an exemplary distributed data processing system in which an aspect of the exemplary embodiment may be implemented. The distributed data processing system 700 may include a network of computers in which an aspect of the exemplary embodiment may be implemented. The distributed data processing system 700 includes at least one network 702, which is a medium used to provide communication links between various devices and computers connected together within the distributed data processing system 700. The network 702 may include connections such as wires, wireless communication links, or fiber optic cables.

[0107] In the illustrated example, servers 704 and 706 are connected to network 702 along with storage unit 708. In addition, clients 710, 712, and 714 are also connected to network 702. These clients 710, 712, and 714 may be, for example, personal computers, network computers, etc. In the illustrated example, server 704 provides data such as boot files, operating system images, and applications to clients 710, 712, and 714. Clients 710, 712, and 714 are clients to server 704 in the illustrated example. The distributed data processing system 700 may include additional servers, clients, and other devices that are not shown.

[0108] In the illustrated example, the distributed data processing system 700 is the Internet, which has a network 702 representing a global collection of networks and gateways that use the Transmission Control Protocol / Internet Protocol (TCP / IP) suite of protocols to communicate with one another. At the heart of the Internet is a backbone of high-speed data communication lines between major nodes or host computers, consisting of thousands of commercial computer systems, government computer systems, educational computer systems, and other computer systems that route data and messages. Naturally, the distributed data processing system 700 may be implemented to include many different types of networks, such as intranets, local area networks (LANs), wide area networks (WANs), etc. As mentioned above, Figure 7 is intended as an example and not as an architectural limitation to different embodiments of the invention, and therefore, the specific elements shown in Figure 7 should not be considered as limitations to the environments in which exemplary embodiments of the invention may be implemented.

[0109] As shown in Figure 7, one or more computing devices, for example, a server 704, may be specifically configured to implement a VQA system that implements the logical reasoning AI mechanism of an exemplary embodiment, including elements of the AI ​​system 100 from Figure 1. That is, the VQA system comprises a video parser 120, a transformer-based action transition model 130, a language / program parser 160, and a program execution unit 170, all of which operate as described herein with respect to one or more exemplary embodiments. The configuration of the computing devices may include providing application-specific hardware, firmware, etc., to facilitate the execution of operations and the generation of outputs described herein with respect to exemplary embodiments. The configuration of the computing devices may similarly or alternatively include providing software applications stored in one or more storage devices and loaded into the memory of a computing device such as a server 704, to cause one or more hardware processors of the computing devices to run software applications that configure the processors to perform operations and generate outputs described herein with respect to exemplary embodiments. Furthermore, any combination of application-specific hardware, firmware, software applications running on the hardware, etc., may be used without departing from the spirit and scope of the exemplary embodiments.

[0110] When a computing device is comprised of one of these methods, it should be understood that the computing device becomes a specialized computing device, not a general-purpose computing device, that is specifically configured to implement the mechanism of the exemplary embodiment. Furthermore, as will be described below, the implementation of the mechanism of the exemplary embodiment improves the functionality of the computing device and provides useful and specific results that facilitate automated AI-based responses to logical reasoning natural language questions about a particular video sequence, such as questions targeting one or more of the following: interaction, sequence, prediction, or feasibility.

[0111] As one example, using the depiction in Figure 7, a user of a client computing device, for example, client computing device 710, can view a video sequence represented by video data 110, which may be stored on a server computing device, in network-attached storage 708, etc. For example, via a web browser application, the user may navigate to server 706 or storage device 708 and access the video data 110, which is then rendered on client computing device 710. Alternatively, in some exemplary embodiments, the video data may be provided by the same computing system implementing the exemplary embodiment's AI system 100, such that the user logs on to server 704 to access the video data, for example.

[0112] A user of the client computing device 710 may wish to ask a logical reasoning question about the video data 110, and therefore may submit a natural language question 150 to the mechanism of the exemplary embodiment through a web browser or a web page or both, or other interface associated with the AI ​​system 100. The AI ​​system 100 operates on the video data 110 and the input question 150 in the manner described above with respect to Figure 1 to generate a final response 180, which is then returned to the client computing device 710. In the client computing device 710, the final response 180 may be output to the user for use. Therefore, the response to the logical reasoning question about the input video sequence may be provided to the user using an automated machine learning-based VQA system that has been extended to process the logical reasoning question by automatically generating a program and a predicted hypergraph data structure for the natural language question and the input video data.

[0113] As described above, the mechanism of the exemplary embodiment utilizes a specifically configured computing device or data processing system to perform an operation that performs a VQA operation using a logical reasoning question response with a predicted hypergraph data structure and generated program specific to the input natural language question being asked. These computing devices or data processing systems may comprise various hardware elements specifically configured to implement one or more of the systems / subsystems described herein through either a hardware configuration, a software configuration, or a combination of hardware and software configurations. Figure 8 is a block diagram of just one exemplary data processing system in which an aspect of the exemplary embodiment may be implemented. The data processing system 800 is an example of a computer, such as the server 704 in Figure 7, in which computer-readable code or instructions implementing the processes and aspects of the exemplary embodiment of the present invention may be arranged and / or executed to achieve the operation, output and external effects of the exemplary embodiment as described herein.

[0114] In the illustrated example, the data processing system 800 utilizes a hub architecture including a northbridge and memory controller hub (NB / MCH) 802 and a southbridge and input / output (I / O) controller hub (SB / ICH) 804. A processing unit 806, main memory 808, and graphics processor 810 are connected to the NB / MCH 802. The graphics processor 810 may be connected to the NB / MCH 802 via an Accelerated Graphics Port (AGP).

[0115] In the illustrated example, a local area network (LAN) adapter 812 connects to the SB / ICH804. An audio adapter 816, a keyboard and mouse adapter 820, a modem 822, a read-only memory (ROM) 824, a hard disk drive (HDD) 826, a CD-ROM drive 830, a Universal Serial Bus (USB) port and other communication ports 832, and a PCI / PCIe device 834 connect to the SB / ICH804 via buses 838 and 840. Examples of PCI / PCIe devices include Ethernet® adapters, add-in cards, and PC cards for notebook computers. PCI uses a CardBus controller, while PCIe does not. ROM 824 may be, for example, a flash basic input / output system (BIOS).

[0116] The HDD826 and CD-ROM drive 830 are connected to the SB / ICH804 via bus 840. The HDD826 and CD-ROM drive 830 may use, for example, an Integrated Drive Electronics (IDE) or Serial Advanced Technology Attachment (SATA) interface. A Super I / O (SIO) device 836 may be connected to the SB / ICH804.

[0117] The operating system runs on the processing unit 806. This operating system coordinates and controls the various components within the data processing system 800 shown in Figure 8. As a client, the operating system may be a commercially available operating system such as Microsoft® Windows® 10. An object-oriented programming system, such as the Java® programming system, may run together with the operating system and provide calls to the operating system from Java® programs or applications running on the data processing system 800.

[0118] As a server, the data processing system 800 may be, for example, an IBM eServer® System p® computer system, a Power® processor-based computer system, etc., running the Advanced Interactive Executive (AIX®) operating system or the LINUX® operating system. The data processing system 800 may be a symmetric multiprocessor (SMP) system including multiple processors within the processing unit 806. Alternatively, a single-processor system may be used.

[0119] An operating system, an object-oriented programming system, and instructions for an application or program may reside on a storage device such as an HDD 826 and be loaded into main memory 808 for execution by the processing unit 806. A process for an exemplary embodiment of the present invention may be executed by the processing unit 806 using computer-readable program code, for example, the computer-readable program code may reside, for example, in main memory 808, memory such as ROM 824, or one or more peripheral devices 826 and 830.

[0120] A bus system such as bus 838 or bus 840, as shown in Figure 8, may consist of one or more buses. Naturally, the bus system may be implemented using any type of communication fabric or architecture that provides data transfer between different components or devices attached to such fabric or architecture. A communication unit such as modem 822 or network adapter 812 in Figure 8 may include one or more devices used to transmit and receive data. Memory may be a cache, such as main memory 808, ROM 824, or one found in NB / MCH 802 in Figure 8.

[0121] As mentioned above, in some exemplary embodiments, the mechanism of the exemplary embodiment may be implemented as application software stored in a storage device such as an HDD 826 and loaded into memory such as main memory 808 for execution by one or more hardware processors such as processing unit 806, such as application-specific hardware, firmware, etc. Accordingly, the computing device shown in Figure 8 will be specifically configured to implement the mechanism of the exemplary embodiment and to perform the operations described herein with respect to the AI ​​system 100 of Figure 1 and to generate output.

[0122] Those skilled in the art will understand that the hardware in Figures 7 and 8 may vary depending on the implementation. Other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disc drives, may be used in addition to or instead of the hardware shown in Figures 7 and 8. Furthermore, the processes of the exemplary embodiments may be applied to multiprocessor data processing systems other than the SMP systems mentioned herein without departing from the spirit and scope of the invention.

[0123] Furthermore, the data processing system 800 may take the form of any of the many different data processing systems, including client computing devices, server computing devices, tablet computers, laptop computers, telephones or other communication devices, personal digital assistants (PDAs), etc. In some exemplary cases, the data processing system 800 may be a portable computing device configured with flash memory that provides non-volatile memory for storing operating system files or user-generated data or both. Essentially, the data processing system 800 may be any known or future-developed data processing system, without architectural limitations.

[0124] As described above, the exemplary embodiment provides an improved artificial intelligence mechanism that performs logical reasoning on a set of input images, such as a video input, to respond to natural language questions that require such logical reasoning, rather than simply responding to questions about what is explicitly shown in the images themselves. The exemplary embodiment makes it possible to supply an artificial intelligence computer model with logical thinking through both vision and cognition, rather than merely object or relation recognition.

[0125] The exemplary embodiments described above operate by generating a context graph data structure for images in an input video sequence, but it should be understood that exemplary embodiments are not limited thereto. Rather, exemplary embodiments may utilize any data representation that maintains temporal and entity-relationship information about entities and relationships detected in the input image data. Embodiments of context graph data structures are intended to be non-limiting examples only.

[0126] In addition, while exemplary encodings and program module structures are used herein for illustrative purposes, it should be understood that exemplary embodiments are not limited to these specific examples. Conversely, any suitable encoding and any suitable set of program modules and program module structures may be used without departing from the spirit and scope of the exemplary embodiments. The encodings and program module structures shown in the drawings and described above are intended to be non-limiting examples only.

[0127] Furthermore, while exemplary embodiments implement specific combinations of transformer-type model architectures and transformer models, LSTMs, etc., exemplary embodiments are not limited to this particular architecture or these specific computer models. Conversely, any suitable set or architecture of machine learning computer models that achieve similar behavior may be used without departing from the spirit and scope of the invention. The architectures and computer models described herein and shown in the drawings are intended to be non-limiting examples and are provided for illustrative purposes to demonstrate one possible architecture and configuration of computer models that may be used to implement exemplary embodiments of the invention.

[0128] As described above, it should be understood that exemplary embodiments may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments that include both hardware and software elements. In one exemplary embodiment, the mechanism of the exemplary embodiment is implemented in software or program code, including but not limited to firmware, resident software, microcode, etc.

[0129] A data processing system suitable for storing and / or executing program code would include at least one processor directly or indirectly coupled to a memory element via a communication bus, such as a system bus. The memory element may include local memory used during the actual execution of the program code, mass storage, and cache memory that provides temporary storage for at least some program code to reduce the number of times the code must be retrieved from mass storage during execution. The memory may be of various types, but is not limited to ROM, PROM, EPROM, EEPROM, DRAM, SRAM, flash memory, solid-state memory, etc.

[0130] Input / output devices, or I / O devices (including, but not limited to, keyboards, displays, pointing devices, etc.), can be coupled to the system either directly or through intermediary wired or wireless I / O interfaces or controllers, or both. I / O devices can take many different forms other than conventional keyboards, displays, pointing devices, etc., including, but not limited to, communication devices coupled via wired or wireless connections, such as smartphones, tablet computers, touchscreen devices, and voice recognition devices. Any known or future-developed I / O devices are intended to fall within the scope of the exemplary embodiments.

[0131] Furthermore, a network adapter may be coupled to the system, enabling the data processing system to be coupled to other data processing systems or remote printers or storage devices via an intervening private or public network. Modems, cable modems, and Ethernet cards are only a fraction of the currently available types of network adapters for wired communication. Wireless communication-based network adapters, including but not limited to 802.11a / b / g / n wireless communication adapters and Bluetooth® wireless adapters, may also be used. Any known or future-developed network adapters are intended to fall within the spirit and scope of the present invention.

[0132] The description of the present invention is presented for illustrative and explanatory purposes and is not intended to be exhaustive or to limit the invention to the forms disclosed herein. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The embodiments have been selected and described to best illustrate the principles and practical applications of the invention and to enable those skilled in the art to understand the invention in terms of various embodiments with various modifications suitable for a particular intended use. The terms used herein have been selected to best illustrate the principles of the embodiments, their practical applications or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for performing artificial intelligence-based video question answering in a data processing system, A step of the data processing system's video parser parsing an input video data sequence to generate one or more situation data structures, each situation data structure including entities identified by the video parser as existing within the images of the input video data sequence, and data elements corresponding to a first relationship between entities. A step of running at least one first machine learning computer model of the data processing system on one or more of the aforementioned situation data structures to predict one or more second relationships between the one or more of the aforementioned situation data structures, A step of determining an executable program to be executed in response to an received input natural language question by at least one second machine learning computer model of the data processing system executed on the received input natural language question, The data processing system executes the determined executable program with respect to the one or more situation data structures and the one or more predicted second relationships between the one or more situation data structures. The data processing system outputs a response to the input natural language question based on the result of executing the determined executable program. Equipped with, The method comprises at least one first machine learning computer model, a situation encoder, and a machine learning-trained dynamics transformer computer model.

2. The method according to claim 1, wherein the at least one first machine learning computer model further comprises a sequence decoder.

3. The step of running the aforementioned at least one first machine learning computer model is: The steps include: encoding one or more status data structures using the status encoder to generate token sequences corresponding to the entities and relationships represented within the one or more status data structures; The steps include: processing the token sequence using the machine learning-trained dynamics transformer computer model to predict missing or subsequent token sequences, and generating a predicted token sequence that includes the predicted missing or subsequent token sequences; The method according to claim 2, having the following characteristics.

4. The method according to claim 3, wherein the sequence decoder generates the predicted one or more second relationships based on the predicted token sequence as one or more predicted hypergraph data structures.

5. The at least one second machine learning computer model comprises a language / program parser and a program execution unit, and the step of determining the executable program is: The steps include: processing the input natural language question using the language / program parser and predicting a plurality of program modules to be executed in response to the input natural language question; The program execution unit combines the multiple program modules to form an executable program that is executed on one or more predicted hypergraph data structures, thereby generating a final response to the input natural language query. The data processing system performs the step of outputting the final response to the input natural language question. The method according to claim 4, having the following characteristics.

6. The method according to claim 4, wherein each hypergraph data structure includes one or more hyperedges connecting a first situation data structure in the one or more situation data structures to at least one second situation data structure, the at least one second situation data structure being either the first situation data structure or a different situation data structure in the one or more situation data structures.

7. The method according to claim 6, wherein each hyperedge in the one or more hyperedges includes a predicted action corresponding to at least one first entity in the first situation data structure, accompanied by at least one second entity in the at least one second situation data structure.

8. The method according to any one of claims 1 to 7, wherein each situation, which includes one or more images of the input video data sequence, has a corresponding situation data structure within the one or more situation data structures.

9. The method according to any one of claims 1 to 7, wherein the one or more status data structures are status graph data structures, and within the status graph data structure, the data elements include nodes corresponding to entities and edges corresponding to relationships between entities.

10. The method according to any one of claims 1 to 7, wherein the input natural language question is a logical reasoning question of any type of interaction question, sequence question, predictive question, or feasibility question.

11. In the data processing system, A procedure comprising: parsing an input video data sequence using a video parser of the data processing system to generate one or more situation data structures, each situation data structure including entities identified by the video parser as existing within the images of the input video data sequence, and data elements corresponding to a first relationship between entities; A procedure for running at least one first machine learning computer model of the data processing system on one or more of the aforementioned situation data structures to predict one or more second relationships between the one or more of the aforementioned situation data structures, A procedure for determining an executable program to be executed in response to an received input natural language question by at least one second machine learning computer model of the data processing system executed on the received input natural language question, The data processing system performs a procedure for executing the determined executable program with respect to one or more situation data structures and one or more predicted second relationships between the one or more situation data structures, The data processing system performs the following steps: outputting a response to the input natural language question based on the result of executing the determined executable program; A computer program for executing, The at least one first machine learning computer model is a computer program comprising a situation encoder and a machine learning-trained dynamics transformer computer model.

12. The computer program according to claim 11, wherein the at least one first machine learning computer model further comprises a sequence decoder.

13. The data processing system includes a procedure for executing the at least one first machine learning computer model, at least A procedure for encoding one or more status data structures using the status encoder to generate token sequences corresponding to the entities and relationships represented within the one or more status data structures, A procedure for processing the token sequence using the machine learning-trained dynamics transformer computer model to predict missing or subsequent token sequences, and for generating a predicted token sequence including the predicted missing or subsequent token sequences. The computer program according to claim 12, which is further executed by...

14. The computer program according to claim 13, wherein the sequence decoder generates one or more predicted second relationships based on the predicted token sequence as one or more predicted hypergraph data structures.

15. The at least one second machine learning computer model comprises a language / program parser and a program execution unit, and the computer program provides the data processing system with at least a procedure for determining the executable program. A procedure for processing the input natural language question using the language / program parser and predicting a number of program modules to be executed in response to the input natural language question, A procedure for generating a final response to an input natural language query by combining the multiple program modules using a program execution unit to create an executable program that is executed on one or more predicted hypergraph data structures, The data processing system provides a procedure for outputting the final response to the input natural language question. The computer program according to claim 14, which is further executed by...

16. The computer program according to claim 14, wherein each hypergraph data structure includes one or more hyperedges connecting a first situation data structure in one or more situation data structures to at least one second situation data structure, the at least one second situation data structure being either the first situation data structure or a different situation data structure in one or more situation data structures.

17. The computer program according to claim 16, wherein each hyperedge in the one or more hyperedges includes a predicted action corresponding to at least one first entity in the first situation data structure, with at least one second entity in the at least one second situation data structure.

18. The computer program according to any one of claims 11 to 17, wherein each situation, which includes one or more images of the input video data sequence, has a corresponding situation data structure within the one or more situation data structures.

19. The computer program according to any one of claims 11 to 17, wherein the one or more status data structures are status graph data structures, and within the status graph data structure, the data elements include nodes corresponding to entities and edges corresponding to relationships between entities.

20. It is a device, At least one processor, At least one memory connected to the at least one processor, Equipped with, When the at least one memory is executed by the at least one processor, the at least one processor receives Parsing an input video data sequence to generate one or more situation data structures, each situation data structure including data elements corresponding to entities identified as existing within the images of the input video data sequence, and a first relationship between entities, The process involves running at least one first machine learning computer model on one or more of the aforementioned situation data structures to predict one or more second relationships between the aforementioned situation data structures, To determine the executable program to run in response to the received input natural language question, Executing the determined executable program with respect to the one or more situation data structures and the one or more predicted second relationships between the one or more situation data structures, Based on the result of executing the determined executable program, the system outputs a response to the input natural language question. Having an order to carry out, The apparatus comprises at least one first machine learning computer model, a situation encoder, and a machine learning-trained dynamics transformer computer model.

Images