Scene-aware video dialog

Abstract

A context-aware dialog system includes an input interface to receive a sequence of video frames, background information, and a query, and a memory configured to store a neural network trained to generate a reply to the input query by analyzing one or a combination of the input sequence of video frames and the input background information. The system also includes a processor configured to detect and classify objects in each video frame of the sequence of video frames, determine relationships between the classified objects in each video frame, extract features representing the classified objects and the determined relationships for each video frame to generate a sequence of feature vectors, and submit the sequence of feature vectors, the input query, and the input background information to the neural network to generate the reply to the input query.

Description

Technical Field

[0001] The present invention relates generally to video scene awareness recognition, and more particularly to methods and systems for generating responses relevant to video queries in a video dialogue system. Background Art

[0002] Everyday communication between humans is closely conveyed through different modalities of information about surrounding objects and events, such as visual information and sound. Multimodal reasoning is required to enable machines to converse with humans about these objects and events. For such multimodal reasoning, it is necessary to generate an effective model of the dynamic audiovisual scene. Based on the model, the machine infers these objects and events and generates responses to human queries. Recently, this multimodal reasoning problem has been modeled in audiovisual scene-aware dialogue (AVSD) systems. Applications of AVSD systems include end-to-end dialogue, visual question answering (VQA), and video captioning. AVSD systems are used to support dialogue between humans and machines. To support dialogue, AVSD systems must be able to jointly model the audiovisual scene and previous dialogue context using multiple modalities (such as audio, video, queries, and multi-turn dialogue history) and generate responses in free-form natural language. Such AVSD tasks are challenging due to the difficulty in capturing diverse information, such as the dynamics of a video, across all modalities. Some modalities may be complementary, while others may be independent or even conflicting. For example, AVSD tasks require modalities such as natural language and video (or audio) information. However, in some cases, video may be independent of natural language and / or natural language information, while sound may contain similar information.

[0003] Typically, neural networks are trained to capture video dynamics by encoding spatiotemporal representations of videos. However, such neural networks are typically trained only for human action recognition, and the corresponding spatiotemporal representations lack information about objects in the video, which is necessary for understanding the scene in the video. Information about objects includes the motion of objects in the scene or the relationships between objects in the scene. Therefore, this information about objects in the scene is important and necessary for representing the scene. Furthermore, responses to video queries can be generated based on features extracted by processing the entire video. These features represent global characteristics that indicate the action in the video. However, processing and analyzing the entire video in its raw format to extract such global features for generating responses is impractical. It is possible to extract features by sampling a few frames from the entire video. However, such features may not be able to model the finer-grained details necessary to understand more detailed information about the objects and actions in the video. This lack of detailed information may result in inaccurate responses to queries.

[0004] Therefore, it is necessary to extract features of the video in an accurate and natural language mode to generate a response to a query. More specifically, it is necessary to extract features that represent the entire scene in the video in order to generate a response to a query from a user. Summary of the Invention

[0005] The Audiovisual Scene-Aware Dialogue (AVSD) system is a visually aware dialogue system for solving problems that conduct conversations in natural language and provide answers to inquiries related to the video. Typically, these inquiries are related to actions performed in the video. For example, a user may ask a question about a person shown in a video (for example, the query may be "What is the person in the video doing?"). In addition, the scenes in the video differ in terms of temporal information and frame-level information. If the scene in the first 10 frames of a video changes after 5 seconds, the user may be required to make a subsequent inquiry again after receiving an answer to the previous inquiry. At this time, the AVSD system is required to generate an answer related to the subsequent inquiry.

[0006] It is an object of some embodiments to provide a system and method for generating responses to video queries. Additionally or alternatively, it is another object of some embodiments to extract features representing a video scene that are optimal for generating relevant responses to queries. The query can ask about actions as well as about objects in the video scene. Additionally or alternatively, it is another object of some embodiments to process a video associated with audio, and previous questions and answers about the video, to automatically generate responses to queries about the video. For example, one query asks about a boy holding a book in a video scene. Another query can ask about an object, such as the book. In this case, the relevant response to the query is a description of the book. In some embodiments, for feature extraction from a video, each video frame is represented separately to capture the objects and their relationships in the scene, and a collection of video frames represents the temporal evolution of the objects and their relationships.

[0007] To this end, some embodiments are based on the recognition that a scene representation suitable for scene understanding captures the features of objects in a scene, the relationships between objects, and the temporal evolution of objects and their relationships or actions to perform spatiotemporal reasoning. The temporal evolution of objects and their relationships is represented by a collection or sequence of video frames. In some embodiments, spatiotemporal reasoning is performed based on a spatiotemporal scene graph representation (STSGR). The STSGR represents objects and their relationships. The STSGR also provides an alternative data structure for efficiently storing object representations in memory. To implement this spatiotemporal feature representation (i.e., STSGR), in addition to and instead of capturing actions in a video, some embodiments detect and classify objects in each video frame of the video. Furthermore, objects in each video frame may have relationships with other objects in the video frame. To this end, some embodiments determine the relationships between the classified objects. Embodiments process the scene graph within a graph neural network framework to perform spatiotemporal reasoning via neural message passing, where neural message passing enables information flow between objects (such as video captions, video conversation history, audio) and relationships in the STSGR. Furthermore, embodiments also use inferred graph labels to control the information flow between queries, conversation history, video, and audio in the STSGR. The inferred graph labels correspond to the labeled nodes and labeled edges with the highest weights computed by a pre-trained neural network (such as a graph-based neural network for node classification). Some embodiments are based on the recognition that scene graph representations at different frame levels have different numbers of nodes. To this end, some embodiments employ graph average pooling and graph max pooling to perform intra-graph reasoning, where intra-graph reasoning corresponds to performing graph reasoning on classified objects and their determined relationships in a single graph of the scene. Intra-graph reasoning is performed to generate a frame memory or graph memory representing features corresponding to the visual and semantic scene graph of a video frame in STSGR.

[0008] Additionally or alternatively, another purpose of some embodiments is to extract features representing classified objects and the relationships between the determined classified objects. To this end, the features extracted from each video frame are represented as a scene graph representation, which represents objects as nodes and the relationships between objects as edges. In addition, a scene graph representation sequence corresponding to the video frame sequence of the video is generated. Some embodiments are based on the following implementation: extracting key video frames from the video to generate a scene graph representation sequence. A variety of key frame extraction methods can be used to extract key frames, such as clustering-based key frame extraction, vision-based key frame extraction, motion analysis-based key frame extraction, etc. Some embodiments extract key frames based on features extracted from a model trained on a dataset for AVSD applications (e.g., the VisualGenome dataset). For example, for AVSD applications in football videos, key frames are extracted based on features extracted from datasets that include players on a football field, footballs held by players, etc. In some other embodiments, key frames are extracted by discarding redundant video frames of the video.

[0009] To this end, features extracted from the video are aligned with the query types accepted by the AVSD system to improve the accuracy of AVSD responses. Some embodiments use a neural network to process the video to generate responses. In some embodiments, a neural network trained to generate such responses is used. The neural network corresponds to an attention-based network architecture. An attention-based neural network calculates attention coefficients for different modalities of input information. The attention coefficients provide weights for the video frames associated with the query when preparing responses to the query. To this end, some embodiments incorporate temporal features into each feature vector of each video frame. A feature vector is an aggregation of multiple feature vectors centered around that feature vector. Thus, each feature vector represents the features and relationships of objects in the corresponding video frame, as well as similar features and relationships of objects in adjacent video frames. Thus, the neural network generates a sequence of feature vectors for a sequence of video frames. The feature vector sequence is aggregated with temporally evolving features. To this end, temporal features are incorporated into each feature vector of each video frame. Furthermore, some embodiments propagate information from each individual feature vector to adjacent feature vectors. Examples of such neural networks include encoder / decoder neural networks with attention-based decoders, transformers, and the like. After aligning the spatiotemporal features with other multimodal features, feature vectors representing the dynamic characteristics of the video are extracted to generate relevant responses.

[0010] Some embodiments are based on the recognition that the performance of AVSD systems can be further improved by tailoring object detection to the specific goals of the AVSD system. For example, in some embodiments, object detection is generalized so that only a specific number of objects are detected and classified. In some embodiments, objects are detected by an object classifier and relationships between objects are determined by a relationship classifier. In one implementation, the object classifier is a neural network trained to detect approximately 2,000 object categories. Therefore, the object classifier does not detect the entire object in the video frame. For example, in a video frame, the first 36 categories detected are selected for object detection. This embodiment is beneficial for general applications of AVSD systems.

[0011] To this end, an object classifier and a relationship classifier are selected from a set of neural network-based classifiers. The selected classifiers can be used for different types of videos, regardless of the video context. In some other embodiments, the object classifier is selected based on context information. For example, the context information may include subtitles indicating the type of video, and the AVSD system selects an object classifier that matches or is at least most relevant to the identified video type. Additionally or alternatively, in some embodiments, the AVSD system determines the type of video based on its context. Additionally or alternatively, in some embodiments, the AVSD system determines the type of video based on the type of inquiry. For example, if the inquiry asks which teams are playing in this football game, the AVSD system determines that the video is likely about a football game.

[0012] Additionally or alternatively, some embodiments vary the object detection for different applications. For example, the object detection for a driving application configured to respond to inquiries related to navigating a vehicle may be different from the object detection configured for sporting events and / or movies. Even within the category of sporting events, different games (such as football or badminton) may need to be configured to recognize different objects and different relationships. To this end, different embodiments use different object detection modules configured to detect and classify different types of objects. Furthermore, some embodiments are configured to select different object detection modules based on the type of inquiry and / or video.

[0013] Thus, one embodiment discloses a scene-aware dialogue system for generating responses to multimodal queries, comprising: an input interface configured to receive a video frame sequence, context information, and a query; a memory configured to store at least one neural network, the at least one neural network comprising a visual scene-aware dialogue neural network, the visual scene-aware dialogue neural network being trained to generate responses to the input query by analyzing one or a combination of an input video frame sequence and input context information provided to the neural network; a processor configured to detect and classify objects in each video frame in the video frame sequence; integrate regions of interest of objects into the video frame sequence to determine a relationship between the classified objects in each video frame, wherein at least one video frame in the video frame sequence includes at least two classified objects, and wherein the relationship between the two classified objects is an intra-frame object relationship restricted to the video frames; extracting features representing the classified objects and the determined relationship for each video frame to generate a sequence of feature vectors, wherein there is one feature vector for each video frame; and submitting the sequence of feature vectors, the input query, and the input context information to the neural network to generate a response to the input query; and an output interface for rendering the response to the input query.

[0014] Another embodiment discloses a method for generating a response to a multimodal query, wherein the method includes receiving a sequence of video frames, context information, and a query; detecting and classifying objects in each video frame in the sequence of video frames; integrating regions of interest of objects into the sequence of video frames to determine a relationship between the classified objects in each video frame, wherein at least one video frame in the sequence of video frames includes at least two classified objects, and wherein the relationship between the two classified objects is an intra-frame object relationship confined within the video frame; extracting features representing the classified objects and the determined relationship for each video frame to produce a sequence of feature vectors, wherein there is one feature vector for one video frame; submitting the sequence of feature vectors, the input query, and the input context information to at least one neural network, the at least one neural network including a visual scene perception dialogue neural network, the visual scene perception dialogue neural network trained to generate a response to the input query by analyzing one or a combination of the input sequence of video frames and the input context information; and rendering the response to the input query through an output interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] [ Figure 1 ]

[0016] Figure 1 A schematic diagram of an audio-visual scene-aware dialogue (AVSD) system for generating responses to queries is shown in accordance with some embodiments.

[0017] [ Figure 2A ]

[0018] Figure 2A A sequence of video frames of a video according to some embodiments is shown.

[0019] [ Figure 2B ]

[0020] Figure 2B A sequence of scene graph representations for a sequence of video frames is shown in accordance with some embodiments.

[0021] [ Figure 2C ]

[0022] Figure 2C A representation depicting the graphical attention network and relation graph network generated for each frame according to some embodiments is shown.

[0023] [ Figure 2D ]

[0024] Figure 2D Shown is a representation of a visual graphical memory sequence depicting a scene graph representation sequence according to some embodiments.

[0025] [ Figure 3 ]

[0026] Figure 3 A schematic diagram of a semantically controlled transformer for generating a response to a query is shown in accordance with some embodiments.

[0027] [ Figure 4 ]

[0028] Figure 4 A schematic overview of the principles of an AVSD system for generating responses to queries according to some embodiments is shown.

[0029] [ Figure 5 ]

[0030] Figure 5 An example scenario is shown for generating a response to a query using a device implemented with an AVSD system according to an example embodiment.

[0031] [ Figure 6 ]

[0032] Figure 6 A schematic diagram of a table including information of a video frame sequence, background information, and an input query is shown according to some embodiments.

[0033] [ Figure 7A ]

[0034] Figure 7A An exemplary scenario for generating a response to an input query according to an example implementation is shown.

[0035] [ Figure 7B ]

[0036] Figure 7B An exemplary scenario for generating a response to an input query according to another example embodiment is shown.

[0037] [ Figure 7C ]

[0038] Figure 7C An exemplary scenario for generating a response to an input query according to another example embodiment is shown.

[0039] [ Figure 8 ]

[0040] Figure 8 A flow chart of a method for generating a response to a query by an AVSD system is shown, according to some embodiments.

[0041] [ Figure 9 ]

[0042] Figure 9 A diagram depicting a method for Figure 1 Table of the ablation analysis of the components of the AVSD system.

[0043] [ Figure 10A ]

[0044] Figure 10A is a table depicting a comparison of the AVSD system 100 with baseline methods on a test split of the AVSD challenge under Dialogue Systems Technology Challenge 7 (DSTC7), according to some embodiments.

[0045] [ Figure 10B ]

[0046] Figure 10B A table depicting a comparison of the AVSD system 100 and a baseline approach on a test partition of the AVSD challenge of DSTC8 is shown, according to some embodiments.

[0047] [ Figure 11 ]

[0048] Figure 11 A detailed schematic overview of the principles of an AVSD system for generating responses to queries according to some embodiments is shown. DETAILED DESCRIPTION

[0049] In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other cases, devices and methods are shown in block diagram form only to avoid obscuring the present disclosure.

[0050] As used in this specification and claims, when used in conjunction with a list of one or more components or other items, the terms "for example" and "such as" and the verbs "include," "have," "comprises," and their other verb forms should each be interpreted as open-ended, meaning that the list should not be construed to exclude other additional components or items. The term "based on" means based at least in part on. Furthermore, it should be understood that the phraseology and terminology used herein are for descriptive purposes only and should not be construed as limiting. Any headings used in this description are for convenience only and have no legal or limiting effect.

[0051] System Overview

[0052] Figure 1 A schematic diagram of an audiovisual scene-aware dialogue (AVSD) system 100 for generating responses to queries, according to some embodiments, is shown. The AVSD system 100 includes an input interface 102, a processor 104, a memory 106, a storage device 114, and an output interface 116. The input interface 102 is configured to receive input data 126, which includes a sequence of video frames of a video, contextual information, and a query. The input data 126 corresponds to multimodal information, such as audio, video, text, and the like. The AVSD system 100 receives the input data 126 via a network 124 using a network interface controller (NIC) 122. In some cases, the video may be a live streaming video received via the network 124. In other cases, the video may be an offline video stream or a recorded video, where the offline video stream may be stored in the storage device 114. The storage device 114 is further configured to store contextual information, where the contextual information includes one or a combination of audio information and textual information about the input video (such as video subtitles, a history of the video question-and-answer dialogue, and the like).

[0053] Processor 104 is configured to detect and classify objects in each video frame of the video frame sequence. Processor 104 is also configured to determine relationships between objects in each video frame of the video frame sequence and extract features representing the objects and their relationships in each video frame to generate a sequence of feature vectors. Each feature vector in the sequence of feature vectors corresponds to a corresponding video frame of the video frame sequence. Processor 104 is also configured to submit the sequence of feature vectors, a query, and context information to one or more neural networks stored in memory 106 to generate a response to the query, wherein the query relates to one or a combination of objects, relationships between objects, and the temporal evolution of objects in the input video frame sequence. The one or more neural networks are trained to generate responses to the query by analyzing one or a combination of the input sequence of video frames and the input context information, such that the one or more neural networks are multimodal neural networks configured to process multimodal information. The one or more trained neural networks include an audiovisual scene perception neural network 108, a feature extraction neural network 110, and an aggregation neural network 112.

[0054] For example, the audio-visual scene perception neural network 108 is trained to process features representing the scene in each video frame of the video frame sequence (e.g., a scene graph representation) to generate a response. Examples of the audio-visual scene perception neural network 108 include one or a combination of a faster region-based convolutional neural network (Faster RCNN) and a three-dimensional (3D) convolutional neural network (CNN). The scene graph representation provides spatial information for each video frame, including features of objects in each video frame and relationships between objects. The processor 104 is further configured to combine the spatial information with the temporal information of each video frame to generate a spatiotemporal scene graph representation (STSGR) model. More specifically, the processor 104 generates an STSGR model for each video frame in the video frame sequence based on the integrated region of interest. Furthermore, one or more neural networks are trained to perform spatiotemporal learning with each STSGR model when training the STSGR model for the video frame sequence, thereby generating relevant responses to the query. Spatiotemporal reasoning captures the flow of visual and semantic information within the video, so that each STSGR model represents each corresponding video frame as a spatiotemporal visual graph stream and a semantic graph stream. This allows object-level graph reasoning to generate responses relevant to the video query.In some embodiments, one or more neural networks are multi-headed reordering transformers for generating object-level graph reasoning, which are capable of reordering the heads of a sequence of feature vectors.

[0055] In some embodiments, the audio-visual scene perception neural network 108 corresponds to an attention-based network architecture that computes an attention coefficient for each edge in a scene graph representation. The attention coefficient provides a value that determines the importance between two linked nodes of a graph (such as an STSGR). The attention coefficient is used to selectively focus on salient features (such as classified objects) and relationships for spatiotemporal reasoning.

[0056] In addition, the processor 104 is configured to extract feature vectors from the generated scene graph representation by utilizing the feature extraction neural network 110. The feature extraction neural network 124 corresponds to a pre-trained neural network, which extracts feature vectors from the generated scene graph representation and generates a feature vector sequence corresponding to the scene graph representation sequence of the video frame sequence. Further, the processor 104 is configured to modify the value of each feature vector in the feature vector sequence using the weighted values ​​of adjacent feature vectors in the feature vector sequence.

[0057] In particular, the feature extraction neural network 110 performs frame-level intra-graph reasoning to extract feature vectors. The feature vectors here correspond to the visual graph memory of the scene graph representation sequence.

[0058] In one implementation, intra-graph reasoning is performed by aggregating node-based features (i.e., object features) of the scene graph representation and aggregating edge-based features (i.e., relationship features) of the scene graph representation. In addition, intra-graph reasoning uses an attention-based neural network to calculate a weight value (i.e., self-attention value) for a pair of linked nodes. The weight value represents the importance of a node being paired with another node. In a similar manner, the weights of other paired nodes are determined using an attention-based neural network. In addition, a weighted sum of adjacent nodes (i.e., adjacent objects) in the scene graph representation is calculated based on all weight values. The processor 104 updates the features of each node in each corresponding scene graph representation based on the weighted sum.

[0059] Furthermore, processor 104 provides the weighted sum as input to a relational neural network. In one implementation, the relational neural network corresponds to a multi-layer, fully connected network that is used to generate relational features from two connected node features in the updated scene graph representation. Thus, the updated scene graph representation is aggregated into a visual graph memory that provides a sequence of feature vectors. The sequence of feature vectors represents both spatial and temporal representations of input data 126. Furthermore, to determine each feature vector in the sequence of feature vectors, aggregation neural network 110 aggregates the values ​​of multiple feature vectors that fit a window centered around the feature vector. A feature vector in such a sequence of feature vectors is a weighted combination of the values ​​of multiple feature vectors that fit a window centered around the feature vector. This allows aggregation of similar features related to objects and their relationships in adjacent video frames. Processor 104 is configured to generate a response to the query using each feature vector in the sequence of feature vectors. Furthermore, the generated response to the query is rendered on output device 118 via output interface 116. Output device 118 includes, but is not limited to, a computer, laptop, tablet, phablet, or any display device. In some implementations, the output device 118 may include an application interface for rendering the reply.

[0060] In some embodiments, to improve the efficiency of the AVSD system 100, the processor 104 can be configured to extract a set of key frames from the video frame sequence. The set of key frames includes video frames that represent transitions in the motion of a person or object in the video frame sequence. The processor 104 can be configured to extract the set of key frames based on a pre-trained model for AVSD applications (e.g., a visual genome). The extraction of the set of key frames enables the processor 104 to efficiently process the video frames (i.e., the set of key frames) to generate responses to video queries because the number of video frames from the video frame sequence that need to be processed is reduced. Therefore, the utilization of key frames helps an efficient AVSD system (i.e., the AVSD system 100) generate responses to video queries.

[0061] Figure 2AA sequence of video frames, such as video frame 202a, video frame 202b, video frame 202c, and video frame 202d of video 200A, according to some embodiments is shown. In some embodiments, when video 200A is provided to the AVSD system 100, the processor 104 detects key frames from the sequence of video frames 202a-202d. In some embodiments, the key frames are all frames of the sequence. In some embodiments, the processor 104 extracts the key frames using a model trained on a dataset. Additionally or alternatively, some embodiments select key frames at equal time steps. Additionally or alternatively, some embodiments select key frames using a temporal attention model. Additionally or alternatively, in some other embodiments, the processor 104 extracts key frames using a key frame extraction technique, such as cluster-based key frame extraction, vision-based key frame extraction, motion analysis-based key frame extraction, and the like. Based on the extraction of key frames from video 200A, processor 104 only needs to process the key frames instead of the entire video frame sequence (ie, video frame sequences 202a-202d) of video 200A. This improves the computational speed and efficiency of AVSD system 100.

[0062] In some embodiments, the processor 104 further detects and classifies objects in each video frame of the video frame sequence 202a-202d. The audio-visual scene perception neural network 108 may include a set of instructions for object detection techniques (such as bounding box techniques). The processor 104 may be configured to execute such instructions to detect objects in each video frame. For example, in video frame 202a, the detected objects are indicated by bounding boxes, such as bounding box 204a and bounding box 204b. An object classifier based on the audio-visual scene perception neural network 108 classifies the detected objects. The object classifier may include a set of instructions for classifying objects based on conventional object classification techniques. The processor 104 may be configured to execute the set of instructions of the object classifier in order to classify the objects in each video frame. The classified objects are labeled ( Figure 2A (not shown). For example, bounding box 204a is labeled as a pot and bounding box 204b is labeled as a stove. In a similar manner, objects are detected and classified in the remaining video frames of the video frame sequence 202b-202d. In addition, the processor 104 determines the relationship between the classified objects to generate a scene graph representation for each video frame (i.e., video frame 202a, video frame 202b, video frame 202c, and video frame 202d), which will be displayed in FIG. Figure 2B Alternatively, the processor 104 may detect and classify objects from key frames of the video 200A based on utilization of the audio-visual scene perception neural network 108 as described above.

[0063] Figure 2BA scene graph representation sequence 200B is shown according to some embodiments. The scene graph representation sequence 200B includes scene graph representations 206a, 206b, 206c, and 206d for a sequence of video frames 202a-202d. In addition, the scene graph representation sequence 206a-206d corresponds to a sequence of features 208a, 208b, 208c, and 208d for classifying objects and determining relationships. The processor 104 is configured to generate the scene graph representation sequence using the audio-visual scene perception neural network 108, such as Figure 1 . Each scene graph representation of the scene graph representation sequence 206a-206d represents the classification objects and the relationships between the objects in each video frame of the video frame sequence 202a-202d. In the scene graph representation 206a, each node represents a classification object, and each edge connecting two nodes represents a relationship between the classification objects. For example, in the scene graph representation 206a, the subject node (e.g., 9 freezer) is directed to the object node (e.g., 13 cabinet). In addition, the edges connecting the nodes are directed edges so as to specify the relationships between the objects. The scene graph representation can be represented as G={(x i , e ij , x j )|x i , x j ∈V,e ij ∈ε}, where V is the set of vertices consisting of a single object, ε is the set of edges consisting of a relationship linking two objects, and (x i , e ij , x j ) represents the topic node x i and object node x j By directed relationship edge e ij connect.

[0064] In some embodiments, the audio-visual scene perception neural network 108 includes a set of classifiers (also referred to as a set of neural network-based classifiers) for extracting features of an object based on visual features or semantic features of the object in each video frame of the video frame sequence 202a-202d. The set of classifiers includes an object classifier and a relationship classifier. The processor 104 is configured to select the object classifier and the relationship classifier from the set of neural network-based classifiers based on an input sequence of video frames, input context information, input query, or a combination thereof. For example, the video frame 202a represented by I, the video frame 202b represented by F, and the video frame 202c represented by F are respectively selected from the object classifier and the relationship classifier. I The object characteristics represented by B I The bounding box represented by S I The semantics of the representation can be extracted by a neural network-based object classifier such as the Faster Recurrent Convolutional Neural Network (R-CNN) object detection model represented as

[0065] F I , B I , S I =RCNN(I) (1)

[0066] in, S I is a list of semantic labels. In some implementations, the object detection model can detect N o = 36 detections, each using feature dimension d o =1024, and uses an object semantic label vocabulary size of 1601.

[0067] In a similar manner, the processor 104 selects a relationship classifier to determine the relationship between the classified objects in each video frame of the video frame sequence 202a-202d. The relationship classifier identifies the visual relationship between the classified objects in the video frame sequence 202a-202d. In addition, the relationship classifier generates a fixed number of relationship proposals, N r has the highest confidence. For example, N r is set to a fixed number of 100. In some implementations, the visual relationships in each video frame (i.e., each of the video frames 202a-202d) are determined using a relationship detection model that embeds objects and relationships into a vector space where both discriminative power and semantic affinity are preserved. The relationship detection model is trained on a video dataset containing 150 objects and 50 relationships represented as predicates. When the relationship detection model is applied to the video frames 202a-202d, a set of subjects S, predicates P, and objects O is obtained, i.e.<S,P,O> is obtained as the output for each video frame. In one embodiment, the original predicate semantics P is discarded because the relation predicates of the relation detection model trained on the video dataset are limited and fixed. Therefore, the relation proposal is based on the implicit relation semantics used to learn the objects.<S,O> Yes. In most cases, the relationship proposal N r It may not include all unmentioned objects. Filter out unmentioned objects by performing graph pruning.

[0068] In addition, the processor 104 integrates the regions of interest of the objects in the video frame sequence 202a-202d to determine the relationship between the two objects of the classification object. More specifically, for the determined<S,O> For, the bounding boxes of S and O (e.g., Figure 2A The union box of the bounding boxes 204a and 204b) is regarded as the predicate region of interest used as an additional node in the new scene graph. The union box, which covers more information than the subject and object bounding boxes, captures the high-order interactions between each of the two objects.

[0069] Therefore, the processor 104 uses the feature extraction neural network 110 to extract feature sequences 208a-208d including features of visual information and semantic information. In addition, the processor 104 uses the scene graph representation 206a-206d to extract the visual memory of the video frames 202a-202d, which will be Figure 2C is further described in the description of .

[0070] Figure 2C A representation 200C is shown depicting a graph attention network 210 and a relation graph network 212 for representing 206a-206d forming a scene graph, according to some embodiments.

[0071] In some embodiments, the processor 104 aggregates the classified objects and the determined relationships to generate a visual memory for each video frame of the video frame sequence 202a-202d. To this end, the processor 104 utilizes a graph attention network 210 and a relationship graph network 212 stored in the memory 106. The graph attention network 210 includes a node 210a representing an attention coefficient value. Each scene graph representation of the scene graph representation sequence 206a-206d is aggregated by performing intra-graph reasoning using the graph attention network 210 and the relationship graph network 212. The processor 104 utilizes the feature extraction neural network 110 of the AVSD system 100 to perform frame-level intra-graph reasoning on the scene graph representations 206a-206d to extract a visual graph memory or a semantic graph memory for each video frame of the video frame sequence 202a-202d.

[0072] The visual graph memory provides higher-level features representing finer-grained information for each node (i.e., object) in the scene graph representations 206a-206d. Higher-level features are extracted based on the attention coefficient of each edge in each scene graph representation of the scene graph representation sequence 206a-206d. The processor 104 determines the attention coefficient through the graph attention network 210. In addition, the processor 104 aggregates the node features of the scene graph representations 206a-206d based on the attention coefficient. The processor 104 calculates additional edge features based on the node features through the relationship graph network 212. In addition, the processor 104 aggregates the additional edge features so that the relationship graph network 212 updates the node features.

[0073] In node-based feature aggregation, for M node features X={x1, x2, ..., x M}, for each pair of linked nodes, perform self-attention. For linked node x i and x j , which represents the attention coefficient of node j’s importance to node i 210a∝ ij Calculated by the following formula

[0074]

[0075] Among them, || represents vertical cascade operation, N i represents the neighboring object nodes of object i, is the weight matrix of the shared linear transformation that transforms the original features into higher-level features, is the attention weight vector. The features of each object node in the scene graph representations 206a-206d are modified based on the weighted sum of the features of the neighboring objects:

[0076]

[0077] Here, the variables k and K define the number of heads in the multi-head attention scheme. i The weighted sum of is used as the input of the relationship graph network 212 for performing edge-based feature aggregation. The relationship graph network 212 is a multi-layer fully connected network, h Λ is used to extract features from two connected nodes (x′ i , x′ j ) Generate edge feature e ij :

[0078] e ij =h Λ (x′ i , x′ j ) (4)

[0079] Among them, h Λ : It is a nonlinear transformation with a learnable parameter Λ. Output object features It is obtained by aggregating the edge features pointing to the object node:

[0080]

[0081] Among them, ε i Represents the set of edges pointing to node i. Therefore, the processor 104 updates the node features within the scene graph representation sequence 206a-206d based on the graph attention network 210 and the relationship graph network 212. In addition, in order to obtain higher-level features for each node of the scene graph representation 206a-206d, the updated graph is aggregated into the visual graph memory. In one implementation, the processor 104 is configured to perform pooling of the visual graph memory based on graph average pooling (GAP) and graph maximum pooling (GMP). GAP and GMP are stored in the memory 106. The processor 104 accesses the GAP pooling and the GMP pooling and provides them to the feature extraction neural network 110 for generating two graph streams representing the visual graph memory. Figure 2D, visual pattern memories such as visual pattern memory 214a, visual pattern memory 214b, visual pattern memory 214c, and visual pattern memory 214d are described in FIG.

[0082] Figure 2D A representation 200D depicting a visual graphic memory sequence is shown according to some embodiments. The visual graphic memory sequence 214a-214d includes visual graphic memory 214a, visual graphic memory 214b, visual graphic memory 214c, and visual graphic memory 214d, which are connected to form a final graphic memory.

[0083] V * =GAP(X * ,∈)||GMP(X * ,∈) (6)

[0084] Wherein, ∈ represents the connection structure of scene graph representation 206a-206d, X * Represents the final node feature

[0085] For a length of L 202a and 202c in window 216b. The processor 104 aggregates the visual graph memories 214a and 214c using the aggregate neural network 112. The visual graph memory sequences 214a-214d are set to For performing self-attention on the entire pattern memory (such as visual pattern memory 214a-214d) within window 214b. The processor 104 calculates the self-attention value using the attention-based neural network stored in the memory 106. The self-attention is calculated by the attention-based neural network using the softmax function:

[0086]

[0087] in, is the weight matrix, is the weight vector, ∝∈R S is the attention weight of video frame 202b (i.e., the center video frame) to video frames 202a-202c in window 216b. Visual pattern memory 214bv c It is updated by aggregating information across windows 216b using the attention coefficient ∝:

[0088] v c =∝f T

[0089] Sliding windows 216a-216c over a visual graphic memory (such as graphic memory 216b) of a center video frame (e.g., video frame 202b) of a video frame sequence 202a-202d provides a final graphic memory sequence 218a, 218b, 218c, and 218d. The final graphic memory sequence 218a-218d may be represented as V = {v1, v2, ..., v L}, which aggregates the spatial and temporal information of the video frames 202a-202d. The final graph memories 218a-218d are provided as input to the self-attention encoder and feed-forward network layer 220. The self-attention encoder and feed-forward network layer 220 extract features represented as feature vectors 220a, 220b, 220c, and 220d. The feature vectors 220a, 220b, 220c, and 220d are submitted to the semantic control transformer for generating a response to the query of the video 200A. The semantic control transformer encodes the background information, which is Figure 3 is further described in .

[0090] Figure 3 A schematic diagram of a semantic control transformer 300 for generating responses to queries, according to some embodiments, is shown. The semantic control transformer 300 includes two multi-head attention (MHA) network layers, such as MHA network layer 308a and MHA network layer 308b. In some embodiments, the memory 106 may be configured to store the semantic control transformer, wherein the semantic control transformer 300 may include a set of instructions for encoding context information of feature vectors 220a, 220b, 220c, and 220d to generate responses to queries. To execute the semantic control transformer 300, the processor 104 receives as input feature vectors 302, context information 304, and a query 306. Feature vectors 302 correspond to a sequence of feature vectors 220a-220d and are represented as (V x D), where D is the feature dimension. Context information 304 corresponds to the history of the video question answering (H x D). Query 306 corresponds to a question about video 200A, represented as (Q x D).

[0091] Specifically, feature vector 302, context information 304, and query 306 are provided as input to MHA network layer 308a. MHA network layer 308a encodes textual information based on context information 304 and learns a conversation model for generating a response to query 306. Furthermore, MHA network layer 308a generates an encoded feature vector 312, encoded context information 314, and an encoded query 316. The encoded feature vector 312, encoded context information 314, encoded query 316, and features (A x D) of the sub-answer 310 are provided as input to another MHA network layer 308b to generate a response to query 306. The response includes feature vectors 318a, 318b, 318c, and 318d, which are generated by shuffling the header vectors of reference answer 310, encoded feature vector 312, encoded context information 314, and encoded query 316, respectively. Since latent features are also extracted, the reordering of the head vectors improves the performance of the semantic control transformer 300. The responses are generated in an iterative manner, as Figure 3 For example, the answer is generated one word at a time, and the partial answer generated so far is encoded and passed as input to generate the next word to form the reference answer 324.

[0092] The head vectors of feature vectors 318a-318d are reordered before being fed to a feedforward network (FFN) module 320, which is then connected. FFN module 320 includes two fully connected layers with a ReLI function between them. The features of background information 304 are fused with the visual features of feature vector 302 to extract feature vector 322. A loss function (L) 326 is implemented between the predicted probability distribution P of feature vector 322 and the ground truth distribution G of the features 324 of the reference answer. In one embodiment, loss function 326 is based on the Kullback-Leibler divergence:

[0093]

[0094] In each iteration, the co-attention transformer of the semantic control transformer 300 is used to generate a word and predict the next word for the answer. In addition, all next token probability distributions are collected in batches to obtain the predicted probability distribution P. In a similar manner, the ground truth token distribution G is obtained from the ground truth answer or response to the query 306.

[0095] Thus, the semantic control transformer 300 learns the conversation model and generates a response to the query 306 .

[0096] Figure 4A schematic overview 400 of the principles of the AVSD system 100 according to some embodiments is shown. The AVSD system 100 receives an input sequence of video frames and context information 304. The processor 104 generates a scene graph representation 404 for the sequence of video frames 202a-202d. The scene graph representation 404 corresponds to Figure 2B The scene graph representations 206a-206d described in the description of FIG. The scene graph representation 404 is a semantic representation of the scene in the frame. Figure 2C As described in the description of FIG, a visual graph memory (e.g., visual graph memories 214a-214d) is generated from a scene graph representation 404 through intra-graph reasoning 306. The processor 104 uses the visual graph memories 214a-214d for inter-graph information aggregation 408 to generate a final graph memory, i.e., graph memories 218a-218d. The final graph memories 218a-218d are provided as input to the semantic control transformer 300. In addition, the processor 104 is configured to perform semantic control transformer reasoning 410 to encode the final graph memories 218a-218d, the context information 304, and the input query 412 to generate an output answer 414. In some embodiments, the input query 412 is provided to the semantic control transformer 300 to perform the semantic control transformer reasoning 410.

[0097] Typically, in language modeling, the words used to answer a sentence are predicted from a vocabulary. In one implementation, the next word prediction for the words in the answer sentence is performed based on the input query 412. The context information 304 includes source sentences, such as video subtitles, conversation history, and reference answers 310 (i.e., already generated answers). For example, the conversation history H = {C, (Q I , A I ),...,(Q l-I , A l-I )}, where C is the video subtitle, Q I It's an inquiry. is a reference answer. The semantic control transformer inference 410 generates a probability distribution of the next token of a word for all word tokens in the vocabulary of the output answer 414. The inference process of the semantic control transformer inference 410 is controlled based on the connected visual graph memories 214a-214d and the final graph memories 218a-218d.

[0098] In the semantic control transformer reasoning 410, the conversation history (H), video subtitles (C), query (Q I ) and reference answers The sentence sources are embedded together using tokenization and word position embedding layers. For example, the text source Tokenized as e h , e c , eq , e a . In one implementation, the text source is tokenized via byte pair encoding (BPE). The tokenized text source is transformed into a representation of an LW-dimensional vector corresponding to the sentence length (L) and the word embedding dimension (W) through a word position embedding layer. Each word of the tokenized text source is encoded into the position embedding space and added to the word embedding layer. In a similar manner, the target sentence is encoded into the position embedding space. The continuous representation S∈R of the text source at the input of the self-attention module is transformed using a linear transformation L×C Translated into key (k), query (q), and value (v). The self-attention module computes the attention value between the key and the query. The attention value between the key and the query enables each word in the text source to aggregate information from other words using the self-attention module.

[0099] In addition, the dimension 2d h The visual graphic memories (i.e., visual graphic memories 214c-214d) are transferred to the d that matches the LW dimension of the text source. h Dimensional feature e v Next, the tokenized reference answer (ie, reference answer 310) is transformed using the self-attention based multi-headed rearrangement transformer (ie, MHA network layer 308a). a Encode to generate the encoded hidden representation (h enc ).

[0100] h enc =FFN(Attention(W q e a , W k e a , W v e a ), (9)

[0101] Among them, W q 、W k 、W v are the weight matrices for query (q), key (k), and value (v), respectively. FFN is a feedforward network module that includes two fully connected layers with an activation function (i.e., rectified linear unit (ReLu)) in between. The encoded hidden representation corresponds to feature vectors 318a-318d.

[0102] The attention coefficient between a key (k) and a query (q) with a value (v) is determined according to the attention function defined below:

[0103]

[0104] in, is a scaling factor used to maintain the scalar in magnitude, d his the dimension of each header in the feature vector (ie, the encoded feature vector 312, the encoded context information 314, and the encoded query 316).

[0105] After encoding the input query 412, each other word and visual embedding e j Perform collaborative attention, where j∈{h,c,q,v}, with the same transformer structure as the Multi-Head Rearrangement Transformer (i.e., MHA network layer 308b):

[0106] h′ enc，j =FFN(Attention(W q h enc , W k e j ;W v e j ), (11)

[0107] Where h′ enc，j is a new encoding feature.

[0108] By concatenating the sentence source features and the visual features (i.e., feature vectors 318a-318d), the feature vector h* is extracted. enc，j . Each head vector in each sentence source feature (i.e., the encoded background information 314 and the encoded query 316) and each visual feature (i.e., the encoded feature vector 312) is rearranged by a multi-head rearrangement transformer (i.e., the MHA network layer 308b). Multi-head rearrangement enables the head vectors of the encoded feature vector 312, the encoded background information 314, and the encoded query 316 to interact from beginning to end, which improves the performance of the semantic control transformer inference 410. The head vectors are rearranged before being fed to the two fully connected layers of the FFN module 320, which are then concatenated. The concatenation fuses the features of the text source and the visual features to extract the final encoded feature vector h* enc，j . Eigenvector h* enc，j The probability distribution (p) for predicting the next token in the vocabulary vocab ). Use FFN with softmax function to predict the next logo probability distribution (p vocab ):

[0109] (p vocab )=softmax(FFN(h* enc )) (12)

[0110] In the VIn test phase, a beam search is performed using b beams to generate the answer sentence. In each step, the first b b flags with the highest confidence scores are selected. The answer ends at the flag of the generated sentence. <eos>Therefore, the processor outputs an output reply 414 to the input query 412 based on the generated answer.

[0111] Figure 5 An example scenario 500 is shown for generating a response 520 to a query 506 using a device 516 implemented with an AVSD system 100 according to an example embodiment. The device 516 can be a television, a laptop, a computer, a smartphone, or any display device. In some embodiments, the device 516 is embedded with the AVSD system 100 and is configured to process and generate a response to a query 506 from a user 502. In this case, the response related to the query 506 is generated by encoding multimodal information (such as contextual information, such as video subtitles of the video, video conversation history of the video, and audio of the video). The multimodal information is encoded as visual and semantic information of the video, such as Figure 3 As described in . Visual and scene information is provided by Figure 2A and Figure 2B The processor 104 of the AVSD system 100 described in .

[0112] In an alternative embodiment, the AVSD system 100 may be remotely coupled to the device 516 via an online connection link over a network, such as the network 124. Additionally, the user 502 is associated with an electronic device 504 that is capable of communicating with the device 516. The electronic device 504 may communicate with the device 516 via a communication link, such as a Bluetooth connection, an infrared connection, a Wi-Fi connection, or the like. In another alternative embodiment, the AVSD system 100 may be remotely coupled to the device 516 via an online connection link over a network, such as the network 124. Figure 5 ) is connected to device 516.

[0113] Furthermore, device 516 may include one or more components, such as a camera 508, a display screen 510, a microphone 512, a speaker 514, and the like. Camera 508 captures user 502 within field of view 518 of camera 508. Additionally or alternatively, camera 508 captures gestures of user 502, such as a gesture pointing to an object in the video. Accordingly, device 516 sends query 506 along with the gesture to AVSD system 100.

[0114] For example, a user 502 is watching a sports game displayed on a display screen 510 of a device 516, and the user 502 provides a query 506 about the sports game through the electronic device 504. The query 506 may be "How many players are playing on the field?" The user 502 may provide the query 506 along with a gesture pointing to the sports game. The query 506 is captured by the electronic device 504, enabling the system 100 to determine what the user 502 is asking. The electronic device 504 sends the query 506 to the device 516. Alternatively, the user 502 may provide the query 506 through the microphone 512 of the device 516. The microphone 512 receives the query 506 and provides it to the AVSD system 100 in the device 516. The AVSD system 100 processes the video of the sports game stored in the storage device 114, the input query 506, and the background information of the sports game to generate a response 520 to the query 506, as described above. Figures 2A to 2D 、 Figure 3 and Figure 4 . Thus, device 516 outputs the generated response 520 through a speaker, where response 520 may be “There are 14 players playing on the field.” Thus, device 516 implemented with AVSD system 100 facilitates a response (e.g., 520) to a query (e.g., 506) from a user (e.g., 502).

[0115] Figure 6 An exemplary representation of a table 600 including information about a video frame sequence, contextual information 402, and an input query 412 is shown, according to some embodiments. Table 600 includes columns such as video 602, video subtitles 604, conversation history 606, query 608, answer 610, and generated answer 612. Table 600 includes rows 614 and 616 indicating information corresponding to the video, contextual information, and query. The video 602 column indicates the video file containing the video frame sequence (e.g., video frames 202a-202d) of video 200A. The video file can be stored in a video file format such as flash (.flv or .swf), .mp4, audio-video interleaved (.avi), etc. The video subtitle column 604 indicates video subtitle or subtitle information for the corresponding video file. In some embodiments, the subtitles can come from a video subtitle system that generates a brief description of the entire video. The conversation history 606 column indicates the history of the video question-and-answer conversation. In some implementations, table 600 is stored in the storage device 114 of the AVSD system 100. For example, when the AVSD system 100 receives an input query such as input query 412, the input query 412 is stored in the query 608 column of the table 600 of the storage device 144. Furthermore, the processor 104 of the AVSD system 100 uses information from the video 602, the video subtitles 604, and the conversation history 606 to extract a feature vector to generate the output response 314. In one embodiment, the output response 314 is generated based on a reference response, and the generated response sentences correspond to the columns of the reference response 610 and the generated response 612, respectively. Another embodiment does not use the reference response to generate the response. The reference response is only used during training.

[0116] Additionally, in row 614, under the column for generated answers 612, one or more answers to the input query are generated. Each of the generated answers 612 is associated with a confidence score. The generated answer with the highest confidence score is selected as the output reply (e.g., Figure 3 In a similar manner, in row 616 corresponding to the generated answers 612, the answer with the highest confidence score is generated as the output answer. Figure 7A and Figure 7B The videos are further described in , namely, Video 1 and Video 2 under the column of Video 602.

[0117] Figure 7A is an example scenario 700 for generating a response 706 to an input query 704 of a video according to an example embodiment. The representation 700 depicts a video (e.g., Figure 6 1). The video is played on a device such as device 516. For example, the video shows a man in a room containing items such as a vacuum cleaner, clothes, a table, and a washing machine. When a query 704 (such as, "What do you think this room is?") is provided to the AVSD system 100, a response 708 is generated based on the received query 704.

[0118] To this end, the AVSD system 100 includes a processor 104 that processes the video frame 702 and extracts visual and semantic information from the video frame 702. In addition, the processor 104 encodes the visual and semantic information (such as the video subtitles of the video frame 702, the video conversation history, and the audio of the video frame 702) using the background information for generating a response 708. The response 708 is generated based on the generated response 612 having the highest confidence score, such as Figure 6 For example, the response 708 to the query 704 may be "It looks like a laundry room."

[0119] Figure 7B is an example scenario 710 showing a response 716 to an input query 714 for a video according to another example embodiment. User 502 is watching a video on device 516. The video corresponds to Figure 6 712 . The user 502 provides a query 714 in voice or text format. The AVSD system 100 in the device 516 includes a processor 104 that processes the video frames 712 to extract visual and semantic features of the video frames 712 . The visual and semantic features are further encoded using multimodal input data (i.e., video subtitles, video dialogue, and audio (or background information)) to generate a relevant response to the query 714. A relevant response, such as response 716, is generated for the query 714. Response 716 corresponds to the generated answer with the highest confidence score, such as Figure 6 As shown in Table 600.

[0120] Figure 7C 7 is an example scenario 720 according to another example embodiment, illustrating a response 726 to a video input query 724. User 502 is driving a vehicle, and a camera, such as a dashboard camera, captures video of a dynamic scene 712 in front of the vehicle. In this embodiment, user 502 may make a navigation query 724 to navigate the vehicle to a predefined destination 730. User 502 provides query 724 via voice. In another embodiment, the query may be collected from a pre-calculated navigation route for destination 730. AVSD system 100 in device 516 includes processor 104, which processes video frames 722 to extract visual and semantic features of video frames 722. The visual and semantic features are further encoded with multimodal input data (i.e., video conversation and audio (or background information such as GPS coordinates, depth map, etc.)) to generate a relevant response to query 724. A relevant response, such as response 726, is generated for query 724.

[0121] In this embodiment, the AVSD system 100 may use an object classifier and a relationship classifier that are configured to detect and classify objects and their relationships relevant to generating navigation instructions. For example, objects may include buildings, cars, pedestrians, utility poles, traffic lights, or any other objects relevant to the driver. Examples of relationships may include front, back, right, left, etc. In this embodiment, the AVSD system 100 is configured to generate navigation instructions using descriptions of the classified objects and their relationships to the navigation route to the destination. For example, the AVSD system 100 may generate a navigation instruction such as "Follow the car in front and turn left after passing the tree in front." In this example, the classified objects are a car and a tree. Their relationship to the navigated vehicle indicates that the car and the tree are both in front of the vehicle. Their relationship to the navigation route to the destination indicates that a left turn is required to follow the navigation route.

[0122] The embodiments are based on the recognition that it is desirable to provide route guidance to the driver of a vehicle based on real-time unimodal or multimodal information about static and dynamic objects in the vicinity of the vehicle. For example, some embodiments aim to provide context-based driving instructions, such as "turn right in front of the brown brick building" or "follow the white car," in addition to or instead of GPS-based instructions (like "turn right a second time in 100 feet onto Johnson Street."). Such context-based driving instructions can be generated based on real-time awareness of the scene in the vicinity of the vehicle. For this reason, context-based navigation is referred to herein as scene-aware navigation, which can be implemented using a dialogue system according to various embodiments.

[0123] Figure 8 A flow chart 800 of a method for generating a response to an input query by the AVSD system 100 according to some embodiments is shown. At block 802, the system 100 receives a sequence of video frames, context information, and a query. The video frame sequence corresponds to a sequence of video frames (e.g., Figure 7A and Figure 7B The background information includes Figure 6 7 . The query corresponds to input query 704 or 706 of FIG. 7 . At block 804, system 100 detects and classifies objects in each video frame of video frame sequence 202 a - 202 d. At block 806, system 100 determines a relationship between the classified objects in each video frame of video frame sequence 202 a - 202 d. Each video frame includes at least two classified objects, and the relationship between the two classified objects is an intra-frame object relationship confined within the video frame.

[0124] At block 808, the system extracts features representing the classified objects and the determined relationships for each video frame to generate a sequence of feature vectors (ie, Figure 4 Each video frame of the video frame sequence 202a-202d has a feature vector (ie, feature h*) encoding information or features of the text and visual source. enc At block 810, the system 100 submits the feature vector sequence 418a-418d, the input query 412, and the input context information to a neural network (i.e., the semantic control transformer inference 310) to generate a response to the input query 412 (i.e., the output response 314). At block 812, the system 100 renders the response 314 to the input query 412 via an output interface (such as the output interface 116).

[0125] Figure 9 Table 900 shows ablation analysis of components of the system 100 according to some embodiments. The ablation analysis demonstrates the effectiveness of the AVSD system 100 using permutations in the transformer structure, whether a graph attention network or a relational graph network is used, whether the region of interest (ROI) is re-cropped for the joint bounding box, whether visual and semantic graphs are used simultaneously, whether temporal information aggregation is performed, etc. The ablation analysis is studied using conventional evaluation metrics such as BLUE-1, BLUE-2, BLUE-3, BLUE-4, METEOR, ROUGE L, and CIDEr. The different components correspond to the permutations in the transformer structure, graph neural network type, joint bounding box, semantic label, and temporal information aggregation of the AVSD system 100.

[0126] like Figure 9 As shown, Table 900 describes ablation studies such as the full model, without (w / o) permutation, without graph attention network (GAT), without relational graph network (e.g., EdgeConv), without joint box features, without semantic graph and without temporal information. The full model uses permutation in the transformer structure, graph attention network, relational graph network, joint bounding box, visual and semantic graph and temporal information aggregation. In row 902, the full model has higher accuracy values ​​when all key components are used. For example, the graph attention network is very useful for aggregating information from neighboring nodes, while the relational graph network (i.e., EdgeConv) can provide a small improvement over the graph attention network. In addition, in the multi-head transformer structure ( Figure 3 The use of reordering in the MHA network layers 308a and 308b of

[15] improves the performance of the AVSD system 100. Furthermore, the combined aggregation of bounding boxes, semantic labels, and temporal information helps stabilize the performance of feature vector extraction for generating responses. Overall, by employing these components, the full model outperforms all ablations.

[0127] Figure 10A Table 1000A is a graph illustrating a comparison of the AVSD system 100 with baseline methods on a test split of the AVSD challenge under the Dialogue Systems Technology Challenge 7 (DSTC7), according to some embodiments. In Table 1000, quantitative results for the DSTC7 test split are shown. The results show that even without audio features, such as VGGish audio features or three-dimensional (3D) convolutional neural network (CNN) features (such as Dilated 3D ConvNet (I3D)), the AVSD system 100 achieves better performance than other methods.

[0128] By evaluating AVSD using objective metrics on DSTC7, the performance of the AVSD system 100 is compared with four baseline methods, such as a baseline method, a multimodal attention method, a simple method, and an MTN method. The baseline method is based on the DSTC challenge of extracting features for different modalities. The extracted features of different modalities are combined using simple concatenation or addition to generate a response to the input query. The multimodal attention method implements a multimodal attention that uses attention to selectively focus on salient features to generate a response. The simple method adds image features to response generation, such as VGG features and factor graph attention. The MTN method applies self-attention and co-attention to aggregate information between video, audio, and multi-round dialogue information. In addition, an answer auto-encoding loss is applied to improve performance.

[0129] Figure 10B Table 1000B shows a comparison of the AVSD system 100 and a baseline method on the test split of the Dialogue Systems Technology Challenge 8 (DSTC8), according to some embodiments. Table 1000B shows quantitative results for the DSTC8 test split with additional human ratings. The results demonstrate that the AVSD system 100 improves upon the previous baseline by leveraging a scene graph to represent the visual information of the video. As shown in Table 1000B, the AVSD system 100 improves the human rating to 3.433, a relative improvement of 12%, compared to the simple method.

[0130] Exemplary embodiments

[0131] Figure 11 A schematic overview 1100 of the principles of the AVSD system 100 according to some embodiments is shown. The AVSD system 100 receives an input video frame sequence 1102a-1102c and context information 1130, 1132, 1134. The processor 104 generates scene graph representations 1104a-1104c for the video frame sequence 1102a-1102c. The scene graph representations 1104a-1104c correspond to Figure 2B The scene graph representations 206a-206c described in the description of FIG. The scene graph representations 1104a-1104c are semantic representations of the scene in the frame. From the scene graph representations 1104a-1104c, visual graph memories (e.g., visual graph memories 1108a-1108c) are generated by intra-graph reasoning 306, such as Figure 2C As described in the description.

[0132] The processor 104 uses the visual graph memories 1108a-1108c for inter-graph information aggregation 1112 to generate a final graph memory, namely, graph memories 1128a-1128c. The final graph memories 1128a-1128c are provided as input to a semantic control transformer 1124. Furthermore, the processor 104 is configured to perform semantic control transformer reasoning 1124 to encode 304 the final graph memories 1128a-1128c, the context information, and the input query 1134 to generate an output response 1140. In some embodiments, the input query 1134 is provided to the semantic control transformer 300 for performing semantic control transformer reasoning 1124. The semantic control transformer reasoning 1124 generates a probability distribution of the next token of a word for all tokens of the word in the vocabulary of the output response 1140. The reasoning process of the semantic control transformer reasoning 1140 is controlled based on the connected visual graph memories 214a-214d and the final graph memories 218a-218d.

[0133] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the following description of exemplary embodiments will provide those skilled in the art with an effective description for implementing one or more exemplary embodiments. Various changes may be made to the function and arrangement of elements without departing from the spirit and scope of the subject matter disclosed in the appended claims.

[0134] Specific details are provided in the following description to provide a thorough understanding of the embodiments. However, it will be appreciated by those skilled in the art that embodiments may be implemented without these specific details. For example, the systems, processes, and other elements of the disclosed subject matter may be shown as components in block diagram form to avoid confusing the embodiments in unnecessary details. In other cases, well-known processes, structures, and techniques may be shown without unnecessary details to avoid confusing the embodiments. In addition, the same reference numerals and names in the various figures represent the same elements.

[0135] Furthermore, various embodiments may be described as processes illustrated as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe operations as sequential processes, many operations may be performed in parallel or simultaneously. Furthermore, the order of operations may be rearranged. A process may terminate when its operations are completed, but may have other steps not discussed or included in the diagram. Furthermore, not all operations in any specifically described process may occur in all embodiments. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the function may correspond to the function returning to the calling function or the main function.

[0136] Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, manually or automatically. Manual or automatic implementations may be performed or at least assisted by the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the necessary tasks may be stored in a machine-readable medium. A processor may perform the necessary tasks.

[0137] The various methods or processes outlined herein may be encoded as software executable on one or more processors employing any of a variety of operating systems or platforms. Furthermore, such software may be written using any of a variety of suitable programming languages ​​and / or programming or scripting tools, and may also be compiled into executable machine language code or intermediate code that is executed on a framework or virtual machine. In general, the functionality of the program modules may be combined or distributed as desired in various implementations.

[0138] The embodiments of the present disclosure may be embodied as a method, an example of which has been provided. The actions performed as part of the method may be ordered in any suitable manner. Thus, embodiments may be constructed in which the actions are performed in a different order than shown, which may include performing some actions simultaneously, even though shown as sequential actions in illustrative embodiments. Furthermore, the use of ordinal terms such as "first" or "second" in a claim to modify a claim element does not itself imply any priority, precedence, or order of one claim element relative to another claim element, or the time order in which the actions of the method are performed, but rather serves merely as a label to distinguish a claim element having a particular name from another element having the same name (but using a serial number) to distinguish the claim elements.

[0139] Although the present disclosure has been described with reference to certain preferred embodiments, it should be understood that various other changes and modifications may be made within the spirit and scope of the present disclosure. Therefore, an aspect of the appended claims is to cover all changes and modifications that fall within the true spirit and scope of the present disclosure.< / eos>

Claims

1. A scene-aware dialogue system, comprising: An input interface configured to receive a sequence of video frames, context information, and a query: a memory configured to store at least one neural network, the at least one neural network comprising a visual scene-aware conversational neural network trained to generate a response to an input query by analyzing one or a combination of an input video frame sequence and input context information provided to the neural network; A processor configured to: detecting and classifying objects in each video frame of the sequence of video frames; integrating a region of interest of an object into the sequence of video frames to determine a relationship between classified objects in each of the video frames, wherein at least one video frame of the sequence of video frames includes at least two classified objects, and wherein the relationship between the two classified objects is an intra-frame object relationship confined within the video frame; Extracting features representing the classified objects and the determined relationships for each of the video frames to generate a feature vector sequence, wherein one feature vector is present for one video frame; and submitting the sequence of feature vectors, an input query, and input context information to the neural network to generate a response to the input query; and an output interface for rendering a response to an input query, Each feature vector in the feature vector sequence corresponds to a corresponding video frame in the video frame sequence, The input query relates to one or a combination of objects, relationships between the objects, and temporal evolution of the objects in the sequence of input video frames, The input context information includes one or a combination of sound information and text information about the video frame sequence.

2. The scene-aware dialogue system according to claim 1, wherein: The neural network is a multimodal neural network configured to process information in multiple modalities; and The processor is further configured to modify the value of each eigenvector in the eigenvector sequence by using weighted values ​​of adjacent eigenvectors in the eigenvector sequence.

3. The scene-aware dialogue system according to claim 2, wherein: The value of each of the eigenvectors is determined as a weighted combination of the values ​​of a plurality of eigenvectors fitting a window centered around the eigenvector.

4. The scene-aware dialogue system according to claim 2, wherein: The at least one neural network stored in the memory includes an audio-visual scene perception dialogue (AVSD) neural network, a feature extraction neural network, and an aggregation neural network, wherein the AVSD neural network is trained to prepare a response to an input query, the feature extraction neural network is trained to represent the objects in the video frame sequence and the correspondence between the objects using a feature vector sequence, and the aggregation neural network is trained to determine the value of each feature vector of the feature vector sequence as a weighted combination of values ​​of multiple feature vectors that fit a window centered on the feature vector; and The AVSD neural network corresponds to an attention-based architecture and includes one or a combination of a faster region-based convolutional neural network, a faster RCNN, and a three-dimensional 3D convolutional neural network (CNN).

5. The scene-aware dialogue system according to claim 1, wherein: The memory stores a set of neural network-based classifiers, the set of neural network-based classifiers including object classifiers configured to detect and classify objects of predefined types in the input video frame sequence and relationship classifiers configured to classify relationships between the classified objects, wherein the processor is configured to select and execute the selected neural network-based classifier to detect and classify the objects and the corresponding relationships between the classified objects in each video frame of the input video frame sequence; and The processor is further configured to select the object classifier and the relationship classifier from the set of neural network-based classifiers based on the input video frame sequence, input context information, input query, or a combination thereof.

6. The scene-aware dialogue system according to claim 1, wherein: The memory stores an object classifier and a relationship classifier configured to detect and classify objects and their relationships relevant to generating navigation instructions for driving a vehicle, and wherein the processor is configured to generate navigation instructions using descriptions of objects and relationships relevant to a navigation route to a destination of the vehicle.

7. The scene-aware dialogue system according to claim 1, wherein: The processor is further configured to generate a spatiotemporal scene graph representation (STSGR) model for each frame of the video frame sequence based on the integrated region of interest and visual memory, and wherein the at least one neural network is trained to perform spatiotemporal relationship learning to generate responses to training queries when training the STSGR model of the video frame sequence; and wherein each STSGR model represents each corresponding video frame as a spatiotemporal visual graph stream and a semantic graph stream, and wherein the at least one neural network is a multi-headed reordering transformer for generating object-level graph reasoning, and the multi-headed reordering transformer is capable of reordering the heads of the feature vector sequence.

8. The scene-aware dialogue system according to claim 1, wherein: The processor is further configured to aggregate the classified objects and the determined relationships for generating a visual memory for each video frame of the sequence of video frames.

9. A scene-aware dialogue method, wherein: The method uses a processor in combination with stored instructions implementing the method, wherein the instructions, when executed by the processor, perform the steps of the method, including: receiving a video frame sequence, background information, and a query; detecting and classifying objects in each video frame of the sequence of video frames; integrating a region of interest of an object into the sequence of video frames to determine a relationship between classified objects in each of the video frames, wherein at least one video frame in the sequence of video frames includes at least two classified objects, and wherein the relationship between the two classified objects is an intra-frame object relationship confined within the video frame; Extracting features representing the classified objects and the determined relationships for each of the video frames to generate a feature vector sequence, wherein one feature vector is present for each video frame; submitting the sequence of feature vectors, the input query, and the input context information to at least one neural network, the at least one neural network comprising a visual scene-aware conversational neural network trained to generate a response to the input query by analyzing one or a combination of an input video frame sequence and input context information; and Rendering responses to input queries via output interfaces, Each feature vector in the feature vector sequence corresponds to a corresponding video frame in the video frame sequence, The input query relates to one or a combination of objects, relationships between the objects, and temporal evolution of the objects in the sequence of input video frames, The input context information includes one or a combination of sound information and text information about the video frame sequence.

10. The method according to claim 9, wherein: The neural network is a multimodal neural network configured to process information of different modalities; The method further comprises: modifying the value of each eigenvector of the eigenvector sequence using weighted values ​​of adjacent eigenvectors in the eigenvector sequence, the value of each eigenvector being determined as a weighted combination of values ​​of a plurality of eigenvectors fitting a window centered on the eigenvector; The at least one neural network includes an audio-visual scene perception dialogue (AVSD) neural network, a feature extraction neural network, and an aggregation neural network, wherein the AVSD neural network is trained to prepare a response to an input query, the feature extraction neural network is trained to use the feature vector sequence to represent the objects in the video frame sequence and the correspondence between the objects, and the aggregation neural network is trained to determine the value of each feature vector of the feature vector sequence as a weighted combination of values ​​of multiple feature vectors that fit a window centered on the feature vector; and The method further includes: selecting an object classifier and a relationship classifier from a set of neural network-based classifiers, and executing the selected object classifier for detecting and classifying objects of predefined types in the input video frame sequence and the relationship classifier for classifying the relationships between the classified objects, wherein the selection of the neural network-based classifier is based on input background information, the input video frame sequence, the input query, or a combination thereof.

11. The method according to claim 9, further comprising: generating a spatiotemporal scene graph representation (STSGR) model for each frame of the video frame sequence, each STSGR model representing each corresponding video frame as a spatiotemporal visual graph stream and a semantic graph stream, wherein the at least one neural network is a multi-headed reordering transformer for generating object-level graph reasoning, and wherein the neural network is trained to perform spatiotemporal relationship learning to generate responses to training queries when training the STSGR model of the video frame sequence; And aggregating the classified objects and the determined relationships to generate a visual memory for each video frame of the video frame sequence.

12. A scene-aware dialogue system, comprising: an input interface configured to receive a sequence of video frames, context information, and a query; a memory configured to store at least one neural network, the at least one neural network comprising a visual scene-aware conversational neural network, the visual scene-aware conversational neural network being trained to generate a response to an input query by analyzing one or a combination of an input video frame sequence and input context information provided to the neural network, wherein the input query relates to one or a combination of objects, relationships between the objects, and a temporal evolution of the objects in the input video frame sequence, and wherein the context information comprises one or a combination of audio information and textual information related to the input video, such that the neural network is a multimodal neural network configured to process multimodal information; A processor configured to: detecting and classifying objects in each video frame of the sequence of video frames; determining a relationship between the classified objects in each of the video frames, wherein at least one video frame in the sequence of video frames includes at least two classified objects, and wherein the relationship between the two classified objects is an intra-frame object relationship confined within the video frame; Extracting features representing the classified objects and the determined relationships for each of the video frames to generate a feature vector sequence, wherein one feature vector is present for one video frame; Modifying the value of each eigenvector in the eigenvector sequence using weighted values ​​of adjacent eigenvectors in the eigenvector sequence; and submitting the sequence of feature vectors, the input query, and the input context information to the neural network to generate a response to the input query; and An output interface renders a response to the input query.

13. A scene-aware dialogue system, comprising: an input interface configured to receive a sequence of video frames, context information, and a query; a memory configured to store: at least one neural network, the at least one neural network comprising a visual scene-aware conversational neural network trained to generate a response to an input query by analyzing one or a combination of an input video frame sequence and input context information provided to the neural network; as well as a set of neural network-based classifiers, the set of neural network-based classifiers comprising: an object classifier configured to detect and classify objects of predefined types in the sequence of input video frames; and a relationship classifier configured to classify relationships between the classified objects; A processor configured to: selecting the object classifier and the relationship classifier from the set of neural network-based classifiers based on the input video frame sequence, the input context information, the input query, or a combination thereof; executing the selected object classifier and the relationship classifier to detect and classify a correspondence between an object and the classified objects in each video frame of the input video frame sequence; determining a relationship between the classified objects in each video frame, wherein at least one video frame in the video frame sequence includes at least two classified objects, and wherein the relationship between the two classified objects is an intra-frame object relationship confined within the video frame; Extracting features representing the classified objects and the determined relationships for each video frame to generate a feature vector sequence, wherein one feature vector is present for one video frame; and submitting the sequence of feature vectors, the input query, and the input context information to the neural network to generate a response to the input query; and An output interface renders a response to the input query.

14. A scene-aware dialogue system, comprising: an input interface configured to receive a sequence of video frames, context information, and a query; a memory configured to store: at least one neural network, the at least one neural network comprising a visual scene-aware conversational neural network trained to generate a response to an input query by analyzing one or a combination of an input video frame sequence and input context information provided to the neural network; as well as an object classifier and a relationship classifier configured to detect and classify objects and their relationships relevant to generating navigation instructions for driving a vehicle; A processor configured to: detecting and classifying objects in each video frame of the sequence of video frames; determining a relationship between the classified objects in each video frame, wherein at least one video frame of the sequence of video frames includes at least two classified objects, and wherein the relationship between the two classified objects is an intra-frame object relationship confined within the video frame; Extracting features representing the classified objects and the determined relationships for each of the video frames to generate a feature vector sequence, wherein one feature vector is present for one video frame; and submitting the sequence of feature vectors, the input query, and the input context information to the neural network to generate a response to the input query; and an output interface that renders a response to the input query, The processor is further configured to generate navigation instructions using the descriptions and relationships of objects related to a navigation route to a destination of the vehicle.

15. A scene-aware dialogue system, comprising: an input interface configured to receive a sequence of video frames, context information, and a query; a memory configured to store at least one neural network, the at least one neural network comprising a visual scene-aware conversational neural network trained to generate a response to an input query by analyzing one or a combination of an input video frame sequence and input context information provided to the neural network; A processor configured to: detecting and classifying objects in each video frame of the sequence of video frames; determining a relationship between the classified objects in each of the video frames, wherein at least one video frame of the sequence of video frames includes at least two classified objects, and wherein the relationship between the two classified objects is an intra-frame object relationship confined within the video frame; Extracting features representing the classified objects and the determined relationships for each of the video frames to generate a feature vector sequence, wherein one feature vector is present for one video frame; and submitting the sequence of feature vectors, the input query, and the input context information to the neural network to generate a response to the input query; and an output interface that renders a response to the input query, Wherein, the processor is further configured to generate a spatiotemporal scene graph representation (STSGR) model for each frame of the video frame sequence based on the integrated region of interest and visual memory, and wherein the at least one neural network is trained to perform spatiotemporal relationship learning when training the STSGR model of the video frame sequence to generate responses to training queries.

16. A scene-aware dialogue method, wherein: The method uses a processor in combination with stored instructions implementing the method, wherein the instructions, when executed by the processor, perform the steps of the method, including: receiving a video frame sequence, context information, and a query, wherein the input query relates to one or a combination of objects in the input video frame sequence, relationships between objects in the input video frame sequence, and a temporal evolution of the objects in the input video frame sequence, and wherein the context information comprises one or a combination of audio information and text information related to the input video; detecting and classifying objects in each video frame of the sequence of video frames; determining a relationship between the classified objects in each of the video frames, wherein at least one video frame of the sequence of video frames includes at least two classified objects, and wherein the relationship between the two classified objects is an intra-frame object relationship confined within the video frame; Extracting features representing the classified objects and the determined relationships for each of the video frames to generate a feature vector sequence, wherein one feature vector is present for one video frame; Modifying the value of each eigenvector of the eigenvector sequence using weighted values ​​of adjacent eigenvectors in the eigenvector sequence, and determining the value of each eigenvector as a weighted combination of values ​​of multiple eigenvectors fitting a window centered on the eigenvector; submitting the sequence of feature vectors, the input query, and the input context information to at least one neural network comprising a visual scene-aware conversational neural network, the visual scene-aware conversational neural network being trained to generate a response to the input query by analyzing one or a combination of an input video frame sequence and the input context information, wherein the at least one neural network is a multimodal neural network configured to process information of different modalities; and A response to the input query is rendered via an output interface.

17. A scene-aware dialogue method, wherein: The method uses a processor in combination with stored instructions implementing the method, wherein the instructions, when executed by the processor, perform the steps of the method, including: receiving a video frame sequence, background information, and a query; detecting and classifying objects in each video frame of the sequence of video frames; determining a relationship between the classified objects in each of the video frames, wherein at least one video frame in the sequence of video frames includes at least two classified objects, and wherein the relationship between the two classified objects is an intra-frame object relationship confined within the video frame; Extracting features representing the classified objects and the determined relationships for each of the video frames to generate a feature vector sequence, wherein one feature vector is present for one video frame; Generate a spatiotemporal scene graph representation (STSGR) model for each frame of the video frame sequence, each STSGR model representing each corresponding video frame as a spatiotemporal visual graph stream and a semantic graph stream; submitting the sequence of feature vectors, the input query, and the input context information to at least one neural network comprising a visual scene-aware conversational neural network, the visual scene-aware conversational neural network trained to generate a response to the input query by analyzing one or a combination of an input video frame sequence and the input context information, wherein the at least one neural network is a multi-headed reordering transformer for generating object-level graphical reasoning, and wherein the neural network is trained to perform spatiotemporal relationship learning when training an STSGR model of the video frame sequence to generate responses to training queries; and A response to the input query is rendered via an output interface.

Images