Method and system for scene aware audiovisual presentation

Abstract

The embodiments disclose a method and system for a scene understanding audiovisual representation of a scene. The scene understanding audiovisual representation corresponds to a graph of nodes connected by edges. The nodes in the graph indicate the video features of objects in the scene. An edge in the graph connecting two nodes indicates the interaction of the corresponding two objects in the scene. In the graph, at least one or more edges are associated with the audio features of the sound generated by the interaction of the corresponding two objects. The graph of the audiovisual representation of the scene can be used to perform a variety of different tasks. Example tasks include one or a combination of action recognition, anomaly detection, sound localization and enhancement, removal of noisy background sounds, and system control.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to audiovisual scene aware recognition, and more particularly, to methods and systems for processing scene aware audiovisual representations. [Background technology]

[0002] Over the years, various techniques have been developed to capture scenes of the surrounding environment. For example, the concept of photography and imaging was developed hundreds of years ago and has gradually migrated itself into the digital domain. Initially, digital images were defined as two-dimensional (2D) representations composed of pixels, each of which has a finite, discrete numerical representation of its intensity or gray level. However, pixel representations are not always convenient for different image-related applications. Therefore, alternative representations of the surrounding environment have been developed.

[0003] For example, distance field image representations are useful in a variety of graphics applications, including anti-aliasing, ray marching, and texture synthesis. As an example, distance fields have been widely used in simulation of numerical control (NC) applications. Scalable Vector Graphics (SVG) is an Extensible Markup Language (XML)-based vector image format for 2D graphics with support for interactivity and animation. SVG-based representations have been found to be advantageous for map-based applications. Additionally or alternatively, various feature extraction and compression techniques have been developed for more compact and more efficient image representation and storage. For example, various object detection and segmentation applications have been developed to analyze features extracted from pixels rather than pixel intensities.

[0004] A similar trend to find different image representations for different applications is observed in the video domain. However, the video domain presents more challenges because the representation should capture not only the spatial variations of objects in the spatial domain, but also their evolution in the time domain. Therefore, many different representations have been developed to capture and store video files. For example, various lossy and lossless compression techniques have been developed to exploit similarities in adjacent video files, various feature extraction methods have been developed to extract features in both spatial and time domains, and various graph-based methods have been developed to represent the temporal and spatial relationships of different objects in a video.

[0005] To further represent environmental scenes, digital audio techniques are used in the recording, manipulation, mass production, and distribution of sounds, including songs, instrumental pieces, podcasts, sound effects, and other sound recordings. Various techniques depend on the appropriate audio representation. For example, different audio coding formats have been developed for storing or transmitting digital audio. Examples of audio coding formats include MP3, AAC, Vorbis, FLAC, and Opus. Also, similar to image processing, various techniques have been proposed to work with features extracted from audio files. These feature-based techniques are widely used in automatic speech recognition systems.

[0006] Scene representation becomes an even more challenging problem when there is a need to provide a common representation for both audio and video modalities. Although these modalities may represent the same scene, these different modalities may not be naturally aligned with each other. Therefore, modern techniques usually address these modalities separately by adding pre-processing and post-processing techniques. Examples of pre-processing techniques include deriving cues from video files to assist audio processing and vice versa. Examples of post-processing techniques include providing audio captions to annotate the video.

[0007] However, none of these techniques are suitable for the complex and rich representation of audiovisual scenes in an environment. Therefore, there is a need for new representations of audiovisual scenes that are suitable for other applications and techniques. In addition, there is a need for systems and methods configured to generate and process such new representations. Summary of the Invention

[0008] It is therefore an objective of some embodiments to provide a system and method for processing an audiovisual representation of a scene in an efficient and accurate manner. Examples of such processing include generating an audiovisual representation and performing application-specific tasks based on the audiovisual representation. Thus, in one embodiment, a graph-based representation is generated that illustrates a scene-grasping audiovisual representation of a video scene. The video may be captured using one or more image / video capture devices, such as one or more cameras. The one or more cameras may be connected to a system, such as a computing system, for generating the scene-grasping audiovisual representation.

[0009] Some embodiments are based on the realization that a scene understanding audiovisual representation comprises a structure that is mapped onto a graph of nodes connected by edges. Each node in the graph represents an object, e.g., a video feature of an object in the scene. Each edge in the graph connects two nodes. Each edge in the graph indicates an interaction of two corresponding objects in the scene. In such a manner, the graph represents objects and their interactions.

[0010] In some embodiments, the graph representation may be generated based on the detection and classification of objects in a scene. Also, the nodes (i.e., objects) and edges (i.e., object interactions) for the graph representation may be selected based on a predefined set of possible objects and a predefined set of possible interactions. The selection of nodes and edges enforces the uniqueness of the objects and interactions. For example, if a group of people does not belong to the set of possible objects, the object may be classified as a person rather than as a group of people.

[0011] Some embodiments are based on the understanding that a scene includes complex interactions of two objects, including one or a combination of spatial and temporal interactions. For example, a captured scene may include dynamic content with different objects showing people, vehicles, or animals, different sounds in the spatial domain, and the evolution of the corresponding objects in the time domain. The scene may also include different interactions or relationships that exist between the objects. In a real-time example scenario, the scene may correspond to a busy street with objects such as people taking different actions, moving vehicles, parked vehicles, running animals, etc. The people taking different actions may include musicians with their own instruments. In this scene, some instruments may be actively played by the corresponding musicians, while some instruments may not be played.

[0012] Additionally or alternatively, an objective of some embodiments is to generate a graph representation with complex interactions, including spatial and temporal interactions of objects. To that end, in some embodiments, the graph representation is composed of nodes that are fully connected by edges, each of which represents an object in the scene, and each of which represents a fully connected edge represents a corresponding interaction between two objects in the spatial and temporal domains of the scene.

[0013] Some embodiments are based on the understanding that in an interaction, some objects may contribute equally to the generation of a sound. In some cases, an object may generate a sound while interacting with another object and may also contribute solely to the generation of a sound. For example, a musician may sing while playing a guitar in a scene. In some other cases, some objects may not contribute equally to the generation of a sound. For example, a musician may play a violin without singing. In such cases, objects that do not contribute equally to the generation of a sound may be defined based on their type of interaction. Thus, in some embodiments, corresponding edges of such objects may include an attribute that defines the type of interaction. In some exemplary embodiments, the type of interaction is indicated by the direction of the edge, and corresponding nodes (i.e., objects) connected by the directed edge may be identified as a main object and a context object. For example, a node indicating a violin played by a musician corresponds to a main object, and a musician playing a violin corresponds to a context object.

[0014] In some implementations, nodes and / or edges in the graph may be associated with audio features of sounds generated by objects or by object interactions. For example, a node of an object such as a dog barking while splashing in water may be associated with a corresponding audio feature. Similarly, edges connecting a primary object and a context object may also be associated with an audio feature. Additionally or alternatively, another objective of some embodiments is to extract features such as video and audio features that include both spatial and temporal information of a scene.

[0015] Some embodiments are based on the understanding that joint extraction of video and audio features can improve the overall performance of the system. Joint extraction of video and audio features can eliminate pre-processing and post-processing steps, which can improve the overall performance.

[0016] Therefore, some embodiments disclose an audio source separation framework that separates individual sounds of an object from an audio mixture. The audio mixture may correspond to a mixed audio spectrogram that includes frequency changes of different audio signals. The sounds may be separated from the audio mixture based on the visuals of the object from the video and the corresponding interactions of the object.

[0017] In some embodiments, the audio source separation framework may include a neural network. The neural network may include multiple sub-networks trained end-to-end to associate audio features with nodes and / or edges of a graph. The multiple sub-networks may include attention-based sub-networks trained to provide different attention to different interactions of objects.

[0018] The trained attention-based sub-network can be used to derive visual features of objects in a scene, which correspond to latent auditory elements that guide the audio source separation framework to separate the sound of the object from the sound mixture. For example, the visual features of a musical instrument such as a violin in the scene can guide the audio source separation framework to separate the sound of the violin from the sound mixture.

[0019] In some embodiments, visual features may be extracted as one or more subgraphs from the graph representation. The one or more subgraphs may provide an embedding vector (i.e., a low-dimensional vector representation) that provides orthogonality to the audio features. Also, the scene may include various sounds for each object. Therefore, the audio features associated with different objects and / or edges are orthogonal to each other. The orthogonality of the audio features enhances the separation of different individual sounds from the various sounds. In some embodiments, the audio features associated with edges may be represented by a binary mask. The binary mask separates the desired sound of the object by separating the corresponding audio features of the object from the sound mixture.

[0020] Some embodiments are based on the understanding that the scene understanding audiovisual representation can serve as an input to other processes or applications. For example, a computing system can provide the generated scene understanding audiovisual representation to a server to perform tasks of corresponding applications, such as action recognition and anomaly detection, sound localization and enhancement, etc. The computing system can be trained with the scene understanding audiovisual representation using various machine learning methods. Thus, the structure of the audiovisual representation can be used as supervised learning to train a neural network to perform a task. Additionally or alternatively, another objective of some embodiments is to train a neural network using the generated scene understanding audiovisual representation.

[0021] In such manner, graphs according to different embodiments may capture rich and complex audiovisual representations of a scene. In addition, the graphs may have identifiable properties that allow the graphs to be generated by trained neural networks, and the generated graphs to be used to train other neural networks.

[0022] Accordingly, some embodiments disclose a non-transitory memory configured to store an audiovisual representation of a scene as a graph of nodes connected by edges. A node in the graph represents a video characteristic of an object in the scene. An edge in the graph connecting two nodes represents an interaction of two corresponding objects in the scene. In the graph, at least one or more edges are associated with an audio characteristic of a sound produced by the interaction of the two corresponding objects.

[0023] Accordingly, some other embodiments disclose a system for generating an audiovisual representation of a scene. The system includes an input interface, a processor, and an output interface. The input interface is configured to accept audio and video frames of the scene. The processor is configured to execute a trained neural network to convert the accepted audio and video frames into a graph of the audiovisual representation of the scene. The graph includes nodes connected by edges, where the nodes in the graph indicate video features of objects in the scene, an edge in the graph connecting two nodes indicates an interaction of two corresponding objects in the scene, and at least one or more edges in the graph are associated with an audio feature of a sound generated by the interaction of the two corresponding objects. The output interface is configured to store the graph of the audiovisual representation of the scene in a memory.

[0024] Accordingly, some other embodiments disclose a system including a memory configured to store an audiovisual representation of a scene as a graph of nodes connected by edges, where the nodes in the graph represent video features of objects in the scene, an edge in the graph connecting two nodes represents an interaction of two corresponding objects in the scene, and at least one or more edges in the graph are associated with an audio feature of a sound produced by the interaction of the two corresponding objects. The system further includes a processor operatively connected to the memory for performing a task using the graph of the audiovisual representation of the scene. [Brief description of the drawings]

[0025] [Figure 1A] FIG. 2 illustrates a graphical representation of an audiovisual representation of a scene according to some embodiments. [Figure 1B] FIG. 13 illustrates a graphical audiovisual representation of a scene according to some other embodiments. [Figure 1C] FIG. 13 illustrates a graphical audiovisual representation of a scene according to some other embodiments. [Figure 1D] FIG. 13 illustrates a graphical audiovisual representation of a scene according to some other embodiments. [Figure 1E] FIG. 13 illustrates a graphical audiovisual representation of a scene according to some other embodiments. [Diagram 2] FIG. 1 is a block diagram of a system for generating an audiovisual representation of a scene, according to some embodiments. [Figure 3A] FIG. 2 is a schematic diagram illustrating the extraction of audio and video features from video frames to generate an audio-visual representation of a scene, according to some embodiments. [Figure 3B] FIG. 2 is a schematic diagram illustrating the extraction of audio and video features from video frames to generate an audio-visual representation of a scene, according to some embodiments. [Figure 3C]FIG. 1 is a schematic diagram illustrating a framework for addressing audio source separation tasks, according to some embodiments. [Figure 4] 1 is a block schematic diagram of a system for performing a task using a graph of an audiovisual representation of a scene, according to some example embodiments. [Diagram 5] 5 illustrates an example scenario corresponding to a task using the system of FIG. 4, according to some example embodiments. [Figure 6] 5 illustrates an example scenario corresponding to a task using the system of FIG. 4, according to some other example embodiments. [Figure 7] 5 illustrates an example scenario corresponding to a task using the system of FIG. 4, according to some other example embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without these specific details. In other instances, the devices and methods are shown only in block diagram form in order to avoid obscuring the present disclosure.

[0027] As used in this specification and claims, the terms "for example," "such as," "including," and the verbs "comprise," "have," "include," and other verb forms thereof, when used in conjunction with a list of one or more components or other items, should each be construed as open-ended. That is, the list should not be considered to exclude other additional components or items. The term "based on" means based at least in part on. It should also be understood that the language and terminology employed herein is for purposes of illustration and should not be considered limiting. Any headings used within this description are for convenience only and have no legal or limiting effect.

[0028] 1A shows a representation 100 of a graph 102 showing a scene understanding audiovisual representation of a scene, according to some embodiments. The scene is represented by a video 108. The graph 102 includes nodes such as node 104A, node 104B, node 104C, node 104D, node 104E, node 104F, and node 104G (hereinafter referred to as nodes 104A-104G). The nodes 104A-104G are connected by edges such as edge 106A, edge 106B, edge 106C, edge 106D, edge 106E, edge 106F, and edge 106G (hereinafter referred to as edges 106A-106G). Each of the nodes 104A-104G represents a video feature of an object in the scene. For example, node 104A indicates a dog in the scene, node 104B indicates water, node 104C indicates a pool, node 104D indicates a walkway, node 102E indicates a plant, node 104F indicates a tree, and node 104G indicates a tank. Each edge of edges 106A-106G connecting two nodes indicates an interaction of the corresponding two objects in the scene. For example, edge 106A connecting node 104A (i.e., dog) and node 104B (i.e., water) indicates an interaction of the dog with the water in the scene. In some cases, at least one of nodes 104A-104G may be associated with an audio feature of a sound generated by the corresponding object without interaction with other objects in the scene. Similarly, edges 106B, 106C, 106D, 106E, 106F, and 106G indicate an interaction of the corresponding two objects as shown in FIG. 1A.

[0029] Also in the graph 102, some of the nodes 104A-104G and / or edges 106A-106G are associated with audio features of sounds. For example, node 104A is associated with audio features of a dog barking in the scene. For example, in this example, edge 106A is associated with audio features of a water splashing sound. In the scene, the water splashing sound is generated due to an interaction between the dog (i.e., node 104A) and the water (i.e., node 104B). The audio features may be obtained from the audio mixture 110.

[0030] In some exemplary embodiments, a scene may include a complex interaction of two corresponding objects. Such a complex interaction may be included in a graph representation of the scene, which will now be described with reference to FIG. 1B.

[0031] 1B shows a representation 112 representing a graph 114 showing a scene understanding audiovisual representation of a scene according to some other embodiments. As shown in FIG. 1B, the scene corresponds to a video 116. In some embodiments, the interaction of two corresponding objects in the scene may include one or a combination of spatial and temporal interactions of the objects. The spatial and temporal interactions of the objects may be shown in a graph 114. As shown in FIG. 1B, the graph 114 includes nodes that are fully connected to each other by edges of the graph 114. The nodes that are fully connected by edges of the graph 114 indicate complex interactions of the objects including spatial and temporal information.

[0032] In some cases, the type of interaction may be indicated, which is now explained with reference to FIG.

[0033] 1C shows a representation 118 of a scene graph 120 according to some other embodiments. The graph 120 corresponds to a scene in a video 122 that includes musicians, instruments, and the like.

[0034] In some cases, several objects in a scene that interact with each other may contribute equally to the production of sound. For example, one musician may sing while playing an instrument such as a xylophone. In some other cases, the objects may not contribute equally to the production of sound. For example, another musician may only play an instrument such as a flute. In such cases, the interaction between the musician and the flute may be indicated by the direction of the edges connecting the objects, i.e., the musician and the flute.

[0035] In the graph 120, node 124A represents a flute and node 124B represents a musician playing the flute. Also, each of the objects, i.e., the flute and the musician, may be identified as a main object and a context object. An edge 126 of the graph 120 associated with an audio feature connects a main object that generates a sound with a context object that causes the main object to generate the sound. Edge 126 has a direction pointing from node 124B of the context object to node 124A of the main object. In particular, the object that generates the sound, i.e., the flute, is defined as the main object, and the object that causes the generation of the sound by playing the flute, i.e., the musician, is defined as the context object. Nodes 124A and 124B are connected by a directed edge, such as edge 126. Also, edge 126 may be associated with an audio feature corresponding to the flute sound generated by the interaction of the musician with the flute. The audio feature corresponding to the flute sound may be associated with edge 126 by determining a corresponding visual feature of the flute sound in the scene.

[0036] In some embodiments, one or more subgraphs of graph 120 may be created for a visual feature in a scene, which is illustrated in FIG.

[0037] 1D shows a representation 128 of a graph 130 of a scene according to some other embodiments. 114 The graph 130 corresponds to an image, e.g. 116 The graph 130 includes nodes well connected by multiple edges that indicate complex interactions of objects corresponding to spatial and temporal information of the scene. The edges may also be associated with audio features corresponding to sounds generated by the object interactions. The audio features are associated with the edges based on visual features of the objects. The visual features may be derived from a subgraph 134 of the graph 130. The subgraph 134 may provide an embedding vector that provides orthogonality to the audio features. Due to the orthogonality of the audio features, the memory space required to store the graph 130 may be reduced. The embedding vector may also be used in separating audio features from an audio mixture of multiple sounds in the scene. The audio mixture may be represented by a mixed audio spectrogram 132.

[0038] Similarly, different speech features for different interactions may be separated using one or more subgraphs, such as subgraph 134, from a complete graph, such as graph 130. Such subgraphs are further illustrated in FIG. 1E.

[0039] 1E illustrates a representation 138 representing one or more subgraphs, such as subgraph 142A and subgraph 142B, of a graph of a scene of video 140, e.g., graph 130, according to some embodiments. Subgraph 142A corresponds to visual features 140A in video 140, and subgraph 142B corresponds to visual features 140B in video 140. For example, visual features 140A include features of objects, such as people, and visual features 140B include features of objects, such as vehicles.

[0040] Thus, subgraph 142A includes nodes that represent people in the scene and edges that represent people interactions. Subgraph 142B includes nodes that represent vehicles, such as cars, trains, etc., in the scene and edges that represent vehicle interactions. The edges of each of graphs 142A and 142B may be associated with audio features such that audio features associated with different edges are orthogonal to each other, which helps in separating the corresponding audio features of people and vehicles from various sounds in video 140.

[0041] Such a graph representation, including information of different objects in a scene and their different interactions, such as spatial and temporal interactions, may be suitable for a complex and rich representation of an audiovisual scene of an environment. The graph representation may be generated by a system, which will now be described with reference to FIG.

[0042] 2 is a block diagram of a system 200 for generating an audiovisual representation of a scene, according to some embodiments. The system 200 includes an input interface 202, a processor 204, an output interface 206, and a memory 208. In some exemplary embodiments, the memory 208 is configured to store a neural network 210. In some other exemplary embodiments, the memory 208 may be further configured to store an audio mixture of a plurality of sounds in the scene.

[0043] The input interface 202 is configured to accept audio and video frames of a scene. Some embodiments are based on the understanding that joint extraction of video and audio features may improve the overall performance of the system 200. Accordingly, the processor 204 may be configured to execute a neural network 210 to convert the accepted audio and video frames into a graph (e.g., graph 102) of an audio-visual representation of the scene including nodes (e.g., nodes 104A-104G) connected by edges (e.g., edges 106A-106G).

[0044] In the graph 102, at least one or more edges (e.g., edge 106A) are associated with the audio features of the sounds generated by the interaction of two corresponding objects (e.g., node 106A and node 106B). Such a multimodal graph (e.g., graph 102) enables seamless inter-modality interaction, which is useful in performing various tasks such as multi-task learning. Therefore, the audio features of the corresponding sounds are separated from the audio mixture (e.g., mixed audio spectrogram 132). The mixed audio spectrogram 132 includes a mixture of multiple sounds of a scene or various sounds for each type of main object. The separation of audio features is described in more detail in Figures 3A and 3B.

[0045] 3A and 3B show a schematic diagram 300 illustrating the separation of audio and video features from a video 302 to generate a scene graph (e.g., graph 102) according to some embodiments. The separation of audio and video features is performed by system 200. Video 302 is provided as an input to system 200.

[0046] In some example embodiments, objects in the video 302 may be detected and classified using object detection and classification techniques that may include, but are not limited to, faster Region-based Convolutional Neural Network (R-CNN) models and ResNet, which generates bounding boxes for object detection in video frames 302. ResNet may be pre-trained based on a dataset, such as the Visual Genome dataset.

[0047] After object detection and classification, video features of the object may be extracted from the video 302. In an exemplary embodiment, the video features may be extracted as feature vectors. For example, the feature vectors of the video features from the video frames 302 may include a 2048 dimensional vector. Also, audio features from the video frames 302 may be detected based on a trained audio dataset. For example, audio features corresponding to a music dataset may be trained on an audio dataset, such as an AudioSet dataset. The audio features may be extracted as feature vectors, such as a 512 dimensional vector. In some embodiments, to maintain consistency of feature dimensions of audio and video features, the dimensional vectors of the video features may be encoded according to the dimensions of the audio features. Thus, the video dimensional vectors may be encoded to the dimensions of the audio features. For example, the 2048 dimensional vectors of the video features are encoded to 512 dimensions. In this way, the video dimensional vectors of the video features are aligned to the dimensional vectors of the audio features, and a common representation for different audio and video modalities is obtained. The alignment of different audio and video modalities also allows a system (e.g., system 200) to perform multiple tasks simultaneously while precluding pre-processing and / or post-processing tasks (e.g., captioning). In some exemplary embodiments, the dimensional vectors are rectified linear unit It may be encoded using an artificial neural network such as a two-layer multilayer perceptron with a rectified linear (ReLU) activation function.

[0048] Also, a graph 304 is constructed from the extracted dimensional vectors of the video frames 302. The graph 304 is provided as an input to a Graph Attention Network (GAN) and a Recurrent Neural Network (RNN) 306. In an example embodiment, the GAN and RNN 306 may process the graph 304 including: i) updating the nodes of the graph 304, ii) capturing pairwise interactions between the nodes of the graph 304, and iii) pooling feature vectors of audio and video features.

[0049] The GANs and RNNs 306 may include components such as graph attention network convolution (GATConv), edge convolution, and feature vector pooling. GATConv updates the features of nodes in the graph 304 based on edge proximity information and multi-headed graph message passing, e.g., four attention heads for message passing.

[0050] After updating the nodes, the GAN and RNN 306 may capture paired interactions, i.e., edges of the nodes, using edge convolution. In an exemplary embodiment, the edge convolution captures the connectivity vector of two nodes of the graph 304. For example, the edge convolution captures the connectivity vector of two nodes (512×2=1024). Thus, the dimension vector for the paired interactions is a 512-dimensional vector. The feature vectors corresponding to the updated nodes and paired interactions are pooled to obtain a graph attention embedding vector 308. The graph attention embedding vector 308 is a single vector representation. In an exemplary embodiment, the feature vectors may be pooled by using a pooling technique such as a Global Max and Average pool technique to obtain the graph attention embedding vector 308.

[0051] The graph attention embedding vector 308 is provided as an input to an audio source separation framework 310. The graph attention embedding vector 308 corresponds to visual features derived from the video 302 (e.g., visual features 140A and 140B in FIG. 1E ) that guide the audio source separation framework 310 to separate audio features of corresponding objects and their interactions.

[0052] The audio source separation framework 310 may include an attention-based neural network architecture that derives visual features corresponding to objects and their interactions. The attention-based neural network architecture may be trained to provide an attention value for each edge, i.e., the interaction of two corresponding objects in the graph 304. The attention value corresponds to the importance of the edge, i.e., the importance of the interaction of the two corresponding objects in the graph. The audio source separation framework 310 may include skip connections 314 that provide the output of one layer as an input to the next layer by skipping the layer in between. The attention-based neural network architecture of the audio source separation framework 310 is further described in FIG. 3C.

[0053] The audio source separation framework 310 also generates a separated mask 316. The separated mask 316 may include an embedding vector that provides orthogonality to the audio features. The orthogonality of the audio features can improve computation speed and allows for unique separation of the desired audio from the mixed audio. In an exemplary embodiment, the audio source separation framework 310 separates the audio sources of the objects and / or interactions from the mixed audio spectrogram 312 based on the embedding vector 308. The mixed audio spectrogram 312 corresponds to the mixed audio spectrogram 132. The separated mask 316 is also provided to a converter 318 for converting the audio features into separated audio. The separated audio is provided as an output 320.

[0054] 3C shows a schematic diagram representing an audio source separation framework 322 corresponding to an audio source separation task according to some embodiments. In one exemplary embodiment, the audio source separation task may correspond to separating a piano sound from an audio mixture of a video, for example, video 302.

[0055] The audio source separation framework 322 includes a neural network 326. The neural network 326 includes multiple sub-networks corresponding to multiple RNNs, such as RNN 326A, RNN 326B, and RNN 326C (i.e., multiple RNNs 326A-326C). The multiple RNNs 326A-326C may be trained end-to-end to associate sounds with interactions of corresponding two objects. The multiple RNNs 326A-326C include attention-based sub-networks trained to place different attentions on different interactions of different pairs of objects. In some exemplary embodiments, the multiple RNNs 326A-326C may be trained using the graph 324. For example, the multiple RNNs 326A-326C may be trained to provide different attentions to different interactions of different pairs of objects in the graph 324. In some embodiments, multiple RNNs 326A-326C may perform supervised learning (particularly self-supervised learning) to provide different attention to different interactions of different pairs of objects in the graph 324.

[0056] Graph 324 is a fully connected graph having nodes such as node 324A, node 324B, node 324C, and node 324D. Nodes 324A, 324B, 324C, and 324D (or nodes 324A-324D) represent objects such as musicians and instruments in a scene. For example, nodes 324A and 324B correspond to musicians, and nodes 324C and 324D correspond to instruments such as piano and guitar, respectively. Nodes 324A-324D are fully connected to each other by edges of graph 324. Such multimodal alignment of audio and video features allows the system to jointly learn multiple tasks.

[0057] The trained sub-networks 326A-C may provide an embedding vector 328 that provides orthogonality to the audio features of the objects and their interactions. The embedding vector may be used in separating audio features from the guitar and piano audio mixture represented by the mixed audio spectrogram 332. The mixed audio spectrogram 332 corresponds to the mixed audio spectrogram 312.

[0058] The embedding vectors 328 include embedding vector 328A (E1), embedding vector 328B (E2), and embedding vector 328C (E3). Each of the embedding vectors 328A, 328B, and 328C corresponds to a distinct audio generating object in the scene. The embedding vectors 328A-328C may associate audio features to nodes or edges. Thus, the embedding vectors 328A-328C generate a spectrogram mask 328 for the audio of the corresponding object or interaction. The spectrogram mask 328 corresponds to the separated mask 316 of FIG. 3B. In some embodiments, the embedding vectors 328A-328C are associated with edges by using a binary mask. The binary mask, when applied to the mixed audio spectrogram 322, generates the associated audio features.

[0059] The spectrogram mask is also provided to the audio encoder-decoder 330 for audio source separation of the object from the mixed audio spectrogram 332. The mixed audio spectrogram 332 corresponds to the mixed audio spectrogram 312. For example, the mixed audio spectrogram 332 includes a mixture of audio corresponding to musical instruments such as a guitar and a piano.

[0060] When the mixed audio spectrogram 332 is passed to the audio encoder-decoder 330, the embedding vector 328 removes the corresponding object audio, such as the piano audio, from the mixed audio spectrogram 332. The separated audio of the piano is obtained as output 334. After training, the multiple sub-networks 326 learn to generate spectrogram masks that remove the object audio from the mixed audio spectrogram 332.

[0061] Thus, the graph 324 of the scene understanding audiovisual representation is provided as an input to a system for performing different application tasks. Because the graph 324 contains well-connected nodes 324A-324D, a system using such a graph becomes able to handle new tasks efficiently with low computational complexity and low storage consumption.

[0062] A system for performing tasks using graphs is now further described in FIG.

[0063] 4 is a block schematic diagram of a system 400 for performing a task using a graph of an audiovisual representation of a scene, according to some exemplary embodiments. The system 400 includes a memory 402 that stores an audiovisual representation graph 404, and a processor 406 operatively connected to the memory 402. The memory 402 may be a non-transitory memory. The audiovisual representation graph 404 corresponds to the graph 324. The processor 406 is configured to perform a task using the audiovisual representation graph 404. The tasks may include one or a combination of action recognition, anomaly detection, sound localization and enhancement, removal of noisy background sounds, and system control.

[0064] One or a combination of action recognition, anomaly detection, sound localization and enhancement, noisy background sound removal, and system control are further described with reference to FIGS. 5, 6, and 7. Exemplary embodiments

[0065] 5 illustrates an example implementation of an example use case 500 of system 400 according to some example embodiments. Use case 500 corresponds to an anomaly detection application, such as a security surveillance application. In an illustrative example scenario, an event occurring at a location 502 may be captured by one or more cameras, such as camera 504A and camera 504B, installed at different points of the location 502. Location 502 may include indoor or outdoor areas, such as private or public property.

[0066] The captured events are provided as input to a system, such as system 400, via a wired or wireless network connection. System 400 processes the captured events to recognize actions and detect anomalous actions or events using a graph of the audiovisual representation of the scene, such as graph 404.

[0067] For example, cameras 504A and 504B capture footage of breaking a window 506. Each of cameras 504A and 504B captures footage of a person 508 breaking the window 506 with a stick. As shown in FIG. 5, camera 504A captures the front side of person 508 and camera 504B captures the back side of person 508. The captured footage of such an event is sent to system 400. System 400 generates a graph corresponding to the scene of the event captured by both cameras 504A and 504B. System 400 recognizes the actions of person 508 and detects anomalous actions based on the graph.

[0068] Additionally or alternatively, system 400 may locate and highlight sounds, such as the sound of window 506 breaking, to detect anomalous events. For example, camera 504B may be occluded to capture a stick in the hand of person 508. In some cases, the atmosphere of location 502 may affect the capture of footage by cameras 504A and 504B. In such cases, the sound of window 506 breaking may be transmitted to system 400. System 400 may locate and highlight the sound of the breaking sound and detect anomalous events based on the located and highlighted breaking sound.

[0069] In a similar manner, the system 400 can be used in a vehicle driver assistance system, which is now illustrated in FIG.

[0070] 6 illustrates an example use case 600 implementation of system 400 according to some other example embodiments. Use case 600 corresponds to a vehicle driver assistance system 602 of a vehicle 604. Vehicle 604 may correspond to an automated vehicle, a manually driven vehicle, or a semi-automated vehicle.

[0071] The vehicle driver assistance system 602 includes a dashboard camera that captures video of the road as the vehicle 604 is driven. 606 The vehicle driver assistance system 602 may also include a rear camera (not shown in FIG. 6).

[0072] The vehicle driver assistance system 602 may transmit the captured video to the system 400. The system 400 may process the captured video and assist in navigation of the vehicle 604. For example, the system 400 may detect and identify a vehicle, such as an ambulance, behind the vehicle 604 based on the sound of the ambulance vehicle. The vehicle driver assistance system 602 may receive information of the detected ambulance vehicle and instruct the driver or operator of the vehicle 604. For example, the driver or operator may be instructed to move aside to give way to the ambulance vehicle. Additionally or alternatively, the instruction may be displayed via an interface of the vehicle driver assistance system 602. In some other cases, the instruction may be displayed by the vehicle driver assistance system 602. system Instructions may be given via an audio output of 602. For example, instructions may include "yield to an ambulance."

[0073] In some cases, an ambulance vehicle may be detected based on sounds produced by the ambulance vehicle, such as sirens, bells, and the like.

[0074] FIG. 7 illustrates a method for controlling a power supply according to some other exemplary embodiments. , S 7 shows an example implementation of an example use case 700 of the system 400. The use case 700 corresponds to a human assistance system 702. In an illustrative example scenario, a person 704 may fall or hit a wall. If the person 704 falls, the falling sound may be soft. Also, the falling sound may be obstructed by an object 706. The object 706 may correspond to a dog. When the person 704 falls, the dog may bark.

[0075] The human assistance system 702 may capture video of the person 704 falling and the dog 706 barking. In some cases, the human assistance system 702 may not be able to detect the person 704 falling due to obstruction by the object 706. In such cases, the human assistance system 702 may transmit the video to the system 400.

[0076] The system 400 may process the video received from the human assistance system 702 using the audio-visual representation graph 404 in the memory 402. For example, the system 400 may reduce disturbing sounds, i.e., the barking of the object 706, and emphasize the falling sound of the person 704 based on the audio-visual representation graph 404. Also, the system 400 may detect an abnormal action or event based on the emphasized falling sound. The system 400 may also detect a fall as an abnormal action and send the detected abnormal action to the human assistance system 702. The human assistance system 702 may trigger an alert to assist the person 704. For example, the human assistance system 702 may send an alert message to the caretaker of the person 704. In some cases, the human assistance system 702 may issue an emergency call for medical assistance of the person 704.

[0077] In such an aspect, the system 400 can be used for different applications using the audiovisual representation graph, which contains and provides information capturing the spatial variations of objects in the spatial domain and their evolution in the time domain. The information of the spatial variations and the evolution of objects in the time domain can improve the accuracy of the results of different applications. Also, the audiovisual representation graph can enable the system 400 to efficiently process new tasks with low computational complexity and low storage consumption.

[0078] End The description of the exemplary embodiments merely provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. End The description will provide one of ordinary skill in the art with an enabling description for implementing one or more exemplary embodiments. It is contemplated that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0079] Specific details may be set forth in order to provide a thorough understanding of the embodiments. EndThe description of the embodiments is given in the description of the embodiments. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Also, the same reference numbers and names in the various drawings refer to the same elements.

[0080] Also, individual embodiments may be described as a process, which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operations as a sequential process, many of the operations may occur in parallel or simultaneously. In addition, the order of operations may be rearranged. A process may terminate when its operations are completed, but may have additional steps not described or included in the drawings. Moreover, not all operations in any specifically described process may occur in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the end of the function may correspond to the function returning to the calling function or to the main function.

[0081] Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, manually or automatically. The manual or automatic implementation may be performed or at least assisted through the use of a machine, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. A processor may perform the necessary tasks.

[0082] The various methods or processes outlined herein may be coded as software executable on one or more processors employing any one of a variety of operating systems or platforms. In addition, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0083] The embodiments of the present disclosure may be embodied as a method, examples of which are provided. The operations performed as part of the method may be ordered in any suitable manner. Thus, while an example embodiment shows sequential operations, embodiments may be constructed in which operations are performed in an order other than that shown, including performing some operations simultaneously. Additionally, the use of ordinal terms such as "first," "second," etc. in the claims to modify claim elements does not, in itself, imply any priority, precedence, or order of a claim element relative to another claim element, or any chronological order in which the operations of the method are performed, but is merely used as a label to distinguish a claim element having a certain name from another element having the same name (except for the use of the ordinal term).

[0084] Although the disclosure has been described with reference to certain preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the disclosure. It is therefore the object of the appended claims to cover all such variations and modifications as fall within the true spirit and scope of the disclosure.

Claims

1. 1. A system for processing an audiovisual representation of a scene, comprising: At least one processor; and a non-transitory memory having instructions stored thereon, the instructions, when executed by the at least one processor, causing the system to: treating an audiovisual representation of a scene as a graph of nodes connected by edges, where the nodes in the graph represent video features of objects in the scene, and an edge in the graph connecting two nodes represents an interaction between two corresponding objects in the scene, and at least one edge in the graph is associated with an audio feature of a sound produced by the interaction of the two corresponding objects; To process the audiovisual representation of the scene, the instructions cause the system to generate the graph of nodes from audio and video signals, store the graph of nodes in the non-transient memory, and perform a task based on the graph of nodes.

2. The system of claim 1 , wherein the interaction of two corresponding objects in the scene includes one or a combination of a spatial interaction and a temporal interaction.

3. The system of claim 1 , wherein at least one node in the graph is associated with an audio feature of a sound produced by a corresponding object without interaction with other objects in the scene.

4. 2. The system of claim 1, wherein the edge associated with an audio feature connects a primary object that generates the sound with a context object that causes the primary object to generate the sound, the edge having a direction pointing from a node of the context object to a node of the primary object.

5. The system of claim 1 , wherein multiple edges in the graph are associated with phonetic features, and the phonetic features associated with different edges are orthogonal to each other.

6. The system of claim 5 , wherein the audio features are associated with a plurality of edges based on visual features of the corresponding objects of the audio features, the visual features corresponding to one or more subgraphs of the corresponding objects from the graph.

7. further configured to store an audio mixture of a plurality of sounds in the scene; The system of claim 1 , wherein the audio features associated with the edges are represented by a binary mask that, when applied to the audio mixture, generates the associated audio features.

8. an input interface configured to accept audio and video frames of the scene, the at least one processor configured to execute a trained neural network to convert the accepted audio and video frames into the graph of nodes connected by the edges; and The system of claim 1 , comprising an output interface configured to store the graph of nodes connected by the edges in the non-transitory memory.

9. The neural network assigns the sound to the interaction of the two corresponding objects.

10. The system of claim 8, comprising a plurality of sub-networks trained end-to-end to associate a plurality of objects, the plurality of sub-networks comprising an attention-based sub-network trained to place different attention on different interactions of different pairs of objects.

10. The system of claim 1 , wherein the at least one processor is configured to perform the task using the graph of nodes connected by the edges that provides the audiovisual representation of the scene.

11. The system of claim 10 , wherein the tasks include one or a combination of action recognition, anomaly detection, sound localization and enhancement, removal of noisy background sounds, and system control.

12. 1. A method for processing an audiovisual representation of a scene, the method using a processor in combination with stored instructions implementing the method, the instructions, when executed by the processor, performing steps of the method, the steps comprising: processing an audiovisual representation of a scene as a graph of nodes connected by edges, wherein nodes in the graph represent video features of objects in the scene, and an edge in the graph connecting two nodes represents an interaction of two corresponding objects in the scene, and at least one edge in the graph is associated with an audio feature of a sound produced by the interaction of the two corresponding objects; The method, wherein the processing step includes the steps of generating the graph of nodes from audio and video signals, storing the graph of nodes in a memory, and performing a task based on the graph of nodes.

13. 13. The method of claim 12, wherein the edge associated with an audio feature connects a primary object that generates the sound with a context object that causes the primary object to generate the sound, the edge having a direction pointing from a node of the context object to a node of the primary object.

14. The method of claim 12 , wherein multiple edges in the graph are associated with phonetic features, and the phonetic features associated with different edges are orthogonal to each other.

15. accepting audio and video frames of the scene; and executing a neural network trained to convert the received audio and video frames into the graph of nodes connected by the edges.

16. performing said task using said graph of nodes connected by said edges providing said audiovisual representation of said scene; The method of claim 12 , wherein the tasks include one or a combination of action recognition, anomaly detection, sound localization and enhancement, removal of noisy background sounds, and system control.

17. A non-transitory computer-readable storage medium having stored thereon a program executable by a processor to perform a method, the method comprising: processing an audiovisual representation of a scene as a graph of nodes connected by edges, wherein nodes in the graph represent video features of objects in the scene, and an edge in the graph connecting two nodes represents an interaction of two corresponding objects in the scene, and at least one edge in the graph is associated with an audio feature of a sound produced by the interaction of the two corresponding objects; A non-transitory computer-readable storage medium, wherein the processing step includes the steps of generating the graph of nodes from audio and video signals, storing the graph of nodes in a memory, and performing a task based on the graph of nodes.

Images