Method and system for scene-aware audio-video representation

Abstract

Implementations disclose a method and system for a scene-aware audio-video representation of a scene. The scene-aware audio-video representation corresponds to a graph of nodes connected by edges. A node in the graph represents a video feature of an object in the scene. An edge in the graph connecting two nodes represents an interaction of the respective two objects in the scene. In the graph, at least one or more edges are associated with an audio feature of a sound emitted by the interaction of the respective two objects. The graph of the audio-video representation of the scene can be used to perform various different tasks. Examples of tasks include one or a combination of action recognition, anomaly detection, sound localization and augmentation, noise background sound removal, and system control.

Description

Technical Field

[0001] This disclosure generally relates to audio-video scene-perception recognition, and more specifically to methods and systems for processing scene-perception audio-video representations. Background Technology

[0002] Over the years, various techniques have been developed to capture scenes of the surrounding environment. For example, the concepts of photography and imaging have evolved over hundreds of years and gradually transitioned into the digital realm. Initially, digital images were defined as a two-dimensional (2D) representation composed of pixels, where the intensity or grayscale level of each pixel has a finite discrete quantity of numerical representation. However, pixel representation is not always convenient for various image-related applications. Therefore, alternative representations of the surrounding environment have been developed.

[0003] For example, distance field image representations are useful in a variety of graph applications, including anti-aliasing, ray stepping, and texture synthesis. Distance fields, for instance, have been widely used in simulations for numerical control (NC) applications. Scalable Vector Graphics (SVG) is a two-dimensional graph vector image format based on Extensible Markup Language (XML) that supports interactivity and animation. SVG-based representations have been found to be advantageous for map-based applications. Additionally or alternatively, various feature extraction and compression techniques have been developed for more compact and efficient image representation and storage. For example, various object detection and segmentation applications have been developed to analyze features extracted from pixels, rather than pixel intensity.

[0004] A similar trend has been observed in the video domain of seeking different image representations for different applications. However, the video domain presents more challenges because the representation must capture not only the spatial changes of objects in the spatial domain but also their evolution in the temporal domain. To this end, many different representations have been developed to capture and store video files. For example, various lossy and lossless compression techniques have been developed to leverage the similarity between adjacent video files, various feature extraction techniques have been developed to extract features in the spatial and temporal domains, and various graph-based techniques have been developed to represent the temporal and spatial relationships between different objects in a video.

[0005] To further represent environmental scenes, digital audio technology is used for the recording, manipulation, mass production, and distribution of sound, including the recording of songs, instrumental works, podcasts, sound effects, and other sounds. Various technologies rely on appropriate audio representations. For example, different audio encoding formats have been developed for the storage or transmission of digital audio. Examples of audio encoding formats include MP3, AAC, Vorbis, FLAC, and Opus. Furthermore, similar to image processing, various techniques have been proposed for processing features extracted from audio files. These feature-based techniques are widely used in automatic speech recognition systems.

[0006] Scene representation becomes a more challenging problem when a common representation is needed for both audio and video modalities. While these modalities can represent the same scene, they may not naturally align with each other. To address this, modern techniques typically handle these modalities separately by adding pre-processing and post-processing techniques. Examples of pre-processing techniques include extracting cues from video files to aid audio processing, and vice versa. Examples of post-processing techniques include providing audio captions for annotating video.

[0007] However, these techniques are not suitable for the complex and rich representations of audio-video scenes in an environment. Therefore, there is a need for new representations of audio-video scenes applicable to other applications and technologies. Furthermore, there is a need for a system and method configured to generate and process such new representations. Summary of the Invention

[0008] Therefore, the aim of some implementations is to provide a system and method for processing audio-video representations of a scene in an efficient and accurate manner. Examples of such processing include generating audio-video representations and performing application-specific tasks based on these representations. To this end, in one implementation, a graph-based representation of the scene-aware audio-video representation indicating the scene is generated. One or more image / video capture devices (such as one or more cameras) can be used to capture the video. One or more cameras can be connected to the system, such as a computing system for generating scene-aware audio-video representations.

[0009] Some implementations are based on the idea that a scene-aware audio-video representation comprises a structure mapped onto a graph of nodes connected by edges. Each node in the graph represents an object, for example, a video feature indicating an object in the scene. Each edge in the graph connects two nodes. Each edge in the graph indicates an interaction between two corresponding objects in the scene. In this way, the graph represents objects and their interactions.

[0010] In some implementations, graph representations can be generated based on the detection and classification of objects in the scene. Furthermore, nodes (i.e., objects) and edges (i.e., object interactions) in the graph representation can be selected based on a predetermined set of possible objects and a set of possible interactions. The selection of nodes and edges ensures the uniqueness of objects and interactions. For example, an object can be classified as a person, rather than a group of people, unless a group of people belongs to the set of possible objects.

[0011] Some implementations are based on the understanding that a scene comprises complex interactions between two objects, including one or a combination of spatial and temporal interactions. For example, a captured scene may include dynamic content with different objects indicating people, vehicles, or animals, different audio in the spatial domain, and the evolution of the corresponding objects in the temporal domain. Furthermore, a scene may also include different interactions or relationships existing between the objects. In a real-time example scene, the scene might correspond to a busy street with objects such as people in different actions, vehicles in motion, vehicles at rest, and running animals. People in different actions could include musicians carrying musical instruments. In the scene, some instruments might be actively played by the corresponding musician, while others might not be played.

[0012] Alternatively or concurrently, some implementations aim to generate graph representations with complex interactions, including spatial and temporal interactions between objects. To this end, in some implementations, the graph representation consists of nodes that are fully connected by edges. Each node indicates an object in the scene, and each fully connected edge indicates an interaction corresponding to two objects in the spatial and temporal domains of the scene.

[0013] Some implementations are based on the understanding that, during interaction, some objects can contribute equally to sound generation. In some cases, an object can generate sound while interacting with another object, or it can contribute to sound generation independently. For example, a musician might sing and play the guitar simultaneously in a scene. In other cases, some objects may not contribute equally to sound generation. For example, a musician might play the violin without singing. In these cases, those objects that do not contribute equally to sound generation can be defined based on their interaction type. To this end, in some implementations, the corresponding edges of such objects may include attributes defining the interaction type. In some example implementations, the type of interaction can be indicated by the direction of the edge, and the corresponding nodes (i.e., objects) connected by the directed edges can be distinguished as main objects and environment objects. For example, the node indicating that the musician is playing the violin corresponds to the main object, and the musician playing the violin corresponds to the environment object.

[0014] In some implementations, nodes and / or edges in the graph can be associated with audio features of sounds emitted by objects or interactions between objects. For example, an object node (such as a dog barking while playing in water) can be associated with a corresponding audio feature. Similarly, edges connecting the main object and environment objects can also be associated with audio features. Additionally or alternatively, another objective of some implementations is to extract features, such as video features and audio features that include spatial and temporal information about the scene.

[0015] Some implementations are based on the understanding that joint extraction of video and audio features can improve the overall performance of the system. Joint extraction of video and audio features can eliminate preprocessing and post-processing steps, which can improve overall performance.

[0016] To this end, some implementations disclose an audio-source separation framework that isolates individual audio from audio mixtures of an object. The audio mixture can correspond to a mixed audio spectrogram comprising frequency variations of different audio signals. Audio can be isolated from audio mixtures based on the visual representation of the object from a video and the corresponding interaction with the object.

[0017] In some implementations, the audio-source separation framework may include a neural network. The neural network may include multiple subnetworks trained end-to-end to associate audio features with nodes and / or edges of a graph. The multiple subnetworks may include attention-based subnetworks trained to focus different attention on different interactions of an object.

[0018] The trained attention-based subnetwork can be used to infer the visual features of objects in a scene. These visual features correspond to latent auditory elements that guide the audio-source separation framework to isolate the object's audio from the mixed audio. For example, the visual features of an instrument such as a violin in a scene can guide the audio-source separation framework to isolate the violin sound from the mixed audio.

[0019] In some implementations, visual features can be extracted from a graph representation into one or more subgraphs. One or more subgraphs can provide embedding vectors (i.e., low-dimensional vector representations) that provide orthogonality to the audio features. Furthermore, the scene can include various sounds for each object. For this purpose, audio features associated with different objects and / or edges are orthogonal to each other. The orthogonality of the audio features implements the separation of the individual audios from the various sounds. In some implementations, the audio features associated with edges can be represented by a binary mask. The binary mask separates the desired object audio by separating the corresponding audio features of the object from the audio mixture.

[0020] Some implementations are based on the understanding that the scene-aware audio-video representation can be used as input to other processes or applications. For example, a computing system can provide the generated scene-aware audio-video representation to a server for performing tasks of a corresponding application, such as action recognition and anomaly detection, sound localization and enhancement, etc. Various machine learning methods can be used to train the computing system on the scene-aware audio-video representation. For this purpose, the structure of the audio-video representation can be used for supervised learning to train a neural network for performing the task. Additionally or alternatively, another objective of some implementations is to use the generated scene-aware audio-video representation to train a neural network.

[0021] In this way, graphs, depending on the implementation, can capture rich and complex audio-video representations of a scene. Furthermore, the graphs can be differentiable, allowing graphs to be generated from trained neural networks and used to train other neural networks.

[0022] Therefore, some implementations disclose a non-transitory memory configured to store an audio-video representation of a scene as a graph of nodes connected by edges. Nodes in the graph indicate video features of objects in the scene. An edge connecting two nodes in the graph indicates an interaction between two corresponding objects in the scene. In the graph, at least one or more edges are associated with audio features of sounds emitted by the interaction between the two corresponding objects.

[0023] Therefore, some other embodiments disclose a system for generating an audio-video representation of a scene. The system includes an input interface, a processor, and an output interface. The input interface is configured to accept audio frames and video frames of the scene. The processor is configured to execute a neural network trained to convert the accepted audio and video frames into a graph of an audio-video representation of the scene. The graph includes nodes connected by edges, wherein nodes in the graph indicate video features of objects in the scene, wherein an edge connecting two nodes in the graph indicates an interaction between two corresponding objects in the scene, and wherein at least one or more edges in the graph are associated with audio features of sounds emitted by the interaction between the two corresponding objects. The output interface is configured to store the graph of the audio-video representation of the scene in memory.

[0024] Therefore, some other embodiments disclose a system including a memory and a processor, the memory being configured to store an audio-video representation of a scene as a graph of nodes connected by edges, wherein nodes in the graph indicate video features of objects in the scene, wherein an edge connecting two nodes in the graph indicates an interaction between two corresponding objects in the scene, and wherein at least one or more edges in the graph are associated with audio features of sounds emitted by the interaction between the two corresponding objects, and the processor is operatively connected to the memory to perform tasks using the graph of the audio-video representation of the scene. Attached Figure Description

[0025] [ Figure 1A ]

[0026] Figure 1A An audio-video representation of a scene as a diagram is shown according to some implementations.

[0027] [ Figure 1B ]

[0028] Figure 1B An audio-video representation of a scene as a diagram is shown according to some other embodiments.

[0029] [ Figure 1C ]

[0030] Figure 1C An audio-video representation of a scene as a diagram is shown according to some other embodiments.

[0031] [ Figure 1D ]

[0032] Figure 1D An audio-video representation of a scene as a diagram is shown according to some other embodiments.

[0033] [ Figure 1E ]

[0034] Figure 1E An audio-video representation of a scene as a diagram is shown according to some other embodiments.

[0035] [ Figure 2 ]

[0036] Figure 2 A block diagram of a system for generating audio-video representations of a scene, according to some embodiments, is shown.

[0037] [ Figure 3A ]

[0038] Figure 3A A schematic diagram is shown illustrating the extraction of audio and video features from video frames to generate an audio-video representation of a scene, according to some implementation methods.

[0039] [ Figure 3B ]

[0040] Figure 3B A schematic diagram is shown illustrating the extraction of audio and video features from video frames to generate an audio-video representation of a scene, according to some implementation methods.

[0041] [ Figure 3C ]

[0042] Figure 3C A schematic diagram of a framework corresponding to an audio source separation task, according to descriptions of some implementation methods, is shown.

[0043] [ Figure 4 ]

[0044] Figure 4 A block diagram of a system that performs tasks by using audio-video representations of use scenarios according to some example implementations is shown.

[0045] [ Figure 5 ]

[0046] Figure 5 The use of some example implementations is shown. Figure 4 Example scenarios of the system corresponding to the tasks.

[0047] [ Figure 6 ]

[0048] Figure 6 The corresponding use according to some other example implementations is shown. Figure 4 Example scenarios of system tasks.

[0049] [ Figure 7 ]

[0050] Figure 7 The corresponding use according to some other example implementations is shown. Figure 4 Example scenarios of system tasks. Detailed Implementation

[0051] In the following description, numerous specific details are set forth for purposes of explanation and to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these specific details. In other instances, apparatus and methods are shown in block diagram form only to avoid obscuring this disclosure.

[0052] As used in this specification and claims, the terms “for example,” “like,” and “such as,” as well as the verbs “comprising,” “having,” “including,” and other verb forms thereof, when used with a list of one or more components or other items, shall be construed as open-ended, meaning that the list should not be considered as excluding other additional components or items. The term “based on” means at least partially based on. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes only and should not be considered limiting. Any headings used in this specification are for convenience only and have no legal or limiting effect.

[0053] Figure 1A A representation 100 according to some embodiments is shown, depicting a scene-perceptual audio-video representation of a scene, as shown in Figure 102. The scene is represented by video 108. Figure 102 includes nodes such as nodes 104A, 104B, 104C, 104D, 104E, 104F, and 104G (hereinafter referred to as nodes 104A-104G). Nodes 104A-104G are connected by edges such as edges 106A, 106B, 106C, 106D, 106E, 106F, and 106G (hereinafter referred to as edges 106A-106G). Each node in nodes 104A-104G indicates video features of objects in the scene. For example, node 104A indicates a dog, node 104B indicates water, node 104C indicates a pool, node 104D indicates a sidewalk, node 102E indicates plants, node 104F indicates a tree, and node 104G indicates a water tank in the scene. Each of the edges 106A-106G connecting two nodes indicates an interaction between the corresponding two objects in the scene. For example, edge 106A connecting node 104A (i.e., the dog) and node 104B (i.e., the water) indicates an interaction between the dog and the water in the scene. In some cases, at least one node among nodes 104A-104G can be associated with the audio characteristics of a sound emitted by the corresponding object without interacting with other objects in the scene. Similarly, edges 106B, 106C, 106D, 106E, 106F, and 106G indicate interactions between the corresponding two objects, such as... Figure 1A As shown.

[0054] Furthermore, in Figure 102, some of nodes 104A-104G and / or edges 106A-106G are associated with audio features of sound. For example, node 104A is associated with the audio features of a dog barking in the scene. For example, in this example, edge 106A is associated with the audio features of splashing sounds. In this scene, splashing sounds are generated due to the interaction between the dog (i.e., node 104A) and the water (i.e., node 104B). The audio features can be obtained from audio mixing 110.

[0055] In some example implementations, a scene may include complex interactions between two corresponding objects. Such complex interactions can be included in a graphical representation of the scene, as will be discussed below. Figure 1B Describe it.

[0056] Figure 1B A representation 112 according to some other embodiments is shown, which depicts a scene-perceived audio-video representation indicating a scene, as shown in Figure 114. This scene corresponds to video 116, as... Figure 1B As shown. In some implementations, the interaction between two corresponding objects in a scene may include one or a combination of spatial and temporal interactions between the objects. Spatial and temporal interactions between the objects can be indicated in Figure 114. Figure 1B As shown, Figure 114 includes nodes that are fully connected to each other through edges in Figure 114. The fully connected nodes through edges in Figure 114 indicate complex interactions of objects, including spatial and temporal information.

[0057] In some cases, the interaction type can be indicated, as will be discussed below. Figure 1C Describe it.

[0058] Figure 1C A representation 118 of FIG120 is shown, depicting a scene according to some other embodiments. FIG120 corresponds to the scene in video 122 that includes musicians, musical instruments, etc.

[0059] In some situations, interacting objects in a scene may contribute equally to producing sound. For example, a musician might sing while playing an instrument (such as a xylophone). In other cases, objects may contribute unequally. For example, another musician might only play an instrument, such as a flute. In this case, the interaction between the musician and the flute can be indicated by the direction of the edge connecting the objects (i.e., the musician and the flute).

[0060] In Figure 120, node 124A indicates a flute, and node 124B indicates a musician playing the flute. Furthermore, each object (i.e., the flute and the musician) can be distinguished as a main object and an environment object. Edge 126 of Figure 120, associated with audio features, connects the main object emitting the sound and the environment object causing the main object to emit the sound. Edge 126 has a direction from node 124B of the environment object to node 125A of the main object. Specifically, the object (i.e., the flute emitting the sound) is defined as the main object, and the object (i.e., the musician causing the sound by playing the flute) is defined as the environment object. Nodes 124A and 124B are connected by directed edges (such as edge 126). Furthermore, edge 126 can be associated with audio features corresponding to the flute sound emitted by the interaction between the musician and the flute. By determining the corresponding visual features of the flute sound in the scene, the audio features corresponding to the flute sound can be associated with edge 126.

[0061] In some implementations, one or more sub-graphs of Figure 120 can be created for visual features in the scene, such as Figure 1D As shown.

[0062] Figure 1D A representation 128 of Figure 130 depicting a scene according to some other embodiments is shown. Figure 130 corresponds to Figure 114. Figure 130 includes nodes fully connected by multiple edges indicating complex interactions of objects corresponding to spatial and temporal information of a video (e.g., video 116). Furthermore, the edges may be associated with audio features corresponding to sounds emitted by the interactions of the objects. The audio features are associated with the edges based on the visual features of the objects. The visual features can be derived from sub-figure 134 of Figure 130. Sub-figure 134 can provide embedding vectors that provide orthogonality to the audio features. Due to the orthogonality of the audio features, the memory space required to store Figure 130 can be reduced. Furthermore, the embedding vectors can be used to separate audio features from an audio mixture of multiple sounds in the scene. The audio mixture can be represented by a mixed audio spectrogram 132.

[0063] Similarly, one or more subgraphs (such as subgraph 134) can be used to separate different audio features for different interactions from a complete graph (such as Figure 130). These subgraphs are further used in... Figure 1E As shown in the image.

[0064] Figure 1EA representation 138 according to some embodiments is shown, depicting one or more sub-figures, such as sub-figures 142A and 142B of a figure (e.g., figure 130 of a scene in video 140). Sub-figure 142A corresponds to visual feature 140A in video 140, and sub-figure 142B corresponds to visual feature 140B in video 140. For example, visual feature 140A includes features of objects such as people, and visual feature 140B includes characteristics of objects such as vehicles.

[0065] To this end, subgraph 142A includes nodes indicating people in the scene and edges indicating human interactions. Subgraph 142B includes nodes indicating vehicles (such as cars, trains, etc.) in the scene and edges indicating vehicle interactions. The edges of each of Figures 142A and 142B can be associated with audio features such that audio features associated with different edges are orthogonal to each other, which helps to separate the corresponding audio features of people and vehicles from the various sounds in video 140.

[0066] Graph representations that include information about different objects in a scene and their various interactions (such as spatial and temporal interactions) can be suitable for complex and rich representations of audio-visual scenes. Graph representations can be generated by the system, as will be discussed below. Figure 2 Describe it.

[0067] Figure 2 A block diagram of a system 200 for generating an audio-video representation of a scene, according to some embodiments, is shown. System 200 includes an input interface 202, a processor 204, an output interface 206, and a memory 208. In some example embodiments, memory 208 is configured to store a neural network 210. In some other example embodiments, memory 208 may also be configured to store an audio mix of multiple sounds in the scene.

[0068] Input interface 202 is configured to accept audio and video frames of the scene. Some implementations are based on the understanding that joint extraction of video and audio features can improve the overall performance of system 200. Therefore, processor 204 can be configured to execute neural network 210 to convert the accepted audio and video frames into a graph of audio-video representation of the scene (e.g., FIG. 102), which includes nodes (e.g., nodes 104A-104G) connected by edges (e.g., edges 106A-106G).

[0069] In Figure 102, at least one or more edges (e.g., edge 106A) are associated with audio features of sounds emitted by the interaction of two corresponding objects (e.g., node 106A and node 106B). Such a multimodal graph (e.g., Figure 102) allows for seamless intermodal interaction, which facilitates the performance of various tasks such as multi-task learning. For this purpose, the audio features of the corresponding sounds are separated from audio mixtures (e.g., a mixed audio spectrogram 132). The mixed audio spectrogram 132 includes multiple sounds from the scene or mixtures of various sounds for each type of main object. The separation of audio features is achieved in… Figure 3A and Figure 3B The following will provide a more detailed explanation.

[0070] Figure 3A and Figure 3B A schematic diagram 300 according to some embodiments is shown, depicting the separation of audio and video features of a graph (e.g., FIG. 102) used to generate a scene from video 302. The separation of audio and video features is performed by system 200. Video 302 is provided as input to system 200.

[0071] In some example implementations, object detection and classification techniques can be used to detect and classify objects in video frame 302. These techniques may include, but are not limited to, faster region-based convolutional neural network (R-CNN) models and ResNet models that generate bounding boxes for detecting objects in video frame 302. ResNet may be pre-trained on a dataset (e.g., a visual genome dataset).

[0072] After object detection and classification, video features of the objects can be extracted from video 302. In an example implementation, video features can be extracted as feature vectors. For example, the feature vector of video features from video frame 302 may include a 2048-dimensional vector. Furthermore, audio features from video frame 302 can be detected based on a trained audio dataset. For example, audio features corresponding to a music dataset can be trained on an audio dataset (such as the AudioSet dataset). Audio features can be extracted as feature vectors, for example, 512-dimensional vectors. In some implementations, the dimensional vectors of video features can be encoded according to the dimensions of the audio features to maintain consistency in the feature dimensions of the audio and video features. For this purpose, the video dimensional vector can be encoded as the dimensions of the audio features. For example, the 2048-dimensional vector of the video features is encoded as a 512-dimensional vector. In this way, the video dimensional vector of the video features is aligned with the dimensional vector of the audio features, and a common representation of different audio and video modalities is obtained. Furthermore, the alignment of different audio and video modalities allows the system (e.g., system 200) to perform multiple tasks simultaneously, while excluding preprocessing and / or post-processing tasks (such as captioning). In some example implementations, dimensional vectors can be encoded using artificial neural networks, such as a two-layer multilayer perceptron with a rectified linear unit (ReLU) activation function.

[0073] Furthermore, a graph 304 is constructed based on the dimension vector of the extracted video frame 302. Graph 304 is provided as input to a Graph Attention Network (GAN) and a Recurrent Neural Network (RNN) 306. In an example implementation, the GAN and RNN 306 can process graph 304 by i) updating the nodes of graph 304, ii) capturing pairwise interactions between nodes in graph 304, and iii) pooling the feature vectors of audio and video features.

[0074] GANs and RNNs 306 can include components such as graph attention network convolution (GATConv), edge convolution, and feature vector pooling. GATConv updates the features of nodes in graph 304 based on edge adjacency information and multi-head graph messaging (e.g., four attention heads for messaging).

[0075] After updating the nodes, GAN and RNN 306 can use edge convolutions to capture pairwise interactions, i.e., the edges of the nodes. In the example implementation, edge convolutions capture the concatenated vectors of two nodes in graph 304. For example, edge convolutions capture the concatenated vectors of two nodes (512 × 2 = 1024). Therefore, the dimension vector of the pairwise interactions is a 512-dimensional vector. The feature vectors corresponding to the updated nodes and pairwise interactions are pooled to obtain the graph attention embedding vector 308. The graph attention embedding vector 308 is a single vector representation. In the example implementation, the feature vectors can be pooled using pooling techniques such as Global Max and Average pooling to obtain the graph attention embedding vector 308.

[0076] The graph attention embedding vector 308 is provided as input to the audio-source separation framework 310. The graph attention embedding vector 308 corresponds to the visual features derived from the video 302 (e.g., Figure 1E The visual features 140A and 140B), and the video 302 guide the audio-source separation framework 310 for separating the audio features of the corresponding object and the interaction of the object.

[0077] The audio-source separation framework 310 may include an attention-based neural network architecture that derives visual features corresponding to objects and their interactions. The attention-based neural network architecture can be trained to provide an attention value to each edge (i.e., the interaction between the corresponding two objects in Figure 304). The attention value corresponds to the importance of the edge, i.e., the importance of the interaction between the corresponding two objects in the graph. The audio-source separation framework 310 may include a skip connection 314 that feeds the output of one layer as input to the next layer by skipping layers between them. The attention-based neural network architecture of the audio-source separation framework 310... Figure 3C Further details are provided below.

[0078] Furthermore, the audio-source separation framework 310 generates a separation mask 316. The separation mask 316 may include embedding vectors that provide orthogonality to the audio features. Orthogonality of the audio features can improve computational speed and allow for the unique separation of the desired audio from the mixed audio. In an example implementation, the audio-source separation framework 310 separates the audio sources of the object and / or interaction from the mixed audio spectrogram 312 based on the embedding vector 308. The mixed audio spectrogram 312 corresponds to the mixed audio spectrogram 132. Furthermore, the separation mask 316 is provided to a transformer 318 for transforming the audio features into separated audio. The separated audio is provided as output 320.

[0079] Figure 3CA schematic diagram of an audio-source separation framework 322 corresponding to an audio source separation task according to some embodiments is shown. In an example embodiment, the audio source separation task may correspond to separating piano sounds from an audio mix of a video (e.g., video 302).

[0080] The audio-source separation framework 322 includes a neural network 326. The neural network 326 includes multiple subnetworks corresponding to multiple RNNs (such as RNN 326A, RNN 326B, and RNN 326C (i.e., multiple RNNs 326A-326C)). The multiple RNNs 326A-326C can be trained end-to-end to associate sound with the interaction of corresponding two objects. The multiple RNNs 326A-326C include attention-based subnetworks that are trained to place different attention on different interactions of different object pairs. In some example implementations, Figure 324 can be used to train the multiple RNNs 326A-326C. For example, the multiple RNNs 326A-326C can be trained to provide different attention to different interactions of different object pairs in Figure 324. In some implementations, multiple RNNs 326A-326C can perform supervised learning (in particular, self-supervised learning) to provide different attention to different interactions of different object pairs in Figure 324.

[0081] Figure 324 is a fully connected graph with nodes (such as nodes 324A, 324B, 324C, and 324D). Nodes 324A, 324B, 324C, and 324D (or nodes 324A-324D) indicate objects (such as musical instruments and musicians in a scene). For example, nodes 324A and 324B correspond to musicians, and nodes 324C and 324D correspond to instruments such as pianos and guitars, respectively. Nodes 324A-324D are fully connected to each other through the edges of Figure 324. This multimodal alignment of audio and video features enables the system to jointly learn multiple tasks.

[0082] Multiple trained subnetworks 326A-326C can provide embedding vectors 328, which provide orthogonality for the audio features of the object and the interaction of the object. These embedding vectors can be used to separate audio features from the audio mixture of a guitar and a piano, represented by a mixed audio spectrogram 332. The mixed audio spectrogram 332 corresponds to a mixed audio spectrogram 312.

[0083] Embedding vector 328 includes embedding vector 328A (E1), embedding vector 328B (E2), and embedding vector 328C (E3). Each of embedding vectors 328A, 328B, and 328C corresponds to a different audio generation object in the scene. Embedding vectors 328A-328C can be associated with audio features to nodes or edges. For this purpose, embedding vectors 328A-328C generate a spectrogram mask 328 for the audio of the corresponding object or interaction. The spectrogram mask 328 corresponds to... Figure 3B The separation mask 316. In some implementations, the embedding vectors 328A-328C are associated with edges using a binary mask. When applied to the mixed audio spectrogram 322, the binary mask produces associated audio features.

[0084] Furthermore, a spectrogram mask is provided to the audio encoder-decoder 330 for separating the audio source of the object from the mixed audio spectrogram 332. The mixed audio spectrogram 332 corresponds to the mixed audio spectrogram 312. For example, the mixed audio spectrogram 332 includes a mixture of audio corresponding to instruments such as guitars and pianos.

[0085] When the mixed audio spectrogram 332 is passed to the audio encoder-decoder 330, the embedding vector 328 removes the audio of the corresponding object (such as the audio of a piano) from the mixed audio spectrogram 332. The separated audio of the piano is obtained as output 334. After training, multiple subnetworks 326 learn to generate spectrogram masks of the audio from which the object has been removed from the mixed audio spectrogram 332.

[0086] Therefore, the scene-aware audio-video representation graph 324 is provided as input to a system for performing tasks for different applications. Since graph 324 includes fully connected nodes 324A-324D, systems using this graph can handle new tasks efficiently with low computational complexity and low storage consumption.

[0087] Next, in Figure 4 The system that uses the graph to perform tasks is further described below.

[0088] Figure 4 A block diagram of a system 400 is shown that performs tasks using an audio-video representation diagram 404 according to some example embodiments of a use scenario. System 400 includes a memory 402 storing an audio-video representation diagram 404 and a processor 406 operatively connected to the memory 402. The memory 402 may be a non-transitory memory. The audio-video representation diagram 404 corresponds to FIG. 324. The processor 406 is configured to perform tasks using the audio-video representation diagram 404. These tasks may include one or a combination of motion recognition, anomaly detection, sound localization and enhancement, background noise removal, and system control.

[0089] refer to Figure 5 , Figure 6 and Figure 7 It further describes one or a combination of motion recognition, anomaly detection, sound localization and enhancement, noise background sound removal, and system control.

[0090] Exemplary Implementation

[0091] Figure 5 An exemplary use case 500 implementation of system 400 according to some example implementations is shown. Use case 500 corresponds to an anomaly detection application (such as a security surveillance application). In the illustrative example scenario, an event occurring at location 502 may be captured by one or more cameras (such as cameras 504A and 504B installed at different points at location 502). Location 502 may include an indoor or outdoor area of ​​private property, public property, etc.

[0092] The captured events are provided as input to a system, such as system 400, via a wired or wireless network connection. System 400 processes the captured events to identify actions and uses an audio-visual representation of the scene (such as Figure 404) to detect unusual actions or events.

[0093] For example, cameras 504A and 504B capture video of window 506 being broken. Each of cameras 504A and 504B captures video of person 508 using a stick to break window 506. Figure 5 As shown, camera 504A captures the front of person 508, and camera 504B captures the back of person 508. The video of this captured event is sent to system 400. System 400 generates a map corresponding to the scene of the event captured by both cameras 504A and 504B. System 400 identifies the actions of person 508 and detects abnormal actions based on the map.

[0094] Alternatively or concurrently, system 400 may locate and amplify sound (such as the sound of breaking window 506) to detect unusual events. For example, camera 504B may be obstructed to capture a stick in the hand of person 508. In some cases, the environment at location 502 may affect the video captured by cameras 504A and 504B. In this case, the sound of window 506 breaking may be sent to system 400. System 400 may perform sound localization and amplification of the breaking sound, and detect unusual events based on the localized and amplified breaking sound.

[0095] In a similar manner, system 400 can be used for vehicle driver assistance systems, and then... Figure 6 This is described in the text.

[0096] Figure 6An exemplary use case 600 implementation of system 400 according to some other example implementations is shown. Use case 600 corresponds to vehicle driver assistance system 602 of vehicle 604. Vehicle 604 may correspond to an autonomous vehicle, a manually operated vehicle, or a semi-autonomous vehicle.

[0097] The vehicle driver assistance system 602 may include one or more cameras (such as a dashboard camera 606 that captures video of the road while the vehicle 604 is in motion). The vehicle driver assistance system 602 may also include a rear camera ( Figure 6 (Not shown in the image).

[0098] The vehicle driver assistance system 602 can send the captured video to system 400. System 400 can process the captured video and assist in the navigation of vehicle 604. For example, system 400 can detect and identify vehicles, such as detecting and identifying an ambulance that may be behind vehicle 604 based on the sound of an ambulance. The vehicle driver assistance system 602 can receive information about the detected ambulance and instruct the driver or operator of vehicle 604. For example, the driver or operator may be instructed to move to the side of the road to make way for the ambulance. Alternatively or additionally, this instruction can be displayed via the interface of the vehicle driver assistance system 602. In some other cases, the instruction can be given via the audio output of the vehicle driver assistance system 602. For example, the instruction may include "Make way for the ambulance".

[0099] In some cases, ambulances can be detected based on the sounds they make (such as sirens, bells, etc.).

[0100] Figure 7 An exemplary use case 700 implementation of system 400 according to some other example implementations is shown. Use case 700 corresponds to human assistance system 702. In an illustrative example scenario, a person 704 may fall or bump into a wall. When person 704 falls, the sound of the fall may be low. At the same time, the sound of the fall may be disturbed by object 706. Object 706 may correspond to a dog. When person 704 falls, the dog may bark.

[0101] Human assistance system 702 can capture video of a person 704 falling and a dog 706 barking. In some cases, due to interference from object 706, human assistance system 702 may fail to detect a fall of person 704. In this case, human assistance system 702 can send the video to system 400.

[0102] System 400 can use the audio-video representation diagram 404 in memory 402 to process the video received from human assistance system 702. For example, system 400 can reduce interfering sounds (i.e., the barking of object 706) and amplify the sound of person 704 falling based on the audio-video representation diagram 404. Furthermore, system 400 can detect abnormal movements or events based on the amplified fall sound. System 400 can also detect a fall as an abnormal movement and send the detected abnormal movement to human assistance system 702. Human assistance system 702 can trigger an alarm to assist person 704. For example, human assistance system 702 can send an alarm message to person 704's caregiver. In some cases, human assistance system 702 can issue an emergency call for medical assistance to person 704.

[0103] In this way, system 400 can be used for various applications utilizing audio-video representations. Audio-video representations include information providing information on the spatial changes of captured objects in the spatial domain and their evolution in the temporal domain. Information on the temporal evolution and spatial changes of objects can improve the accuracy of results for various applications. Furthermore, audio-video representations enable system 400 to handle new tasks efficiently with low computational complexity and low storage consumption.

[0104] The above description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the above description of the exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing one or more exemplary embodiments. Various changes to the function and arrangement of elements are contemplated without departing from the spirit and scope of the subject matter disclosed as set forth in the appended claims.

[0105] Specific details are set forth in the foregoing description to provide a comprehensive understanding of the embodiments. However, those skilled in the art will understand that embodiments can be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other instances, well-known processes, structures, and techniques may be shown without unnecessary details to avoid confusion with the embodiments. Furthermore, the same reference numerals and reference numerals in the various figures indicate the same elements.

[0106] Furthermore, various implementations can be described as processes, which may be depicted as work diagrams, flowcharts, data flow diagrams, structural diagrams, or block diagrams. Although flowcharts can describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Moreover, the order of operations can be rearranged. A process may terminate upon completion of its operations, but there may be additional steps not discussed or included in the diagram. Furthermore, not all operations in any particular described process may occur in all implementations. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, the termination of the function can correspond to the function returning to the calling function or the main function.

[0107] Furthermore, implementations of the disclosed subject matter can be implemented, at least partially, manually or automatically. Manual or automatic implementation can be performed, or at least assisted by, the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments for performing the necessary tasks can be stored in a machine-readable medium. The processor can then perform the necessary tasks.

[0108] The various methods or processes outlined herein can be encoded as software executable on one or more processors using any of a variety of operating systems or platforms. Furthermore, such software can be written using any of a variety of suitable programming languages ​​and / or programming or scripting tools, and can also be compiled into executable machine language code or intermediate code that executes on a framework or virtual machine. Typically, in various implementations, the functionality of program modules can be combined or distributed as needed.

[0109] Embodiments of this disclosure can be embodied as a method, examples of which have been provided. Actions performed as part of the method can be ordered in any suitable manner. Therefore, embodiments can be constructed that perform actions in a different order than those illustrated, which may include performing some actions simultaneously, even if these actions are shown as sequential in the exemplary embodiments. Furthermore, the use of ordinal numbers such as "first" or "second" to modify claim elements in the claims does not imply any priority, precedence, or order of one claim element relative to another, nor does it imply a temporal order of the actions of the method; rather, it serves merely as a label (but for use with ordinal terms) to distinguish one claim element with a specific name from another with the same name, thus differentiating claim elements.

[0110] Although this disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adjustments and modifications can be made within the spirit and scope of this disclosure. Therefore, aspects of the appended claims cover all such variations and modifications within the true spirit and scope of this disclosure.

Claims

1. A system for processing audio-video representation of a scene, the system comprising: At least one processor; and non-volatile memory, on which instructions are stored, which, when executed by the at least one processor, cause the system to: The audio-video representation of a scene is processed as a graph of nodes connected by edges, wherein nodes in the graph indicate video features of objects in the scene, wherein an edge connecting two nodes in the graph indicates an interaction between two corresponding objects in the scene, and wherein at least one or more edges in the graph are associated with audio features of sounds emitted by the interaction between the two corresponding objects. Specifically, in order to process the audio-video representation of the scene, the instructions cause the system to generate a graph of the nodes from the audio and video signals, store the graph of the nodes in the memory, and perform tasks or combinations thereof based on the graph of the nodes. In the graph, multiple edges are associated with audio features, and the audio features associated with different edges are orthogonal to each other. The audio feature is associated with multiple edges based on the visual features of the corresponding object of the audio feature, and the visual feature corresponds to one or more subgraphs of the corresponding object from the graph.

2. The system according to claim 1, wherein, The interaction between the two objects in the scenario includes one or a combination of spatial and temporal interactions.

3. The system according to claim 1, wherein, At least one node in the graph is associated with the audio characteristics of a sound emitted by the corresponding object without interacting with other objects in the scene.

4. The system according to claim 1, wherein, An edge associated with an audio feature connects the main object that emits the sound and the environment object that causes the main object to emit the sound, wherein the edge has a direction from a node of the environment object to a node of the main object.

5. The system of claim 1, further configured to store an audio mix of multiple sounds in the scene, wherein, The audio features associated with the edge are represented by a binary mask, and the binary mask produces associated audio features when applied to the audio mix.

6. The system according to claim 1, further comprising: An input interface configured to accept audio and video frames of the scene, wherein the at least one processor is configured to execute a neural network trained to convert the received audio and video frames into a graph of nodes connected by the edges; and An output interface configured to store a graph of the nodes connected by the edges in the memory.

7. The system according to claim 6, wherein, The neural network includes multiple subnetworks that are trained end-to-end to associate the sound with the interaction of two corresponding objects, and wherein the multiple subnetworks include an attention-based subnetwork that is trained to focus different attention on different interactions of different object pairs.

8. The system according to claim 1, wherein, The at least one processor is configured to perform the task using a graph of nodes connected by edges provided by the audio-video representation of the scene.

9. The system according to claim 8, wherein, The task includes one or a combination of motion recognition, anomaly detection, sound localization and enhancement, noise and background noise removal, and system control.

10. A method for processing audio-video representations of a scene, wherein, The method uses a processor coupled with stored instructions that implement the method, wherein the instructions, when executed by the processor, perform steps in the method, the method comprising the following steps: The audio-video representation of a scene is processed as a graph of nodes connected by edges, wherein nodes in the graph indicate video features of objects in the scene, wherein an edge connecting two nodes in the graph indicates the interaction between two corresponding objects in the scene, and wherein at least one or more edges in the graph are associated with audio features of sounds emitted by the interaction between the two corresponding objects. The processing includes generating a graph of the nodes from audio and video signals, storing the graph of the nodes in memory, and performing one or a combination of tasks based on the graph of the nodes. In the graph, multiple edges are associated with audio features, and the audio features associated with different edges are orthogonal to each other. The audio feature is associated with multiple edges based on the visual features of the corresponding object of the audio feature, and the visual feature corresponds to one or more subgraphs of the corresponding object from the graph.

11. The method according to claim 10, wherein, The edge associated with the audio feature connects the main object that emits the sound and the environment object that causes the main object to emit the sound, and wherein the edge has a direction from a node of the environment object to a node of the main object.

12. The method according to claim 10, further comprising the step of: Accept audio and video frames from the scene; as well as A neural network is executed, which is trained to convert the received audio and video frames into a graph of nodes connected by the edges.

13. The method according to claim 10, further comprising the following step: The task is performed using a graph of nodes connected by edges that provide the audio-video representation of the scene, wherein the task includes one or a combination of action recognition, anomaly detection, sound localization and enhancement, noise background noise removal, and system control.

14. A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including a program executable by a processor for performing a method, the method comprising: The audio-video representation of a scene is processed as a graph of nodes connected by edges, wherein nodes in the graph indicate video features of objects in the scene, wherein an edge connecting two nodes in the graph indicates an interaction between two corresponding objects in the scene, and wherein at least one or more edges in the graph are associated with audio features of sounds emitted by the interaction between the two corresponding objects, and wherein the processing includes generating the graph of nodes from audio and video signals, storing the graph of nodes in memory, and performing one or a combination of tasks based on the graph of nodes. In the graph, multiple edges are associated with audio features, and the audio features associated with different edges are orthogonal to each other. The audio feature is associated with multiple edges based on the visual features of the corresponding object of the audio feature, and the visual feature corresponds to one or more subgraphs of the corresponding object from the graph.

Images