Audio-visual retrieval and video-to-audio generation with enhanced synchronicity

Abstract

Methods, apparatus, programs, and storage media for audio-visual contrastive learning are described. The method includes receiving an audio-visual sequence comprising a plurality of corresponding video data samples and audio data samples. A token sequence is generated from the audio data sample. A plurality of tokens in the token sequence are masked to generate a masked token sequence. A conditioning sequence is generated based on the video data sample. The masked tokens in the masked token sequence are unmasked by a transformer conditioned on the conditioning sequence. A loss function is evaluated based on the unmasked token sequence. The transformer is trained based on the loss function.

Description

AUDIO-VISUAL RETRIEVAL AND VIDEO-TO-AUDIO GENERATION WITH ENHANCED SYNCHRONICITY CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from Spanish Application No. P202430161 filed on 5 March 2024, U.S. Provisional Application No. 63 / 631,343 filed on 8 April 2024, and U.S. Provisional Application No. 63 / 760,505 filed on 19 February 2025, each of which is incorporated by reference herein in its entirety. TECHNICAL FIELD

[0002] The present disclosure relates to training and inference of neural networks for contrastive learning and for video-to-audio generation. BACKGROUND

[0003] Contrastive learning has been traditionally used in self-supervised unimodal representation learning, as in vision pretraining and audio pretraining. Recently it has also achieved impressive results in multimodal representation learning, by utilizing large amounts of (weakly) supervised data, as in vision-language, audio-language and recently also expanded to more than two modalities.

[0004] The general correspondence of the audio and visual modalities in the natural world has led to the application of contrastive learning in audio-visual representation learning through unlabeled web-scale video datasets. Early works use this audio-visual correspondence in a discriminative approach by predicting whether a visual and an audio frame originate from the same point in the video, while using this correspondence by focusing on a temporal synchronicity between visual and audio clips. A fully contrastive framework was adopted, which proposed to improve the quality of negative examples and targeted the issues of false positives and false negatives in the contrastive framework. Some frameworks used both local and global information to enhance the generalizability of the learned representations.

[0005] Contrastive learning is usually applied on temporally pooled visual and audio representations, which is disregarding their sequential nature. Thus, it is questionable whether itcan take advantage of fine-grained information within the sequence, which can be crucial in differentiating among edge cases of semantically close examples.

[0006] Thus, there is a need for an improved method for contrastive learning, which can consider the sequential nature of the visual and audio representations.

[0007] Further, current neural network-based audio generation algorithms are conditioned on a variety of modalities, including text, images, or other audios. In particular, video-to-audio (V2A) generation leverages visual-only video features to render plausible sounds that match the scene. But more importantly, the generated sound onsets should also match the visual actions that are aligned with them, otherwise unnatural synchronization artifacts arise. Recently the progression of conditioning sound generators on still images and then video features has been explored. This has either been done by focusing on quality and semantic matching while ignoring synchronization, or sacrificing some amount of quality to focus on improving synchronization only.

[0008] Thus, there is a need for an improved method for V2A generation, which considers synchronization and quality at the same time. SUMMARY

[0009] In view of the above, the present disclosure provides methods, apparatus, and programs, as well as computer-readable storage media for neural network based audio-visual contrastive leaning, audio-visual retrieval and video-to-audio generation, having the features of the respective independent claims.

[0010] According to a first aspect of the disclosure, a method of audio-visual contrastive learning is provided. An audio-visual sequence including a plurality of corresponding video data samples and audio data samples may be received. Then, for each audio data sample and each video data sample in the sequence the following steps may be performed. Visual features may be determined (e.g., extracted) from the video data sample and audio features may be determined (e.g., extracted) from the audio data sample. A visual latent representation may be obtained by applying a neural image encoder to the visual features and an audio latent representation may be obtained by applying a neural audio encoder to the audio features. The latent representations may represent temporal sequences of features in a latent space. A sequential contrastive loss functionmay be applied directly to the plurality of visual and audio latent representations. Applying the sequential contrastive loss function directly to the plurality of video and audio latent representations may include applying a distance function directly on the two temporal sequences in the latent space and using the result of the distance function for the sequential contrastive loss function. The neural image encoder and the neural audio encoder may be trained based on the sequential contrastive loss function. Therefore, the neural image encoder and the neural audio encoder may be trained for maximizing a distance between dissimilar audio data samples and video data samples and to minimize a distance between similar audio data samples and video data samples in the latent space.

[0011] By applying a sequential contrastive loss function directly to the visual and audio latent representations, i.e., without mean-pooling the visual and audio latent representations, differentiation between semantically close examples, e.g. in the visual or audio domain, can be drastically improved.

[0012] In some embodiments, the video data sample may be represented by video frames and the audio data sample may be represented in the Mel-spectrum. The video data sample may include ^ video frames of size ℎ × ^ and three color channels, and the audio data sample may include a sequence of length ^, with ^ Mel-filter bank features per entry. Alternatively, the audio data sample may be represented as an audio waveform. The audio waveform may be a discretized audio waveform of length ^.

[0013] In some embodiments, determining visual features from the video data sample may include encoding the video data sample by a pre-trained neural visual encoder, and determining audio features from the audio data sample may include encoding the audio data sample by a pre-trained neural audio encoder. The pre-trained neural visual encoder may be a CLIP encoder, and / or the pre-trained neural audio encoder may be a BEATS encoder. In this case, the output of the pre-trained neural visual encoder may be a sequence of length ^, with ^^^^^visual features per entry, and / or the output of the neural pre-trained audio encoder may be a sequence of length ^, with ^^^^^^audio features per entry.

[0014] In some embodiments, the determined visual and audio features may be projected to a shared multi-dimensional space. The shared multi-dimensional space may be of size d. The projection may be performed by a multi-layer perceptron. The multi-layer perceptron may also betrained based on the sequential contrastive loss function.

[0015] In some embodiments, obtaining the visual latent representation by applying the neural image encoder to the visual features may include adding a positional encoding to the visual features before being processed by the neural image encoder. Additionally or alternatively, obtaining the audio latent representation by applying the neural audio encoder to the audio features may include adding a positional encoding to the audio features before being processed by the neural audio encoder. The positional encoding of the visual features may be a sinusoidal positional encoding of the number of frames ^. The positional encoding of the audio features may be a sinusoidal positional encoding of the sequence length ^.

[0016] In some embodiments, the neural image encoder may be a first transformer with ^^layers. Further the neural audio encoder may be a second transformer with ^^layers. Both transformers may be transformers with a neural network structure of the original transformer.

[0017] In some embodiments, the sequential contrastive loss function may be based on a distance between a visual latent representation and an audio latent representation in the plurality of visual and audio latent representations. In particular, the distance may be defined as ^^,^=^(^^^ , ^^^ ), if the visual latent representation corresponding to element ^ of thesequence is matrix ^^^ of size ^^ × ^, and the audio latent representation corresponding toelement of the audio-visual sequence is matrix ^^^ of size ^!" ^.

[0018] The sequential contrastive loss function may then be defined as ℒ$%& = − ()|ℬ| ∑|ℬ|123(5^689:77 / <) 123(5^6D9E77 / <)^B( [log123 89: + log |>| 123 D9E], wherein|ℬ|parameter, ^6 GHI = ^5J1KL89:(^) 6^H^ ^5J1KLD9E(^)MNO89:(^) and ^ = MNOD9E(^) .In some embodiments,Euclidian distance, a dynamic time warping, or a Wasserstein distance.

[0019] In some embodiments, the method may further include applying a frame-level contrastive loss function to the plurality of visual and audio latent representations. Further, the neural image encoder and the neural audio encoder may be trained based on the frame-wisecontrastive loss function. Therefore, the neural encoders (and any other optional trainable neural network element) may be trained based on a hybrid approach, combining a contrastive loss based on a mean-pooled and a non-mean-pooled version of the representation of visual and audio features in latent space.

[0020] According to a second aspect of the disclosure, a neural network-based method of audio-visual retrieval is provided. A video data sample and a plurality of audio data samples, or a plurality of video data samples and an audio data sample, or a plurality of video data samples and a plurality of audio data samples may be received, depending on the retrieval use case, i.e., retrieving an audio sample corresponding to a video data sample, retrieving a video data sample corresponding to an audio sample, or matching multiple pairs of audio and video data samples. Then, for each audio data sample and each video data sample the following steps may per performed. Visual features may be determined (e.g., extracted) from the video data sample and audio features may be determined (e.g., extracted) from the audio data sample. A visual latent representation may be obtained by applying a neural image encoder to the visual features and an audio latent representation may be obtained by applying a neural audio encoder to the audio features. The neural image encoder and the neural audio encoder may have been trained according to any embodiment of the first aspect. The latent representations may represent temporal sequences of features in a latent space. A matching pair of an audio data sample and a video data sample, or multiple matching pairs of audio data samples and video data samples may be determined based on the corresponding visual latent representation and audio latent representation.

[0021] By determining a matching pair of an audio data sample and a video data sample based on a neural network (i.e., the neural audio encoder and the neural image encoder) trained with a sequential contrastive loss function, retrieval results may be improved, especially if semantically close data samples (audio or visual) exist.

[0022] In some embodiments, the video data sample may be represented by video frames and the audio data sample may be represented in the Mel-spectrum. The video data sample may include ^ video frames of size ℎ × ^ and three color channels, and the audio data sample may include a sequence of length ^, with ^ Mel-filter bank features per entry. Notably, the format of video and audio data may be identical to the format of audio and video data used for training.

[0023] Alternatively, the audio data sample may be represented as an audio waveform. The audio waveform may be a discretized audio waveform of length ^.

[0024] In some embodiments, determining visual features from the video data sample may include encoding the video data sample by a pre-trained neural visual encoder, and determining audio features from the audio data sample may include encoding the audio data sample by a pre-trained neural audio encoder. The pre-trained neural visual encoder may be a CLIP encoder, and / or the pre-trained neural audio encoder may be a BEATS encoder. In this case, the output of the pre-trained neural visual encoder may be a sequence of length ^, with ^^^^^visual features per entry, and / or the output of the neural pre-trained audio encoder may be a sequence of length ^, with ^^^^^^audio features per entry.

[0025] In some embodiments, the determined visual and audio features may be projected to a shared multi-dimensional space. The shared multi-dimensional space may be of size d. The projection may be performed by a multi-layer perceptron.

[0026] In some embodiments, obtaining the visual latent representation by applying the neural image encoder to the visual features may include adding a positional encoding to the visual features before being processed by the neural image encoder. Additionally or alternatively, obtaining the audio latent representation by applying the neural audio encoder to the audio features may include adding a positional encoding to the audio features before being processed by the neural audio encoder. The positional encoding of the visual features may be a sinusoidal positional encoding of the number of frames ^. The positional encoding of the audio features may be a sinusoidal positional encoding of the sequence length ^.

[0027] In some embodiments, the neural image encoder may be a first transformer with ^^layers. Further the neural audio encoder may be a second transformer with ^^layers. Both transformers may be transformers with a neural network structure of the original transformer.

[0028] In some embodiments, determining the matching pair of audio data sample and video data sample may include determining the audio data sample and the video data sample with a minimal distance for the corresponding audio latent representation and visual latent representation. The distance may be measured by one or more of an interpolated Euclidian distance, a dynamic time warping distance, or a Wasserstein distance.

[0029] In some embodiments, a type of distance used for measuring the distance may beequal to a type of distance used for training the neural image encoder and the neural audio encoder.

[0030] In some embodiments, determining the matching pair of audio data sample and video data sample may include determining a collapsed visual latent representation from the visual latent representation and a collapsed audio latent representation from the audio latent representation. Further, the matching pair of audio data sample and video data sample may be determined based on a cosine similarity of the collapsed visual latent representation and the collapsed audio latent representation.

[0031] Thereby, retrieval may be based on a mean-pooled version of the latent representation, while training may be based on the non-mean-pooled version, i.e., with a sequential contrastive loss function. Thereby, retrieval results may be improved, without increasing the complexity of the retrieval process.

[0032] In some embodiments, the method may further include outputting the matching pair of the audio data sample and the video data sample, or the plurality of matching pairs of audio and video data samples for joined playback.

[0033] According to a third aspect of the disclosure, a method of training a neural network for video-to-audio generation is provided. Audio-visual content may be received. The audio-visual content may include a video data sample and a corresponding audio data sample. A token sequence may be generated (e.g., determined or computed) from the audio data sample. A plurality of tokens in the token sequence may be masked to generate a masked token sequence. Further, a conditioning sequence may be generated (e.g., determined or computed) based on the video data sample. Then, the masked tokens in the masked token sequence may be unmasked by a transformer conditioned on the conditioning sequence. A loss function may be evaluated based on the unmasked token sequence, and the transformer may be trained based on the loss function. Therefore, the transformer may be trained to correctly predict the masked tokens.

[0034] By training a transformer to unmask a masked tokens sequence corresponding to an input audio data sample and conditioning the transformer on features (e.g., semantic and temporal features) of the video data sample, the transformer (and any other trainable neural network element) may be trained for generating audio data from an input (silent) video data sample, wherein the audio sample may be of high quality and at the same time highlysynchronized with events in the input (silent) video data sample.

[0035] In some embodiments, the audio data sample may be an audio waveform of P samples and the video data sample may be a sequence of Q video frames.

[0036] In some embodiments, generating the token sequence from the audio data sample may include applying a pre-trained descript audio codec to the audio data sample to generate the token sequence. The token sequence may therefore correspond to a plurality of codewords. The plurality of codewords may be structured as R sequences of codewords, each with a length of S codewords. The descript audio codec may lead to down-sampling of the audio waveform by a ^ factor ofT.

[0037] In some embodiments, generating the token sequence from the audio data sample may further include embedding and aggregating the token sequence to generate an embedded token sequence. In this case, generating the masked token sequence may be based on the embedded token sequence. Further, a positional encoding may be added to the embedded token sequence.

[0038] In some embodiments, masking the plurality of tokens in the token sequence to generate the masked token sequence may include determining positions to be masked in the token sequence based on a masking scheduler. The masking scheduler may be a cosine scheduler, for example.

[0039] In some embodiments, generating the conditioning sequence based on the video data sample may include generating the conditioning sequence from the video data sample based on a first neural encoder tuned for extracting semantic features from the video data sample and a second neural encoder for extracting temporal alignment features from the video data sample. The first neural encoder may be an image feature encoder and the second neural encoder may be a 3D convolutional video encoder, for example. Specifically, the first neural encoder may be a pre-trained CLIP encoder and the second neural encoder may be a pre-trained S3D encoder. Additionally, each output of the first neural encoder and the second neural encoder may be projected to a multi-dimensional space on which the transformer operates by a first and second multi-layer perceptron, respectively. The first and second multi-layer perceptron may be trained based on the loss function.

[0040] By generating the conditioning sequence based on both semantic and temporalfeatures, the neural network may be better trained for generation of an audio data sample with high quality and better synchronicity with the input video data sample.

[0041] In some embodiments, the conditioning sequence may be length adjusted by a length adapter neural block to be of equal length to the masked token sequence. In some embodiments, the loss (loss function) may be further based on the masked token sequence and the conditioning sequence.

[0042] In some embodiments, training the transformer based on the loss function may include minimizing a negative log-likelihood of masked positions in the unmasked token sequence.

[0043] In some embodiments, generating the conditioning sequence is further based on a visual front-end of a contrastive audio-visual model. The contrastive audio-visual model may be pre-trained based on training according to any embodiment of the first aspect. The output of the visual front-end may be projected to a multi-dimensional space on which the transformer operates by a third multi-layer perceptron. The third multi-layer perceptron may be trained based on the loss function.

[0044] In some embodiments, the transformer may include U AdaLN blocks.

[0045] Alternatively, the transformer may include U cross-attention blocks. In this case, the outputs of the first and second neural encoder may be matched in sequence length by interpolation, combined and projected to a multi-dimensional space on which a transformer encoder operates to generate the conditioning sequence by the transformer encoder. Further, a positional encoding may be added before generating the conditioning sequence by the transformer encoder. The transformer encoder may also be trained based on the loss function.

[0046] Alternatively, the transformer may include U / 2 interleaved cross-attention blocks and AdaLN blocks. In this case, the outputs of the first and second neural encoder may be matched in sequence length by interpolation, combined and projected to a multi-dimensional space on which a transformer encoder operates to generate the conditioning sequence by the transformer encoder. Further, a positional encoding may be added before generating the conditioning sequence by the transformer encoder. The transformer encoder may also be trained based on the loss function.

[0047] In some embodiments, the method may further include evaluating a second loss function based on a mean squared error between the conditioning sequence and an encoded audio data sample. The encoded audio data sample may be generated from the audio data sample based on a pre-trained neural audio encoder. The pre-trained neural audio encoder may be a BEATS encoder, for example.

[0048] In some embodiments, the method may further include evaluating a third loss function based on a contrastive audio-visual model. The contrastive audio-visual model may be pre-trained based on training according to any embodiment of the first aspect. Alternatively, the contrastive audio-visual model may be pre-trained based on a cosine similarity.

[0049] According to a fourth aspect, a neural network-based method for video-to-audio generation is provided. A (silent) video data sample may be received. A conditioning sequence may be generated (e.g., determined or computed) based on the video data sample. A fully masked token sequence may be provided to a transformer conditioned on the conditioning sequence and trained according to any embodiment of the third aspect. A total number of steps may be determined for iteratively unmasking the fully masked token sequence. The transformer may then be initialized with the fully masked token sequence. For each step of the total number of steps with exception of the last step, a plurality of tokens may be predicted (e.g., based on a probability) by the transformer and the predicted tokens may be provided to the transformer as input for a subsequent step. An audio data sample may be generated from a fully unmasked token sequence output by the transformer in a last step corresponding to the total number of steps. The audio data sample may finally be output as a paired data sample to the video data sample.

[0050] By using a trained transformer to unmask a masked tokens sequence corresponding to an input audio data sample and conditioning the transformer on features (e.g., semantic and temporal features) of the video data sample, the generated audio sample may be of high quality and at the same time highly synchronized with events in the input (silent) video data sample.

[0051] In some embodiments, the video data sample may be a sequence of Q video frames.

[0052] In some embodiments, generating the conditioning sequence based on the video data sample may include generating the conditioning sequence from the video data sample basedon a first neural encoder tuned for extracting semantic features from the video data sample and a second neural encoder for extracting temporal alignment features from the video data sample. The first neural encoder may be an image feature encoder and the second neural encoder may be a 3D convolutional video encoder, for example. Specifically, the first neural encoder may be a pre-trained CLIP encoder and the second neural encoder may be a pre-trained S3D encoder.

[0053] Additionally, each output of the first neural encoder and the second neural encoder may be projected to a multi-dimensional space on which the transformer operates by a first and second trained multi-layer perceptron, respectively.

[0054] By generating the conditioning sequence based on both semantic and temporal features, the trained neural network may generate an audio data sample with high quality and better synchronicity with the input video data sample.

[0055] In some embodiments, generating the conditioning sequence is further based on a visual front-end of a contrastive audio-visual model. The contrastive audio-visual model may be pre-trained based on training according to any embodiment of the first aspect. The output of the visual front-end may be projected to a multi-dimensional space on which the transformer operates by a third trained multi-layer perceptron.

[0056] In some embodiments, determining the total number of steps for iteratively unmasking the fully masked token sequence may depend on an available computational budget or a time constraint.

[0057] In some embodiments, the fully masked token sequence may represent an empty codeword representation for the audio data sample to be generated. The codeword representation may include empty positions for R sequences of codewords, each with a length of S codewords.

[0058] In some embodiments, predicting a plurality of tokens by the transformer and providing the predicted tokens to the transformer as input for a subsequent step may include estimating a probability of each codeword for all empty codeword positions in the codeword representation by the transformer. Further, the probability of each codeword may be sampled for each empty codeword position to determine a candidate codeword for each empty codeword position. A confidence score may be determined for each candidate codeword. Then, W candidate codewords with the lowest confidence score may be re-masked for generating a partially empty codeword representation. W may decrease with increasing number of steps being carried out. Thepartially empty codeword representation may be provided to the transformer for the subsequent step. Further, in the last step all remaining codewords on the empty codeword positions may be determined to generate the fully unmasked token sequence.

[0059] In some embodiments, sampling the probability of each codeword for each empty codeword position may be based on a weighted guidance. The weighted guidance may be based on an output of the transformer with the conditioning sequence and an output of the transformer without the conditioning sequence. In other words, the weighted guidance may be based on a confidence score with the conditioning sequence and a confidence score without the conditioning sequence.

[0060] In some embodiments, sampling the probability of each codeword may be based on a temperature factor.

[0061] In some embodiments, the probability of each codeword may be represented by a multinomial distribution.

[0062] In some embodiments, the confidence score may be based on a probability of the candidate codeword given all candidate codewords and the conditioning sequence. The confidence score may be further based on additive noise. The additive noise may be an i.i.d. noise sample from a (0,1) Gumbel distribution multiplied by a diversity factor decreasing for increasing number of steps being carried out.

[0063] In some embodiments, generating the audio data sample from the fully unmasked token sequence output by the transformer in the last step corresponding the total number of steps may include decoding the fully unmasked token sequence to generate the audio data sample. The decoding may be performed by a descript audio codec decoder. The descript audio codec decoder may be chosen such the encoding performed by the descript audio codec encoder used during training may be reverted.

[0064] In some embodiments, the transformer may include U AdaLN blocks.

[0065] Alternatively, the transformer may include U cross-attention blocks. In this case, the outputs of the first and second neural encoder may be matched in sequence length by interpolation, combined and projected to a multi-dimensional space on which a trained transformer encoder operates to generate the conditioning sequence by the trained transformer encoder. Further, a positional encoding may be added before generating the conditioningsequence by the trained transformer encoder.

[0066] Alternatively, the transformer may include U / 2 interleaved cross-attention blocks and AdaLN blocks. In this case, the outputs of the first and second neural encoder may be matched in sequence length by interpolation, combined and projected to a multi-dimensional space on which a trained transformer encoder operates to generate the conditioning sequence by the trained transformer encoder. Further, a positional encoding may be added before generating the conditioning sequence by the trained transformer encoder.

[0067] In some embodiments, before outputting the audio data sample as the paired data sample to the video data sample, any variant of the method may be repeated X times to generate X candidate audio data samples. As a final step, a best matching pair of the X candidate audio data samples and the video data sample may be determined based on a method of audio-visual retrieval according to any embodiment of the second aspect. The candidate audio data sample of the best matching pair may be provided as the audio data sample.

[0068] In some embodiments, the method may further include providing the paired audio data sample and video data sample for playback.

[0069] Aspects of the present disclosure may be implemented via an apparatus. The apparatus may include a processor and memory coupled to the processor. The processor may be adapted carry out the method according to aspects and embodiments of the present disclosure.

[0070] Aspects of the present disclosure may be implemented via a program. When instructions of the program are executed by a processor, the processor may carry out aspects and embodiments of the present disclosure. A computer-readable storage medium may store the program. Such computer-readable storage media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc.. Accordingly, some innovative aspects of the subject matter described in this disclosure can be implemented via one or more computer-readable storage media having software stored thereon.

[0071] It will be appreciated that apparatus features and method steps may be interchanged in many ways. In particular, the details of the disclosed method(s) can be realized by the corresponding apparatus (or system), and vice versa, as the skilled person will appreciate. Moreover, any of the above statements made with respect to the method(s) are understood tolikewise apply to the corresponding apparatus (or system), and vice versa. BRIEF DESCRIPTION OF DRAWINGS

[0072] Example embodiments of the disclosure are explained below with reference to the accompanying drawings, wherein

[0073] Fig. 1 is a flowchart illustrating an example of a method of audio-visual contrastive learning according to embodiments of the disclosure,

[0074] Fig. 2 schematically illustrates an example of a neural network for audio-visual contrastive learning according to embodiments of the disclosure,

[0075] Fig. 3 is a flowchart illustrating an example of a neural network-based method of audio-visual retrieval according to embodiments of the disclosure,

[0076] Fig. 4 illustrates a performance comparison of a hybrid approach for audio-visual retrieval according to embodiments of the disclosure,

[0077] Fig. 5 is a flowchart illustrating an example of a method of training a neural network for video-to-audio generation according to embodiments of the disclosure,

[0078] Figs. 6A to 6C schematically illustrate three examples of a neural network for video-to-audio generation according to embodiments of the disclosure,

[0079] Fig. 7 schematically illustrates neural network structure of a AdaLN Block and a Cross-attention Block according to embodiments of the disclosure,

[0080] Fig. 8 is a flowchart illustrating an example of a neural network-based method for video-to-audio generation according to embodiments of the disclosure,

[0081] Fig. 9 (Parts A and B) schematically illustrates training and inference of a neural network for video-to-audio generation according to embodiments of the disclosure,

[0082] Fig. 10 schematically illustrates an example of an apparatus for multimodal audio- visual retrieval and video-to-audio generation according to embodiments of the disclosure.DETAILED DESCRIPTION 1. Audo-Visual Contrastive Learning and Retrieval

[0083] Audio-visual representation learning is at the core of several recent advancements such as multimodal LLMs and generative models used in video-to-audio, audio-to-video, joint audio-visual, and any-to-any synthesis. Contrastive learning, which aims to distinguish between similar and dissimilar pairs of data points, has emerged as an effective methodology for learning audio-visual representations, by relying on the co-occurrence of the two modalities in unlabeled web-scale video datasets. The standard approach for contrastive learning is to contrast between single embedding vectors that should ideally capture all the semantics of each example. Although such a representation may be sufficient for static modalities, like images, this approach potentially over-compresses for modalities of dynamic nature, like videos, hindering the effectiveness of contrastive learning, and the robustness of the multimodal space.

[0084] In particular, given a batch of visual and audio representations ℬ = YZ^^ ^^ , ^^ [\extracted from videos, where ^^^ ∈ ℝ_` × a and ^^^ ∈ ℝb` × a, the vanilla or vector-basedcontrastive loss is defined as: ℒ^%^ = − ( ∑|ℬ|[123(5c`` / <)123(5c`` / <))|ℬ| ^B( log ∑|>| 123( + log∑|>| ],? 5c`? / <) 123(5c?` / <)

[0085] |ℬ| is a pairwisesimilarity matrix, computed on the mean-pooled audio-visual representation: f^ ^^! = sim(meanpool(^^ ), meanpoolZ^^ [)

[0086] Where “meanpool” indicates temporal mean-pooling and sim : ℝa × ℝa → ℝ isthe cosine similarity function between two d-dimensional vectors: op^ sim(o, p) =

[0087] The goal of the vector-is to pull together in the shared d- dimensional embedding space the mean-pooled audio-visual representations that originate from the same example, and push away the ones originating from different ones. A caveat of this approach is that it disregards the sequential nature of the visual and audio modalities, since it is applied on mean-pooled representations. Although the similarity function can match the generalsemantics through these representations, it is doubtful whether it can adherently model the relations between different temporal sub-spaces within them. Take for example a visual scenewhere three events are happening in order, as in ( ), and two semantically similar, buttemporally different audio scenes ( ) and (

[0088] The vector-based contrastivethe correct signal to the model as to which representations to pull / push, but the bottlenecked mean-pooled space potentially does not have the capacity to do so, even though the complexity of the model can be very high. Thus, it becomes questionable whether the vanilla formulation of the contrastive learning can accurately learn how to disambiguate between the correct and false pairings, in such scenarios, which are not so rare in web-scale video datasets.

[0089] To overcome this shortcoming, a Sequential Audio-Visual Contrastive (SAVC) learning approach is proposed, which operates directly on the non-compressed representation space, and can thus take advantage of the fine-grained semantic information available in the sequences. The method aims to obtain a robust sequential and multimodal representation space that can be useful for several tasks, such as generation or retrieval. Different sequential distance metrics for the proposed SAVC method are disclosed, varying both in performance and computation.

[0090] The Figures (Figs.) and the following description relate to preferred embodiments by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be employed without departing from the principles of what is claimed. Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying figures. It is noted that wherever practicable similar or like reference numbers may be used in the figures and may indicate similar or like functionality. The figures depict embodiments of the disclosed system (or method) for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

[0091] A method 100 is provided for audio-visual contrastive learning as depicted in the flowchart of Fig. 1.

[0092] In step S101, an audio-visual sequence comprising a plurality of corresponding video data samples and audio data samples is received. In other words, each data sample for audio-visual contrastive learning may be a video sample and an audio sample, matching the content of the video sample. The data sample may be an unlabeled video. To retrieve the video data sample s^and the corresponding audio data sample ^^, they may be extracted from the unlabeled video. Alternatively, s^and ^^may be received directly by method 100. s^∈ ℝ_` ×t×I×u may be ^ visual frames of size b` ×^^ ℎ × ^ with three color channels and^ ∈ ℝMel-filter bank features of the audio sample with length ^^and ^These specific representations of the audio and video data shouldnot be construed as limitation to the disclosed embodiments. For example, the video data sample may alternatively only have two color channels and / or represented in the frequency domain. The audio sample may also be represented differently, i.e., as FFT coefficients instead of Mel-filter bank features. Different representations may have different advantages and drawbacks. Components in the proposed neural network on which method 100 may be performed may need to be adapted to the dimension and representation of the audio and video data sample.

[0093] In step S102, for each audio data sample and each video data sample in the sequence, visual features are determined from the video data sample and audio features are determined from the audio data sample (sub-step S102a).

[0094] An example implementation for determining the visual and audio features is depicted in the neural network for audio-visual contrastive learning of Fig. 2. In this example, visual and audio features are obtained / determined via pretrained models Contrastive Language- Image Pre-Training (CLIP) and Bidirectional Encoder representation from Audio Transformers (BEAT)s. In other words, the CLIP and BEAT models may have been trained previously and their weights may be fixed / frozen during the contrastive learning method 100.

[0095] CLIP is a vision-language model that has been pretrained with contrastive learning on a large dataset of paired images and textual descriptions. It thus serves as a state-of-the-art visual recognition front-end. To determine the visual features, the image encoder from CLIP, which is a VisionTransformer (ViT) with ^^^^^layers and a dimensionality of ^^^^^, may be used. Each frame of the video data samples may be processed separately, and then the features may beconcatenated to obtain a sequence of semantic visual representations swx^3 _` ×aDE`y^ ∈ ℝ , i.e.,swx^3^ = [z{|}-^^^Zs^,([, … , z{|}-^^^Zs^,_`[]BEATs is a state-of-the-art audio tagger that has been pretrained with self-supervised learning, and finetuned on large scale audio event classification. Different from other audio frontends that are limited to providing a single semantic representation for the whole audio, outputs from BEATs are fine-grained per-frame predictions of audio events. Thus, a semantically rich sequential audio representation may be provided that can be beneficial in cases where multiple events are happening within an audio sample. To determine the audio features, the encoder of BEATs is used that also follows the ViT architecture, consisting of ^^%^^$transformer layers with a dimensionality of ^^%^^$. The linear classification layer may be discarded and thus the final hidden representation of the encoder ^^%^^$may be used, thus obtaining a sequence of audiofeatures ^^%^^$ b^ ∈ ℝ ` ×a^^^^^, i.e.,^^%^^$^ = ^^^^c-^^z(^^)

[0096] It is noted that CLIP and BEATs encoder used in the implementation depicted in Fig. 2 should not be construed as to limit the invention. Other existing or future pretrained neural network based audio and visual feature extractors may be used for determining the visual and audio features, as long as they provide a similar quality of features. Alternatively, features extractors that are not based on trained neural networks may also be used if they provide a similar quality of feature extraction.

[0097] In sub-step S102b, a visual latent representation is obtained by applying a neural image encoder to the visual features and an audio latent representation is obtained by applying a neural audio encoder to the audio features.

[0098] Before obtaining the visual latent representation and the audio latent representation, swx^3^ and ^^^%^^$may be projected to a shared dimensionality ^, if the dimensionality is different for the audio and visual features. The projection may be performed by a multilayer perceptron (MLP) for the audio features and the visual features. The MLPs may comprise a linear projection to ^^T^, a GELU activation function, and another linear projection to the common dimensionality ^. The result of both MLPs may be denoted as s^wx^3 = ^- wx^3^ ^{}(s^ )v^^1KNM = ^- ^1KNM^ ^{}(v^ )

[0099] Optionally, a sinusoidal positional encoding may be added to visual and audio features with common dimensionality ^ before being input to the neural image encoder and the neural audio encoder, respectively. The neural image encoder ^-^^^ may have ^^layers and the neural audio encoder ^-^^^ may have ^^layers. For the contrastive learning, both encoders may be initialized with random weights. Both encoders may have dimensionality ^ and their neural network structure may follow from the original transformer as described in “Attention is all you need, NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems, Pages 6000 - 6010, 2017”, which is incorporated herein by reference in its entirety. It is noted that the term encoder and transformer may be used interchangeably for referring to a neural network-based vector encoding in a latent space based on an input feature set. The design choices for the neural image encoder and the neural audio encoder may vary since the audio representation may already be a well-formed sequence and thus requires less complex processing than the visual one, which may come from independently extracted frame representations. The complexity may for example differ in the number of layers ^^and ^^.

[0100] The visual latent representation and the audio latent representation may then be expressed as ^^ = ^-^^^(s^wx^3 + ^^ ^ ^ ∙ Pos(^^))^^ = ^-^^^(v^^1KNM ^^ ^ + ^ ∙ Pos(^^))

[0101] Where Pos(·) may be the positional encoding function and ^^, ^^> 0 may be learnable parameters.

[0102] In step S103, a sequential contrastive loss function is directly applied to the plurality of video visual and audio latent representations.

[0103] Traditionally, for contrastive learning, the latent representations ^^^, ^^^ would bemean-pooled to obtain representations ^^ ^ a^ , ^^ ∈ ℝ , for which the contrastive loss function in abatch B of examples is applied. In this embodiment, instead of mean-pooling, the contrastive loss function is directly applied to the latent representations ^^^, ^^^ . More precisely, the similarity / distance function may be directly applied to the latent representations ^^^, ^^^ . The sequential contrastive loss function can then be expressed as:ℒ = − ( 123(5^689: / <) 6D9E$%&|ℬ|77 123(5^ 7 / <))|ℬ| ∑ [log 7^B( 123 + log123 5^6D9E] ^GHI and 6originatefrom ^ ∈ ℝ|ℬ|×|ℬ| after row-wise and column-wise normalization. F may be a tunablesequential distance matrices may be determined as: ^6 GHI = ^5J1KL89:(^) ^5J1KL (^9: a 6^H^ D9E )MNO8 (^) nd ^ = MNOD9E(^) .

[0104] Each entrysimilarity / distance function, i.e., D ^ ^ _^,^ = ^(^^ , ^^ ) ∶ ℝ `×a × ℝb?×a → ℝ«.

[0105] Themay influence performance of the learning process and any inference process afterwards. Without any limitation, three example sequential distance functions are presented in the following. Interpolated Euclidean:

[0106] Since the two sequences have in general different sampling rates (^^ ≠ ^^), apossible method to get a sequential distance measure may be to linearly interpolateof the sequences to the length of the other one, and then calculate the mean squared Euclidean distance (MSE) between every corresponding point in the sequences.

[0107] There may be different options for the interpolation. For example, the sequences may be interpolated from visual to audio (v → a), or from audio to visual (a → v). Further, the level of interpolation may vary, i.e., it may be low (at the pretrained features) or it may be high (at the shared representation space).

[0108] The entries of the matrix ^ can then be expresses as: b?^^^­→^1 ¯° ^→^ ^^ ))Where ^^→^ ∈ ℝb ×a and ^^^^→^ = interpolate ^^ (^^ , m^)^^→^ = int ^^ erpolate(^^ , n^)° ∶ ℝa × ℝa → ℝ« is the squared Euclidean distance between two normalized d−dimensionalvectors.

[0109] An alternative may be to interpolate the sequences to the same length before applying the corresponding encoders, which may be more efficient, since it can be interpolated to the length of the shortest sequence, saving compute from the quadratic complexity of the Transformer. In that case the interpolated sequences would be: ^^→^ = ^-^^^(interpolate(s^wx^3, m ) + ^^^ ^ ^ ∙ Pos(^!))∙ Pos(^^))Dynamic Time Warping:

[0110] Dynamic Time Warping (DTW) is defined as the distance between the shortest alignment path between the two sequences. Since DTW is not differentiable softDTW may be used.

[0111] The entries of the matrix ^ can then be expresses as: D^5³^´^,^ = min ¯ °(^ ^^,± , ^^ )Where · is the collection ofthe soft-min operator, and γ ≥ is a hyperparameter, with γ → 0 approaching the DTW distance. Wasserstein:

[0112] Finally, the application of the Wasserstein distance using Optimal Transport isexamined. Assuming two uniform probability distributions ¹(^), º(!), with ¹(^)² = 1 / ^^ and, º(!) = 1 / ^ that define the mass in each position of theacost matrix »(^^) ∈ ℝ_`×b? (^^) ^ ^« , where ¼½^ = °(^^,± , ^^,x ), the Wasserstein distance may bedefined as the minimum transportation cost over all possible transportation plans ¾(^^) ∈ ℝ-`×b?«between the two representations. To make it differentiable and efficient tothe upper- bound regularized version is used and evaluated with the Sinkhorn algorithm. D^^^ = (^^) ^ ^ (^^)^,^́ m¾(i7=n) ¯ ¿²^ ∙ °Z^^,± , ^^,x [ − À ∙ H(¾ )Where À > 0 is aof a matrix.

[0113] Since the Wasserstein distance is permutation invariant, and thus by default is a distance measure between two set of points, to make it sequential, two positional regularizationterms Â(^) ∈ ℝ_` and Ã(Ä) ∈ ℝb? may be concatenated in order to penalize the transportation ofThus, the Wasserstein distance may be applied using themodified representations ^^̅ _`×(a«() ^ b?×(a«()^ ∈ ℝ and ^^̅ ∈ ℝ :^^ ^ (^) (^) ²5(^̅ = [^^ ; Ç Â ], where ɲ = _`5(Where Ç > 0 may be aDepending on the sequential loss function, the result of the contrastive loss function ℒ $%& mayvary.

[0114] Returning back to Fig. 1, in step S104 the neural image encoder and the neural audio encoder are trained based on the sequential contrastive loss function. Training the neural image encoder and the neural audio encoder may involve evaluating the sequential contrastive loss function, and, depending on the result of the evaluation, weights of the neural image encoder and the neural audio encoder may be updated. Any suitable optimizer may be used for updating the weights. Optimization, i.e., minimization of the contrastive loss function, may continue until a maximum number of batches or until a target loss is achieved.

[0115] After training of the neural network, i.e., the neural image encoder and the neural audio encoder and any other trainable element in the framework, e.g., the MLPs in Fig. 2, has been finished, the neural network may be used for retrieval, i.e., for retrieving an audio datasample, matching a video data sample out of plurality of video data samples, or retrieving a video data sample, matching an audio data sample out of plurality of audio data samples. A method 300 is provided for neural network-based audio-visual retrieval as depicted in the flowchart of Fig. 3.

[0116] In step S301 a video data sample and a plurality of audio data samples, or a plurality of video data samples and an audio data sample, or a plurality of video data samples and a plurality of audio data samples are received. For example, a single video / audio data sample may be input by a user and plurality of audio / video data samples are input by a user, downloaded from a database or received in any other suitable way.

[0117] In step S302, for each audio data sample and each video data sample, visual features are determined from the video data sample and audio features are determined from the audio data sample (sub-step S302a). Steps S302 and 302a may be identical to steps S102 and 102a. Therefore, for a detailed description, it is referred to method 100.

[0118] In sub-step S302b, a visual latent representation is obtained by applying a neural image encoder to the visual features and an audio latent representation is obtained by applying a neural audio encoder to the audio features. Step 302b may be identical to step 102b, with the only difference that the neural image encoder and the neural audio encoder, and any other optional trainable neural network element may have been trained, i.e., the weights of the neural network elements may be optimized, for example by executing any variant of method 100.

[0119] In step S303, a matching pair of an audio data sample and a video data sample out of the plurality of audio data samples and the video data sample, the plurality of video data samples and the audio sample, or the plurality of audio data samples and the plurality of video data samples is determined based on the corresponding visual latent representation and audio latent representation.

[0120] As the neural image encoder and the neural audio encoder have been trained, similar video and audio samples may have a small distance in latent space, while dissimilar pairs may have a large distance in latent space.

[0121] The trained neural network according to method 100 may be used for classic vector-based retrieval, sequence-based retrieval or a combination thereof. All three options will be presented in the following.Vector-based Retrieval:

[0122] To use the trained model for audio-visual cross-modal retrieval, mean-poolingmay be applied to the representations ^^ ^^ , ^^ in order to obtain collapsed (non-sequential)embeddings ^^, ^^ ∈ ℝa for the whole test set Ì , and then q ^^ ^ uerying with visual embedding ^^ ,we retrieve the audio embedding ^^^ with the highest cosine similarity Í. =argmax Í(^^ ^^ , ^N )^∈Î

[0123] Similarly, audio embedding ^^^ may be queried to find the closest video embedding. Sequence-based Retrieval: As contrastive learning is performed with sequential distances, the sequential distances can also be applied for retrieval. Thus, querying with visual representation ^^^, the audio representation ^^^ with the lowest sequential distance ^ is retrieved. =argmin ^(^^, ^^^ N )^∈Î

[0124] Where ^ is any of the sequential distance function mentioned for method 100, i.e., the Interpolated Euclidean, DTW or Wasserstein. Preferably, retrieval uses the same sequential distance function as used for contrastive learning. As the underlying distance may always be the squared Euclidean, it may also be possible to use different sequential distance functions for retrieval and contrastive learning. This may offer a useful flexibility, since DTW and Wasserstein distances are more computationally intensive than the interpolated Euclidean. The normalization for the sequential distance that may have been applied during training does not need to be applied during retrieval, since it does not affect the ordering. Hybrid Retrieval:

[0125] A hybrid approach, by combining both vector-based and sequence-based retrieval may also be possible. A preselection of top-Ï candidates may be made by the relatively fast vanilla vector retrieval and then the final selection may be made on these samples using the sequence-based approach presented in this application. The hybrid approach may provide a flexibility in the trade-off between speed and accuracy, and can be useful for large-scale retrieval, where indexing all candidates with the sequence-based distance measure becomes costly.Experimental Results

[0126] The following example setup values and results should not be construed as limiting the disclosed invention in any way. All example setup values and results merely serve as illustration of performance of the proposed method. Experimental Setup:

[0127] The data from VGGSound (containing 10-second clips form Youtube videos) is used to train the proposed models, e.g., the model depicted in Fig, 2. More specifically, the training data may comprise 153,000 videos. 550 sample videos from the training set are used for validation during training. For testing, 13,000 videos from the test split are used, where examples with some missing files and completely silent examples may be excluded. For out-of-distribution testing the test split of the Music dataset is used, which contains 1,900 videos of 10 seconds each. This may be particularly challenging for audio-visual retrieval, since each video originates from a longer music clip, and thus there are several videos with very similar semantics in both the visual and the acoustic sources.

[0128] For each video sample, its RGB visual frames are extracted. Each frame is processed independently with a visual frontend, which is a CLIP image encoder. For the audio sample of its corresponding video sample, it is resampled to 16kHz, and 128-dimensional Mel- filter bank features are extracted with a window of 25ms. For the audio frontend the BEATs encoder is used, which has been pretrained with self-supervised learning and then finetuned on audio classification with AudioSet-2M. It is a Transformer with 12 layers and dimensionality of 768, and has in total 90M parameters. The audio classification head is removed and the final hidden representation is used for the audio features. The input to BEATs is patched to 16 × 16, along the filter-bank dimension, and flattened. Its output is reconstructed to a monotonically sequence by stacking the patches of each time-step along the feature dimension, resulting in an output dimensionality of 6144. The visual and audio MLPs have hidden dimensions of 2048 and768, accordingly, and project to common dimensionality of ^ = 512. The scale of the positionalencodings is learned and initialized to 1 / Ñ(512). The visual and audio Transformers have Nv= 8 and Na= 2 layers, both with dimensionality of 512, hidden dimension of 2048, 8 attention heads and pre-layer norm structure. The total number of training parameters for the proposed model (e.g., all weights of trainable elements in Fig. 2) is 38M.

[0129] For training, AdamW (^( = 0.95, ^) = 0.98) is used with a base learning rate of7e-4, and a cosine scheduler with warm-up. Experiments with smaller batch sizes (< 100 examples) are trained for 250k steps with 10k warm-up, while experiments with larger ones are trained for 150k steps with 5k warm-up. Dropout is set to 0.1 for all modules. For the contrastive loss, when training with the vanilla (vector-based) setting the representations are mean-pooled tosingle 512-dimensional embeddings, and the learnable temperature is initialized to F = 0.07. Forthe sequential contrastive learning, the temperature is initialized with F = 1. For soft-DTW Õ = 1is used, while for Wasserstein À = 1 is used for the entropy regularization and Ç = 1 for thepositional regularization. For frame level contrastive learning, F^^^^ = 0.25 and F^%^^$ = 1 areused for obtaining the soft labels. A lower temperature is chosen for CLIP, since a much higher self-similarity than BEATs was observed. The proposed models are trained on an NVIDIA V100- 32GB using full precision.

[0130] For evaluation the 10 best checkpoints are averaged according to validation set performance. The trained models are evaluated on the test sets of VGGSound and Music using both vanilla (vector-based) and sequence-based retrieval, and a → v and v → a recall are measured at 1 and 5.

[0131] The effectiveness of the proposed contrastive learning with sequential distances is investigated on bidirectional retrieval in VGGSound test (≈ 13k examples). In the left part of Table 1 the recall@1 is presented in a setting where the models have been trained with a small batch size of 32 examples. Firstly, it is observed that for the models that used sequential contrastive learning, if the same distance is used also at inference time for retrieval, recall@1 improves dramatically by several points. Secondly, among the sequential distances, the interpolated Euclidean outperforms soft-DTW and Wasserstein, while also being simpler and more efficient. Thirdly, even though models trained with sequential contrastive learning have not been explicitly optimized to perform vanilla (vector-based) retrieval, they can do fairly well, being competitive with the vanilla (vector-based) models. Since batch size is a special hyperparameter in contrastive learning, where usually the higher the better, in the right part of Table 1, the same experiments with maxing-out the batch size are performed. Due to the size of the pairwise sequential distance matrices, most experiments cannot scale beyond a batch size of 64 in a 32GB GPU. Thus, it is observed that the configuration where pre interpolation is applied from video-to-audio (Euclideanpre / v→a), when fully scaled to a batch size of 256, can surpass theother methods. For a fair comparison the same interpolation is also performed for the vanilla contrastive, which can then be scaled to 1024 examples per batch, but no improvement could be noticed. Due to its simplicity, scalability and performance, the remaining experiments are conducted with the interpolated Euclidean.Table 1. contrastive losses. Each model is run with a small batch size of 32, and a maxed-out batch size. For each model Vanilla retrieval (temporal mean-pooling and cosine similarity) and Distance-based retrieval with the corresponding distance used in contrastive training are performed. Models trained with vanilla contrastive learning use the interpolated Euclidean for Distance-based retrieval. The subscript in the method name indicates the interpolation used, if any.

[0132] In Table 2 the results of the proposed method in VGGSound and also in the zero- shot and out-of-distribution data of the Music dataset are provided. The proposed models are compared to two strong baselines from the literature: ImageBind, which added other modalities to the CLIP space, having around 700M parameters, and CAVMAE, which was trained with masked data modeling and (vanilla) contrastive learning in AudioSet-2M. In VGGSound, it is observed that the proposed sequential contrastive method can surpass the previously proposed methods by large margins, indicating that it can make better use of the fine-grained sequential information in the audio-visual representations. More specifically, it improves recall@1 by 10-12 points compared to ImageBind and 14-15 points compared to CAV-MAE. It’s noteworthy that the implementation of the vanilla contrastive method proposed in method 300 is competitive with previous works, which may be due to the use of BEATs as an audio front-end. In the Music test set, for which all models are evaluated zero-shot, the proposed method outperforms again previous works by even larger margins in relative comparisons. More specifically, the post- interpolation Euclidean contrastive is more than 3× better than ImageBind and CAV-MAE in terms of recall@1. The Music dataset is not only out-of-distribution due to its domain, but also much more challenging due to the presence of multiple videos with almost identical visual andacoustic features. Thus, the higher performance of the proposed method in this challenging setting, re-enforces the importance of a representation space with proper intra-sequence semantic modelling. Table 2. Bidirectional retrieval results in VGGSound and Music (Zero-shot) test. Upper part of the Table presents performance of previous methods, and lower part presents performance of the proposed methods. † use Distance retrieval.

[0133] Although the benefits of retrieval with sequential distances compared to mean- pooling and cosine similarity are evident, there is an extra computational overhead. This overhead is not an issue when the 13,000 examples of VGGSound test are used for retrieval, but it can be problematic in large-scale retrieval settings of millions of examples. To close this gap, the hybrid retrieval approach that combines both the efficiency of the vanilla and the accuracy of the proposed distance-based is analyzed. Practically, a pre-selection of k candidates is performed using vanilla retrieval and then the sequence-based approach is used in this reduced pool, which has a negligible cost for a small enough k. This approach can work for the proposed models trained with sequential contrastive learning, due to their emergent vanilla retrieval capabilities (see Table 1). In Fig. 4 this hypothesis is tested with a range of values for k in VGGSound and Music test sets. The results indicate that hybrid method is robust and can quickly achieve theupper-bound recall score of the fully distance-based with Ï = 100. Results also drasticallyimprove for smaller values, such as Ï ^ 10. This showcases a useful flexibility in the trade-offbetween efficiency and accuracy, which is not present in models trained with vanilla (vector- based) contrastive learning.

[0134] Following, in Table 3 a speed test is performed with the hybrid retrieval approach. The time to retrieve is tested for different values of k, 1,000 queries in a test set of 10,000candidates. Ï ^ 10Ö means vector-based retrieval, while Ï ^ 10× is the fully sequence-basedretrieval. Results indicate that although sequence-based can be costly, the hybrid approach mayindeed be very efficient. More specifically, at Ï ^ 10), where according to Fig. 4 the retrievalscore of sequence-based is achieved, the hybrid approach has only ×1.5 and ×1.8 computational overhead compared to the vanilla (vector-based) approach. GPU CPU Table 3. Absolute (in seconds) and relative inference time for hybrid retrieval of 1k queries in a test set of 10k candidates. 2. Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity

[0135] V2A generation has some immediate and impactful applications for the media production industry. First, as a technology to accelerate, improve, and / or simplify Foley sound effect generation. Traditionally, an artist recreates sounds that would match a silent video with physical objects while watching the video. In the V2A generative case, an audio effect could be automatically generated in order to mix it with a silent video recording, or even many possible candidates could be generated for an artist to select from them. Additionally, tasks that feature both synchronization with respect to a visual input, and also a textually guided conditioning (including transcripts of contents to be spoken), like automatic dubbing, can benefit greatly from a synchronized V2A generative model that features multi-modal conditioning.

[0136] Early neural V2A approaches proposed sound synthesis from videos as a way to study physical interactions of materials within a visual scene of limited diversity. Similarly, other early works started tackling V2A inside a crossmodality generative adversarial framework, where both video-to-audio and audio-to-video were tackled as a joint problem. A generative model was proposed based on SampleRNN in order to directly generate audio waveforms from unconstrained videos, although still limiting their study to 10 classes collected under their VEGAS dataset. This is built in a source-specific manner, where one model is trained per sound class. Similarly, RegNet is another source-specific V2A model which tackles the problem ofaudio-visual alignment by leveraging motion feature extraction to drive the sound generation. Moreover, they also introduce a regularizer mechanism to palliate the generation of visually irrelevant sounds. Motivated by the need to scale V2A as a source-agnostic problem, SpecVQGAN was proposed as a first multi-class visually guided sound generator model. SpecVQGAN is built upon an autoregressive transformer that learns to generate sequences of codewords that represent mel-spectrograms through a VQGAN lossy compression. Then, a neural vocoder is used to invert the mel-spectrogram back into an audio waveform. Im2Wav is another transformer-based audio language model conditioned on image representation to perform V2A. In this case, a pre-trained CLIP model is used to extract the sequence of visual features coming from the video frames. Then, the approach predicts the discrete tokens that a VQ-VAE model yields. Similarly, CLIPSonicIQ leverages the CLIP features of individual visual frames to drive a sound generator. In this case, the generative approach follows a diffusion strategy that generates mel- spectrograms. This skips the usage of a lossy compression of the mel-spectrogram, but still leverages a neural vocoder to produce the audio waveforms. Except for the usage of a pre-trained CLIP encoder, all these proposals train multiple modules from scratch with their own limited data collections. Recently, the potential of multiple prior mapping models between audio-visual modalities has been investigated, leveraging the inter-connection of pre-trained foundation models of each modality. In particular, V2A-mapper bridge the domain gap between an average CLIP embedding, which summarizes the input video sequence, and a CLAP embedding, which drives an AudioLDM generative model. Many of the aforementioned works leverage visual encoders that were pre-trained for individual image recognition tasks. This hinders the process of modeling the visual dynamics intrinsic to the video scene and its audio-visual synchronicity. Recently, Diff-Foley was proposed as a latent-diffusion generative model which is driven by a contrastive audio-visual pre-trained (CAVP) encoder to improve this. The CAVP explicitly learns to distill audio onset features into the video encoder through self-supervised training, fine-tuning a video encoder to extract alignment-sensitive visual cues to drive the V2A generation. On the other hand, FoleyGen proposes specific architectural attention patterns pre-designed to enforce audiovisual alignment in their generative model. Some works have also shown the effectiveness of separating the audio onset prediction task upon the video in order to drive an audio synthesis model for impact sounds. Other works have also leveraged physics parameters estimated from real-world impact sounds in order to synthesize impact sounds synchronized with a video input. In a related task to enhance the temporal placement of generated sounds in a composition, avideo-synchronized text-to-audio model was proposed by building a Transformer encoder that drives a latent diffusion TTA whilst contrastively pairing audio and video features. Audio-Visual Alignment Representations

[0137] A crucial aspect of V2A is the synchronization (temporal alignment) between an input video and the generated audio. This is often achieved with the help of an audio-visual alignment representation model. AVST (Audio-Visual Synchronization with Transformers) detects audio-visual synchronization in a self-supervised manner and predicting the class as either sync or off-sync. SparseSync considers that the audio-visual correspondence may only be available at sparse events. The proposed SparseSelector compresses the audio and visual input tokens into two small sets of learnable selectors. These selectors form an input to a transformer which predicts the temporal offset between the audio and visual streams. It formulates audio- visual synchronization as a classification task onto a set of offsets (for example, 21 classes between -2 / +2 sec.). Diff-Foley adopts CAVP (Contrastive Audio-Visual Pretraining) to learn more temporally and semantically aligned features, then it trains a latent diffusion model (LDM) with CAVP-aligned visual features on spectrogram latent space. That is, it leverages CAVP for (1) generating audio that is temporally aligned with the visual events, and (2) deriving the Alignment Accuracy metric. Masked Audio Token Generation

[0138] Early works proved that waveform-based generative modeling was possible with explicit maximum-likelihood autoregressive strategies, as in WaveNet or SampleRNN. These proposals suffered from inefficiencies inherent to their autoregressive nature, which was palliated by subsequent works like WaveRNN or parallel WaveNet. The advancement in neural audio codecs also facilitated the use of language modeling strategies for generative audio, and one of their strong advantages over previous models is the lower framerate featured in the codec spaces compared to the raw waveforms. Based on the SoundStream codec, AudioLM is the first to take a language modelling approach to audio generation, which combines semantic and acoustic tokens in a hierarchical fashion to achieve long-term consistency and high quality. Based on Encodec, AudioGen is an autoregressive generative model that generates audio samples conditioned on text inputs. Following AudioLM, MusicLM tackles conditional music generation by means of a hierarchical sequence-to-sequence modeling approach based on MuLan audio tokens in addition to the semantic tokens and acoustic tokens. Following AudioGen, MusicGen consists in anautoregressive transformer-based decoder conditioned on a text or melody representation. It is able to generate consistent music with a single-stage language model through an efficient codebook interleaving strategy. Despite promising results are obtained in the aforementioned models, the autoregressive sequence length grows quadratically, easily forming an extremely long sequence due to the temporally dense nature of audio and the multiple levels of VQ codebooks. SoundStorm is one of the first to adapt a parallel decoding scheme like MaskGIT to predict masked audio tokens produced by SoundStream. Based on DAC, VampNet follows a similar approach for music audio generation. Through different prompting techniques, VampNet can operate in a continuum between compression and generation. Based on Encodec, MAGNet proposes to further improve the efficiency and quality by means of predicting spans of masked tokens, scoring the prediction confidence with a pretrained model, and fusing AR and non-AR generation. MaskVAT

[0139] To overcome the previous shortcomings, a V2A generative model is proposed, named MaskVAT, that interconnects a full-band high-quality general audio codec with a masked language generative model. This combination allows modeling both quality and synchronicity at the same time only based on architecture and input visual feature traits. Further, contrastive audio-visual learning (see section 1) may help improving MaskVAT’s semantic and time alignment with the video as additional input features, regularization loss and as a post-sampling selection model.

[0140] The proposed framework may embed 3 main interconnected blocks: (1) an audio tokenizer that discretizes the audio waveform into a token sequence, (2) the MaskVAT transformer model, (3) and a conditioner used as front-end to MaskVAT.

[0141] The Figures (Figs.) and the following description relate to preferred embodiments by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be employed without departing from the principles of what is claimed.

[0142] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying figures. It is noted that wherever practicable similar or like reference numbers may be used in the figures and may indicate similar or like functionality. Thefigures depict embodiments of the disclosed system (or method) for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

[0143] A method 500 is provided for training a neural network for video-to-audio generation as depicted in the flowchart of Fig. 5.

[0144] In step S501, audio-visual content is received, wherein the audio-visual content comprises a video data sample and a corresponding audio data sample. It should be noted that step S501 may describe a single set of audio-visual content for training the neural network. The neural network may be trained with a plurality of pairs of video data sample and audio data sample, i.e., the audio-visual content for training the neural network may comprise a plurality of training samples.

[0145] The audio-visual content may be an unlabeled video. To retrieve the video data sample s^and the corresponding audio data sample ^^, they may be extracted from the unlabeledvideo. Alternatively, s and ^ may be rece _` ×t×I×u^ ^ ived directly by method 500. s^ ∈ ℝ maybe ^^ visual frames of size ℎ × ^ with three color channels and v^ ∈may be an audiowaveform of P^samples. The audio waveform may be sampled at 44.1 kHz, for example. These specific representations of the audio and video data should however not be construed as limitation to the disclosed embodiments. For example, the video data sample may alternatively only have two color channels and / or represented in the frequency domain. The audio sample may also be represented differently, i.e., as FFT coefficients. Different representations may have different advantages and drawbacks. Components in the proposed neural network may need to be adapted to the dimension and representation of the audio and video data sample.

[0146] In step S502, a token sequence is generated from the audio data sample. An example framework for generating the token sequence is depicted in Figs. 6A to 6C, which shows three example neural network frameworks (a to c) for video-to-audio generation, in particular, the training of the video-to-audio generation. To generate the token sequence, the audio waveform may be first processed by a (pre-trained) neural codec to generate a codegram, i.e., a coded representation, from the audio waveform. A particular example for the neural codec may be a Descript audio codec (DAC).

[0147] The DAC may feature an improved residual vector-quantization (RVQ) compared to previous neural audio codecs, which is used to control the rate-distortion trade-off while maximizing the usage of full-band information present in the raw audio. Additionally, the DAC may improve upon previous neural audio codecs by leveraging periodic inductive biases inside their fully convolutional architecture.

[0148] In the example embodiment depicted in Figs. 6A to 6C, the DAC takes the audiowaveform of P samples v ∈ ℝ^ and returns the codegram, which is a tensor z^ = DAC(v),where z^ ∈ ℝT×Ú. Aof the DAC may be the framerate reduction it features,converting the waveform at for example 44.1kHz to R token sequences of 86.1Hz. Therefore, its ^ time down-sampling rate may beT = 512. The number of parallel channels in z^ refers to theamount of RVQ levels used tothe codec bitrate while maintaining the same S sequence length. In a particular example, the number of token sequences generated by the DAC may be equal to 9.

[0149] For the codegram z^ ∈ ℝT×Ú coming from the tokenizer, previously ahierarchical strategy has been imposed to deal with the different codebook levels R. Either the codebook levels are predicted sequentially for the whole sequence length S, hence predicting thelevel Ï entirely before going to Ï + 1 in the codegram, or separate models have been used todeal with coarse codebook levels and fine codebook levels in two stages, conditioning the fine token prediction upon the coarse one.

[0150] For the proposed framework, no hierarchical design is proposed, but instead, the model may learn the structure. Hence the output of our MaskVAT transformer may directly yield the probabilities for the whole codegram, spanning R levels and S time-steps, all in parallel. This is represented in Fig. 9. In order to reconstruct the audio waveform, the DAC decoder may take all the codeword levels R at time-step Û, embeds them through a linear projection back into a real-valued embedding Ü²Ý ∈ ℝ(Ö)× per codebook level Ï, and summs all of them to reconstruct thefull-band embedding from the residual hierarchy additive aggregation. Hence DAC decoder’sinput is a real-valued tensor ¾ T& ∈ ℝ , where ¾& = ∑Ú ² ² T²B( ¾& and ¾& ∈ ℝ . Since thesummation of RVQ levels may intrinsically represent a full-band acoustic composition, this strategy may be used inject the codegram z^into our MaskVAT transformer. Further, it may be beneficial to initialize the codegram embeddings with the pre-trained ones by DAC. Thiscodegram embedding and aggregation is depicted after the DAC Encoder block in each MaskVAT design of Figs. 6A to 6C. Therefore, there may be as many DAC Embed parallel layers as R codegram levels coming from the audio tokenizer.

[0151] Next, in step S503, a plurality of tokens in the token sequence are masked to generate a masked token sequence. For example, a subset of the L × K tokens generated in step S502 may be masked with a special token [MASK], as shown in the Fig. 9A. The mask positionsto be replaced by [MASK] in the codegram U ∈ Þ0,1ßT×Ú may be determined by a maskingscheduler function. For example, a cosine scheduler may be used for as the masking schedular.Therefore, the probability of each position to be masked may be computed as à = ^áâ(ã),where ã~ å[0, ¶)], from which U^,² = Bernoulli(à) may be obtained. The masked codegramresulting from applying theU to the codegram z^may then be denoted as zç^. The cosine schedular may merely serve as an example for determining the masked positions. Other schedulers or different ways to determine the masked positions may be used.

[0152] Further, in step S504, a conditioning sequence is generated based on the video data sample. Step S504 may be performed in parallel to steps S502 and S503. To generate the conditional sequence, a first neural encoder tuned for extracting semantic features from the video data sample and a second neural encoder for extracting temporal alignment features from the video data sample may be used. In Figs. 6A to 6C, three example configurations based on neural encoders for semantic features and temporal alignment features are shown for generating the conditioning sequence. In the example of Fig. 6A, a pretained CLIP encoder, a pre-trained 3D convolutional video (S3D) encoder, and the visual output of the pre-trained contrastive audio- visual model (CAV) of Fig. 2 may be used to generate the conditioning sequence. The CLIP encoder may be the CLIP encoder described with respect to Fig. 2. The CLIP encoder may be suitable for extracting semantic features from the video data sample. The conditioning sequence may additionally be generated based on a text input or an image input. The text input or the image input may provide a context for generating the audio sample, e.g., a type of an instrument. The CLIP encoder may be used to extract semantic features from the text input or the image input.

[0153] The S3D encoder may be beneficial for detection of audio-visual temporal offsets (temporal alignment features), i.e. detecting temporal shifts between the two modalities. The S3D encoder may be pretrained based on the SparseSync work. The S3D encoder may yield a spatio-temporal tensor of features s^u³ ∈ ℝèéêë×ì()×téêë×Iéêë , which may be average-pooledspatially to yield s^ ∈ ℝèéêë×ì(). These extractedbe considered to besensitive to alignment, since their pre-training task required synchronizing video activity events with the appearance of audio event onsets. For all three video feature extractors, an MLP may be used to project the feature into a dimensionality of the transformer. The transformer may get a channel-wise concatenation of the extracted feature sequences once they are resampled to have the same lengths, resulting in the visual conditioning tensor (conditioning sequence): ^^ [íèëòîèîïðñ 4s è èóôõö^; í ëòîèéêë (s^^u³); í ëòîèîò÷ (sóøù)]Where íèè9û^`úymay be the nearest neighbor resampling operator between input ^^_^and output ^Hü^length, respectively.

[0154] In the example of Figs. 6B and 6C, the CLIP and S3D encoder may also be used for extracting features. Contrary to the implementation of Fig. 6A, said features / sequence of features is first embedded through a transformer encoder (sequence-to-sequence encoder), before being input to the MaskVAT transformer. In this case, the input to the sequence-to-sequence encoder may be denoted as: ^^ [s èîïðñóôõö; íèéêë (s^^u³)]where S3D features are adjusted

[0155] Further, in step S505, the masked tokens in the masked token sequence are unmasked by a transformer conditioned on the conditioning sequence.

[0156] Three possible transformer designs are depicted in Figs. 6A to 6C. These specific designs should however not be construed as limiting the scope of the invention. Other transformer topologies may also be used, if they are suitable for unmasking masked tokens based on a conditioning sequence.

[0157] In the design of Fig. 6A, M AdaLN blocks may be stacked to build the Transformer structure. The conditioning front-end outputs may be adjusted to have the same length as the Transformer input token sequence through the length adapter, which may be a nearest neighbor interpolation layer, and may get concatenated channel-wise after-wards, as described with respect to step S504. Then the conditioning sequence may be served to the AdaLNblocks. The AdaLN blocks itself may be an adaptation of the AdaLN block proposed in diffusion. A detailed configuration of the proposed AdaLN block is depicted in Fig.7 (a). The proposed modification adapts the AdaLN modulation to deal with conditioning sequences, hence featuring a temporal dimension of information. Both input tokens and conditioning sequence may feature the same length in this case. Leveraging AdaLN this way enforces the alignment based on an external length adaptation procedure which is not learned.

[0158] The second MaskVAT transformer design is depicted in Fig. 6B. In this case, a stack of M cross-attention blocks may act as a parallel decoder in order to mix the conditioning with the main token sequence. A single cross-attention block is depicted in Fig. 7 (b), which is the same decoder block as in the original Transformer.

[0159] An advantage of this approach may be the possibility of introducing auxiliar losses that enforce a semantic / alignment proximity with respect to other audio features, similar to what the CAV represents, but in an end-to-end fashion. To achieve this, a mapping on the output of the Transformer encoder depicted in Fig. 6B is performed upon linear projections of BEATs features,i.e. Linear4BEATs4v^^ ∈ ℝè^^^^^×^^8ú , where v may be the audio sample introduced in stepS501, ^^%^^$may be the length of BEATs time-patch sequence, and ^^G_may be the hidden size of the transformer encoder. It should be noted that differently from the CAV pre-training, much more proximity between the Transformer encoder features and BEATs may be enforced, since there may be no need to build a common audio-visual space but instead try to distill BEATs information into driving the generative process of MaskVAT. In this case, the length adapter module may resample the Transformer encoder outcomes to match the temporal resolution of BEATs projections, as done in the CAV pre-training.

[0160] Finally, in Fig. 6C, a hybrid approach is shown that mixes the approaches depicted in Fig. 6A and 6B. In this approach the end-to-end learnable component of distilling BEATs into a Transformer encoder that processes the visual features (see Fig. 6B) is combined with the alignment enforcement of using AdaLN blocks depending on the most alignment-sensitive features, the ones coming from S3D.

[0161] All these MaskVAT models (Fig. 6A to 6C) may end with a linear head operatorthat yields a 3D grid of dimensions S × R × ^, representing the logits over DAC codewords. Inthe specific examples shown in Fig. 6A to 6C, R ^ 9 and ^ ^ 1024 due to the intrinsicconfiguration of DAC. Different numbers may however be used for R and ^, for example by adapting the DAC.

[0162] Unmasking the masked tokens may comprise predicting a candidate token for the masked positions. As this process may also be used for inference of the neural network, it is referred to the description of Fig. 8, in particular step S805, for more details.

[0163] Then, in step S506, a loss function may be evaluated based on the unmasked token sequence.

[0164] Generally, the target may be to minimize a negative log-likelihood through a cross-entropy loss, for the masked positions. Therefore, the loss function may be defined as: ℒb^$² ^ −^[ ¯ log à(^^̂,²|z^ç , ^^] Where ^^̂,²may be

[0165] For different implementations, additional loss functions may be added, to further improve the training process.

[0166] For example, in the sequence-to-sequence and hybrid setups of Fig 6B and Fig. 6C, respectively, a combination of a regression and contrastive loss between the visual embedding sequence after the Transformer encoder and its corresponding BEATs-projected audio embedding sequence may be used additionally. For regression, an MSE minimization between the pairs of sequences may be applied. For the contrastive loss, a [CLS] token may be pre-pended before injecting the sequence into the Transformer encoder, and this position may be selected as the pooled embedding representative to contrast against the average projected BEATs embedding in a CLIP-like contrastive setup. The total loss to train MaskVAT then may be: ℒb^$²^^^_$%&)$%& ^ ℒb^$² + ÇG%^ℒ^^^ + Ç^H_^ℒ^H_^G^$^^^%Where ÇG%^andof the regression and contrastive regularizations, respectively.

[0167] On the other hand, for the setup of Fig 6 (a), the additional training objective / loss function may be the CAV pre-training based on contrastive learning upon sequential distances as defined in any of the implementations of method 100.

[0168] Finally, in step S507, the transformer is trained based on the loss function. In particular, the transformer may be trained based on any combination of the loss functions defined in step S506. Further, not only the transformer, but all trainable elements depicted in, for example Figs. 6A to 6C (marked with thick line), may be trained based on the loss function.

[0169] Training the transformer and any other trainable neural network element may involve evaluating the total loss function (i.e. the combination of the applicable loss functions of the implementation), and, depending on the result of the evaluation, weights of the neural network elements may be updated. Any suitable optimizer may be used for updating the weights. Optimization, i.e., minimization of the total loss function, may continue until a maximum number of batches or until a target loss is achieved.

[0170] After training of the neural network, i.e., the transformer and any other trainable element in the framework, e.g., thick lined elements in Figs. 6A to 6C, has been finished, the neural network may be used for sampling, i.e., for generating an audio sample, given a particular video sample.

[0171] A method 800 is provided for neural network-based video-to-audio generation as depicted in the flowchart of Fig. 8.

[0172] In step S801, a video data sample may be received. The video data sample may have the same format as described with respect to step S501. Notably, the video data sample may not have any audio data attached to it.

[0173] In step S802, a conditioning sequence may be generated based on the video data sample. Step S802 may be identical to step S504, but with the trainable neural network elements, e.g., the transformer encoder, being pretrained according to method 500. Therefore, for a detailed description, it is referred to method 500.

[0174] In step S803 a fully masked token sequence is provided to a trained transformer conditioned on the conditioning sequence. In particular, the transformer may be trained according to method 500. The fully masked token sequence may represent an empty codeword representation for an audio data sample to be generated. The codeword representation may comprise empty positions for K sequences of codewords, each with a length of L codewords.

[0175] In step S804 a total number of steps for iteratively unmasking the fully masked token sequence is determined. The total number of steps and the number of masked tokens mayprovide an indication how many tokens are unmasked in each step. Therefore, the total number of steps may influence both precision and speed of the audio generation. The total number of steps ^$^%^$may thus be chosen depending on a computational budget or a time constraint.

[0176] Then, in step S805, for each step, a plurality of tokens are predicted by the transformer and the predicted tokens are provided to the transformer as input for a subsequent step. In other words, in each step a certain number of tokens may be predicted and the partially unmasked token sequence may be provided to the transformer for a next iteration.

[0177] Predicting tokens by the transformer may comprise estimating a probabilitydistribution of each codegram position 4Û, Ï^ over the codewords of the Ï-th codebook at eachstep ^ ∈ [0, ^$^%^$ − 1]. While computing these probabilities, a classifier-free guidance uponthe logits output by the transformer may also be provided, introducing the coefficient Õ. This technique may improve generation quality at the expense of sample diversity.In particular, if Û^_ = ℳ(z^^ç,L , ^) ) may denote the output logits of MaskVAT ℳ in conditionalform, and Ûü_ = ℳ(z^ç^,L) may denote the unconditional logits that only depend on the estimated and partially-masked codegram z^ç^,L, the guidance-weighted logits may result:Û^ ^ ü_ ^ 41 + Õ)Û_ + ÕÛ_Where Õ ≥ 0, and Õ = 0 may bemode.

[0178] Then, for each masked position (Û, Ï) at step ^, one may sample from themultinomial distribution with a temperature factor F . Thereby, a c ^ $ andidate token ^^̂,²,_may be generated per masked position at step ^. In a next step, a confidence of each of these sampledtokens may be determined based on the log-probability of each position 4Û, Ï^.In this case, the confidence for each estimation at step ^ may be calculated as: confidenceZ^^̂^,²,_ [ ^ log àZ^^̂^,²,_ qz^^ç,L , ^[ + °_ ∙ ^Where ° =_«( _° ∙ (1 −è^^^y^) may be a diversity term and ^ may be an i.id. noise drawn fromGumbel(0, 1). In other words, °_ ∙ ^ may add noise to the confidence, which decreases inamplitude with an increasing number of steps performed. This diversity technique may enhance the generation quality, especially when the number of total steps ^$^%^$is increased. As a nextstep, the next ^ number of tokens may be selected to mask at the next sampling iteration ^ + 1.In particular, this may be performed by taking the W estimated tokens with the lowest confidence score and building a new mask by placing the [MASK] values in these low confidence positions (token positions with low confidence score). For generating the mask, e.g. choosing the number W, the scheduler described in step 503 of method 500 or any other suitable scheduler may be used. The remaining positions may be kept as successfully unmasked in the estimated codegramz^^ç,L«( at the ^ + 1 sampling step. This whole block of operations may be repeated until ^ =^$^%^$ − 1. This operation is depicted in Fig. 9.

[0179] Further, in step 806, an audio data sample is generated from a fully unmasked token sequence output by the transformer in a last step corresponding to the total number of steps.In particular, in the last step, i.e. ^ = ^$^%^$, in contrast to the preceding steps, all remainingmasked positions may be unmasked. Therefore, no confidence score needs to be determined and no remasking needs to be applied to the tokens. Thereby, the fully unmasked token sequence z^^may be output by the transformer in a last step. To generate the audio data sample, a DAC decoder may be applied to the unmasked token sequence z^^.

[0180] Through the introduced probability in the sampling scheme, e.g. the noise in the confidence score or the scheduler, the generated audio data sample may very each time the sampling is performed on the same video data sample. Therefore, some generated audio samples may match better the input video in terms of semantic contents and alignment especially than others.

[0181] To improve the consistency of the generated audio sample, the sampling may be run a plurality of times, and a best matching audio sample may be determined. To determine the best matching audio sample, the pre-trained CAV may be applied to measure the distance in latent space between each generated audio sample and the input video data sample. Finally, the audio sample with the minimal distance to the video sample in latent space may be chosen as the final audio sample for output. This optional improvement of the generated audio sample is depicted in Fig. 9B, with 3 audio samples generated by the sampling process for comparison.

[0182] Finally, in step S807, the generated audio sample is output as a paired data sample to the video data sample. For example, the paired video and audio data sample may be output for playback on a suitable device (e.g., a device capable of outputting video and audio signals) or forstorage in a database. Experimental Results

[0183] The following example setup values and results should not be construed as limiting the disclosed invention in any way. All example setup values and results merely serve as illustration of performance of the proposed method. Datasets

[0184] Both the MaskVAT and CAV model variations are trained on the VGGSound dataset, which contains videos curated to maximize the audio-visual correspondence in the videos while remaining unconstrained in the nature of their content. Originally, the dataset contained 200k video clips in the training partition, but since many videos are not available anymore and videos are filtered based on quality heuristics, a copy of approximately 155k video clips is used for training. The pre-processing heuristics involve removing videos with silent audio or whose audio length does not match a minimum of 10 seconds, as well as videos featuring less than 15 video frames per second (FPS). Each video ends up being 10s long, with the audio sampled at 44kHz. In order to build the validation split, 535 video clips re-purposed from the original train split are selected, which amount to approximately 1,5 hours of content.

[0185] Three test partitions are used to evaluate different aspects of performance for all models. First, a subset of the original VGGSound test split is used to evaluate the generated audio quality and semantic matching with the video. In this VGGSound-test spilt 12,639 video clips are used after following the same process as in train split. Secondly, the temporal alignment is only assessed on a subset of VGGSound-test, specifically filtered to contain only sparse in time and space synchronization signals. This subset contains videos whose audio events and their onsets exhibit strong alignments sparsely in time, like a dog barking in-camera, or a tennis player hitting the ball. This split is named VGGSound-test-sparse. Third, each model’s capabilities are tested out-of-distribution (OOD) on the music synthesis domain by leveraging the MUSIC dataset. The nature of these videos also requires strong audio onset detection from the close-up camera recording of someone playing a musical instrument, therefore this dataset is used to evaluate audio quality, semantic matching, and temporal alignment. In this case 1,908 test video clips are extracted, each spanning 10s duration, from a nonoverlapped sliding window applied upon 103 test videos effectively downloaded from the MUSIC21-solo test partition.Implementation Details

[0186] All models are trained until convergence, tracking the aggregated score of the token prediction accuracy, the average FD scores, and the WavCLIP score on the VGGSound validation partition. The MaskVAT^a^Tèmodel variation is trained with an effective batch size of 200 (across 4 GPUs). The MaskVATSeq2Seq and MaskVATHybrid approaches are trained with an effective batch size of 400 (across 8 GPUs). Larger batches helped stabilizing convergence in this case, probably due to the contrastive component upon the CLIP+S3D encoder in ℒ^H_^G^$^^^%. All models are trained with AdamW, applying a weight decay of 105ì, a learning rate warmup andpolynomial decay between 105^ and 2 · 105×. Warm-up is applied on the first 3,000 trainingiterations. All models are trained with 10% conditioning dropout, so that CLIP, S3D, and CAV front-end features, are independently replaced by learnable [NULL] tokens representing an absence of that information. This enables the application of classifier-free guidance during sampling for each type of conditioning independently.

[0187] Training weights are chosen as ÇG%^ = 1 and Ç^H_^ = 1. Sampling temperaturedefaults to F$ = 1 throughout allexplored grid values:Õ ∈ [0,16]. Diversity explored grid values: ° ∈ [0,16]. Sampling ^$^%^$ explored grid values:^$^%^$ ∈ [8,128], sweet spot found at 32 across MaskVAT variations.Objective Metrics

[0188] For the V2A experiments three axes of performance are evaluated: (1) generated audio quality, (2) semantic matching between the generated audio and the original audio and video, and (3) temporal alignment between the generated audio and the original audio / video. For an objective measurement of quality, it is relied on computing the Fréchet distance (FD) upon different audio feature extractors. FD is supposed to rate the trade-off between quality and diversity attained in the generated audio. Moreover, each audio feature extractor used to compute a different FD offers a different focus on aspects of the generated audio that fit those of the ground truth. three types of embeddings are leveraged to compute FDs. First, VGGish is used to yield the more standardized Fréchet audio distance (FAD) for better comparability with the state of the art. This is a classifier working on magnitude filter-bank representations of the audio, with a receptive field of one second, that operates on 16kHz signals. Secondly, an MFCC representation is used to obtain the FDM metric. This representation is frame-based, with eachframe containing a window of 2048 samples and a shift of 512 samples. 128 filter-banks and 64 MFCCs are extracted, so the embeddings to compute the FDM are 64-dimensional. Finally, the DAC codec 8-dimensional embeddings across the K RVQ levels after quantization are leveraged, prior to the residual summation at the input of the decoder. This is the FDD metric, and thedimensionality of the embeddings are 8 × R, which is 8 × 9 = 72 in the default pre-trainedDAC. This is also a frame-based representation, with a wider receptive field than the MFCC one. Importantly, both MFCC and DAC front-ends operate on 44.1kHz signals, hence measuring the statistical distance in the full-band scenario, which is important for a general audio synthesis situation.

[0189] To assess the semantic matching, two metrics that measure the proximity of the signals in the highly semantic CLIP space are proposed. Further, the audio waveform front-end Wav2CLIP is leveraged to project the generated outcomes into CLIP space. Wav2CLIP was precisely trained to project 16kHz audio waveforms of variable length into a fixed embedding in the CLIP space from audio-video data. Then, the cosine similarity is measured between the two projected embeddings. Two ways of projecting through Wav2CLIP are implemented to measure the proximity: First, both the generated audio and the ground truth audio that came originally with the video are projected. Then the cosine similarity of both embeddings L2 normalized in CLIP space are measured. This metric is named WaveCLIP (WC), where higher values imply closer semantic audio-vs-audio. Secondly, only the generated audio is projected and compared against the average video CLIP embedding, both L2 normalized. This metric is named CyleCLIP (CC), since the metric evaluated how aligned the generated waveform is with the original visual content.

[0190] Finally, the degree of alignment of generated audios is measured with three metrics. First, the self-similarity-based audio novelty is computed, reported as novelty score (NS). This is obtained as the Pearson correlation coefficient between the self-similarity audio novelty curves of the BEATs-encoded sequences for the generated and ground truth audio signals. Note that video prompts are not involved in this metric, hence it is an audio-to-audio comparison. Secondly, two SparseSync (SS) metrics are computed, based on the synchronization model proposed. On one hand, the mean offset prediction is computed between the prompted video with the generated audio: (videoi, genaudioi). On the other hand, the predicted offset probability (21 classes between -2.0 and 2.0 secs with 0.2-sec step size) is used to evaluate theKL-divergence between the ground truth and the generated audio in SSKL. Since SS absolute shift predictions on ground truth data sometimes fail, it is further analyzed whether MaskVAT yields similar temporal shifts to the ones detected by SS on ground truth videos. Results

[0191] The objective results for the VGGSound test are presented in Table 4. Note that NS results show that DAC reconstruction have a very high correlation with the ground truth audios, which is expectable since the codec is transparent to any shift.

[0192] Time synchronicity is improved when using the S3D features with AdaLN blocks in the architecture, hence in MaskV ATAdaLNand MaskV ATHybridcases. Model Quality Semantic Alignment FDD ↓ FDM ↓ FAD ↓ WC ↑ CC ↑ NS ↑ SS ↓ SS KL ↓DACrecon. 0.04 0.10 1.06 0.90 0.126 0.97 0.46 0.07Diff-Foley 1.09 30.8 8.60 0.35 0.087 0.07 0.57 1.19Diff-Foley-BWE 1.22 22.4 7.54 – – – – – Im2Wav 0.45 11.9 6.21 0.45 0.116 0.00 0.68 1.39 Im2Wav-BWE 0.45 7.24 7.89 – – – – – SpecVQGAN 0.26 7.75 5.27 0.33 0.080 0.02 0.67 1.18 SpecVQGAN-BWE 0.42 8.10 5.75 – – – – – V2A-Mapper 0.45 14.1 0.89 0.47 0.124 -0.01 0.68 1.43 V2A-Mapper-BWE 0.24 2.72 0.84 – – – – – MaskVAT AdaLN-A 0.06 1.21 3.83 0.48 0.123 0.05 0.60 1.21 MaskVAT AdaLN-B 0.05 0.88 3.39 0.50 0.123 0.16 0.43 0.88 MaskVAT AdaLN-C 0.05 0.80 2.95 0.52 0.126 0.16 0.48 0.92 MaskVAT Seq2Seq 0.06 0.60 1.51 0.55 0.140 0.05 0.63 1.24 MaskVAT Hybrid 0.08 0.88 2.04 0.55 0.136 0.17 0.40 0.84 Table 4 Further, the objective results for the MUSIC test are presented in Table 5.ModelQuality Semantic AlignmentFDD ↓ FDM ↓ FAD ↓ WC ↑ CC ↑ NS ↑ SS ↓ SS KL ↓ DACrecon. 0.03 0.17 7.99 0.88 0.131 0.94 ? ? Diff-Foley 0.63 24.2 46.3 0.43 0.09 0.02 ? ? Diff-Foley-BWE 0.52 22.48 47.7 – – – – – Im2Wav 0.49 14.1 38.4 0.38 0.08 0.00 ? ? Im2Wav-BWE 0.49 6.63 44.7 – – – – – SpecVQGAN 0.27 7.06 43.2 0.29 0.07 0.01 ? ? SpecVQGAN-BWE 0.41 7.18 44.5 – – – – – V2A-Mapper 0.55 14.4 12.8 0.56 0.124 0.01 ? ? V2A-Mapper-BWE 0.30 4.81 12.1 – – – – MaskVAT AdaLN-A 0.08 1.60 22.8 0.53 0.123 0.02 ? ? MaskVAT AdaLN-B 0.07 1.15 25.3 0.57 0.123 0.16 ? ? MaskVAT AdaLN-C 0.07 1.28 22.6 0.59 0.130 0.15 ? ? MaskVATSeq2Seq0.07 1.02 15.8 0.63 0.137 0.06 ? ?MaskVAT Hybrid 0.09 1.23 19.7 0.62 0.135 0.16 ? ? Table 5 Apparatus

[0193] Finally, the present disclosure likewise relates to an apparatus (e.g., computer- implemented apparatus or apparatus having processing capability in general) for performing or implementing methods and techniques described throughout the present disclosure. For example, this apparatus may relate to an audio decoder. The audio decoder may be a low latency audio decoder, for example, that may be applicable to general audio.

[0194] Fig. 10 shows an example of such apparatus 1000. In particular, apparatus 1000 comprises a processor 1010 and a memory 1020 coupled to the processor 1010. The memory 1020 may store instructions for the processor 1010. The processor 1010 may also receive, among others, suitable input data 1030 (e.g., input frames of time-frequency coefficients, etc.), depending on use cases and / or implementations. The processor 1010 may be adapted to carry out or implement the methods / techniques described throughout the present disclosure and to generate corresponding output data 1040, depending on use cases and / or implementations.

[0195] The present disclosure likewise relates to corresponding computer programs, computer program products, and computer-readable storage media storing such computer programs or computer program products. Interpretation

[0196] A computing device implementing the techniques described above can have thefollowing example architecture. Other architectures are possible, including architectures with more or fewer components. In some implementations, the example architecture includes one or more processors (e.g., dual-core Intel® Xeon® Processors), one or more output devices (e.g., LCD), one or more network interfaces, one or more input devices (e.g., mouse, keyboard, touch- sensitive display) and one or more computer-readable mediums (e.g., RAM, ROM, SDRAM, hard disk, optical disk, flash memory, etc.). These components can exchange communications and data over one or more communication channels (e.g., buses), which can utilize various hardware and software for facilitating the transfer of data and control signals between components.

[0197] The term “computer-readable medium” refers to a medium that participates in providing instructions to processor for execution, including without limitation, non-volatile media (e.g., optical or magnetic disks), volatile media (e.g., memory) and transmission media. Transmission media includes, without limitation, coaxial cables, copper wire and fiber optics. Computer-readable medium can further include operating system (e.g., a Linux® operating system), network communication module, audio interface manager, audio processing manager and live content distributor. Operating system can be multi-user, multiprocessing, multitasking, multithreading, real time, etc. Operating system performs basic tasks, including but not limited to: recognizing input from and providing output to network interfaces and / or devices; keeping track and managing files and directories on computer-readable mediums (e.g., memory or a storage device); controlling peripheral devices; and managing traffic on the one or more communication channels. Network communications module includes various components for establishing and maintaining network connections (e.g., software for implementing communication protocols, such as TCP / IP, HTTP, etc.).

[0198] Architecture can be implemented in a parallel processing or peer-to-peer infrastructure or on a single device with one or more processors. Software can include multiple software components or can be a single body of code.

[0199] The described features can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used, directly or indirectly, in a computer toperform a certain activity or bring about a certain result. A computer program can be written in any form of programming language (e.g., Objective-C, Java), including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, a browser-based web application, or other unit suitable for use in a computing environment.

[0200] Suitable processors for the execution of a program of instructions include, by way of example, both general and special purpose microprocessors, and the sole processor or one of multiple processors or cores, of any kind of computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer will also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, ASICs (application-specific integrated circuits).

[0201] To provide for interaction with a user, the features can be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor or a retina display device for displaying information to the user. The computer can have a touch surface input device (e.g., a touch screen) or a keyboard and a pointing device such as a mouse or a trackball by which the user can provide input to the computer. The computer can have a voice input device for receiving voice commands from the user.

[0202] The features can be implemented in a computer system that includes a back-end component, such as a data server, or that includes a middleware component, such as an application server or an Internet server, or that includes a front-end component, such as a client computer having a graphical user interface or an Internet browser, or any combination of them. The components of the system can be connected by any form or medium of digital data communication such as a communication network. Examples of communication networksinclude, e.g., a LAN, a WAN, and the computers and networks forming the Internet.

[0203] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

[0204] A system of one or more computers can be configured to perform particular actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

[0205] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination.

[0206] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should beunderstood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0207] Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the present invention discussions utilizing terms such as “processing”, “computing”, “calculating”, “determining”, “analyzing” or the like, refer to the action and / or processes of a computer or computing system, or similar electronic computing devices, that manipulate and / or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.

[0208] Reference throughout this invention to “one example embodiment”, “some example embodiments” or “an example embodiment” means that a particular feature, structure or characteristic described in connection with the example embodiment is included in at least one example embodiment of the present invention. Thus, appearances of the phrases “in one example embodiment”, “in some example embodiments” or “in an example embodiment” in various places throughout this invention are not necessarily all referring to the same example embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this invention, in one or more example embodiments.

[0209] As used herein, unless otherwise specified the use of the ordinal adjectives “first”, “second”, “third”, etc., to describe a common object, merely indicate that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner. Also, it is to be understood that the phraseology and terminology used herein are for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having” and variations thereof are meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Unless specified or limited otherwise, the terms “mounted”, “connected”, “supported”, and “coupled” and variations thereof are used broadly and encompass both direct and indirect mountings, connections, supports, and couplings.

[0210] In the claims below and the description herein, any one of the terms comprising, comprised of or which comprises is an open term that means including at least the elements / features that follow, but not excluding others. Thus, the term comprising, when used inthe claims, should not be interpreted as being limitative to the means or elements or steps listed thereafter. For example, the scope of the expression a device comprising A and B should not be limited to devices consisting only of elements A and B. Any one of the terms including or which includes or that includes as used herein is also an open term that also means including at least the elements / features that follow the term, but not excluding others. Thus, including is synonymous with and means comprising.

[0211] It should be appreciated that in the above description of example embodiments of the present invention, various features of the present invention are sometimes grouped together in a single example embodiment, Fig., or description thereof for the purpose of streamlining the present invention and aiding in the understanding of one or more of the various inventive aspects. This method of invention, however, is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed example embodiment. Thus, the claims following the Description are hereby expressly incorporated into this Description, with each claim standing on its own as a separate example embodiment of this invention.

[0212] Furthermore, while some example embodiments described herein include some but not other features included in other example embodiments, combinations of features of different example embodiments are meant to be within the scope of the present invention, and form different example embodiments, as would be understood by those skilled in the art. For example, in the following claims, any of the claimed example embodiments can be used in any combination.

[0213] In the description provided herein, numerous specific details are set forth. However, it is understood that example embodiments of the present invention may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0214] Thus, while there has been described what are believed to be the best modes of the present invention, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the present invention, and it is intended to claim all such changes and modifications as fall within the scope of the present invention. For example, any formulas given above are merely representative of procedures that may be used.Functionality may be added or deleted from the block diagrams and operations may be interchanged among functional blocks. Steps may be added or deleted to methods described within the scope of the present disclosure.

[0215] Various aspects and implementations of the present disclosure may also be appreciated from the following enumerated example embodiments (EEEs), which are not claims.

[0216] EEE 1. A method of audio-visual contrastive learning, the method comprising: receiving an audio-visual sequence comprising a plurality of corresponding video data samples and audio data samples; for each audio data sample and each video data sample in the sequence: determining visual features from the video data sample; determining audio features from the audio data sample; obtaining a visual latent representation by applying a neural image encoder to the visual features; obtaining an audio latent representation by applying a neural audio encoder to the audio features; applying a sequential contrastive loss function directly to the plurality of visual and audio latent representations; training the neural image encoder and the neural audio encoder based on the sequential contrastive loss function.

[0217] EEE 2. The method of EEE 1, wherein the visual latent representation represents a temporal sequence of visual features in a latent space and the audio latent representation represents a temporal sequence of audio features in the latent space.

[0218] EEE 3. The method of EEE 2, wherein applying the sequential contrastive loss function directly to the plurality of video and audio latent representations comprises applying a distance function directly on the temporal sequence of visual features and audio features in the latent space and using the result of the distance function for the sequential contrastive loss function.

[0219] EEE 4. The method of any previous EEE, wherein the video data sample is represented by video frames and the audio data sample is represented in the Mel-spectrum or the audio data sample is represented as an audio waveform.

[0220] EEE 5. The method of EEE 4, wherein the video data sample comprises ^ video frames of size ℎ " ^ and three color channels, and the audio data sample comprises a sequence of length ^, with ^ Mel-filter bank features per entry, or the audio data sample is a discretized audio waveform of length ^.

[0221] EEE 6. The method of any previous EEE, wherein determining visual features from the video data sample comprises encoding the video data sample by a pre-trained neural visual encoder, and wherein determining audio features from the audio data sample comprises encoding the audio data sample by a pre-trained neural audio encoder.

[0222] EEE 7. The method of EEE 6, wherein the pre-trained neural visual encoder is a CLIP encoder, and / or wherein the pre-trained neural audio encoder is a BEATS encoder.

[0223] EEE 8. The method of EEE 7 when depending on EEE 5, wherein the output of the pre-trained neural visual encoder is a sequence of length ^, with ^^^^^visual features per entry, and / or the output of the neural pre-trained audio encoder is a sequence of length ^, with ^^^^^^audio features per entry.

[0224] EEE 9. The method of any previous EEE, wherein determining the visual and audio features further comprises projecting the audio features and the visual features to a shared multi-dimensional space.

[0225] EEE 10. The method of EEE 9 when depending on EEE 8, wherein the shared multi-dimensional space is of size ^, and the projection is performed by a multi-layer perceptron, wherein optionally the multi-layer perceptron is trained based on the sequential contrastive loss function.

[0226] EEE 11. The method of any previous EEE, wherein obtaining the visual latent representation by applying the neural image encoder to the visual features comprises adding a positional encoding to the visual features before being processed by the neural image encoder, and / or obtaining the audio latent representation by applying the neural audio encoder to the audio features comprises adding a positional encoding to the audio features before being processed by the neural audio encoder.

[0227] EEE 12. The method of EEE 11 when depending on EEE 5, wherein the positional encoding of the visual features is a sinusoidal positional encoding of the number of frames ^, and / or wherein the positional encoding of the audio features is a sinusoidal positional encoding of the sequence length ^.

[0228] EEE 13. The method of any previous EEE, wherein the neural image encoder is a first transformer with ^^layers, and / or the neural audio encoder is a second transformer with ^^layers.

[0229] EEE 14. The method of any previous EEE, wherein the sequential contrastive loss function is based on a distance between a visual latent representation and an audio latent representation in the plurality of visual and audio latent representations.

[0230] EEE 15. The method of EEE 14, wherein the visual latent representation corresponding to element ^ of the audio-visual sequence is matrix ^^^ of size ^ " ^, and the audio latent representation corresponding to element of the audio-visual sequence is matrix ^^^ of size^ " ^, and the distance is defined as ^ ^^,^ = ^(^^ , ^^^ ).

[0231] EEE 16. The method of EEE 15, wherein the sequential contrastive loss function is defined as 6ℒ $%& ^ − (89:6D9E)|ℬ|∑|ℬ| ^B( [log 123(5^ 77 / <)12345^77 / <) ∑ |>|123( 689:+ log ∑|>|6D9E], ?@A5^ 7= / <) ?@A123(5^ =7 / <)wherein|ℬ|is the length of the audio-visual sequence, F is a tunable parameter, ^6 GHI =^5J1KL89:(^)6 ^H^^5J1KLD9E(^) MNO89:(^)and ^ =MNOD9E(^) .

[0232] EEE 17. The method of EEEs 14 to 16, wherein the distance is an interpolated Euclidian distance, a dynamic time warping, or a Wasserstein distance.

[0233] EEE 18. The method of any previous EEE, wherein the method further comprises: applying a frame-level contrastive loss function to the plurality of visual and audio latent representations; and additionally training the neural image encoder and the neural audio encoder based on the frame-wise contrastive loss function.

[0234] EEE 19. The method of any previous EEE, wherein the neural image encoder and the neural audio encoder are trained for maximizing a distance between dissimilar audio data samples and video data samples and to minimize a distance between similar audio data samples and video data samples in a latent space.

[0235] EEE 20. A neural network-based method of audio-visual retrieval, the method comprising: receiving a video data sample and a plurality of audio data samples, or a plurality of video data samples and an audio data sample, or a plurality of video data samples and a plurality of audio data samples; for each audio data sample and each video data sample: determining visual features from the video data sample; determining audio features from the audio data sample; obtaining a visual latent representation by applying a neural image encoder to the visual features; obtaining an audio latent representation by applying a neural audio encoder to the audio features, wherein the neural image encoder and the neural audio encoder have been trained according to any one of EEEs 1 to 19;determining a matching pair of an audio data sample and a video data sample out of the plurality of audio data samples and the video data sample, the plurality of video data samples and the audio sample, or the plurality of audio data samples and the plurality of video data samples based on the corresponding visual latent representation and audio latent representation.

[0236] EEE 21. The method of EEE 20, wherein determining the matching pair of audio data sample and video data sample comprises determining the audio data sample and the video data sample with a minimal distance for the corresponding audio latent representation and visual latent representation, wherein the distance is measured by one or more of an interpolated Euclidian distance, a dynamic time warping, or a Wasserstein distance.

[0237] EEE 22. The method of EEE 20 or 21. wherein determining the matching pair of audio data sample and video data sample comprises: determining a collapsed visual latent representation from the visual latent representation and a collapsed audio latent representation from the audio latent representation; determining the matching pair of audio data sample and video data sample based on a cosine similarity of the collapsed visual latent representation and the collapsed audio latent representation.

[0238] EEE 23. The method of EEE 22 when depending on EEE 21, wherein a type of distance used for measuring the distance is equal to a type of distance used for training the neural image encoder and the neural audio encoder.

[0239] EEE 24. The method of any one of EEEs 20 to 23, wherein the method further comprises outputting the matching pair of the audio data sample and the video data sample for joined playback.

[0240] EEE 25. A method of training a neural network for video-to-audio generation, the method comprising:receiving audio-visual content, wherein the audio-visual content comprises a video data sample and a corresponding audio data sample; generating a token sequence from the audio data sample; masking a plurality of tokens in the token sequence to generate a masked token sequence; generating a conditioning sequence based on the video data sample; unmasking the masked tokens in the masked token sequence by a transformer conditioned on the conditioning sequence; evaluating a loss function based on the unmasked token sequence; and training the transformer based on the loss function.

[0241] EEE 26. The method of EEE 25, wherein the audio data sample is an audio waveform of P samples and the video data sample is a sequence of Q video frames.

[0242] EEE 27. The method of any one of EEEs 25 to 26, wherein generating the token sequence from the audio data sample comprises: applying a pre-trained descript audio codec to the audio data sample to generate the token sequence, wherein the token sequence corresponds to a plurality of codewords.

[0243] EEE 28. The method of EEE 27, wherein the plurality of codewords are structured as R sequences of codewords, each with a length of S codewords, wherein the descript audio ^ codec leads to down-sampling of the audio waveform by a factor of T.

[0244] EEE 29. The method of any one of EEEs 25 to 28, wherein generating the token sequence from the audio data sample further comprises: embedding and aggregating the token sequence to generate an embedded token sequence; and generating the masked token sequence based on the embedded token sequence.

[0245] EEE 30. The method of EEE 29, wherein generating the token sequence from the audio data sample further comprises:adding a positional encoding to the embedded token sequence.

[0246] EEE 31. The method of any one of EEEs 25 to 30, wherein masking the plurality of tokens in the token sequence to generate the masked token sequence comprises: determining positions to be masked in the token sequence based on a masking scheduler.

[0247] EEE 32. The method of EEE 31, wherein the masking scheduler is based on a cosine scheduler.

[0248] EEE 33. The method of any one of EEEs 25 to 32, wherein generating the conditioning sequence based on the video data sample comprises generating the conditioning sequence from the video data sample based on a first neural encoder tuned for extracting semantic features from the video data sample and a second neural encoder for extracting temporal alignment features from the video data sample.

[0249] EEE 34. The method of EEE 33, wherein the first neural encoder is an image feature encoder and / or the second neural encoder is a 3D convolutional video encoder.

[0250] EEE 35. The method of any one of EEEs 33 to 34, wherein the first neural encoder is a pre-trained CLIP encoder and / or the second neural encoder is a pre-trained S3D encoder.

[0251] EEE 36. The method of any one of EEEs 33 to 35, wherein the conditioning sequence is length adjusted by a length adapter neural block to be of equal length to the masked token sequence.

[0252] EEE 37. The method of any one of EEEs 25 to 36, wherein the loss is further based on the masked token sequence and the conditioning sequence.

[0253] EEE 38. The method of any one of EEEs 25 to 37, wherein training the transformer based on the loss function comprises minimizing a negative log-likelihood of maskedpositions in the unmasked token sequence.

[0254] EEE 39. The method of any one of EEEs 33 to 38 when depending on EEE 33, wherein generating the conditioning sequence from the video data sample based on the first neural encoder tuned for extracting semantic features from the video data sample and the second neural encoder for extracting temporal alignment features from the video data sample comprises: projecting each output of the first neural encoder and the second neural encoder to a multi-dimensional space on which the transformer operates by a first and second multi-layer perceptron, respectively, wherein the first and second multi-layer perceptron are trained based on the loss function.

[0255] EEE 40. The method of any one of EEEs 33 to 39 when depending on EEE 33, wherein generating the conditioning sequence is further based on a visual front-end of a contrastive audio-visual model, wherein the contrastive audio-visual model is pre-trained based on training according to any one of EEEs 1 to 19.

[0256] EEE 41. The method of EEE 40, wherein the output of the visual front-end is projected to a multi-dimensional space on which the transformer operates by a third multi-layer perceptron, wherein the third multi-layer perceptron is trained based on the loss function.

[0257] EEE 42. The method of any one of EEEs 25 to 41, wherein the transformer comprises U AdaLN blocks.

[0258] EEE 43. The method of any one of EEEs 33 to 41 when depending on EEE 33, wherein the transformer comprises U cross-attention blocks and wherein the outputs of the first and second neural encoder are matched in sequence length by interpolation, combined and projected to a multi-dimensional space on which a transformer encoder operates to generate the conditioning sequence by the transformer encoder, wherein optionally a positional encoding is added before generating the conditioning sequence by the transformer encoder, and wherein the transformer encoder is trained based on the loss function.

[0259] EEE 44. The method of any one of EEEs 33 to 41 when depending on EEE 33, wherein the transformer comprises U / 2 interleaved cross-attention blocks and AdaLN blocks, wherein the outputs of the first and second neural encoder are matched in sequence length by interpolation, combined and projected to a multi-dimensional space on which a transformer encoder operates to generate the conditioning sequence by the transformer encoder, wherein optionally a positional encoding is added before generating the conditioning sequence by the transformer encoder; and wherein the transformer encoder is trained based on the loss function.

[0260] EEE 45. The method of any one of EEEs 43 to 44, wherein the method further comprises: evaluating a second loss function based on a mean squared error between the conditioning sequence and an encoded audio data sample, wherein the encoded audio data sample is generated from the audio data sample based on a pre-trained neural audio encoder, wherein the pre-trained neural audio encoder is optionally a BEATS encoder; evaluating a third loss function based on a contrastive audio-visual model, wherein the contrastive audio-visual model is pre-trained based on training according to any one of EEEs 1 to 19 or the contrastive audio-visual model is pre-trained based on a cosine similarity; determining a total loss by calculating a weighted sum of the loss function, the second loss function and the third loss function; and training the transformer based on the total loss.

[0261] EEE 46. The method of any one of EEEs 25 to 45, wherein the transformer is trained to correctly predict the masked tokens.

[0262] EEE 47. A neural network-based method for video-to-audio generation, the method comprising: receiving a video data sample; generating a conditioning sequence based on the video data sample;providing a fully masked token sequence to a transformer conditioned on the conditioning sequence and trained according to any one of EEEs 25 to 46; determining a total number of steps for iteratively unmasking the fully masked token sequence; initializing the transformer with the fully masked token sequence; for each step, predicting a plurality of tokens by the transformer and providing the predicted tokens to the transformer as input for a subsequent step, generating an audio data sample from a fully unmasked token sequence output by the transformer in a last step corresponding to the total number of steps; outputting the audio data sample as a paired data sample to the video data sample.

[0263] EEE 48. The method of EEE 47, wherein generating the conditioning sequence based on the video data sample comprises generating the conditioning sequence from the video data sample based on a first neural encoder tuned for extracting semantic features from the video data sample and a second neural encoder for extracting temporal alignment features from the video data sample.

[0264] EEE 49. The method of EEE 48, wherein the first neural encoder is an image feature encoder and / or the second neural encoder is a 3D convolutional video encoder.

[0265] EEE 50. The method of any one of EEEs 48 to 49, wherein the first neural encoder is a pre-trained CLIP encoder and / or the second neural encoder is a pre-trained S3D encoder.

[0266] EEE 51. The method of any one of EEEs 48 to 50, wherein generating the conditioning sequence based on the video data sample further comprises generating the conditioning sequence based on a text input or an image input for providing a context for generating the audio data sample.

[0267] EEE 52. The method of EEE 51 when depending on EEE 50, wherein the pre- trained CLIP encoder extracts semantic features from the text input or the image input for generating the conditioning sequence.

[0268] EEE 53. The method of any one of EEEs 48 to 52, wherein the conditioning sequence is length adjusted by a length adapter neural block to be of equal length to the fully masked token sequence.

[0269] EEE 54. The method of any one of EEEs 48 to 53, wherein generating the conditioning sequence from the video data sample based on the first neural encoder tuned for extracting semantic features from the video data sample and the second neural encoder for extracting temporal alignment features from the video data sample comprises: projecting each output of the first neural encoder and the second neural encoder to a multi-dimensional space on which the transformer operates by a first and second multi-layer perceptron, respectively, wherein the first and second multi-layer perceptron are trained according to EEE 39.

[0270] EEE 55. The method of any one of EEEs 48 to 54, wherein generating the conditioning sequence is further based on a visual front-end of a contrastive audio-visual model, wherein the contrastive audio-visual model is pre-trained based on training according to any one of EEEs 1 to 19.

[0271] EEE 56. The method of any one of EEEs 47 to 55, wherein determining the total number of steps for iteratively unmasking the fully masked token sequence depends on an available computational budget or a time constraint.

[0272] EEE 57. The method of any one of EEEs 47 to 56, wherein the fully masked token sequence represents an empty codeword representation for the audio data sample to be generated, wherein the codeword representation comprises empty positions for R sequences of codewords, each with a length of S codewords.

[0273] EEE 58. The method of EEE 57, wherein predicting a plurality of tokens by the transformer and providing the predicted tokens to the transformer as input for a subsequent stepcomprises: estimating a probability of each codeword for all empty codeword positions in the codeword representation by the transformer; sampling the probability of each codeword for each empty codeword position to determine a candidate codeword for each empty codeword position; determining a confidence score for each candidate codeword; re-masking W candidate codewords with the lowest confidence score for generating a partially empty codeword representation; providing the partially empty codeword representation to the transformer for the subsequent step.

[0274] EEE 59. The method of EEE 58, wherein sampling the probability of each codeword for each empty codeword position is based on a weighted guidance, wherein the weighted guidance is based on an output of the transformer with the conditioning sequence and an output of the transformer without the conditioning sequence.

[0275] EEE 60. The method of any one of EEEs 58 to 59, wherein the sampling the probability of each codeword is based on a temperature factor.

[0276] EEE 61. The method of any one of EEEs 58 to 60, wherein the probability of each codeword is represented by a multinomial distribution.

[0277] EEE 62. The method of any one of EEEs 58 to 61, wherein the confidence score is based on a probability of the candidate codeword given all candidate codewords and the conditioning sequence.

[0278] EEE 63. The method of EEE 62, wherein the confidence score is further based on additive noise.

[0279] EEE 64. The method of EEE 63, wherein the additive noise is an i.i.d. noisesample from a (0,1) Gumbel distribution multiplied by a diversity factor decreasing for increasing number of steps being carried out.

[0280] EEE 65. The method of any one of EEEs 58 to 64, wherein R decreases with increasing number of steps being carried out.

[0281] EEE 66. The method of any one of EEEs 58 to 65, wherein in the last step all remaining codewords on the empty codeword positions are determined to generate the fully unmasked token sequence.

[0282] EEE 67. The method of any one of EEEs 49 to 66, wherein generating the audio data sample from the fully unmasked token sequence output by the transformer in the last step corresponding the total number of steps comprises: decoding the fully unmasked token sequence to generate the audio data sample.

[0283] EEE 68. The method of EEE 67, wherein decoding is performed by a descript audio codec decoder.

[0284] EEE 69. The method of any one of EEEs 47 to 68, wherein the transformer comprises U AdaLN blocks.

[0285] EEE 70. The method of any one of EEEs 48 to 68 when depending on EEE 48, wherein the transformer comprises U cross-attention blocks and wherein the outputs of the first and second neural encoder are matched in sequence length by interpolation, combined and projected to a multi-dimensional space on which a trained transformer encoder operates to generate the conditioning sequence by the trained transformer encoder, wherein optionally a positional encoding is added before generating the conditioning sequence by the trained transformer encoder.

[0286] EEE 71. The method of any one of EEEs 48 to 68 when depending on EEE 48,wherein the transformer comprises U / 2 interleaved cross-attention blocks and AdaLN blocks, wherein the outputs of the first and second neural encoder are matched in sequence length by interpolation, combined and projected to a multi-dimensional space on which a trained transformer encoder operates to generate the conditioning sequence by the trained transformer encoder, wherein optionally a positional encoding is added before generating the conditioning sequence by the trained transformer encoder.

[0287] EEE 72. The method of any one of EEEs 47 to 70, wherein before outputting the audio data sample as the paired data sample to the video data sample, the method further comprises: repeating the following processing X times to generate X candidate audio data samples: providing the fully masked token sequence to the transformer conditioned on the conditioning sequence and trained according to any one of EEEs 25 to 46; for each step, predicting the plurality of tokens by the transformer and providing the predicted tokens to the transformer as input for the subsequent step, generating the ^-th candidate audio data sample from the fully unmasked token sequence output by the transformer in the last step corresponding the total number of steps; determining a best matching pair of the X candidate audio data samples and the video data sample based on the network-based method of audio-visual retrieval according to any one of EEEs 20 to 23; and providing the candidate audio data sample of the best matching pair as the audio data sample.

[0288] EEE 73. The method of any one of EEEs 47 to 72, wherein the method further comprises providing the paired audio data sample and video data sample for playback.

[0289] EEE 74. An apparatus, comprising a processor and a memory coupled to the processor, wherein the processor is adapted to carry out the method according to any one of EEEs 1 to 73.

[0290] EEE 75. A program comprising instructions that, when executed by a processor, cause the processor to carry out the method according to any one of EEEs 1 to 73.

[0291] EEE 76. A computer-readable storage medium storing the program according to EEE 75.

Claims

CLAIMS 1. A method of audio-visual contrastive learning, the method comprising: receiving an audio-visual sequence comprising a plurality of corresponding video data samples and audio data samples; for each audio data sample and each video data sample in the sequence: determining visual features from the video data sample; determining audio features from the audio data sample; obtaining a visual latent representation by applying a neural image encoder to the visual features; obtaining an audio latent representation by applying a neural audio encoder to the audio features; applying a sequential contrastive loss function directly to the plurality of visual and audio latent representations; training the neural image encoder and the neural audio encoder based on the sequential contrastive loss function.

2. The method of claim 1, wherein the visual latent representation represents a temporal sequence of visual features in a latent space and the audio latent representation represents a temporal sequence of audio features in the latent space.

3. The method of claim 2, wherein applying the sequential contrastive loss function directly to the plurality of video and audio latent representations comprises applying a distance function directly on the temporal sequence of visual features and audio features in the latent space and using the result of the distance function for the sequential contrastive loss function.

4. The method of any previous claim, wherein the video data sample is represented by video frames and the audio data sample is represented in the Mel-spectrum or the audio data sample is represented as an audio waveform.

5. The method of claim 4, wherein the video data sample comprises ^ video frames of size ℎ " ^ and three color channels, and the audio data sample comprises a sequence of length ^,with ^ Mel-filter bank features per entry, or the audio data sample is a discretized audio waveform of length ^.

6. The method of any previous claim, wherein determining visual features from the video data sample comprises encoding the video data sample by a pre-trained neural visual encoder, and wherein determining audio features from the audio data sample comprises encoding the audio data sample by a pre-trained neural audio encoder.

7. The method of claim 6, wherein the pre-trained neural visual encoder is a CLIP encoder, and / or wherein the pre-trained neural audio encoder is a BEATS encoder.

8. The method of claim 7 when depending on claim 5, wherein the output of the pre- trained neural visual encoder is a sequence of length ^, with ^^^^^visual features per entry, and / or the output of the neural pre-trained audio encoder is a sequence of length ^, with ^^^^^^audio features per entry.

9. The method of any previous claim, wherein determining the visual and audio features further comprises projecting the audio features and the visual features to a shared multi- dimensional space.

10. The method of claim 9 when depending on claim 8, wherein the shared multi- dimensional space is of size ^, and the projection is performed by a multi-layer perceptron, wherein optionally the multi-layer perceptron is trained based on the sequential contrastive loss function.

11. The method of any previous claim, wherein obtaining the visual latent representation by applying the neural image encoder to the visual features comprises adding a positional encoding to the visual features before being processed by the neural image encoder, and / or obtaining the audio latent representation by applying the neural audio encoder to the audio features comprises adding a positional encoding to the audio features before being processed by the neural audio encoder.

12. The method of claim 11 when depending on claim 5, wherein the positional encoding of the visual features is a sinusoidal positional encoding of the number of frames ^, and / or wherein the positional encoding of the audio features is a sinusoidal positional encoding of the sequence length ^.

13. The method of any previous claim, wherein the neural image encoder is a first transformer with ^^layers, and / or the neural audio encoder is a second transformer with ^^layers.

14. The method of any previous claim, wherein the sequential contrastive loss function is based on a distance between a visual latent representation and an audio latent representation in the plurality of visual and audio latent representations.

15. The method of claim 14, wherein the visual latent representation corresponding to element ^ of the audio-visual sequence is matrix ^^^ of size ^ " ^, and the audio latent representation corresponding to element of the audio-visual sequence is matrix ^^^ of size^ " ^, and the distance is defined as ^^,^ = ^(^ ^^ , ^^^ ).

16. The method of claim 15, wherein the sequential contrastive loss function is defined as ℒ$%& = −( )ℬ ∑|ℬ|123(5^68779: / <)12345^6D779E / <) ^B( [log|>|123 5^689:+ log|>|123],parameter, ^6 GHI =^5J1KL89:(^) NO89:(^)a 6 ^H^^5J1KLD9E(^) Mnd ^ =MNOD9E(^) .

17. The method of claims 14 to 16, wherein the distance is an interpolated Euclidian distance, a dynamic time warping, or a Wasserstein distance.

18. The method of any previous claim, wherein the method further comprises: applying a frame-level contrastive loss function to the plurality of visual and audio latent representations; andadditionally training the neural image encoder and the neural audio encoder based on the frame-wise contrastive loss function.

19. The method of any previous claim, wherein the neural image encoder and the neural audio encoder are trained for maximizing a distance between dissimilar audio data samples and video data samples and to minimize a distance between similar audio data samples and video data samples in a latent space.

20. A neural network-based method of audio-visual retrieval, the method comprising: receiving a video data sample and a plurality of audio data samples, or a plurality of video data samples and an audio data sample, or a plurality of video data samples and a plurality of audio data samples; for each audio data sample and each video data sample: determining visual features from the video data sample; determining audio features from the audio data sample; obtaining a visual latent representation by applying a neural image encoder to the visual features; obtaining an audio latent representation by applying a neural audio encoder to the audio features, wherein the neural image encoder and the neural audio encoder have been trained according to any one of claims 1 to 19; determining a matching pair of an audio data sample and a video data sample out of the plurality of audio data samples and the video data sample, the plurality of video data samples and the audio sample, or the plurality of audio data samples and the plurality of video data samples based on the corresponding visual latent representation and audio latent representation.

21. The method of claim 20, wherein determining the matching pair of audio data sample and video data sample comprises determining the audio data sample and the video data sample with a minimal distance for the corresponding audio latent representation and visual latent representation, wherein the distance is measured by one or more of an interpolated Euclidian distance, a dynamic time warping, or a Wasserstein distance.

22. The method of claim 20 or 21. wherein determining the matching pair of audio data sample and video data sample comprises: determining a collapsed visual latent representation from the visual latent representation and a collapsed audio latent representation from the audio latent representation; determining the matching pair of audio data sample and video data sample based on a cosine similarity of the collapsed visual latent representation and the collapsed audio latent representation.

23. The method of claim 22 when depending on claim 21, wherein a type of distance used for measuring the distance is equal to a type of distance used for training the neural image encoder and the neural audio encoder.

24. The method of any one of claims 20 to 23, wherein the method further comprises outputting the matching pair of the audio data sample and the video data sample for joined playback.

25. A method of training a neural network for video-to-audio generation, the method comprising: receiving audio-visual content, wherein the audio-visual content comprises a video data sample and a corresponding audio data sample; generating a token sequence from the audio data sample; masking a plurality of tokens in the token sequence to generate a masked token sequence; generating a conditioning sequence based on the video data sample; unmasking the masked tokens in the masked token sequence by a transformer conditioned on the conditioning sequence; evaluating a loss function based on the unmasked token sequence; and training the transformer based on the loss function.

26. The method of claim 25, wherein the audio data sample is an audio waveform of P samples and the video data sample is a sequence of Q video frames.

27. The method of any one of claims 25 to 26, wherein generating the token sequence from the audio data sample comprises: applying a pre-trained descript audio codec to the audio data sample to generate the token sequence, wherein the token sequence corresponds to a plurality of codewords.

28. The method of claim 27, wherein the plurality of codewords are structured as R sequences of codewords, each with a length of S codewords, wherein the descript audio codec ^ leads to down-sampling of the audio waveform by a factor of T.

29. The method of any one of claims 25 to 28, wherein generating the token sequence from the audio data sample further comprises: embedding and aggregating the token sequence to generate an embedded token sequence; and generating the masked token sequence based on the embedded token sequence.

30. The method of claim 29, wherein generating the token sequence from the audio data sample further comprises: adding a positional encoding to the embedded token sequence.

31. The method of any one of claims 25 to 30, wherein masking the plurality of tokens in the token sequence to generate the masked token sequence comprises: determining positions to be masked in the token sequence based on a masking scheduler.

32. The method of claim 31, wherein the masking scheduler is based on a cosine scheduler.

33. The method of any one of claims 25 to 32, wherein generating the conditioning sequence based on the video data sample comprises generating the conditioning sequence from the video data sample based on a first neural encoder tuned for extracting semantic features from the video data sample and a second neural encoder for extracting temporal alignment features from the video data sample.

34. The method of claim 33, wherein the first neural encoder is an image feature encoder and / or the second neural encoder is a 3D convolutional video encoder.

35. The method of any one of claims 33 to 34, wherein the first neural encoder is a pre- trained CLIP encoder and / or the second neural encoder is a pre-trained S3D encoder.

36. The method of any one of claims 33 to 35, wherein the conditioning sequence is length adjusted by a length adapter neural block to be of equal length to the masked token sequence.

37. The method of any one of claims 25 to 36, wherein the loss is further based on the masked token sequence and the conditioning sequence.

38. The method of any one of claims 25 to 37, wherein training the transformer based on the loss function comprises minimizing a negative log-likelihood of masked positions in the unmasked token sequence.

39. The method of any one of claims 33 to 38 when depending on claim 33, wherein generating the conditioning sequence from the video data sample based on the first neural encoder tuned for extracting semantic features from the video data sample and the second neural encoder for extracting temporal alignment features from the video data sample comprises: projecting each output of the first neural encoder and the second neural encoder to a multi-dimensional space on which the transformer operates by a first and second multi-layer perceptron, respectively, wherein the first and second multi-layer perceptron are trained based on the loss function.

40. The method of any one of claims 33 to 39 when depending on claim 33, wherein generating the conditioning sequence is further based on a visual front-end of a contrastive audio- visual model, wherein the contrastive audio-visual model is pre-trained based on training according to any one of claims 1 to 19.

41. The method of claim 40, wherein the output of the visual front-end is projected to a multi-dimensional space on which the transformer operates by a third multi-layer perceptron, wherein the third multi-layer perceptron is trained based on the loss function.

42. The method of any one of claims 25 to 41, wherein the transformer comprises U AdaLN blocks.

43. The method of any one of claims 33 to 41 when depending on claim 33, wherein the transformer comprises U cross-attention blocks and wherein the outputs of the first and second neural encoder are matched in sequence length by interpolation, combined and projected to a multi-dimensional space on which a transformer encoder operates to generate the conditioning sequence by the transformer encoder, wherein optionally a positional encoding is added before generating the conditioning sequence by the transformer encoder, and wherein the transformer encoder is trained based on the loss function.

44. The method of any one of claims 33 to 41 when depending on claim 33, wherein the transformer comprises U / 2 interleaved cross-attention blocks and AdaLN blocks, wherein the outputs of the first and second neural encoder are matched in sequence length by interpolation, combined and projected to a multi-dimensional space on which a transformer encoder operates to generate the conditioning sequence by the transformer encoder, wherein optionally a positional encoding is added before generating the conditioning sequence by the transformer encoder; and wherein the transformer encoder is trained based on the loss function.

45. The method of any one of claims 43 to 44, wherein the method further comprises: evaluating a second loss function based on a mean squared error between the conditioning sequence and an encoded audio data sample, wherein the encoded audio data sample is generated from the audio data sample based on a pre-trained neural audio encoder, wherein the pre-trained neural audio encoder is optionally a BEATS encoder; evaluating a third loss function based on a contrastive audio-visual model, wherein the contrastive audio-visual model is pre-trained based on training according to any one of claims 1 to 19 or the contrastive audio-visual model is pre-trained based on a cosine similarity;determining a total loss by calculating a weighted sum of the loss function, the second loss function and the third loss function; and training the transformer based on the total loss.

46. The method of any one of claims 25 to 45, wherein the transformer is trained to correctly predict the masked tokens.

47. A neural network-based method for video-to-audio generation, the method comprising: receiving a video data sample; generating a conditioning sequence based on the video data sample; providing a fully masked token sequence to a transformer conditioned on the conditioning sequence and trained according to any one of claims 25 to 46; determining a total number of steps for iteratively unmasking the fully masked token sequence; initializing the transformer with the fully masked token sequence; for each step, predicting a plurality of tokens by the transformer and providing the predicted tokens to the transformer as input for a subsequent step, generating an audio data sample from a fully unmasked token sequence output by the transformer in a last step corresponding to the total number of steps; outputting the audio data sample as a paired data sample to the video data sample.

48. The method of claim 47, wherein generating the conditioning sequence based on the video data sample comprises generating the conditioning sequence from the video data sample based on a first neural encoder tuned for extracting semantic features from the video data sample and a second neural encoder for extracting temporal alignment features from the video data sample.

49. The method of claim 48, wherein the first neural encoder is an image feature encoder and / or the second neural encoder is a 3D convolutional video encoder.

50. The method of any one of claims 48 to 49, wherein the first neural encoder is a pre- trained CLIP encoder and / or the second neural encoder is a pre-trained S3D encoder.

51. The method of any one of claims 48 to 50, wherein generating the conditioning sequence based on the video data sample further comprises generating the conditioning sequence based on a text input or an image input for providing a context for generating the audio data sample.

52. The method of claim 51 when depending on claim 50, wherein the pre-trained CLIP encoder extracts semantic features from the text input or the image input for generating the conditioning sequence.

53. The method of any one of claims 48 to 52, wherein the conditioning sequence is length adjusted by a length adapter neural block to be of equal length to the fully masked token sequence.

54. The method of any one of claims 48 to 53, wherein generating the conditioning sequence from the video data sample based on the first neural encoder tuned for extracting semantic features from the video data sample and the second neural encoder for extracting temporal alignment features from the video data sample comprises: projecting each output of the first neural encoder and the second neural encoder to a multi-dimensional space on which the transformer operates by a first and second multi-layer perceptron, respectively, wherein the first and second multi-layer perceptron are trained according to claim 39.

55. The method of any one of claims 48 to 54, wherein generating the conditioning sequence is further based on a visual front-end of a contrastive audio-visual model, wherein the contrastive audio-visual model is pre-trained based on training according to any one of claims 1 to 19.

56. The method of any one of claims 47 to 55, wherein determining the total number of steps for iteratively unmasking the fully masked token sequence depends on an available computational budget or a time constraint.

57. The method of any one of claims 47 to 56, wherein the fully masked token sequence represents an empty codeword representation for the audio data sample to be generated, wherein the codeword representation comprises empty positions for R sequences of codewords, each with a length of S codewords.

58. The method of claim 57, wherein predicting a plurality of tokens by the transformer and providing the predicted tokens to the transformer as input for a subsequent step comprises: estimating a probability of each codeword for all empty codeword positions in the codeword representation by the transformer; sampling the probability of each codeword for each empty codeword position to determine a candidate codeword for each empty codeword position; determining a confidence score for each candidate codeword; re-masking W candidate codewords with the lowest confidence score for generating a partially empty codeword representation; providing the partially empty codeword representation to the transformer for the subsequent step.

59. The method of claim 58, wherein sampling the probability of each codeword for each empty codeword position is based on a weighted guidance, wherein the weighted guidance is based on an output of the transformer with the conditioning sequence and an output of the transformer without the conditioning sequence.

60. The method of any one of claims 58 to 59, wherein the sampling the probability of each codeword is based on a temperature factor.

61. The method of any one of claims 58 to 60, wherein the probability of each codeword is represented by a multinomial distribution.

62. The method of any one of claims 58 to 61, wherein the confidence score is based on a probability of the candidate codeword given all candidate codewords and the conditioning sequence.

63. The method of claim 62, wherein the confidence score is further based on additive noise.

64. The method of claim 63, wherein the additive noise is an i.i.d. noise sample from a (0,1) Gumbel distribution multiplied by a diversity factor decreasing for increasing number of steps being carried out.

65. The method of any one of claims 58 to 64, wherein R decreases with increasing number of steps being carried out.

66. The method of any one of claims 58 to 65, wherein in the last step all remaining codewords on the empty codeword positions are determined to generate the fully unmasked token sequence.

67. The method of any one of claims 49 to 66, wherein generating the audio data sample from the fully unmasked token sequence output by the transformer in the last step corresponding the total number of steps comprises: decoding the fully unmasked token sequence to generate the audio data sample.

68. The method of claim 67, wherein decoding is performed by a descript audio codec decoder.

69. The method of any one of claims 47 to 68, wherein the transformer comprises U AdaLN blocks.

70. The method of any one of claims 48 to 68 when depending on claim 48, wherein the transformer comprises U cross-attention blocks and wherein the outputs of the first and second neural encoder are matched in sequence length by interpolation, combined and projected to amulti-dimensional space on which a trained transformer encoder operates to generate the conditioning sequence by the trained transformer encoder, wherein optionally a positional encoding is added before generating the conditioning sequence by the trained transformer encoder.

71. The method of any one of claims 48 to 68 when depending on claim 48, wherein the transformer comprises U / 2 interleaved cross-attention blocks and AdaLN blocks, wherein the outputs of the first and second neural encoder are matched in sequence length by interpolation, combined and projected to a multi-dimensional space on which a trained transformer encoder operates to generate the conditioning sequence by the trained transformer encoder, wherein optionally a positional encoding is added before generating the conditioning sequence by the trained transformer encoder.

72. The method of any one of claims 47 to 70, wherein before outputting the audio data sample as the paired data sample to the video data sample, the method further comprises: repeating the following processing X times to generate X candidate audio data samples: providing the fully masked token sequence to the transformer conditioned on the conditioning sequence and trained according to any one of claims 25 to 46; for each step, predicting the plurality of tokens by the transformer and providing the predicted tokens to the transformer as input for the subsequent step, generating the ^-th candidate audio data sample from the fully unmasked token sequence output by the transformer in the last step corresponding the total number of steps; determining a best matching pair of the X candidate audio data samples and the video data sample based on the network-based method of audio-visual retrieval according to any one of claims 20 to 23; and providing the candidate audio data sample of the best matching pair as the audio data sample.

73. The method of any one of claims 47 to 72, wherein the method further comprises providing the paired audio data sample and video data sample for playback.

74. An apparatus, comprising a processor and a memory coupled to the processor, wherein the processor is adapted to carry out the method according to any one of claims 1 to 73.

75. A program comprising instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 1 to 73.

76. A computer-readable storage medium storing the program according to claim 75.

Images