Generate synchronized sound from video

Abstract

Embodiments herein describe an audio forwarding regularizer and an information bottleneck for use when training a machine learning (ML) system. The audio forwarding regularizer receives audio training data and identifies visually irrelevant sounds and visually relevant sounds in the training data. By controlling the information bottleneck, the audio forwarding regularizer forwards data primarily related to visually irrelevant sounds to a generator while filtering out visually relevant sounds. The generator also receives data about visual objects from a visual encoder, which is derived from the visual training data. Thus, when being trained, the generator receives data about visual objects and data about visually irrelevant sounds (but little or no data about visually relevant sounds). Thus, during the execution phase, the generator can generate sounds related to visual objects without adding visually irrelevant sounds to the video.

Description

Technical Field

[0001] The present invention relates to identifying visually relevant and visually irrelevant sounds from training data, and more particularly, to using visually irrelevant sounds to train a sound generation system. Background Art

[0002] Various visual events in daily life are usually accompanied by different sounds. In many cases, visual events and sounds are so closely related that one can instinctively infer what the sound will be by observing the visual event. Although the correlation between visual events and sounds is instinctive for humans, it is a difficult task for machine learning applications. In other words, it is difficult to use machine learning applications to derive sounds from silent videos (i.e., visual data without synchronized audio), but there are many real-world applications, such as video editing automation, generating sounds for silent movies, and helping people with visual impairments.

[0003] One reason it's difficult to derive sounds from visual data is because the training data used to train machine learning (ML) systems often has audio data that's unrelated to the visual data (referred to as visually irrelevant sounds). For example, the visual training data may show a dog barking, while the audio training data includes the dog barking and applause from a crowd that isn't shown in the visual training sounds (e.g., visually irrelevant sounds). That is, the audio training data includes both visually relevant data (e.g., the sound of a dog barking) and visually irrelevant sounds. Using audio training data that includes visually irrelevant sounds to train an ML system may confuse the ML system into associating visually irrelevant sounds with objects in the visual training data, even if they are unrelated. However, finding training data that doesn't have any visually irrelevant sounds is difficult. Therefore, it's necessary to train an ML system using training data that may include both visually relevant and visually irrelevant sounds.

[0004] Therefore, there is a need in the art to solve the above problems. Summary of the Invention

[0005] From a first aspect, the present invention provides a method for identifying visually relevant sounds, the method comprising: receiving visual training data at a visual encoder comprising a first machine learning (ML) model; using the first ML model to identify data corresponding to a visual object in the visual training data; receiving audio training data synchronized with the visual training data at an audio forward regularizer comprising a second ML model, wherein the audio training data comprises visually relevant sounds and visually irrelevant sounds, both synchronized to the same frame in the visual training data containing the visual object, wherein the visually relevant sounds correspond to the visual object but the visually irrelevant sounds are generated by an audio source that is not visible in the same frame; filtering data corresponding to the visually relevant sounds from an output of the second ML model using an information bottleneck; and training the first ML model and a third ML model downstream of the second ML model using the data corresponding to the visual object and the data corresponding to the visually irrelevant sounds.

[0006] From another aspect, the present invention provides a computer program product for identifying visually related sounds, the computer program product comprising: a computer-readable storage medium having computer-readable program code implemented therewith, the computer-readable program code being executable by one or more computer processors to perform operations, the operations comprising: receiving visual training data at a visual encoder comprising a first ML model; using the first ML model to identify data corresponding to a visual object in the visual training data; receiving audio training data synchronized with the visual training data at an audio forwarding regularizer comprising a second ML model, wherein the audio training data comprises visually related sounds and visually irrelevant sounds, both synchronized to the same frame in the visual training data containing the visual object, wherein the visually related sounds correspond to the visual object, but the visually irrelevant sounds are generated by an audio source that is not visible in the same frame; filtering data corresponding to the visually related sounds from the output of the second ML model using an information bottleneck; and training a third ML model downstream of the first ML model and the second ML model using the data corresponding to the visual object and the data corresponding to the visually irrelevant sounds.

[0007] From another aspect, the present invention provides a system comprising: a processor; and a memory, the memory comprising a program that, when executed by the processor, performs operations comprising: receiving visual training data at a visual encoder comprising a first ML model; using the first ML model to identify data corresponding to a visual object in the visual training data; receiving audio training data synchronized with the visual training data at an audio forwarding regularizer comprising a second ML model, wherein the audio training data comprises visually relevant sounds and visually irrelevant sounds, both synchronized to the same frame in the visual training data containing the visual object, wherein the visually relevant sounds correspond to the visual object, but the visually irrelevant sounds are generated by an audio source that is not visible in the same frame; filtering data corresponding to the visually relevant sounds from the output of the second ML model using an information bottleneck; and training a third ML model downstream of the first ML model and the second ML model using the data corresponding to the visual object and the data corresponding to the visually irrelevant sounds.

[0008] Viewed from another aspect, the present invention provides a computer program product for identifying visually associated sounds, the computer program product comprising a computer-readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit to perform a method for performing the steps of the present invention.

[0009] Viewed from another aspect, the invention provides a computer program stored on a computer readable medium and loadable into the internal memory of a digital computer, the computer program comprising software code portions for performing the steps of the invention when said program is run on a computer.

[0010] An embodiment of the present invention is a method comprising: receiving visual training data at a visual encoder comprising a first ML model, using the first ML model to identify data corresponding to a visual object in the visual training data, and receiving audio training data synchronized with the visual training data at an audio forward regularizer comprising a second ML model, wherein the audio training data comprises visually relevant sounds and visually irrelevant sounds, both synchronized to the same frame in the visual training data containing the visual object, and wherein the visually relevant sounds correspond to the visual object, but the visually irrelevant sounds are produced by an audio source that is not visible in the same frame. The method also comprises filtering data corresponding to the visually relevant sounds from an output of the second ML model using an information bottleneck, and training a third ML model downstream of the first and second ML models using the data corresponding to the visual object and the data corresponding to the visually irrelevant sounds. One advantage of this embodiment over previous solutions is that the third ML model can take visually irrelevant sounds into account during training. Thus, the third ML model can be accurately trained using training data that includes both visually relevant and irrelevant sounds.

[0011] Another embodiment of the present invention includes the above embodiment and may further include, after training a third ML model, performing an execution phase, the execution phase comprising receiving a silent video at a first ML model, identifying data corresponding to a second visual object in the silent video using the first ML model, generating visually relevant sound to synchronize with at least one video frame of the silent video containing the second visual object using the third ML model, wherein the data corresponding to the second visual object is input to the third ML model, and generating a media presentation based on the synchronized visually relevant sound and video frames in the silent video. One advantage of this embodiment over previous solutions is that the likelihood of the third ML model selecting visually irrelevant sound to synchronize with video frames of the silent video is greatly reduced.

[0012] Another embodiment of the present invention includes the above embodiment, and when executing the execution state, the second ML model in the audio forwarding regularizer may not be used. Advantageously, not using the second ML model during the execution phase can improve the performance of the ML system because these components can be used during training rather than during execution.

[0013] In any of the above embodiments, the information bottleneck may be achieved by limiting the dimensionality of the output of the second ML model, wherein limiting the dimensionality prevents data corresponding to the visually associated sound from reaching the third ML model.

[0014] In any of the above embodiments, the second ML model can be implemented using a sequence-to-sequence ML model. Advantageously, the sequence-to-sequence ML model provides improved results compared to other ML models when processing time-dependent information such as audio and visual training data.

[0015] In the above embodiment, the sequence-to-sequence ML model can optionally output a bottleneck audio frame based on the audio training data and the information bottleneck. Furthermore, the method can optionally include replicating the bottleneck audio frame according to T time segments in the visual training data, and sending the replicated bottleneck audio frame to the third ML model. Advantageously, having as many replicas of the bottleneck audio frame as visual frames can improve the third ML model's ability to distinguish between visual objects identified by the first ML model and visually unrelated sounds identified by the second ML model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention will now be described, by way of example only, with reference to preferred embodiments as illustrated in the following drawings:

[0017] Figure 1 is a system for training a machine learning model to synchronize sound with visual data according to one embodiment described herein.

[0018] Figure 2 is a flowchart for training a machine learning model to synchronize sound with visual data according to one embodiment described herein.

[0019] Figure 3 is a system for synchronizing sound with visual data using a trained machine learning model according to one embodiment described herein.

[0020] Figure 4 is a flow chart for synchronizing sound with visual data according to one embodiment described herein.

[0021] Figure 5 is a system for training a machine learning model to synchronize sound with visual data according to one embodiment described herein.

[0022] Figure 6 is a system for training a machine learning model to synchronize sound with visual data according to one embodiment described herein. DETAILED DESCRIPTION

[0023] Embodiments herein describe a machine learning (ML) system for deriving sounds that can be synchronized with visual data. One advantage of this embodiment is that the ML system can take visually irrelevant sounds into account when training the ML system. This means that the ML system can be trained with audio data that includes irrelevant sounds, which greatly increases the number and diversity of media presentations that can be used to train the ML system.

[0024] In one embodiment, an ML system includes an audio forwarding regularizer and an information bottleneck for use when training the ML system. The audio forwarding regularizer (or just "regularizer") includes an ML model that receives audio training data and identifies visually irrelevant sounds and visually relevant sounds in the training data. By controlling the information bottleneck, the ML system causes the audio forwarding regularizer to forward data to the generator that is primarily related to visually irrelevant sounds, while visually relevant sounds are filtered out. In parallel, the generator also receives data about visual objects from the visual encoder, which is derived from the visual training data. As such, when trained, the generator receives data about visual objects in the training media presentation from the visual encoder and receives data about visually irrelevant sounds (but little or no data about visually relevant sounds) from the audio forwarding regularizer. Thus, the generator learns to distinguish between visual objects (i.e., objects in the visual training data) and visually irrelevant sounds (i.e., sounds that are not related to the visual objects). Thus, when executing the ML system to infer sounds for a silent video, the generator may generate sounds that are relevant to the visual objects (e.g., visually relevant sounds) without adding or synchronizing visually irrelevant sounds to the video.

[0025] Figure 1 1 is an ML system 100 for training an ML model to synchronize sound with visual data, according to one embodiment described herein. ML system 100 includes a visual encoder 110, an audio forwarding regularizer 115, and a generator 125, each of which includes one or more ML models. Details of these ML models are described in subsequent figures.

[0026] As shown, visual encoder 110 receives visual training data 105 as input. In one embodiment, visual training data 105 includes a plurality of sequential video frames. In contrast, audio forward regularizer 115 receives audio training data 107, which includes sounds synchronized with visual training data 105. For example, visual training data 105 can be visual information presented from training media, while audio training data 107 includes sounds presented by the training media. However, as discussed above, audio training data 107 can include both visually relevant sounds (i.e., sounds related to visual objects in visual training data 105) and visually irrelevant sounds (i.e., sounds that are not related or unrelated to the synchronized visual objects in visual training data 105). In one embodiment, the audio source of the visually relevant sounds is off-screen, i.e., not visible in the corresponding video frame. Using the techniques herein, the ML system 100 may be trained such that the generator 125 can distinguish between visually relevant sounds and visually irrelevant sounds, such that during execution, when only silent video (e.g., visual data not synchronized with any sound) is received, the generator 125 may generate or derive visually relevant sounds, which may then be synchronized with the silent video to generate a media presentation that includes audio synchronized with the visual data.

[0027] Typically, the goal of the visual encoder 110 is to identify visual objects in the visual training data 105, such as dogs, people, cars, etc. The visual encoder 110 can also use its ML model (or multiple models) to identify the actions that the visual objects are performing, such as a barking dog, a person shooting a basketball, or a car braking suddenly. These visual objects and their actions can then be transmitted to the generator 125.

[0028] The audio forwarding regularizer 115 uses its ML model (or models) to identify visually relevant and visually irrelevant sounds in the audio training data 107. For example, visually relevant sounds can include sounds generated by visual objects in the visual training data 105, such as a dog barking, a basketball bouncing, or the squeal of tires caused by a car braking suddenly. The audio training data 107 also includes visually irrelevant sounds that are unrelated or independent of the visual objects in the data 105 or their actions. Examples of visually irrelevant sounds can be the applause of the audience (when the audience is not shown in the visual training data 105), the commentator's narration about the basketball game, or the sound made by an animal that is currently off-screen (e.g., not within a synchronized frame in the visual training data 105). For example, at one point in time, when the source of a sound in the audio training data 107 is currently visible in a synchronized frame of the visual training data 105, the sound can be a visually relevant sound, but if its source is no longer visible in a synchronized frame of the visual training data 105, the sound can later be a visually irrelevant sound.

[0029] The goal of the generator 125 is to reproduce the audio training data 107 (which includes both visually relevant and visually irrelevant sounds) using the input provided by the visual encoder 110 and the audio forward regularizer 115. However, simply sending data about visually relevant and irrelevant sounds to the generator 125 during training can cause the generator 125 to associate both visually relevant and irrelevant sounds with visual objects identified by the visual encoder 110, which causes the generator 125 to add both visually relevant and irrelevant sounds to silent videos during the execution phase (i.e., after training is complete). To prevent this, the ML system 100 includes an information bottleneck 120 that limits the amount of data about visually relevant sounds sent to the generator 125 and instead focuses on sending visually irrelevant sounds to the generator 125.

[0030] While it may seem counterintuitive to send data about visually irrelevant sounds to the generator 125 during training (since visually irrelevant sounds should be ignored during the execution phase), doing so advantageously leads to improved results, wherein the generator 125 is able to distinguish between visual objects and visually irrelevant sounds during training rather than falsely correlating them. For example, if the information bottleneck 120 is too wide, then a large amount of data about visually relevant sounds as well as visually irrelevant sounds is provided to the generator 125. In this case, the generator 125 can "cheat" and use the input received from the audio forward regularizer as the predicted sound 130. As a result, the generator 125 does not learn that the visually irrelevant sounds are irrelevant to the visual data provided by the visual encoder 110.

[0031] However, as the information bottleneck 120 tightens, the amount of data corresponding to visually relevant sounds transmitted to the generator 125 decreases. As a result, the generator 125 can no longer simply use the sounds it receives from the regularizer 115 as the predicted sounds 130 (which should include both visually relevant and visually irrelevant sounds). Consequently, in order to generate visually relevant sounds for the predicted sounds 130 (e.g., when attempting to reconstruct the audio training data 107), the one or more ML models in the generator 125 are forced to determine that the data provided by the audio-forward regularizer 115 does not include visually relevant sounds, but rather derives those sounds from the visual objects provided by the visual encoder 110. For example, the generator 125 may access a database to identify sounds related to the visual objects identified by the visual encoder 110. As a result of this process, the generator 125 learns that the data regarding visually irrelevant sounds received from the audio-forward regularizer 115 is, in fact, unrelated to the visual objects generated by the visual encoder 110, thereby achieving the desired effect of training the generator 125 to recognize visually irrelevant sounds. For example, the generator 125 may determine that applause from unseen audience members is unrelated to a visible barking dog, or that commentary from an off-screen announcer is unrelated to an on-screen basketball player bouncing the ball.

[0032] During the training phase, the generator 125 has the goal of outputting a predicted sound 130 that contains both visually relevant and irrelevant sounds. That is, the predicted sound 130 should be as close as possible to the sound in the audio training data 107 (which serves as ground truth). During this process, and as described above, one or more ML models in the generator 125 learn to recognize visually irrelevant sounds.

[0033] In one embodiment, the visual encoder 110, the audio forwarding regularizer 115, the information bottleneck 120, and the generator 125 are stored in a memory as program code that is executed by a processor in at least one computing system. For example, the visual encoder 110, the audio forwarding regularizer 115, the information bottleneck 120, and the generator 125 can be implemented using an ML framework (e.g., an ML software application) stored in the memory and executed by a processor in the computing system.

[0034] Figure 2 2 is a flow chart of a method 200 for training an ML model to synchronize sound with visual data according to one embodiment described herein. In this embodiment, the ML system receives training media data 205 including visual training data 105 (e.g., video frames) and audio training data 107 (e.g., synchronized sounds or soundtracks of the training media). That is, the training media data 205 can be separated such that its video data (e.g., visual training data 105) is provided to a visual encoder, while the corresponding audio data (e.g., audio training data 107) is provided to an audio forward regularizer.

[0035] At block 210 , the visual encoder identifies objects for training the first ML model. In one embodiment, the visual encoder identifies visual objects in the visual training data 105 . In addition to identifying the visual object 225 , the visual encoder may also identify the action that the visual object 225 is currently performing or some other metadata associated with the object, such as the type of object (e.g., the age of a person or the breed of a dog). The visual encoder may include any type of ML model suitable for identifying the visual object 225 .

[0036] At block 215 , the audio forwarding regularizer uses a second ML model to identify visually relevant sounds and visually irrelevant sounds. That is, the second ML model in the regularizer identifies various sounds in the audio training data 107 . As part of this process, at block 220 , an information bottleneck filters out visually irrelevant sounds from the visually relevant sounds represented in the output of the second ML model. Rather than allowing the second ML model to output all of its generated output data, the information bottleneck limits the output of the second ML model so that data regarding visually irrelevant sounds 230 is primarily transmitted to the generator, while most (or all) data regarding visually relevant sounds is filtered out (i.e., not forwarded to the generator). The audio forwarding regularizer may include any type of ML model suitable for identifying visually irrelevant sounds 230 .

[0037] At block 235, the generator recovers the training media to train the third ML model. As part of training the third ML model, the generator attempts to generate (or recover or reproduce) the audio training data 107 received by the regularizer. As described above, the generator receives visually irrelevant sounds from the audio forward regularizer, but not visually relevant sounds. Thus, the third ML model attempts to recover visually relevant sounds from the visual objects 225 in order to reproduce the audio training data 107 and the training media data 205. As part of this process, the third ML model learns that the visual objects 225 are different from the visually irrelevant sounds 230. Thus, during a later execution phase, the third ML model can avoid generating visually irrelevant sounds when generating sounds for silent videos.

[0038] Figure 3 is an ML system 300 for synchronizing sound with visual data using a trained ML model according to one embodiment described herein. Figure 1 The ML system is illustrated for training an ML model, but the ML system 300 is used during the execution phase to generate sound for a silent video after the ML model has been trained (eg, after the method 200 has been executed).

[0039] The ML system 300 includes the visual encoder 110 and generator 125 as described above. However, the system 300 lacks the audio forward regularizer and the information bottleneck. Figure 1 The input of the generator 125 used during the training phase of the ML system 300 receives the output of the regularizer and the information bottleneck instead, receiving a zero vector 310 (e.g., a vector of zeros). Thus, the regularizer and the information bottleneck are not used during the execution phase of the ML system 300, which advantageously improves the performance of the ML system because these components can only be used during training and not during execution. Instead, the generator 125 relies on its trained ML model to recognize the sounds of the visual objects recognized by the visual encoder 110.

[0040] During the execution phase, the visual encoder 110 receives silent video 305—e.g., a series of video frames without corresponding or synchronized sound for the frames. In one embodiment, the goal of the ML system 300 during the execution phase is to generate sounds that correspond to the silent video 305. For example, if the silent video 305 depicts a barking dog or fireworks, the ML system 300 can use the visual encoder 110 and the trained ML model in the generator 125 to generate visually relevant sounds—e.g., a dog barking or the roar and pop of fireworks.

[0041] As in the training phase, the visual encoder 110 identifies visual objects in the silent video 305. The visual encoder 110 may also identify the action of the visual object or the type or characteristics of the visual object. This information is then forwarded to the generator 125.

[0042] Generator 125 uses its trained ML model to identify synchronized sounds for silent video 305. For example, if the visual object depicts a barking dog, generator 125 may determine to synchronize the barking sounds with the video frames showing the barking dog. Because the ML model in generator 125 is trained to distinguish between visual objects and visually irrelevant sounds in the training media data, during execution, generator 125 is less likely to add and synchronize visually irrelevant sounds to the visual objects identified in silent video 305, resulting in an advantage over previous solutions that do not use an audio forwarding regularizer during the training phase. In other words, generator 125 is more likely to synchronize only visually relevant sounds with the visual objects and their actions depicted in silent video 305. Generator 125 outputs predicted sounds 315 that include visually relevant sounds synchronized with the frames of the silent video. ML system 300 can generate a new media presentation that includes sounds synchronized with the video frames of silent video 305.

[0043] Figure 44 is a flow chart of a method 400 for synchronizing sound with visual data, according to one embodiment described herein. An ML system receives a silent video 305, which may include a series of video frames where at least a portion of the video 305 (or all of the video 305) lacks corresponding audio data or sound. Method 400 may be used to generate sound that is synchronized with a portion of the video 305 that previously lacked audio data.

[0044] At block 405, the visual encoder uses the trained first ML model to identify objects in frames of video data. The visual encoder may perform the same Figure 2 The same technique is performed at box 210 to send data corresponding to the visual object 410 to the generator.

[0045] At block 415, the generator uses the trained third ML model to generate visually relevant sounds corresponding to the visual objects in the frames of the video data. That is, the generator uses the visual objects 410 to identify sounds related to these objects. These sounds are synchronized with the frames of the silent video 305.

[0046] At block 420, the ML system outputs a media presentation that includes video frames of the silent video that have now been synchronized with the visually relevant sounds identified by the generator. Thus, when the media presentation is played, the user sees the visual object 410 and the synchronized sounds associated with the visual object 410. Advantageously, the embodiments discussed herein reduce the chance that the generator will select visually irrelevant sounds for inclusion in the media presentation.

[0047] Figure 5 is an ML system 500 for training an ML model to synchronize sound with visual data according to one embodiment described herein. In general, the ML system 500 is Figure 1 An embodiment of ML system 100 is shown. Similar to ML system 100 , ML system 500 includes a visual encoder 110 , an audio forward regularizer 115 , and a generator 125 , each of which includes at least one ML model trained using visual training data 105 and audio training data 107 .

[0048] As shown, the feature extractor extracts frame features from the visual training data 105. These frame features are then used as input to the ML model 505A in the visual encoder 110. In one embodiment, the ML model 505A represents one or more convolutional neural networks (CNNs) that generate one or more vertical vectors. For example, if the visual training data 105 has 24 frames per second, each CNN may output 24 vertical vectors per second. However, in one embodiment, the output of the ML model 505A is a single vector per time (e.g., per second), which is then fed into the ML model 505B.

[0049] In one embodiment, ML model 505B specializes in processing time-dependent information, such as video frames in visual training data. In one embodiment, ML model 505A is a sequence-to-sequence ML model, which can be a long short-term memory (LSTM), a bidirectional LSTM, a recurrent neural network (RNN), a 1D convolutional neural network (convnet), or other sequence learning methods. In one embodiment, ML model 505B outputs one vertical frame per second. Thus, assuming that visual training data 105 has T seconds, ML model 505B outputs T visual frames 515.

[0050] exist Figure 5 , the audio training data 107 is converted into a spectrogram representing the ground truth. To do so, any feature extraction model can be used to transform the audio training data 107. The spectrogram is then used as input to the ML model 505C in the audio forward regularizer 115. In one embodiment, similar to the ML model 505B in the visual encoder 110, the ML model 505C is a sequence-to-sequence ML model, which can be an LSTM, a bidirectional LSTM, an RNN, a 1D convolutional network, or other sequence learning method. Advantageously, the sequence-to-sequence ML model provides improved results compared to other ML models when processing time-dependent information such as audio and visual training data.

[0051] An information bottleneck is implemented by constraining the output of the ML model 505C. For example, reducing the output dimensionality of the ML model 505C can implement an information bottleneck. As shown, the ML model 505C uses the spectrogram as input to generate a bottleneck audio frame 510. The audio forward regularizer 115 then replicates the frame 510 T times, so that advantageously, there are as many copies of the bottleneck audio frame 510 as there are visual frames 515, which can improve the ability of the generator 125 to distinguish between visual objects identified by the visual encoder 110 and visually irrelevant sounds identified by the audio forward regularizer 115.

[0052] Reducing the dimensionality of the audio frame 510 to implement an information bottleneck inherently filters out some (or all) of the data corresponding to visually relevant sounds. In other words, the dimensionality of the audio frame 510 (i.e., the output of the ML model 505C) can be adjusted so that the frame 510 primarily contains data corresponding to visually irrelevant sounds but not visually relevant sounds, thereby achieving the information bottleneck as discussed above.

[0053] The combiner 520 combines the visual frame 505 generated by the visual encoder 110 with the replicated audio frame 510 generated by the audio forward regularizer 115 and forwards the resulting information to the generator 125. The generator 125 includes an ML model 505D downstream of the ML models in the encoder 110 and the regularizer 115. The ML model 505D attempts to replicate the sounds in the audio training data 107 by outputting a predicted spectrogram that should match the ground truth spectrogram. In one embodiment, the ML model 505D may include a mixture of different layers, such as transposed convolution layers, convolution layers, batch normalization (BN) layers, rectified linear unit (ReLU) layers, etc. These layers can be combined into one ML model or into several daisy-chained ML models. Further, although not shown, the generator 125 may include a post-network coupled to the output of the ML model 505D that generates a predicted spectrogram.

[0054] As discussed above, during the training phase, the generator 125 receives data corresponding to visual objects identified by the visual encoder 110 (i.e., visual frames 515) and data corresponding to visually irrelevant sounds (i.e., bottleneck audio frames 510). Based on this information, the training ML model 505D is trained to distinguish between visual objects and visually irrelevant sounds in the audio training data 107. Thus, during the execution phase, the likelihood that the ML model 505D associates visually irrelevant sounds with visual objects in a silent video is advantageously reduced.

[0055] In one embodiment, training the ML system 500 to distinguish between visually relevant and visually irrelevant sounds can be formulated mathematically. In the following discussion, uppercase letters denote random variables (unbold) or random vectors (bold); lowercase letters denote deterministic values. E[] denotes expectation. H() denotes (discrete) Shannon entropy. Further, (V(t); S(T)) denotes a visual-sound pair, where V(t) denotes the visual signal (vectorized) at each video frame t, and S(T) denotes the sound representation (waveform or spectrogram) at each audio frame T. Different frame indices t and T are used because the visual and sound signals have different sampling rates.

[0056] Assume that the audio can be decomposed into correlated and uncorrelated signals:

[0057] S(τ)=S r (τ)+S i (τ) (1)

[0058] The subscript r denotes relevant sounds, while i denotes irrelevant sounds. It is also assumed that there is a relationship only between the video and relevant sounds, denoted as f(). Irrelevant sounds are independent of both relevant sounds and visual features. These relationships can be expressed as:

[0059] S r (τ)=f(V(t)),S i (τ)⊥S r (τ), S i (τ)⊥V(t) (2)

[0060] In one embodiment, the goal is to decouple the two components and generate only the visually relevant component S from the visual signal V(t). r (τ).

[0061] The visual encoder 110 receives the video signal V(t) as input and outputs a set of video features. The audio forward regularizer 115 receives the sound signal S(τ) as input and outputs audio forward information (i.e., the replicated bottleneck audio frame 510). The generator 125 then predicts (or recovers) S(τ). There are two different types of predictions, with or without audio forward. The prediction with audio forward is predicted from both the video features and the audio forward information, denoted as By using e.g. Figure 3 The zero vector 310 shown in sets the input of the audio forwarding regularizer to 0 to predict the prediction of no audio forwarding, which is represented as

[0062] During training, the generator 125 attempts to minimize the following loss involving predictions using audio forwarding:

[0063]

[0064] Where the first term in Equation 3 is the reconstruction error, and the second term is the adversarial loss.

[0065] Figure 6 FIG6 is an ML system 600 for training an ML model to synchronize sound with visual data, according to one embodiment described herein. ML system 600 is identical to ML system 500, except for an additional communication link 605 between visual encoder 110 and audio forwarding regularizer 115. Specifically, link 605 shows the output of ML model 505A being sent to ML model 505C in regularizer 115.

[0066] Omitting link 605, as shown, from ML system 500 can advantageously improve operational efficiency, as ML system 500 can execute faster or use fewer resources than an ML system 600 that utilizes link 605. However, link 605 can improve ML system 600's ability to distinguish between visually relevant and irrelevant sounds relative to ML system 500. For applications where a single type of visual object is typically present, omitting link 605 can result in satisfactory results. For example, if the training data only includes videos of fireworks, omitting link 605 when training the ML system may be sufficient. That is, ML system 500 can accurately distinguish between visually relevant and irrelevant sounds without link 605. However, if the training data includes multiple different visual objects, link 605 can provide substantial performance improvements by using the output of ML model 505A as input to ML model 505C, allowing audio forwarding regularizer 115 to better distinguish between visually relevant and irrelevant sounds.

[0067] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found in the marketplace, or to enable those of ordinary skill in the art to understand the embodiments disclosed herein.

[0068] In the above, reference is made to the embodiments presented in the present disclosure. However, the scope of the present disclosure is not limited to the specifically described embodiments. On the contrary, any combination of the features and elements discussed above, whether or not related to different embodiments, is contemplated to implement and practice the contemplated embodiments. Furthermore, although the embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether a particular advantage is achieved by a given embodiment does not limit the scope of the present disclosure. Therefore, the aspects, features, embodiments and advantages discussed above are merely illustrative and are not considered to be elements or limitations of the appended claims unless expressly stated in a claim. Likewise, reference to "the present invention" should not be construed as a generalization of any inventive subject matter disclosed herein and should not be considered to be elements or limitations of the appended claims unless expressly stated in a claim. Likewise, reference to "the present invention" should not be construed as a generalization of any inventive subject matter disclosed herein and should not be considered to be an element or limitation of the appended claims unless expressly stated in a claim.

[0069] Various aspects of the present invention may be implemented as a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or a combination of hardware and software embodiments, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0070] The present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute various aspects of the present invention.

[0071] Computer-readable storage media can be a tangible device that can retain and store the instructions used by the instruction execution device.Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disc read-only memories (CD-ROM), digital versatile discs (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards, or projection structures in the grooves with instructions recorded thereon, and any suitable combination of the above. Computer-readable storage media as used herein should not be interpreted as temporary signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses passing through fiber optic cables), or electrical signals emitted by wires.

[0072] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or downloaded to an external computer or external storage device. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0073] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or object code written in any combination of one or more programming languages, these programming languages include object-oriented programming languages (such as Smalltalk, C++ etc.) and conventional procedural programming languages (such as " C " programming languages or similar programming languages). The computer-readable program instructions can be performed completely on the user's computer, partly on the user's computer, performed as an independent software package, partly on the user's computer, partly on a remote computer or fully on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer by any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (for example, using an internet service provider through the internet). In certain embodiments, the electronic circuit comprising for example programmable logic circuit, field programmable gate array (FPGA) or programmable logic array (PLA) can be personalized to perform the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to perform the electronic circuit, so as to perform various aspects of the present invention.

[0074] The present invention will be described below with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0075] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in the flowchart and / or block diagram or multiple blocks. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in the flowchart and / or block diagram or multiple blocks.

[0076] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in the flowchart and / or block diagram or multiple boxes.

[0077] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to different embodiments of the present invention. To this end, each box in the flowchart or block diagram may represent a module, segment or portion of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions annotated in the box may not occur in the order annotated in the figure. For example, depending on the functions involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the opposite order. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0078] While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope of the invention is determined by the claims that follow.

Claims

1. A method for identifying visually associated sounds, the method comprising: receiving visual training data at a visual encoder comprising a first ML model; identifying, using the first ML model, data corresponding to visual objects in the visual training data; receiving, at an audio forward regularizer comprising a second ML model, audio training data synchronized with the visual training data, wherein the audio training data comprises visually relevant sounds and visually irrelevant sounds that are both synchronized with a same frame in the visual training data comprising the visual object, wherein the visually relevant sounds correspond to the visual object but the visually irrelevant sounds are generated by an audio source that is not visible in the same frame; filtering data corresponding to the visually relevant sound from an output of the second ML model using an information bottleneck so that data corresponding to the visually irrelevant sound is primarily output from the information bottleneck; as well as A third ML model downstream of the first and second ML models is trained using the data corresponding to the visual objects and the data corresponding to the visually unrelated sounds received from the information bottleneck.

2. The method of claim 1 , further comprising, after training the third ML model: An execution phase is performed, wherein the execution phase includes: receiving, at the first ML model, a silent video; identifying, using the first ML model, data corresponding to a second visual object in the silent video; generating, using the third ML model, visually associated sound to synchronize with at least one video frame of the silent video containing the second visual object, wherein data corresponding to the second visual object is input to the third ML model; as well as A media presentation is generated based on the synchronized visually related sounds and video frames from the silent video.

3. The method according to claim 2, wherein: The second ML model in the audio forwarding regularizer is not used during the execution phase.

4. The method of any one of claims 1-3, wherein the information bottleneck comprises limiting the dimensionality of the output of the second ML model, wherein limiting the dimensionality prevents data corresponding to the visually associated sounds from reaching the third ML model.

5. The method according to any one of claims 1 to 3, wherein The second ML model includes a sequence-to-sequence ML model.

6. The method according to claim 5, wherein: The sequence-to-sequence ML model outputs a bottleneck audio frame based on the audio training data and the information bottleneck, the method further comprising: replicating the bottleneck audio frames according to the number of T time segments in the visual training data; and The copied bottleneck audio frame is transmitted to the third ML model.

7. The method according to claim 6, further comprising: Visually relevant sounds and visually irrelevant sounds in the audio training data are predicted based on receiving the replicated bottleneck audio frame and data corresponding to the visual object using the third ML model.

8. A system for identifying visually associated sounds, comprising: processor; as well as a memory including a program that, when executed by the processor, performs operations comprising: receiving visual training data at a visual encoder comprising a first ML model; identifying, using the first ML model, data corresponding to visual objects in the visual training data; receiving, at an audio forward regularizer comprising a second ML model, audio training data synchronized with the visual training data, wherein the audio training data comprises visually relevant sounds and visually irrelevant sounds that are both synchronized with a same frame in the visual training data comprising the visual object, wherein the visually relevant sounds correspond to the visual object but the visually irrelevant sounds are generated by an audio source that is not visible in the same frame; filtering data corresponding to the visually relevant sounds from an output of the second ML model using an information bottleneck so that data corresponding to the visually irrelevant sounds is primarily output from the information bottleneck; and A third ML model downstream of the first and second ML models is trained using the data corresponding to the visual objects and the data corresponding to the visually unrelated sounds received from the information bottleneck.

9. The system according to claim 8, wherein: The operations further include, after training the third ML model: An execution phase is performed, wherein the execution phase includes: receiving, at the first ML model, a silent video; identifying, using the first ML model, data corresponding to a second visual object in the silent video; generating visually relevant sound using the third ML model to synchronize with at least one video frame of the silent video containing the second visual object, wherein data corresponding to the second visual object is input to the third ML model; and A media presentation is generated based on the synchronized visually related sounds and video frames from the silent video.

10. The system according to claim 9, wherein: The second ML model in the audio forwarding regularizer is not used during the execution phase.

11. The system of any one of claims 8 to 10, wherein the information bottleneck comprises limiting the dimensionality of the output of the second ML model, wherein limiting the dimensionality prevents data corresponding to the visually associated sounds from reaching the third ML model.

12. The system of any one of claims 8 to 10, wherein the second ML model comprises a sequence-to-sequence ML model, and wherein the sequence-to-sequence ML model outputs a bottleneck audio frame based on the audio training data and the information bottleneck, the operations further comprising: replicating the bottleneck audio frames according to the number of T time segments in the visual training data; as well as The copied bottleneck audio frame is transmitted to the third ML model.

13. The system according to claim 12, wherein: The operations further include: Visually relevant sounds and visually irrelevant sounds in the audio training data are predicted based on receiving the replicated bottleneck audio frame and data corresponding to the visual object using the third ML model.

14. A computer program product for identifying visually associated sounds, the computer program product comprising instructions for execution by a processing circuit to perform the method according to any one of claims 1 to 7.

Images