Generate synchronized audio from video

Abstract

Embodiments herein describe an audio transfer regularizer and information bottleneck for use in training a machine learning (ML) system. The audio transfer regularizer receives audio training data and identifies visually irrelevant and relevant sounds in the training data. By controlling the information bottleneck, the audio transfer regularizer transfers data to a generator primarily related to visually irrelevant sounds while filtering out visually relevant sounds. The generator also receives data related to visual objects from a vision encoder derived from the visual training data. In this way, during training, the generator receives data related to visual objects and data related to visually irrelevant sounds (but little data related to visually relevant sounds). In this way, sounds related to visual objects can be generated without adding visually irrelevant sounds to the video during the execution phase.

Description

[Technical field]

[0001] The present invention relates to discriminating between visually related and visually unrelated sounds from training data, and more particularly to training a sound production system using visually unrelated sounds. [Background technology]

[0002] Various visual events in daily life are usually accompanied by different sounds. In many cases, visual events and sounds are so closely correlated that we can intuitively infer what sound is what by observing the visual events. The correlation between visual events and sounds is instinctive for humans, but is a challenging task for machine learning applications. That is, deriving sounds from silent videos (i.e., visual data without synchronized audio) using machine learning applications is difficult, but has numerous real-world applications, such as automating video editing, generating sounds for silent films, and assisting the visually impaired.

[0003] Training data used to train machine learning (ML) systems often has audio data that is unrelated to the visual data (referred to as visually unrelated sounds), making it difficult to derive sounds for the visual data. For example, the visual training data may show a dog barking, while the audio training data includes both the dog barking as well as applause from a crowd that is not shown in the visual training sounds (e.g., visually unrelated sounds). That is, the audio training data includes both visually relevant data (e.g., the sound of a dog barking) and visually unrelated sounds. Training an ML system using audio training data that includes visually unrelated sounds can confuse the ML system and cause the visually unrelated sounds to be associated with objects in the visual training data even when they are unrelated. However, it is difficult to find training data that does not have any visually unrelated sounds. Therefore, it is necessary to train an ML system using training data that may include both visually relevant and unrelated sounds.

[0004] Therefore, there is a need in the art to address the above problems. Summary of the Invention

[0005] Viewed from a first aspect, the present invention provides a method for identifying visually relevant sounds, the method comprising: receiving visual training data at a vision encoder including a first machine learning (ML) model; identifying data in the visual training data corresponding to a visual object using the first ML model; receiving audio training data synchronized to the visual training data at an audio transfer regularizer including a second ML model, the audio training data including visually relevant sounds and visually unrelated sounds both synchronized to a same frame in the visual training data including the visual object, the visually relevant sounds corresponding to the visual object while the visually unrelated sounds are generated by an audio source not visible in the same frame; and extracting visually relevant sounds from an output of the second ML model using an information bottleneck. Irrelevant The method includes filtering the data corresponding to the sounds, and training a third ML model downstream from the first and second ML models using the data corresponding to the visual object and the data corresponding to the visually unrelated sounds.

[0006] Viewed from yet another aspect, the present invention provides a computer program product for identifying visually relevant sounds, the computer program product including a computer readable storage medium having computer readable program code embodied therein, the computer readable program code being executable by one or more computer processors to perform operations including receiving visual training data at a visual encoder including a first ML model; identifying data corresponding to a visual object in the visual training data using the first ML model; receiving audio training data synchronized to the visual training data at an audio transfer regularizer including a second ML model, the audio training data including visually relevant sounds and visually unrelated sounds both synchronized to a same frame in the visual training data including the visual object, the visually relevant sounds corresponding to the visual object while the visually unrelated sounds are generated by an audio source not visible in the same frame; and extracting visually relevant sounds from an output of the second ML model using an information bottleneck. Irrelevant The method includes filtering the data corresponding to the sounds, and training a third ML model downstream from the first and second ML models using the data corresponding to the visual object and the data corresponding to the visually unrelated sounds.

[0007] Viewed from yet another aspect, the invention comprises a system including a processor and a memory including a program that, when executed by the processor, performs operations including receiving visual training data at a visual encoder including a first ML model, identifying data in the visual training data that corresponds to a visual object using the first ML model, receiving audio training data synchronized to the visual training data at an audio transfer regularizer including a second ML model, the audio training data including visually relevant sounds and visually unrelated sounds both synchronized to a same frame in the visual training data that includes the visual object, the visually relevant sounds corresponding to the visual object while the visually unrelated sounds are generated by an audio source not visible in the same frame, and extracting visually relevant sounds from an output of the second ML model using an information bottleneck. Irrelevant The method includes filtering the data corresponding to the sounds, and training a third ML model downstream from the first and second ML models using the data corresponding to the visual object and the data corresponding to the visually unrelated sounds.

[0008] Viewed from yet another aspect, the present invention provides a computer program product for identifying visually related sounds, the computer program product including a computer readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising steps of the present invention.

[0009] Viewed from yet another aspect, the present invention provides a computer program stored on a computer readable medium and loadable into the internal memory of a digital computer comprising software code portions for performing the steps of the present invention when run on the computer.

[0010] One embodiment of the present invention is a method that includes receiving visual training data at a vision encoder that includes a first ML model, and using the first ML model to identify data in the visual training data that corresponds to a visual object, and receiving audio training data synchronized to the visual training data at an audio transfer regularizer that includes a second ML model, the audio training data including visually relevant sounds and visually unrelated sounds that are both synchronized to the same frame in the visual training data that includes the visual object, the visually relevant sounds corresponding to the visual object, while the visually unrelated sounds are generated by an unseen sound source in the same frame. The method also includes using an information bottleneck to extract visually relevant sounds from the output of the second ML model. Irrelevant and filtering the data corresponding to the sounds, and training a third ML model downstream from the first and second ML models using the data corresponding to the visual object and the data corresponding to the visually unrelated sounds. One advantage of this embodiment over previous solutions is that the third ML model can take into account the visually unrelated sounds when it is trained. Thus, the third ML model can be accurately trained using training data that includes both visually relevant and unrelated sounds.

[0011] Another embodiment of the present invention may include the above embodiment and further include performing an execution step after training a third ML model, the execution step including receiving a silent video at a first ML model, identifying data corresponding to a second visual object in the silent video using the first ML model, generating visually relevant sounds using a third ML model to synchronize to at least one video frame of the silent video containing the second visual object, where the data corresponding to the second visual object is input to the third ML model, and generating a media presentation based on the synchronized visually relevant sounds and video frames in the silent video. One advantage of this embodiment over previous solutions is that the possibility of the third ML model selecting visually irrelevant sounds to synchronize to video frames of the silent video is significantly reduced.

[0012] Another embodiment of the present invention includes the above embodiment, and when performing the run state, the second ML model in the voice transfer regularizer may not be used. Advantageously, not using the second ML model during the run phase can increase the performance of the ML system because these components are used during training but not during run time.

[0013] In any of the above embodiments, the information bottleneck may be implemented by limiting the magnitude of the output of the second ML model, where limiting this magnitude prevents data corresponding to visually relevant sounds from reaching the third ML model.

[0014] In any of the above embodiments, the second ML model may be implemented using a sequence-to-sequence ML model, which advantageously provides improved results compared to other ML models when processing time-dependent information, such as audio and visual training data.

[0015] In the above embodiment, the sequence-to-sequence ML model can optionally output a bottleneck speech frame based on the speech training data and the information bottleneck.Furthermore, the method can optionally include duplicating the bottleneck speech frame according to the number of T periods in the visual training data, and sending the duplicated bottleneck speech frame to a third ML model.Advantageously, there are numerous copies of the bottleneck speech frame as visual frames that can improve the ability of the third ML model to distinguish between visual objects identified by the first ML model and visually unrelated sounds identified by the second ML model.

[0016] The invention will now be described, by way of example only, with reference to preferred embodiments thereof, as illustrated in the following figures. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 illustrates a system for training a machine learning model to synchronize sound with visual data, according to one embodiment described herein. [Diagram 2] 1 is a flow chart for training a machine learning model to synchronize sound with visual data, according to one embodiment described herein. [Diagram 3] FIG. 1 illustrates a system for synchronizing sound with visual data using a trained machine learning model, according to one embodiment described herein. [Figure 4] 1 is a flow chart for synchronizing sound with visual data according to one embodiment described herein. [Diagram 5] FIG. 1 illustrates a system for training a machine learning model to synchronize sound with visual data, according to one embodiment described herein. [Figure 6] FIG. 1 illustrates a system for training a machine learning model to synchronize sound with visual data, according to one embodiment described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The embodiments herein describe a machine learning (ML) system for deriving sounds that can be synchronized with visual data. One advantage of the embodiments is that the ML system can take visually irrelevant sounds into account when training the ML system. This means that the ML system can be trained with audio data that includes irrelevant sounds, which greatly increases the number and variety of media presentations that can be used to train the ML system.

[0019] In one embodiment, the ML system includes an audio transfer regularizer and an information bottleneck that are used in training the ML system. The audio transfer regularizer (or simply "regularizer") includes an ML model that receives audio training data and identifies visually irrelevant and relevant sounds in the training data. By controlling the information bottleneck, the ML system causes the audio transfer regularizer to transfer data to the generator that is primarily related to visually irrelevant sounds, while visually relevant sounds are filtered out. In parallel, the generator also receives data on visual objects from the visual encoder that is derived from the visual training data. Thus, when being trained, the generator receives data on visual objects in the training media presentation from the visual encoder and data on visually irrelevant sounds from the audio transfer regularizer (but little data on visually relevant sounds). As a result, the generator learns to distinguish between visual objects (i.e., objects in the visual training data) and visually irrelevant sounds (i.e., sounds unrelated to the visual objects). In this manner, when implementing an ML system to derive sounds for a silent video, the generator can generate sounds that are related to visual objects (e.g., visually relevant sounds) without adding or synchronizing visually unrelated sounds to the video.

[0020] 1 illustrates an ML system 100 for training an ML model for synchronizing audio with visual data, according to one embodiment described herein. The ML system 100 includes a visual encoder 110, an audio transfer regularizer 115, and a generator 125, each of which includes one or more ML models. Details of these ML models are provided in the figures below.

[0021] As shown, the visual encoder 110 receives as input visual training data 105. In one embodiment, the visual training data 105 includes a plurality of consecutive video frames. In response, the audio transport regularizer 115 receives audio training data 107, which includes sounds synchronized to the visual training data 105. For example, the visual training data 105 can be visual information from a training media presentation, while the audio training data 107 contains sounds of the training media presentation. However, as discussed above, the audio training data 107 can include both visually relevant sounds (i.e., sounds that correlate to visual objects in the visual training data 105) as well as visually irrelevant sounds (i.e., sounds that are not correlated or related to synchronized visual objects in the visual training data 105). In one embodiment, the source of the visually relevant sounds is off-screen - i.e., not visible - in the corresponding video frame. Using the techniques herein, the ML system 100 can be trained to enable the generator 125 to distinguish between visually relevant sounds and unrelated sounds, such that, at run-time, when only silent video (e.g., visual data that is not synchronized to any sound) is received, the generator 125 can generate or derive visually relevant sounds that can be synchronized to the silent video to generate a media presentation that includes audio synchronized to the visual data.

[0022] In general, the goal of the visual encoder 110 is to identify visual objects in the visual training data 105, such as dogs, people, cars, etc. The visual encoder 110 can also use its ML model (or models) to identify activities being performed by the visual objects, such as a barking dog, a person bouncing a basketball, a car braking, etc. These visual objects and their activities can then be transmitted to a generator 125.

[0023] The audio transfer regularizer 115 uses its ML model (or models) to distinguish between visually relevant and irrelevant sounds in the audio training data 107. For example, visually relevant sounds may include sounds produced by visual objects in the visual training data 105, such as a dog barking, a basketball bouncing, or the high-pitched sound of tires caused by a car braking suddenly. The audio training data 107 also includes visually irrelevant sounds that are unrelated or independent of the visual objects in the data 105 or their actions. Examples of visually irrelevant sounds could include audience applause (if the audience is not shown in the visual training data 105), a commentator talking about a basketball game, or sounds made by an animal that is currently off-screen (e.g., not in a synchronized frame of the visual training data 105). For example, at a given point in time, a sound in the audio training data 107 may be a visually relevant sound when the sound source is currently visible in a synchronized frame of the visual training data 105, but may be a visually irrelevant sound when the sound source is no longer visible in the synchronized frame of the visual training data 105.

[0024] The goal of the generator 125 is to reproduce the audio training data 107 (including both visually relevant and irrelevant sounds) using inputs provided by the visual encoder 110 and the audio transfer regularizer 115. However, by only sending data regarding visually relevant and irrelevant sounds to the generator 125 during training, both the visually relevant and irrelevant sounds can be correlated to visual objects identified by the visual encoder 110, which causes the generator 125 to add both the visually relevant and irrelevant sounds to the silent video during the execution phase (i.e., after training is completed). To prevent this, the ML system 100 includes an information bottleneck 120 that limits the amount of data regarding visually relevant sounds sent to the generator 125 and instead focuses on sending visually irrelevant sounds to the generator 125.

[0025] Although it may seem counterintuitive to feed data on visually unrelated sounds to the generator 125 during training (because visually unrelated sounds should be ignored during the execution phase), doing so advantageously allows the generator 125 to distinguish between visual targets and visually unrelated sounds during training, rather than erroneously correlating them. For example, if the information bottleneck 120 is too large, a significant amount of data on visually relevant sounds as well as visually unrelated sounds is provided to the generator 125. In this scenario, the generator 125 can "fool" into using the input received from the audio transfer regularizer as the predicted sounds 130. As a result, the generator 125 does not learn that visually unrelated sounds are not correlated to the visual data provided by the visual encoder 110.

[0026] However, when the information bottleneck 120 tightens, the amount of data corresponding to visually relevant sounds sent to the generator 125 is reduced. Therefore, the generator 125 cannot simply use the sounds it receives from the regularizer 115 as the predicted sounds 130 (which should include both visually relevant and unrelated sounds). Thus, in order to generate visually relevant sounds for the predicted sounds 130 (e.g., when trying to reconstruct the audio training data 107), the ML model or models of the generator 125 have no choice but to determine that the data provided by the audio transfer regularizer 115 does not include visually relevant sounds, but instead derives those sounds from the visual objects provided by the vision encoder 110. For example, the generator 125 may have access to a database for identifying sounds related to the visual objects identified by the vision encoder 110. As a result of this processing, generator 125 learns that the data received from audio transport regularizer 115 regarding visually irrelevant sounds is in fact unrelated to the visual objects generated by visual encoder 110, thus achieving the desired effect of training generator 125 to recognize visually irrelevant sounds. For example, generator 125 may determine that the applause of an off-screen audience is unrelated to a visible barking dog, or that the commentary provided by an off-screen announcer is unrelated to an on-screen basketball player bouncing the ball.

[0027] During the training phase, the generator 125 has the goal of outputting predicted sounds 130 that include both visually relevant and unrelated sounds. That is, the predicted sounds 130 should be as close as possible to the sounds in the audio training data 107 (which serve as the ground truth). During this process, and as described above, the ML model or models in the generator 125 learn to identify visually unrelated sounds.

[0028] In one embodiment, the visual encoder 110, the audio transfer regularizer 115, the information bottleneck 120, and the generator 125 are stored in memory as program code executed by a processor in at least one computing system. For example, the visual encoder 110, the audio transfer regularizer 115, the information bottleneck 120, and the generator 125 may be implemented using an ML framework (e.g., an ML software application) stored in memory and executed by a processor in the computing system.

[0029] 2 is a flow chart of a method 200 for training an ML model to synchronize sound to visual data, according to one embodiment described herein. In this embodiment, the ML system receives training media data 205 that includes both visual training data 105 (e.g., video frames) as well as audio training data 107 (e.g., synchronized sounds or soundtracks of the training media). That is, the training media data 205 can be split such that its video data (e.g., visual training data 105) is provided to a visual encoder while the corresponding audio data (e.g., audio training data 107) is provided to an audio transfer regularizer.

[0030] At block 210, the visual encoder identifies objects for training the first ML model. In one embodiment, the visual encoder identifies visual objects in the visual training data 105. In addition to identifying the visual objects 225, the visual encoder may also identify the action currently being performed by the visual object 225, or some other metadata associated with the object, such as the type of object (e.g., the age of a person, or the breed of a dog). The visual encoder may include any type of ML model suitable for identifying visual objects 225.

[0031] In block 215, the audio transfer regularizer uses a second ML model to distinguish between visually relevant and unrelated sounds. That is, the second ML model in the regularizer distinguishes between various sounds in the audio training data 107. As part of this process, in block 220, the information bottleneck filters the visually unrelated sounds from the visually relevant sounds represented in the output of the second ML model. Rather than allowing the second ML model to output all of its generated output data, the information bottleneck limits the output of the second ML model such that data relating to the visually unrelated sounds 230 is primarily sent to the generator, while most (or all) of the data relating to the visually relevant sounds is filtered out (i.e., not forwarded to the generator). The audio transfer regularizer can include any type of ML model suitable for distinguishing visually unrelated sounds 230.

[0032] In block 235, the generator recovers the training media to train the third ML model. As part of training the third ML model, the generator attempts to generate (or recover or regenerate) the audio training data 107 received by the regularizer. As explained above, the generator receives visually irrelevant sounds from the audio transfer regularizer, but does not receive visually relevant sounds. Thus, the third ML model attempts to recover visually relevant sounds from the visual objects 225 in order to regenerate the audio training data 107 and the training media data 205. As part of this process, the third ML model learns that the visual objects 225 are different from the visually irrelevant sounds 230. Thus, during the execution phase described below, the third ML model can avoid generating visually irrelevant sounds when generating sounds for silent videos.

[0033] 3 illustrates an ML system 300 for synchronizing sound with visual data using a trained ML model, according to one embodiment described herein. While FIG. 1 illustrates an ML system for training an ML model, the ML system 300 is used during a run-time phase to generate sound for a silent video after the ML model has been trained (e.g., after method 200 has been performed).

[0034] The ML system 300 includes the visual encoder 110 and the generator 125, as described above. However, the system 300 lacks the audio transfer regularizer and the information bottleneck. Instead, the input of the generator 125 used during the training phase of FIG. 1 receives a zero vector 310 (e.g., a vector with zeros) instead. As such, the regularizer and the information bottleneck are not used during the execution phase of the ML system 300, thereby advantageously improving the performance of the ML system, since these components are used only during training, but not during execution. Instead, the generator 125 relies on its trained ML model to identify the sounds of the visual targets identified by the visual encoder 110.

[0035] During the execution phase, the visual encoder 110 receives a silent video 305, e.g., a series of video frames without sound that correspond or are synchronized frame-to-frame. In one embodiment, the goal of the ML system 300 during the execution phase is to generate sounds that correspond to the silent video 305. For example, if the silent video 305 depicts a barking dog or fireworks, the ML system 300 can use the ML models trained in the visual encoder 110 and the generator 125 to generate visually relevant sounds, e.g., a dog barking or growling and crackling sounds for fireworks.

[0036] Similar to the training phase, the visual encoder 110 identifies visual objects in the silent video 305. The visual encoder 110 may also identify the motion of the visual objects or the type or characteristics of the visual objects. This information is then forwarded to the generator 125.

[0037] The generator 125 uses its trained ML model or models to identify synchronized sounds for the silent video 305. For example, if the visual object shows a barking dog, the generator 125 may determine to synchronize the barking sound to the video frames showing the barking dog. In training the ML model in the generator 125, the model is taught to distinguish between visual objects and visually unrelated sounds in the training media data, so that during execution, the generator 125 is less likely to add and synchronize visually unrelated sounds to the visual objects identified in the silent video 305, thereby providing an advantage over previous solutions that do not use an audio forward regularizer during the training phase. In other words, the generator 125 is more likely to synchronize only visually relevant sounds to the visual objects, whose operation is shown in the silent video 305. The generator 125 outputs predicted sounds 315 that include visually relevant sounds synchronized to the frames of the silent video. The ML system 300 may generate new media presentations that include sounds synchronized to the video frames of the silent video 305.

[0038] 4 is a flow chart of a method 400 for synchronizing sound with visual data, according to one embodiment described herein. The ML system receives a silent video 305, which may include a series of video frames where at least a portion of the video 305 (or all of the video 305) lacks corresponding audio data or sound. The method 400 may be used to generate sound synchronized to the portion of the video 305 that previously lacked the audio data.

[0039] In block 405, the visual encoder uses the trained first ML model to identify objects in frames of video data. The visual encoder may perform the same techniques as performed in block 210 of Figure 2 to send data corresponding to visual objects 410 to the generator.

[0040] In block 415, the generator uses the trained third ML model to generate visually associated sounds that correspond to visual objects in the frames of the video data (i.e., the generator uses the visual objects 410 to identify sounds associated with these objects. These sounds are synchronized to frames of the silent video 305.

[0041] In block 420, the ML system outputs a media presentation that includes video frames of the silent video that are synchronized to the visually relevant sounds identified by the generator, such that when the media presentation is played, the user sees the visual object 410 and the synchronized sounds associated with the visual object 410. Advantageously, the embodiments described herein mitigate the chance that the generator selects visually irrelevant sounds for inclusion in the media presentation.

[0042] Figure 5 is an ML system 500 for training an ML model for synchronizing sound with visual data, according to one embodiment described herein. In general, the ML system 500 is one embodiment of the ML system 100 illustrated in Figure 1. Similar to the ML system 100, the ML system 500 includes a visual encoder 110, an audio transfer regularizer 115, and a generator 125, which includes at least one ML model trained using visual training data 105 and audio training data 107.

[0043] As shown, the feature extractor extracts frame features from the visual training data 105. These frame features are used as input to the ML model 505A of the visual encoder 110. In one embodiment, the ML model 505A represents one or more convolutional neural networks (CNNs) that generate one or more vertical vectors. For example, if the visual training data 105 has 24 frames per second, each CNN may output 24 vertical vectors per second. However, in one embodiment, the output of the ML model 505A is a single vector per time (e.g., per second), which is then fed to the ML model 505B.

[0044] In one embodiment, ML model 505B specializes in processing time-dependent information, such as video frames in the visual training data. In one embodiment, ML model 505A is a sequence-to-sequence ML model, which may be a long short-term memory (LSTM), bidirectional LSTM, recurrent neural network (RNN), 1D convolutional neural network (convnet), or other sequence learning method. In one embodiment, ML model 505B outputs a vertical frame every second. So, assuming there are t seconds in the visual training data 105, ML model 505B outputs T visual frames 515.

[0045] In FIG. 5, the audio training data 107 is transformed into a spectrogram representing the ground truth. To do so, the audio training data 107 may be transformed using any feature extraction model. The spectrogram is then used as an input to the ML model 505C in the audio transfer regularizer 115. In one embodiment, similar to the ML model 505B in the visual encoder 110, the ML model 505C is a sequence-to-sequence ML model, which may be an LSTM, bidirectional LSTM, RNN, 1D convection, or other sequence learning method. Advantageously, the sequence-to-sequence ML model provides improved results compared to other ML models when processing time-dependent information such as audio training data and visual training data.

[0046] The information bottleneck is implemented by suppressing the output of the ML model 505C. For example, the information bottleneck can be implemented by reducing the magnitude of the output of the ML model 505C. As shown, the ML model 505C uses the spectrogram as input to generate a bottleneck audio frame 510. The audio transfer regularizer 115 then replicates the frame 510 T times, so that advantageously, numerous copies of the bottleneck audio frame 510 exist as visual frames 515, which can enhance the ability of the generator 125 to distinguish between visual targets identified by the visual encoder 110 and visually unrelated sounds identified by the audio transfer regularizer 115.

[0047] By reducing the dimensions of the audio frame 510 to implement an information bottleneck, some (or all) of the data corresponding to visually relevant sounds is essentially filtered out. In other words, the dimensions of the audio frame 510 (i.e., the output of the ML model 505C) are adjusted such that the frame 510 contains primarily data corresponding to visually irrelevant sounds but not visually relevant sounds, thereby implementing an information bottleneck as described above.

[0048] The combiner 520 combines the visual frames 505 generated by the visual encoder 110 and the replicated audio frames 510 generated by the audio transfer regularizer 115 and transfers the resulting information to the generator 125. The generator 125 includes the ML model of the encoder 110 and an ML model 505D downstream of the regularizer 115 that attempts to replicate the sounds in the audio training data 107 by outputting predicted spectrograms that should match the ground truth spectrograms. In one embodiment, the ML model 505D may include a mixture of different layers, such as transposed convolutional layers, convolutional layers, batch normalization (BN) layers, rectified linear unit (ReLU) layers, etc. These layers may be combined into one ML model or several daisy-chained ML models. Additionally, although not shown, the generator 125 may include a post network coupled to the output of the ML model 505D that generates the predicted spectrograms.

[0049] As described above, during the training phase, the generator 125 receives data corresponding to visual objects identified by the vision encoder 110 (i.e., the visual frames 515) and data corresponding to visually irrelevant sounds (i.e., the bottleneck audio frames 510). From this information, the ML model 505D is trained to distinguish between visual objects and visually irrelevant sounds in the audio training data 107. Thus, during the execution phase, the likelihood that the ML model 505D will associate visually irrelevant sounds with visual objects in silent videos is advantageously reduced.

[0050] In one embodiment, training the ML system 500 to distinguish between visually relevant and unrelated sounds can be mathematically formulated. In the following description, capital letters represent random variables (not bold) or random vectors (bold), and lowercase letters represent deterministic values. E[] refers to expectation. H() refers to the (discrete) Shannon entropy. Furthermore, (V(t);S(τ)) represents a visual-audio pair, where V(t) represents the visual signal (vectorized) at each video frame t, and S(τ) represents the sound representation (waveform or spectrogram) of each audio frame τ. Different frame indices t and τ are used because the visual and audio signals have different sampling rates.

[0051] Speech is assumed to be decomposed into relevant and irrelevant signals.

[0052]

number

[0053] The subscript r refers to related sounds, while i refers to irrelevant sounds. We also assume that the relationship, denoted as f(), exists only between the video and related sounds. The irrelevant sounds are independent of both related sounds and visual features. These relationships can be expressed as follows:

[0054]

number

[0055] During the ceremony,

[0056]

number

[0057] refers to independence. In one embodiment, the goal is to separate the two components and extract the visually relevant component S from the visual signal V(t). r The aim is to generate only (τ).

[0058] The visual encoder 110 receives as input a video signal V(t) and outputs a set of video features. The audio transfer regularizer 115 receives as input a sound signal S(τ) and outputs audio transfer information (i.e., the duplicated bottleneck audio frames 510). The generator 125 then predicts (or restores) S(τ). There are two different types of predictions, with or without audio transfer:

[0059]

number

[0060] The prediction with audio transfer, denoted as , is predicted from both video features and audio transfer information.

[0061]

number

[0062] A prediction without speech transfer, denoted as , is predicted by setting the input to the speech transfer regularizer to zero, for example using the zero vector 310 illustrated in FIG.

[0063] During training, the generator 125 seeks to minimize the following loss involving prediction with speech transfer:

[0064]

number

[0065] The first term in Equation 3 is the reconstruction error and the second term is the adversarial loss.

[0066] 6 is an ML system 600 for training an ML model for synchronizing sound with visual data, according to one embodiment described herein. ML system 600 is the same as ML system 500, except for an additional communication link 605 between the visual encoder 110 and the audio transfer regularizer 115. In particular, link 605 indicates sending the output of ML model 505A to ML model 505C of the regularizer 115.

[0067] Omitting this link 605 shown in ML system 500 may advantageously improve operational efficiency in that ML system 500 may run faster or use fewer resources than an ML system (600) that uses link 605. However, link 605 may increase the ability of ML system 600 to distinguish between visually relevant and irrelevant sounds compared to ML system 500. In applications where there is typically one type of visual object, omitting link 605 may provide satisfactory results. For example, if the training data includes only videos of fireworks, it may be sufficient to omit link 605 when training the ML system. That is, ML system 500 is able to accurately distinguish between visually relevant and irrelevant sounds without using link 605. However, when the training data contains multiple distinct visual objects, link 605 may provide a substantial improvement in performance by enabling audio transfer regularizer 115 to use the output of ML model 505A as input to ML model 505C to better distinguish between visually relevant and unrelated sounds.

[0068] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements beyond the technology found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0069] As explained above, reference is made to the embodiments presented in this disclosure. However, the scope of the disclosure is not limited to the specific described embodiments. Instead, any combination of the features and elements described above, whether associated with different embodiments or not, is contemplated to implement and practice the contemplated embodiments. Furthermore, while the embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether or not a particular advantage is achieved by a given embodiment does not limit the scope of the disclosure. As such, the above-mentioned aspects, features, embodiments, and advantages are merely illustrative and are not to be construed as elements or limitations of the appended claims, except as expressly recited in the claims. Similarly, reference to the "invention" should not be construed as a generalization of the inventive subject matter disclosed herein, and should not be construed as elements or limitations of the appended claims, except as expressly recited in the claims.

[0070] Aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally herein as a "circuit," "module," or "system."

[0071] The present invention may be a system, method, or computer program product, or a combination thereof. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0072] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read-only memories (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge-in-groove structures having instructions recorded thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, should not be construed as being themselves transitory signals such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted over wires.

[0073] The computer readable program instructions described herein can be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and sends the computer readable program instructions for storage in a computer readable storage medium in each computing / processing device.

[0074] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or partially on the remote computer or entirely on the remote computer. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions according to state information of the computer readable program instructions to implement aspects of the invention.

[0075] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0076] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to generate a machine such that the instructions, executed via the processor of the computer or other programmable data processing apparatus, generate means for implementing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium that can cause a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, and thus a computer readable storage medium having instructions stored therein may include an article of manufacture including instructions that implement aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0077] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams, causing a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process.

[0078] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block of the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function. In some alternative embodiments, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, as well as combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or implements a combination of dedicated hardware and computer instructions.

[0079] While the foregoing is directed to embodiments of the present invention, other or further embodiments of the invention may be devised without departing from the basic scope thereof, which scope is determined by the claims that follow.

Claims

1. 1. A method for identifying visually related sounds, comprising: receiving visual training data at a visual encoder including a first machine learning (ML) model; using the first ML model to identify data in the visual training data that corresponds to a visual object; receiving audio training data synchronized to the visual training data at an audio transfer regularizer including a second ML model, the audio training data including visually relevant sounds and visually unrelated sounds both synchronized to a same frame in the visual training data including the visual object, the visually relevant sounds corresponding to the visual object, while the visually unrelated sounds are generated by a sound source not visible in the same frame; filtering data corresponding to the visually irrelevant sounds from an output of the second ML model using an information bottleneck; and training a third ML model downstream from the first and second ML models using data corresponding to the visual object and data corresponding to the visually unrelated sounds. The method includes:

2. After training the third ML model, The method further includes performing an execution step, the execution step comprising: receiving silent video at the first ML model; using the first ML model to identify data corresponding to a second visual object in the silent video; generating a visually associated sound using the third ML model to be synchronized with at least one video frame of a silent video containing the second visual object, wherein data corresponding to the second visual object is input into the third ML model; and generating a media presentation based on synchronized visually related sound and video frames within said silent video; The method of claim 1 , comprising:

3. The method of claim 2 , wherein the second ML model in the voice transfer regularizer is not used during the execution stage.

4. 4. The method of claim 1, wherein the information bottleneck comprises limiting a magnitude of an output of the second ML model, the limiting the magnitude preventing data corresponding to the visually relevant sounds from reaching the third ML model.

5. The method of any of claims 1 to 4, wherein the second ML model comprises a sequence-to-sequence ML model.

6. The sequence-to-sequence ML model outputs a bottleneck speech frame based on the speech training data and the information bottleneck, and the method further comprises: replicating bottleneck speech frames according to a number of T periods in the visual training data; and sending the duplicated bottleneck speech frame to the third ML model; The method of claim 5 further comprising:

7. predicting the visually relevant and irrelevant sounds in the speech training data based on receiving the replicated bottleneck speech frames and the data corresponding to a visual object using the third ML model. The method of claim 6 further comprising:

8. 1. A system for identifying visually related sounds, comprising: A processor; and a memory containing a program, the program performing operations when executed by the processor, the operations including: receiving visual training data at a visual encoder including a first ML model; using the first ML model to identify data in the visual training data that corresponds to a visual object; receiving audio training data synchronized to the visual training data at an audio transfer regularizer including a second ML model, the audio training data including visually relevant sounds and visually unrelated sounds both synchronized to a same frame in the visual training data including the visual object, the visually relevant sounds corresponding to the visual object, while the visually unrelated sounds are generated by a sound source not visible in the same frame; filtering data corresponding to the visually irrelevant sounds from an output of the second ML model using an information bottleneck; and training a third ML model downstream from the first and second ML models using data corresponding to the visual object and data corresponding to the visually unrelated sounds. Including, the system.

9. The operations, after training the third ML model, include: The method further includes performing an execution step, the execution step comprising: receiving silent video at the first ML model; using the first ML model to identify data corresponding to a second visual object in the silent video; and generating a visually associated sound using the third ML model to be synchronized with at least one video frame of a silent video containing the second visual object, wherein data corresponding to the second visual object is input into the third ML model; and generating a media presentation based on synchronized visually related sound and video frames within said silent video; The system of claim 8 , comprising:

10. The system of claim 9 , wherein the second ML model in the voice transfer regularizer is not used during the execution stage.

11. 11. The system of claim 8, wherein the information bottleneck includes limiting a magnitude of an output of the second ML model, the limiting the magnitude preventing data corresponding to the visually relevant sounds from reaching the third ML model.

12. The second ML model includes a sequence-to-sequence ML model, the sequence-to-sequence ML model outputs a bottleneck speech frame based on the speech training data and the information bottleneck, and the operation comprises: replicating bottleneck speech frames according to a number of T periods in the visual training data; and sending the duplicated bottleneck speech frame to the third ML model; The system of claim 8 , further comprising:

13. The operation comprises: predicting the visually relevant and irrelevant sounds in the speech training data based on receiving the replicated bottleneck speech frames and the data corresponding to a visual object using the third ML model. The system of claim 12 further comprising:

14. A computer-readable recording medium having recorded thereon a computer program for causing a computer to execute the method according to any one of claims 1 to 7.

15. A computer program comprising: A computer program for carrying out the method described.

Images