Translation adaptation for automatic video dubbing

Abstract

A video content item includes a video portion and an audio portion to be translated to a first language from a second language. The audio portion in the first language is caused to be visually correlated with the video portion in the second language. Causing the audio portion in the first language to be visually correlated with the video portion in the second language comprises providing translation content representing the audio portion in the first language as input to a machine learning model, generating, using an output of the machine learning model, one or more variations of the audio portion in the first language, and selecting, using a comparison of a speech component of an audio portion in the second language to a speech component of each of the audio portion in the first language and the one or more variations of the audio portion in the first language.

Description

TRANSLATION ADAPTATION FOR AUTOMATIC VIDEO DUBBINGTECHNICAL FIELD

[0001] Aspects and implementations of the present disclosure relate to a duration and lip shape aware translation adaptation for video dubbing and subtitling.BACKGROUND

[0002] A platform (e.g., a social media platform, media content platform, streaming platform, etc.) can provide users with media content. The platform can include media content in multiple languages, and some media content can be available in multiple languages.Translating speech in a video from an originally recorded language to another language may involve labor-intensive efforts of video dubbing. Video dubbing can refer to combining additional or supplementary speech (dubbed speech, often in a foreign language) with original speech to create the finished soundtrack for the video. However, the dubbed speech may differ from the original speech and may not align with start and end times of the original speech. As a result, the translated audio may appear out of sync and may not be appealing to viewers.SUMMARY

[0003] The below summary is a simplified summary of the disclosure in order to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the disclosure. It is intended neither to identify key or critical elements of the disclosure, nor delineate any scope of the particular implementations of the disclosure or any scope of the claims. Its sole purpose is to present some concepts of the disclosure in a simplified form as a prelude to the more detailed description that is presented later.

[0004] An aspect of the disclosure provides a computer-implemented method that includes identifying a video content item comprising a video portion and an audio portion. The audio portion is to be translated to a first language from a second language. The method further includes causing the audio portion in the first language to be visually correlated with the video portion. Causing the audio portion in the first language to be visually correlated with the video portion includes: providing, as input to a machine learning model, translation content. The translation content represents the audio portion in the first language. The method further includes generating, using an output of the machine learning model, one or more variations of the audio portion in the first language, and selecting, using a comparison of aspeech component of an audio portion in the second language to a speech component of each of the audio portion in the first language and the one or more variations of the audio portion in the first language, the audio portion in the first language for the video content item.

[0005] In some aspects, the audio portion in the first language includes a first speech segment in the first language, and the audio portion in the second language includes a second speech segment in the second language.

[0006] In some aspects, the translation content includes the first speech segment and at least one of the second speech segment or a speech component of the second speech segment.

[0007] In some aspects, the audio portion in the first language is selected for the video content item from the first speech segment and one or more variations of the first speech segment.

[0008] In some aspects, selecting the audio portion in the first language for the video content item includes selecting, from the first speech segment and the one or more variations of the first speech segment, a target speech segment having a speech duration that most closely matches a speech duration of the second speech segment.

[0009] In some aspects, the speech duration of the target speech segment is determined by applying a text-to-speech (TTS) model to the target speech segment. The speech duration of the second speech segment is determined by applying the TTS model to the second speech segment.

[0010] In some aspects, selecting the audio portion in the first language for the video content item includes selecting, from the first speech segment and the one or more variations of the first speech segment, a target speech segment associated with a first sequence of lip shapes with a lowest dynamic time warping (DTW) distance to a second sequence of lip shapes associated with the second speech segment.

[0011] In some aspects, generating the one or more variations of the audio portion in the first language includes translating a first variation of the one or more variations of the first speech segment into at least one of the second language or a third language to obtain a third speech segment. The first variation is output by the machine learning model, and translating the third speech segmentback into the first language to obtain a second variation of the one or more variations of the first speech segment.

[0012] In some aspects, at least a subset of variations of the one or more variations include a combination of words that is different than a combination of words in the first speech segment.

[0013] In some aspects, the speech component includes one or more of a speech duration, a sequence of lip shapes, or a speech structure.

[0014] In some aspects, at least some of the one or more variations of the audio portion in the first language are generated using one or more labels indicative of a predefined relationship between the speech component of the audio portion in the first language and a speech component of a respective variation of the audio portion in the first language.

[0015] An aspect of the disclosure provides a system including a memory and a processor communicatively coupled to the memory. The processor to perform operations including obtaining translation content representing a first training audio portion of a video content item in a first language. The first training audio portion is translated from a second language. The operations further include obtaining one or more variations of the first training audio portion in the first language, obtaining, for each of the one or more variations of the first training audio portion in the first language, a label indicating a relationship between a speech component of the first training audio portion in the first language and a respective speech component of a corresponding variation of the one or more variations of the first training audio portion in the first language, and generating training data to train a machine learning model (MLM). Training data is generated to train the MLM to generate one or more variations of a predicted audio portion of a second video content item in the first language, and indicate a relationship between a speech component of the predicted audio portion in the first language and a speech component of a respective variation of the predicted audio portion in the first language. The training data includes (i) training input data comprising the first training audio portion in the first language and the one or more variations of the first training audio portion in the first language, and (ii) training output data comprising one or more labels of the one or more variations of the first training audio portion in the first language. In some aspects, the training input data further includes a second training audio portion of the video content item. The second training audio portion is in the second language and corresponds to the first training audio portion in the first language.

[0016] In some aspects, the operations further include selecting a translated audio portion for the second video content item from the predicted audio portion and the one or more variations of the predicted audio portion.

[0017] In some aspects, the training input data includes historical output data obtained from the MLM.

[0018] In some aspects, the speech component includes one or more of a speech duration, a sequence of lip shapes, or a speech structure.

[0019] In some aspects, the machine learning model includes at least one of a regression model, a neural network, a supervised model, or an unsupervised model.

[0020] An aspect of the disclosure provides a non-transitory computer readable storage medium comprising instructions for a server that, when executed by a processing device, cause the processing device to perform operations including identifying a video content item comprising a video portion and an audio portion. The audio portion is to be translated to a first language from a second language. The method further includes causing the audio portion in the first language to be visually correlated with the video portion. Causing the audio portion in the first language to be visually correlated with the video portion includes: providing, as input to a machine learning model, translation content. The translation content represents the audio portion in the first language. The operations further include generating, using an output of the machine learning model, one or more variations of the audio portion in the first language, and selecting, using a comparison of a speech component of an audio portion in the second language to a speech component of each of the audio portion in the first language and the one or more variations of the audio portion in the first language, the audio portion in the first language for the video content item.

[0021] In some aspects, the audio portion in the first language includes a first speech segment in the first language, and the audio portion in the second language includes a second speech segment in the second language.

[0022] In some aspects, the speech component includes one or more of a speech duration, a sequence of lip shapes, or a speech structure.BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Aspects and implementations of the present disclosure will be understood more fully from the detailed description given below and from the accompanying drawings of various aspects and implementations of the disclosure, which, however, should not be taken to limit the disclosure to the specific aspects or implementations, but are for explanation and understanding only.

[0024] FIG. 1 illustrates an example system architecture, in accordance with implementations of the present disclosure.

[0025] FIG. 2 depicts a flow diagram of an example method 200 for designating a translation of an original speech segment in a translated language, in accordance with implementations of the present disclosure.

[0026] FIG. 3 depicts a block diagram of an example system architecture for duration and lip shape aware translation adaptation for video dubbing and subtitling, in accordance with implementations of the present disclosure.

[0027] FIGs. 4A and 4B depict timing tables of lip shape alignment between an example English speech segment and example Spanish speech segments in accordance with implementations of the present disclosure.

[0028] FIG. 5 depicts a flow diagram of an example method for duration and lip shape aware translation adaptation for video dubbing and subtitling, in accordance with implementations of the present disclosure.

[0029] FIG. 6 depicts a flow diagram of an example method for training a model for duration and lip shape aware translation adaptation for video dubbing and subtitling, in accordance with implementations of the present disclosure.

[0030] FIG. 7 is a block diagram illustrating an exemplary computer system, in accordance with implementations of the present disclosure.DETAILED DESCRIPTION

[0031] Aspects of the present disclosure relate to a duration and lip shape aware translation adaptation for video dubbing and subtitling. A platform, such as a media content platform (e.g., social media, media content sharing platforms, etc.) can present users with media content (e.g., video content) in various languages. Some video content can be provided in multiple languages by reusing the same video portion and translating the audio portion (e.g., the dialogue spoken by characters) in the video content. Often the translated audio portion and the original video portion do not match up, which can reduce the quality of the user’s viewing experience. Media presented with misaligned audio and visual portions (e.g., lip movement) is “non-isochronic.” Isochrony (audio that properly aligns with the visual portion) is important to video content presentation because when the viewer sees lips moving without hearing a voice, or hears a voice without seeing lips move, (i.e., the audio and visual portions are non-isochronic) the viewer can be distracted. Viewer distractions can lead to a lower enjoyment, intelligibility, and / or viewership of the video content. Some attempts to address this problem include speeding up or slowing down the translated dialogue to meet the time constraints of the dialogue in the original language, or allowing for extra time in a visual shot for the translated dialogue to continue after the original language dialogue would have ended. However, changing the speed of the voice, seemingly at random, and / or includingextended visual shots solely for dialogue translation can result in poor quality of provided content and again lead to a lower enjoyment and / or viewership of the video content.

[0032] Other attempts to address misaligned audio and lip movements include careful translation of each line of dialogue. Dialogue for the audio translation can be translated by humans and / or machines. Human translation of each audio portion or line of dialogue (e.g., speech segment) can be time consuming, costly, and restrictive. Machine translation (MT) is the automatic translation of words and audio portions from one language into another using a model (e.g., a machine translation model (MTM)). Using MTMs can be more cost effective, especially when combined with a human translation element (e.g., a machine translation is performed and a human reviews and / or edits the machine translation). MTMs can be optimized to provide a relatively high adequacy of the translation (e.g., semantic equivalence) and relatively high fluency of the translation (e.g., the accuracy of the message). However, optimizing for these two objectives alone can be insufficient for dialogue and audio translation for video content (e.g., “dubbing”). Additional objectives for dubbing include lip movements and sounds being isochronous (occurring at the same time), and time-aligning translated audio portions with corresponding speech segments to original language video content. Most MTMs do not account for these additional objectives.

[0033] Some MTMs attempt to account for additional objectives to provide duration- aware machine translations. These duration-aware MTMs can provide a machine translation with corresponding timing information. Inputs to duration-aware machine translators can include the audio portions and / or speech segments in the original language as well as phonemes and related durations. However, a duration-aware MTM can require a large amount of isochronic parallel data (e.g., audio portion, speech segments, text, phonemes, facial structure, etc.) whose production can consume significantly more processing resources than obtaining parallel data without any timing constraints. Due to difficulty in obtaining such data, training of these duration-aware or multi-faceted MTMs can suffer from data sparsity, which can also lead to a relatively poor adequacy of translation and / or relatively poor fluency of translation.

[0034] Aspects of the present disclosure address the above noted and other deficiencies by using a duration and lip shape aware translation system for video dubbing and subtitling. The duration and lip shape aware translation (DALSAT) system can apply a two-phase approach to translating audio portions and corresponding speech segments from an original language into a target language (e.g., a translated language) as a target audio portion and corresponding target speech segment. In the first phase, an MTM can translate the originalaudio portion into a translated audio portion. During the second phase, a post-processing system including a paraphrasing sub-system and a ranking sub-system can apply one or more post-processing operations based on the translated audio portion to generate and select a translation of the original audio portion that is more isochronic with the original audio portion.

[0035] Advantages of the two-phase approach include the ability to provide low-cost voice dubbing of good quality by (1) using high-performance MTMs trained on an abundance of (non-isochronic) parallel texts, and (2) modifying the translation in post-processing to be more isochronic with the facial motions. The post-processing phase can include such operations as a paraphrasing operation and a ranking operation. It should be noted that while the figures and examples below demonstrate translations between English and Spanish, those two languages are used only illustratively, and the disclosed systems and techniques can also be successfully applied to any other languages.

[0036] FIG. 1 illustrates an example system architecture 100, in accordance with implementations of the present disclosure. The system architecture 100 (also referred to as “system” herein) includes client devices 102A-N, a data store 110, a platform 120, and / or server machines 130-150, each connected to a network 108. In some implementations, network 108 can include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), a wired network (e.g., Ethernet network), a wireless network (e.g., an 802. 11 network or a Wi-Fi network), a cellular network (e.g., a Long Term Evolution (LTE) network), routers, hubs, switches, server computers, and / or a combination thereof.

[0037] In some implementations, data store 110 can be a persistent storage capable of storing data as well as data structures to tag, organize, and index the data. A data item can include video content 121, in accordance with implementations described herein. Data store 110 can be hosted by one or more storage devices, such as main memory, magnetic or optical storage based disks, tapes, or hard drives, and so forth. In some implementations, data store 110 can be a network-attached file server or some other type of persistent storage such as an object-oriented database, a relational database, and so forth, that may be hosted by platform 120 or one or more different machines coupled to the platform 120 via network 108.

[0038] The client devices 102A-N can each include computing devices such as personal computers (PCs), laptops, mobile phones, smart phones, tablet computers, netbook computers, network-connected televisions, etc. In some implementations, client devices 102A-N may also be referred to as “user devices.” Each client device 102 can include amedia content viewer. In some implementations, the media content viewer can be an application that provides a graphical user interface (GUI) for users to view, edit, and / or create video content 121 . For example, the content viewer can be a web browser that can access, retrieve, present, and / or navigate video content 121 served by a web server. The media content viewer can render, display, and / or present the video content 121 to a user. In one example, the media content viewer can be a standalone application (e.g., a mobile application or app) that allows users to view, edit, and / or create video content 121 . In some implementations, the media content viewer can be a media content platform application for users to view, generate, edit, and / or upload the video content 121 to platform 120. As such, the media content viewers can be provided to the client devices 102A-N by platform 120.

[0039] In some implementations, platform 120 and / or server machines 130-150 can be one or more computing devices (such as a rackmount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memories, databases), networks, software components, and / or hardware components that may be used to provide a user with access to video content 121 and / or provide the video content 121 to the user. For example, platform 120 can be a video content platform. The video content platform can allow a user to create, edit, access, or share with other users, video content stored at data store 110. Platform120 can also include a website (e.g., a webpage) or application back-end software that can be used to provide a user with access to video content 121 . Platform 120 can include video content 121. Video content 121 can be made accessible through platform 120. In some implementations, platform 120 can facilitate the access of video content 121 by a client device 102, but not include video content 121. For example, a client device 102 can request a particular video content 121 from platform 120. Platform 120 can identify the video content121 (e.g., stored in data store 110) and can determine how to present the client device 102 with the requested video content 121. In some implementations, platform 120 can provide the client device 102 with access to the file via the GUI of the media content viewer, as described above.

[0040] Platform 120 can include a duration and lip shape aware translation (DALSAT) system 160 that facilitates the translation of audio portions of video content 121 from an original language into a translated language. The DALSAT system 160 can include a paraphrasing sub-system and a ranking sub-system (not shown). The DALSAT system 160 can produce, in part or in whole, translated audio for dubbed video content 123 using one or more models 170A-N. The models 170A-N canbe trained to generate translated audio basedon audio in an original language (e.g., audio from video content 121). For example, the translated audio can be produced one speech segment at a time. For each audio portion and corresponding speech segment in the original language, the DALSAT system 160 can generate variations of a translated audio portion (e.g., a speech segment in the translated language that was translated from the original speech segment in the original language), and select which translation has a speech component closest to the speech component of the original language. For example, the DALSAT system 160 can generate variations of a translated audio portion, and then the translation (e.g., translated speech segment or a variation of the translated speech segment) which has a speaking duration closest to the speaking duration of the original audio portion can be selected as the product of DALSAT system 160. The one or more models 170A-N can be trained based on speech component labels which indicate a relationship between a speech component (e.g., a duration) of the translated audio portion and a speech component of a variation of the translated audio portion. Speech component labels can be pre-defined or can be determined by the DALSAT system 160. For example, the DALSAT system 160 can have predefined boundary conditions that label a variation of a translated audio portion having a speech duration of 10% less than the speech duration of the translated audio portion, as “shorter,” and a variation of the translated audio portion having a speech duration of 20% greater than the speech duration of the translated audio portion as “longer.” In another example, the DALSAT system 160 can determine the boundaries of the speech duration labels based on patterns identified by the DALSAT system 160 between different speaking durations of variations of various translated speech segments in the training dataset. Further details regarding training and using models 170A-N are provided herein.

[0041] Server machine 130 can include a training set generator 131 that is capable of generating training data (e.g., a set of training inputs and a set of target outputs) to train models 170A-N. Training data can be associated with training a model 170 to produce a translation of an original audio portion with an associated speech component that corresponds to a speech component associated with the original audio portion. Training set generator can obtain variations of a translated audio portion and associated speech component labels from a set of training data. In some implementations, the set of training data can be populated with historical variations of translated audio portions and respective historical speech components previously produced by the DALSAT system 160. In some implementations, the training set generator 131 can determine the associated speech component labels using the speech components produced by the DALSAT system 160. Training set generator 131 can prependthe speech component label to the translated audio portion (e.g., a source speech segment) to generate the training corpus for the model 170 (e.g., a length aware paraphrasing model). Training set generator 131 (or another component of system 100) can store training audio portions, training translated audio portions, and associated speech component labels at data store 110.

[0042] Training set generator 131 can generate training data for the DALSAT system 160 to produce length-aware paraphrases (e.g., variations) of a translated audio portion through modified round-trip translation techniques. Round-trip translation (e.g., translating a speech segment from language “A” to language “B,” and then translating the resulting speech segment back to language “A,”) can be used to verify the accuracy of a translation. However, due to the various ambiguities of translation between languages, performing a round-trip translation can also generate alternative variations of an audio portion which have different words and / or structure that affect the speaking duration of the audio portion without altering the message. These differences can be magnified when a model 170 is trained to modify the translations according to a speech component label (e.g., by prepending the speech component label to the translated audio portion). For example, for the translated audio portion “gracias por venir hoy,” a model 170 can generate the translated audio portion variation “gracias por pasar,” based on a speech component label of “shorter.” In another example, for the translated audio portion “gracias por venir hoy,” a model 170 can generate the translated audio portion variation “gracias por venir el dia de hoy” based on a speech component label of “longer.”

[0043] In some implementations, model training can be supervised, and each set of training data can include a subset of training inputs and target outputs based on the identified data. To train a supervised model, training set generator 131 can generate training data including a subset of training inputs and a subset of target outputs. The subset of training inputs can include variations of a training audio portion (e.g., included in data store 110, as described above), and a subset of target outputs can include labels associated with each variation of the training audio portion. In some implementations, model training can be unsupervised. To train an unsupervised model, training set generator 131 can generate training data by clustering groups of previously provided media items (e.g., included in data store 110) based on similarities between the media items, through dimensionality reduction by reducing the number of features in the data while retaining as much relevant information about the media items as possible, by generating synthetic or partially synthetic data that resembles the original data, through anomaly detection by identifying parts of media itemsthat are significantly different from the rest of the data, or through data augmentation by applying mathematical transformations to the training dataset.

[0044] Server machine 140 can include a training engine 141. Training engine 141 can train a machine learning model 170A-N using the training data from training set generator 131. The machine learning model 170A-N can refer to the translation model artifact that is created by the training engine 141 using the training data that includes training inputs. The model 170 A-N can be composed of one or more layers, e.g., a single level of linear or nonlinear operations (e.g., a support vector machine (SVM) or may be a deep network, i.e., a machine learning model that is composed of multiple levels of non-linear operations). An example of a deep network is a neural network with one or more hidden layers, and a neural network can be trained by, for example, adjusting the weights of the neural network in accordance with a backpropagation learning algorithm or the like. For convenience, the remainder of this disclosure will refer to the translation model as a neural network, even though some implementations might employ an SVM or other type of learning machine instead of, or in addition to, a neural network. In some implementations, the training set is obtained by training set generator 131 hosted by server machine 130.

[0045] Server machine 150 can include DALSAT system 160 that provides a translated audio portion as input to one or more models 170 A-N to obtain one or more outputs. Based on the outputs of the model 170, DALSAT system 160 can designate a translation of the original audio portion in the translated language. The designated translation can be used in translated audio for dubbed video content 123. In response to designating the translation of the original audio portion, the DALSAT system 160 can generate a mapping between the translated audio portion and the translation of the original audio portion. In some implementations, the DALSAT system 160 can generate a mapping between the original audio portion and the designated translation of the original audio portion. The DALSAT system 160 can store generated mapping data in data store 110, a memory associated with a client device 102, and / or another memory associated with system 100. In some implementations, DALSAT system 160 can provide an original audio portion as part of the input to a model 170. In some implementations, DALSAT system 160 can additionally provide a speech component (e.g., a speaking duration value) and / or a speech component label (e.g., the relationship between a speech component of a translated audio portion and a speech component of a variation of the translated audio portion) as part of an input to the model 170.

[0046] The DALSAT system 160 can obtain variations of a translated audio portion from model 170. The DALSAT system 160 can label each variation of the translated audio portion based on a relationship between a speech component of the translated audio portion and a respective speech component of the variation of the translated audio portion. For example, as described above, a variation of a translated audio portion with a speaking duration shorter than the speaking duration of the translated audio portion can receive the label, “shorter.” In another example, as described above, a variation of a translated audio portion with a speaking duration longer than the speaking duration of the translated audio portion can receive the label, “longer.” In this way, the DALSAT system 160 can categorize the variations of the translated audio portion obtained as output from the model 170. The DALSAT system 160 can select the translation (e.g., translated audio portion or variation of the translated audio portion) with a speech component closest to a speech component of the original audio portion. In some implementations the DALSAT system 160 can select the translation of the original audio portion using another model 170A-N.

[0047] The DALSAT system 160 can determine speech components of original phases (e.g., from the audio of video content 121). For example, the DALSAT system 160 can determine the speaking duration of an original audio portion. In some implementations, the speaking duration of translated audio portions, variations of translated audio portions, and the original audio portion can be determined, in part, based on a text-to-speech (TTS) model. Original audio portions, translated audio portions, and variations of translated audio portions can be associated with multiple underlying speech components. Speech components can include, for example, the language of the speech (e.g., an original language), the accent of the speech, the speed of the speech, the cadence of the speech, the familiarity of the speech (e.g., formal, informal, colloquial, etc.), the tone of the speech, the duration of the speech, the shapes of the mouth used to form the speech, etc. As described above, speech component labels can indicate a relationship between two speech components (i.e., between the speech component of the translated audio portion and a speech component of a variation of the translated phase).

[0048] In some implementations, the DALSAT system 160 can configure the size of an original audio portion that will be translated. For example, DALSAT system 160 might limit the size of the audio portion to a certain number of words, speech segments, phrases, sentences, or subjects. In another example, DALSAT system 160 might limit the size of the audio portion to a length of time that a character in video content 121 is speaking. In some implementations, DALSAT system 160 can segment the audio to be translated (e.g., audioobtained from video content 121) into smaller portions (e.g., speech segments) to facilitate translation and / or model training. In some implementations, the DALSAT system 160 can accept user-identified speech segments from the audio of video content 121 as inputs to model 170.

[0049] It should be noted that in some implementations, the functions of server machines 130, 140, and 150 or platform 120 may be provided by a fewer number of machines. For example, in some implementations the server machines 130 and 140 maybe integrated into a single machine, while in other implementations the server machines 130, 140, and 150 may be integrated into multiple machines. In addition, in some implementations one or more of server machines 130, 140, and 150 maybe integrated into platform 120. In general, functions described in implementations as being performed by platform 120 and / or server machines 130-150 can also be performed on the client devices 102A-N in other implementations, if appropriate. In addition, the functionality attributed to a particular component can be performed by different or multiple components operating together. Platform 120 can also be accessed as a service provided to other systems or devices through appropriate application programming interfaces, and thus is not limited to use in websites.

[0050] In some implementations of the disclosure, a “user” can be represented as a single individual. However, other implementations of the disclosure encompass a “user” being an entity controlled by a set of users and / or an automated source. For example, a set of individual users federated as a community in a social network can be considered a “user.” In another example, an automated consumer can be an automated ingestion pipeline, such as a topic channel, of platform 120. Further to the descriptions above, a user may be provided with controls allowing the user to make an election as to both if and when systems, programs, or features described herein may enable collection of user information (e.g., information about a user’s social network, social actions, or activities, profession, a user’s preferences, or a user’s current location), and if the user is sent content or communications from a server. In addition, certain data can be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user’s identity can be treated so that no personally identifiable information can be determined for the user, or a user’s geographic location can be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user can have control over what information is collected about the user, how that information is used, and what information is provided to the user.

[0051] FIG. 2 depicts a flow diagram of an example method 200 for designating a translation of an original audio portion in a target language (e.g., a translated language), in accordance with implementations of the present disclosure. Method 200 can be performed by processing logic that can include hardware (circuitry, dedicated logic, etc.), software (e.g., instructions run on a processing device), or a combination thereof. In one implementation, some or all operations of method 200 can be performed by one or more components of system 100 of FIG. 1. In some implementations, some or all of the operations of method 600 can be performed by training engine 141 in connection with training set generator 131, as described above. In some implementations, operations of method 200 can be performed by DALSAT system 160 as described above.

[0052] At operation 210, processing logic performing the method 200 obtains a translated audio portion in a translated language of an original audio portion in an original language. As shown in the illustrative example, the translated phase, “gracias por venir hoy” is a translation in the translated language (i.e., Spanish) of the original audio portion, “thankyou for coming out today” in the original language (i.e., English). The translated audio portion can be obtained using a machine translation model (MTM). In some embodiments, the translated audio portion can be obtained using a dataset of translated audio portion inputs.

[0053] At operation 220, processing logic generates translation variations of the translated audio portion. As shown in the illustrative example, variations of the translated audio portion, “gracias por venir hoy,” can include “gracias por pasar,” and “gracias por venir el dia de hoy.” These translation variations (e.g., variations of the translated audio portion) can be generated by a paraphrasing model such as model 170 in connection with the DALSAT system 160, as described above. In some implementations, the model 170 can also accept, as part of the input, duration information corresponding to translated audio portion inputs. In some implementations, the model 170 can also accept, as part of the input, the original audio portion in the original language. In some implementations, operation 220 can be performed by a paraphrasing sub-system of a duration aware system such as DALSAT system 160 as described above. Additional details regarding the paraphrasing sub-system are described below with respect to FIG. 3.

[0054] At operation 230, processing logic selects the translation with a duration closest to the duration of the audio portion in the original language (e.g., a duration that corresponds to the duration of the original audio portion). As shown in the illustrative example, processing logic determines that the original audio portion in the original language, “thank you for coming out today,” has a duration of 1.8 seconds, the translated audio portion, “gracias porvenir hoy,” has a duration of 1.3 seconds, one variation of the translated audio portion, “gracias por pasar,” has a duration of 1.1 seconds, and the other variation of the translated audio portion, “gracias por venir el dia de hoy,” has a duration of 1.9 seconds. In some implementations, as described above, the duration of each audio portion (e.g., the original audio portion, the translated audio portion, and variations of the translated audio portion) can be determined using a TTS model.

[0055] Processing logic identifies that the variation of the translated audio portion in the translated language “gracias por venir el dia de hoy,” with a duration of 1.9 seconds is closest to (e.g., corresponds to) the duration of the original audio portion in the original language “thank you for coming out today,” with a duration of 1.8 seconds. Processing logic can select “gracias por venir el dia de hoy” as a designated translation for “thank you for coming out today.” In some implementations, processing logic can designate a translation in the translated language of the original audio portion because the respective translation is more isochronic with the original audio portion and can maintain semantic integrity better than other translation variations. In some implementations, more than a single translated audio portion may be selected for additional screening, e.g., a subset of translated audio portions with a duration within a certain tolerance compared with a duration of the original audio portion, e.g., 10%, 15%, and / or the like. In some implementations, additional (to the duration) metrics may be used for a final selection of the translated audio portion, e.g., an audio portion of the selected subset that has the closest number of vowels (to the original audio portion) or an audio portion with the most matching timing of vowels may be selected. In some implementations, a final selection of the translated audio portion may be performed based on a weighted combination of multiple metrics (e.g., total duration, number of vowels, vowel locations, and / or the like), with empirically selected weights, which may be languagespecific.

[0056] FIG. 3 depicts a block diagram of an example system architecture 300 for duration and lip shape aware translation adaptation for video dubbing and subtitling, in accordance with implementations ofthe present disclosure. System architecture 300 accepts original audio portion 301 and produces output 350. System architecture 300 includes machine translator 310, translated audio portion 311, and post-processing system 320. Postprocessing system 320 includes paraphrasing sub-system 330 and ranking sub-system 340. Machine translator 310 and post-processing system 320 can be separate systems. In some implementations, a model trained to perform post-processing system 320 can also perform the function of machine translator 310.

[0057] Machine translator 310 accepts original audio portion 301 as input, and generates translated audio portion 311 as output. The original audio portion 301 can be provided from a data storage location, such as data store 110 as described with respect to FIG. 1, and the translated audio portion 311 can be stored in a data storage location such as data store 110. Machine translator 310 can include a general-purpose MTM trained on parallel corpora. Parallel corpora (also known as a multilingual corpora) can include a collection of aligned audio portions (e.g., words, sentences, etc.). Each aligned audio portion can include an audio portion in one language and a corresponding translated audio portion in other language(s) associated with the parallel corpora. The parallel corpora can be aligned by word, speech segment, sentence, subject, idea, etc. In some implementations, parallel corpora can be textbased. Parallel corpora can include pivot language(s). A “pivot language” is a default language that can act as a bridge between two languages that do not have a corpora of direct translations (e.g., language translations that did not have related translation pair data to train the machine translator 310). The pivot language can be used by translating the audio portion in an original language into the pivot language and then translating the pivot-segment into the target language. For example, to translate an audio portion from Spanish to Japanese, a machine translator 310 using English as a pivot language might first translate the Spanish audio portion to an English audio portion, and then translate the English audio portion to the Japanese audio portion. In some embodiments, there may be multiple pivot languages. For example, to translate an audio portion from Portuguese to Japanese, a machine translator 310 might first translate the Portuguese audio portion to a Spanish audio portion, then translate the Spanish audio portion to an English audio portion, and then translate the English audio portion to the Japanese audio portion.

[0058] The paraphrasing sub-system 330 of post-processing system 320 accepts translated audio portion 311 as input and generates a variation 331 of the translated audio portion 311. Variations 331 of the translated audio portion 311 can include audio portions with shorter speaking durations, audio portions with longer speaking durations, and audio portions with similar speaking durations. Paraphrasing sub-system 330 can produce variation A 331 A and variation Z 331Z as an output. Paraphrasing sub -system 330 can include a machine learning model (e.g., such as model 170 described above with respect to FIG. l) that is trained on various types of input data associated with given words or audio portions, including text, audio, speech duration, lip shapes, facial shapes, phonemes, word ordering, prosodic pauses, etc. In some implementations, the machine learning model included in paraphrasing sub-system 330 can be a duration-aware model that accepts a translated audioportion and duration information associated with the translated audio portion as input. In the illustrative example, the translation model of machine translator 310 and the model of paraphrasing sub-system 330 are separate, however in some implementations, as noted above, the MTM of the machine translator 310 can be included in the model of paraphrasing subsystem 330.

[0059] Paraphrasing sub-system 330 can generate variations 331 of the translated audio portion. In some implementations, these variations can be generated based on mappings of speech component labels (e.g., “much shorter,” “shorter,” “longer,” “much longer,” etc.).These speech component labels and associated mappings can be generated as training data to train the paraphrasing sub-system 330. During inference, the model in paraphrasing subsystem 330 that is trained on the generated training data can map the input sentence to its variations based on the speech component labels. In some embodiments, the training data (e.g., the speech component labels and associated mappings) can be generated with a roundtrip translation technique as described with respect to FIG. 2. In some implementations, the round-trip translation technique to generate variations 331 can be based on generated or predefined speech component labels (e.g., “much shorter,” “shorter,” “longer,” “much longer,” etc.). For example, prepending the label “shorter” to the translated audio portion 311 “gracias porvenir hoy” can result in the paraphrasing sub-system 330 generating variation A 331 A, “gracias por pasar.” In another example, prepending the label “longer” to the translated audio portion 311 “gracias por venir hoy” can result in the paraphrasing sub-system 330 generating variation Z 331Z, “gracias por venir el dia de hoy.” In some implementations, the paraphrasing sub-system 330 can “blindly” generate variations 331 (i.e., randomly or without prepending a speech component label). In these implementations, after generating the variations 331 of the translated audio portion 311, the paraphrasing sub-system can categorize each variation 331 based on a speech duration (e.g., a speech component value) and label each variation 331 based on the relationship between a respective speaking duration of a given variation 331 and the speaking duration of the translated audio portion 311. For example, after generating variation A 311 A without using a label, paraphrasing sub-system 330 can determine the speaking duration of the variation A 331 A, “gracias por pasar,” is shorter than the speaking duration of the translated audio portion 331, “gracias por venir hoy,” and label variation A 331 A with a “shorter” duration label. Speech component labels can indicate a relationship between a speech component (e.g., a speaking duration) of the translated audio portion 311 and a respective speech component of a variation 331. Speech components can correspond to various types of data associated with a given audio portion orspeech segment, including text, audio, speech duration, lip shape, facial shapes, phonemes, word ordering, prosodic pauses, etc.

[0060] In some implementations, paraphrasing sub-system 330 can accept, as an input, a duration of the original audio portion. The model in the paraphrasing sub-system 330 can be trained to use the duration of the original audio portion when prepending labels to the translated audio portion 311 to generate variations 331. For example, in the illustrative example, the original audio portion duration 343 is 1.8 seconds and the translated audio portion duration 341N is 1.3 seconds. The paraphrasing sub-system 330 can determine that the 1.8 second duration of the original audio portion is longer than the 1 .3 second duration of the translated audio portion and only generate variations 331 that are likely to have a duration longer than 1.3 seconds. The duration of the original audio portion can be determined by the paraphrasing sub-system 330. In some embodiments, the paraphrasing sub-system 330 can determine the original audio portion duration 343 by measuring the time-domain portion of the waveform corresponding to the original audio portion. In some embodiments, the paraphrasing sub-system 330 can use a text-to- speech (TTS) model to determine the original audio portion duration 343. In some embodiments, the paraphrasing sub-system 330 can use an automated speech recognition (ASR) model to determine the duration of the original audio portion. An ASR model can transcribe spoken language into written text. An ASR model can be trained based on datasets of acoustic modeling and language modeling to transcribe spoken language. In some embodiments, the ASR model can be trained to determine a speaking duration of the input spoken language. For example, an ASR model can be trained to accept as input an audio clip including spoken dialogue from video content and produce the outputs of a dialogue transcription and a duration of the spoken dialogue in the audio clip. In some embodiments, the machine learning model included in paraphrasing sub-system 330 can be trained on durational training data which includes speaking durations that were determined using a TTS model. During operation of the paraphrasing sub -system 330, the paraphrasing sub-system 330 can determine the original audio portion duration 343 with an ASR model. In some embodiments, an ASR model can be used during operation of the paraphrasing sub-system 330 to improve the accuracy of the generated variations 331.

[0061] Ranking sub-system 340 can accept translated audio portion 311 and outputs from the paraphrasing sub-system 330 (e.g., variation A 331 A and variation Z 331Z) as inputs. In the illustrative example, the paraphrasing sub-system 330 does not determine a duration (e.g., speech component) of variations 331 or translated audio portion 311. In the illustrative example, the ranking sub-system 340 determines a variation A duration 341 A for variation A331 A, a translated audio portion duration 341N for translated audio portion 311, and a variation Z duration 341Z for variation Z 331Z. The variation A duration 341 A is 1.1 seconds, which is shorter than the translated audio portion duration 341N of 1 .3 seconds. The variation Z duration 341Z is 1.9 seconds, which is longer than the translated audio portion duration 34 IN of 1.3 seconds. The ranking sub-system 340 can also accept original audio portion 301 as an input. The ranking sub-system can determine the original audio portion duration 343, which in the illustrative example is 1 .8 seconds. In some implementations, as described above, the ranking sub-system 340 can determine the variation A duration 341 A, the translated audio portion duration 341N, the variation Z duration 341Z and the original audio portion duration 343 using a text-to-speech (TTS) algorithm. Each audio portion or variation can be converted from a text format to an audio format with an associated speaking duration. In some implementations, to minimize noise (e.g., resulting in inaccurate speaking duration determinations), the same TTS model can be used to convert each audio portion from text to audio.

[0062] Ranking sub-system 340 includes selector 348, which can select the respective duration 341 associated with the variation 331 or translated audio portion 311 that corresponds to (e.g., is closest to) the original audio portion duration 343. Selector 348 can designate which output 350 post-processing system 320 should produce. Post-processing system 320 can produce as an output 350, the variation 331 or translated audio portion 311 with the associated duration 341 selectedby the selector 348. In the illustrative example, the selection criterion for selector 348 is based on the original audio portion duration 343. In some implementations, the selection criterion for selector 348 canbe a smallest duration 341 (e.g., variation A duration 341 A). In some implementations, the selection criterion for selector 348 can be a largest duration 341 (e.g., variation Z duration 341Z). In some implementations, if a variation A duration 341 A or variation Z duration 341Z is the same duration as the translated audio portion duration 34 IN, the selector 348 can select the translated audio portion 311 over a variation 331.

[0063] In some implementations, the selector 348 can include secondary selection criteria to be applied when durations 341 are similar or the same. Secondary selection criteria can include one or more criterion based on any speech components such as speech duration, lip shapes, facial shapes, phonemes, word ordering, prosodic pauses, etc. In at least one implementation, the secondary selection criteria can include a criterion based on how well the lip shapes of the original audio portion 301 match to lip shapes of a variation 331 or lipshapes of translated audio portion 311. This lip shape criterion is further described below with respect to FIG. 4A and FIG. 4B.

[0064] FIGs. 4A and 4B depict timing table 400 and timing table 450 respectively of lip shape alignment between an example English audio portion 410 and example Spanish audio portions 421 and 422, in accordance with implementations of the present disclosure. It should be noted that the languages Spanish and English are used here only illustratively, and that the following description can apply to other language pairings.

[0065] As described above, lip shape alignment can be part of the translation selection operation of post-processing, such as in ranking sub-system 340 described with respect to FIG. 3. Specifically, lip shape alignment can be included as one of a secondary selection criteria in selector 348 to determine which output 350 should be produced by post-processing system 320. Thus, a selection criterion based on lip shape alignment can be applied (with respect to post-processing system 320) when the duration 341 of multiple variations 331 and / or translated audio portion 311 are the same or similar (e.g., when multiple translations satisfy the first selection criterion).

[0066] In the illustrative example, the English audio portion 410 “all right?,” is converted from audio to a sequence of English lip shapes 403. The Spanish audio portion 421 “^esta bien?” is converted from audio to a sequence of Spanish lip shapes 404A. The lip shape alignment path 431 is determined using dynamic time warping (DTW) to align the English lip shapes to the Spanish lip shape sequences, and indicates how closely the English lip shapes 403 matches the Spanish lip shapes 404A. The darker line color indicates a closer match between the two lip shapes, and the lighter line color indicates a less close match between the two lip shapes. DTW can be used to measure the similarity (e.g., “distance”) between two temporal sequences with a time-dependent cost matrix. The cost matrix can be created by measuringthe similarity between each elementin one sequence (e.g., English lip shapes 403) and every element of the other sequence (e.g., Spanish lip shapes 404). An optimization model (e.g., minimization model) can be applied to the resulting cost matrix to find the optimal alignment path with the minimum cumulative cost. The alignment path can represent the best matches between the two temporal sequences, and can allow for variations in the time axis. In FIGs. 4A and 4B, the alignment path is illustratively shown as lip shape alignment path 431 and lip shape alignment path 432 respectively.

[0067] In the illustrative timing table 400, the English audio portion 410 “all right?” has a duration 401 of .1 seconds. The Spanish audio portion 421 “^esta bien?” has a duration 402 A of .36 seconds. The lip shape alignment path 431 shows that the English lip shapes 403initially correlate to the Spanish lip shapes 404A. But, by the end of the sequence of English lip shapes 403, the sequence of Spanish lip shapes 404 A have a poor correlation to the English lip shapes 403.

[0068] In the illustrative timing table 450, the English audio portion 410 “all right?” has a duration 401 of .1 seconds. The Spanish audio portion 422 “^vale?” has a duration 402B of .13 seconds. The lip shape alignment path 432 shows that the sequence of English lip shapes 403 correlates well with the sequence of Spanish lip shapes 404B. Thus, in the illustrative example, because the Spanish lip shapes 404B corresponding to the Spanish audio portion 422 have a better correlation to the English lip shapes 403 than the Spanish lip shapes 404A corresponding to the Spanish audio portion 421, the Spanish audio portion 422 would be designated as the translation of the English audio portion 410 (e.g., designated as output 350 by selector 348 in post-processing system 320).

[0069] FIG. 5 depicts a flow diagram of an example method 500 for duration and lip shape aware translation adaptation for video dubbing and subtitling, in accordance with implementations of the present disclosure. Method 500 can be performed by processing logic that can include hardware (circuitry, dedicated logic, etc.), software (e.g., instructions run on a processing device), or a combination thereof. In one implementation, some or all of the operations of method 500 can be performed by one or more components of system 100 of FIG. 1. In some implementations, some or all of the operations of method 500 can be performed by DALSAT system 160 as described above.

[0070] At operation 510, the processing logic performing the method 500 identifies a video content item comprising a video portion and an audio portion, the audio portion to be translated to a first language from a second language.

[0071] At operation 520, the processing logic causes the audio portion in the first language to be visually correlated with the video portion, wherein causing the audio portion in the first language to be visually correlated with the video portion comprises the following operations, 521, 522, and 523 respectively.

[0072] At operation 521 the processing logic provides, as input to a machine learning model, translation content, wherein the translation content represents the audio portion in the first language. In some implementations, the translation content includes a first speech segment in a first language. The first speech segment can be translated from a second speech segment in the second language. In some implementations, the translation content can include the audio portion in the second language in the second language. In some implementations, the translation content can include the speech component of the audio portion in the secondlanguage. In some implementations, the machine learning model can be trained on previous inputs to the machine learning model and corresponding previous outputs. In some implementations, the machine learning model can be trained by identifying correlations between previous sets of inputs.

[0073] At operation 522 the processing logic generates, using an output of the machine learning model, one or more variations of the audio portion in the first language. In some implementation, processing logic can generate one or more variations of a speech segment associated with the audio portion in the first language. In some implementations, output of the machine learning model can include indications of speech components for the audio portion in the first language, the audio portion in the second language, and / or the one or more variations of the audio portion in the first language. Speech components can include speech duration, lip shape, facial shape(s), speech structure, phonemes, word ordering, prosodic pauses, etc. The output of the machine learning model can include speech component labels. Speech component labels can indicate a relationship between the speech component of the audio portion in the first language and a respective speech component of one of the one or more variations of the audio portion in the first language. In some implementations, the outputs can include a mapping between the speech components of the audio portion in the second language and respective speech components of the one or more variations of the audio portion in the first language.

[0074] In some implementations, the model can generate one or more variations of the audio portion in the first language by paraphrasing the audio portion in the first language. “Paraphrasing” here can refer to expressing the same ideas or communicated information in other words, while retaining the original meaning. Paraphrases of the audio portion in the first language can communicate the expression of the original message (i.e., the expression, idea, information, etc.) of the audio portion in the first language without copying the audio portion in the first language verbatim. In some implementations, to paraphrase the audio portion in the first language, the model can add, subtract, or exchange word(s) for synonyms or other words with similar meanings with respect to the input audio portion (e.g., an audio portion in the first language containing a first speech segment). In some implementations, the model can paraphrase the audio portion in the first language using round-trip translations as described above with respect to FIG. 3. Round-trip translations can use a third language (e.g., a pivot language) to translate an original audio portion into the pivot language, and then translate the pivot-language audio portion back into the original language. For example, a round-trip translation of a Spanish audio portion would include translating the Spanish audio portioninto an English audio portion, then translating the resulting English audio portion back into Spanish as a variation of the Spanish audio portion. As described above, round-trip translation techniques can be used to verify the accuracy of a translation, however as applied here, round-trip translation techniques can be used to generate variations of an input audio portion (e.g., the Spanish audio portion containing a Spanish speech segment). In some embodiments, a round-trip translation can include additional transition languages (e.g., pivot languages). For example, a Spanish audio portion might be translated into English, then the resulting English audio portion translated into Japanese, and the resulting Japanese audio portion then translated back into Spanish. In some embodiments, using additional transitional languages in the round-trip translation technique to generate variations of an input audio portion can increase the variation of the resulting output audio portions.

[0075] As described above, a pivot language can be a default language that can act as a bridge between two different languages (e.g., Spanish -> English -> Japanese) languages that do not have direct translations (e.g., language translations that did not have a sufficient corpora of related translation pair data). A pivot language can also function as a bridge between the same language, such as when used in round-trip translation techniques (e.g., Spanish-> English -> Spanish, Spanish-> Portuguese -> English -> Spanish, etc.), to generate variations of an input speech segment. The pivot language can be used by translating the speech segment in an original language into the pivot language and then translating the resulting pivot language-speech segment into a target language as a target speech segment. In some implementations, the model can generate a third speech segment by translating the first speech segment into the third language (e.g., the pivot language). The model can then translate the third speech segment from the third language into the first language. In some implementations, the model can translate the third speech segment into the first language based on a speech component label. In some implementations, the round-trip translation can be modified using speech component labels, which indicate a relationship between speech components of the translated speech segment (e.g., the input) and actual or intended speech components of a respective variation of the translated speech segment.

[0076] For example, an English speech segment (e.g., as found in the original audio portion 301 as described with respect to FIG. 3) can be translated into a Spanish speech segment (e.g., as found in the translated audio portion 311). Using the above described methods, the Spanish speech segment can then be translated back into a second English speech segment (which may or may not match the original English speech segment), and then re-translated back into Spanish to generate variations of the Spanish speech segment (e.g.,variations 331). As explained in this example, and in some implementations, the third language (e.g., pivot language) can be the same language as the second language. In at least one implementation, the pivot language can be English. In some embodiments, multiple pivot languages can be used.

[0077] At operation 523, the processing logic selects, using a comparison of a speech component of an audio portion in the second language to a speech component of each of the audio portion in the first language and the one or more variations of the audio portion in the first language, the audio portion in the first language for the video content item. In some implementations, the processing logic can determine from respective speech components of the variations of the audio portion and the speech component of the audio portion in the first language, the speech component that is closest to the speech component of the audio portion in the second language. For example, if the speech components associated with the variations of the audio portion in the first language are all a worse match to the second speech components of the audio portion in the second language than the first speech components of the audio portion in the first language, the processing logic can designate the audio portion in the first language (with associated second components) as the translation of the audio portion in the second language.

[0078] In some implementations where the speech component is a speech duration, processing logic can determine the first speech component as a first audio duration of the audio portion in the first language, the second speech component as a second audio duration of the audio portion in the second language, and the one or more third speech components as one or more third audio durations of the one or more variations. Processing logic can select from a set including the first audio duration and the one or more third audio durations, an audio duration having a value that corresponds to (e.g., is closest to) a value of the second audio duration. The audio portion with the audio duration that corresponds to the second audio duration can be designated by processing logic as the translation of the audio portion in the second language.

[0079] In some implementations where the speech component is a speech duration, the processing logic can convert a first speech segment into an audio portion in the first language using a text-to-speech (TTS) system, and similarly, the second speech segment into an audio portion in the second language, and the one or more variations of the first speech segment into one or more variations of audio portions (e.g., one or more variations of the audio portion in the first language). Processing logic can determine the first audio duration of the first audio, the second audio duration of the second audio, and the third one or more audiodurations from the one or more third audios. As described above, processing logic can use these speech components (e.g., the first, second, and one or more third audio durations) to designate a translation of the second phase. The processing logic can select the audio portion associated with the audio duration that corresponds to (e.g., is closest to) the second audio duration. In some implementations, speech segments can be accepted into the post-processing model as audio portions. In some implementations, input accepted to the post-processing model can be converted to audio using methods other than a TTS system. In implementations using a TTS system, the TTS system can be used to standardize the speech segment audio data in respective generated audio portions. For example, the TTS system can convert the speech segments to consistent-speed translations.

[0080] In some implementations where the speech component is a lip shape, processing logic can determine for the audio portion in the first language, the audio portion in the second language, and / or the one or more variations of the audio portion in the first language, the respective speech components as respective sequence of lip shapes. The processing logic can then align the sequence of lip shapes of the audio portion in the first language and the sequence of lip shapes of the one or more variations of the audio portion in the first language to the sequence of lip shapes of the audio portion in the second language. The processing logic can select from the sequence of lip shapes of the audio portion in the first language and the sequence of lip shapes of the one or more variations of the audio portion in the first language, the sequence of lip shapes with the lowest dynamic time warping (DTW) distance to the sequence of lip shapes of the audio portion in the second language. The audio portion with the lip shapes having the lowest DTW distance to the second sequence of lip shapes can be designated by the processing logic as the translation of the audio portion in the second language.

[0081] FIG. 6 depicts a flow diagram of an example method 600 for training a model for duration and lip shape aware translation adaptation for video dubbing and subtitling, in accordance with implementations of the present disclosure. Method 600 can be performed by processing logic that can include hardware (circuitry, dedicated logic, etc.), software (e.g., instructions run on a processing device), or a combination thereof. In one implementation, some or all of the operations of method 600 can be performed by one or more components of system 100 of FIG. 1. In some implementations, some or all of the operations of method 600 can be performed by training engine 141 in connection with training set generator 131 as described above. In some implementations, some, or all of the operations of method 600 can be performed by DALSAT system 160 as described above. In some implementations, themachine learning model can be a regression model, a neural network, a supervised model, an unsupervised model, etc.

[0082] At operation 610, processing logic performing the method 600 can obtain translation content representing a first training audio portion of a video content item in a first language, wherein the first training audio portion is translated from a second language. In some implementations, the translation content can include the audio portion in the second language in the second language. In some implementations, the translation content can include the second speech component of the audio portion in the second language.

[0083] At operation 620, processing logic obtain one or more variations of the first training audio portion in the first language. The one or more variations of the first training audio portion can be obtained from a database. In some implementations, the one or more variations of the first training audio portion can be generated by a machine learning model.

[0084] At operation 630, processing logic obtains, for each of the one or more variations of the first training audio portion in the first language, a label indicating a relationship between a speech component of the first training audio portion in the first language and a respective speech component of a corresponding variation of the one or more variations of the first training audio portion in the first language. The speech component can include a speech duration, lip shape, facial shape, phoneme, word ordering, prosodic pause, etc. In some implementations, processing logic can generate labels based on indications (e.g., values) of one or more speech components associated with respective audio portions. In some embodiments, labels can indicate a relationship between a speech component of a first training speech segment and a respective speech component of a corresponding variation of the one or more variations of the first training speech segment. In some implementations, labels can be based on pre-defined ratios or multipliers that are applied to indications of one or more speech components.

[0085] At operation 640, processing logic generates training data to train a machine learning model (MLM) to: generate one or more variations of a predicted audio portion of a second video content item in the first language, and indicate a relationship between a speech component of the predicted audio portion in the first language and a speech component of a respective variation of the predicted audio portion in the first language, wherein the training data comprises (i) training input data comprising the first training audio portion in the first language and the one or more variations of the first training audio portion in the first language, and (ii) training output data comprising one or more labels of the one or more variations of the first training audio portion in the first language. As described above,variations of the predicted audio portion in the first language can be generated by paraphrasing the audio portion in the first language in the first language. Paraphrasing techniques can include the modified round-trip translation techniques described with respect to FIG. 5, (where audio portions are translated from the initial translation into a pivot language and then back into the translated language), and other techniques to alter the combination of words that express the same or similar ideas.

[0086] The machine learning model can be trained to generate variations of the audio portion in the first language in the first language. In some implementations, the machine learning model can be trained to select from the generated variations, the audio portion that has speech components that match the speech components associated with the audio portion in the second language. In some implementations, training input data can include historical input and output data obtained from the previously trained machine learning model. That is, the machine learning model can be continuously trained based on outputs generated in response to non-training inputs.

[0087] FIG. 7 is a block diagram illustrating an example computer system 700, in accordance with implementations of the present disclosure. The computer system 700 can correspond to platform 120 and / or client devices 102A-N, described in FIG. 1. Computer system 700 can operate in the capacity of a server or an endpoint machine in endpoint-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a television, a personal computer (PC), a tablet PC, a set- top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a server, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0088] The computer system 700 includes a processing device 702 (e.g., a processor), a main memory 704 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), double data rate (DDR SDRAM), or DRAM (RDRAM), etc.), a static memory 706 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 716, which communicate with each other via a bus 730.

[0089] Processing device 702 represents one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, processingdevice 702 can be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. The processing device 702 can also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processing device 702 is configured to execute translation processing logic 722 (e.g., for generating variations of a translated audio portion) for performingthe operations discussed herein. The processing device 702 can be configured to execute instructions 726 stored in volatile memory 704.

[0090] The computer system 700 can further include a network interface device 708. The computer system 700 also can include a video display unit 710 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an input device 712 (e.g., a keyboard, and alphanumeric keyboard, a motion sensing input device, touch screen), a cursor control device 714 (e.g., a mouse), and a signal generation device 718 (e.g., a speaker).

[0091] The data storage device 716 can include a computer-readable storage medium 724 (e.g., a non-transitory machine-readable storage medium) 724 on which is stored one or more sets of instructions 726 (e.g., for generating variations of a translated audio portion) embodying any one or more of the methodologies or functions described herein. The instructions can also reside, completely or at least partially, within the main memory 704 and / or within the processing device 702 during execution thereof by the computer system 700, the main memory 704 and the processing device 702 also constituting machine-readable storage media. The instructions can further be transmitted or received over a network 720 via the network interface device 708.

[0092] While the computer-readable storage medium 724 (machine-readable storage medium) is shown in an exemplary implementation to be a single medium, the terms “computer-readable storage medium” and “machine-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store the one or more sets of instructions. The terms “computer-readable storage medium” and “machine-readable storage medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure. The terms “computer-readablestorage medium” and “machine-readable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical media, and magnetic media.

[0093] Reference throughout this specification to “one implementation,” “one embodiment,” “an implementation,” or “an embodiment,” means that a particular feature, structure, or characteristic described in connection with the implementation and / or embodiment is included in at least one implementation and / or embodiment. Thus, the appearances of the phrase “in one implementation,” or “in an implementation,” in various places throughout this specification can, but are not necessarily, referring to the same implementation, depending on the circumstances. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more implementations.

[0094] To the extent that the terms “includes,” “including,” “has,” “contains,” variants thereof, and other similar words are used in either the detailed description or the claims, these terms are intended to be inclusive in a manner similar to the term “comprising” as an open transition word without precluding any additional or other elements.

[0095] As used in this application, the terms “component,” “module,” “system,” or the like are generally intended to refer to a computer-related entity, either hardware (e.g., a circuit), software, a combination of hardware and software, or an entity related to an operational machine with one or more specific functionalities. For example, a component can be, but is not limited to being, a process running on a processor (e.g., digital signal processor), a processor, an object, an executable, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a controller and the controller can be a component. One or more components can reside within a process and / or thread of execution and a component can be localized on one computer and / or distributed between two or more computers. Further, a “device” can come in the form of specially designed hardware; generalized hardware made specialized by the execution of software thereon that enables hardware to perform specific functions (e.g., generating interest points and / or descriptors); software on a computer readable medium; or a combination thereof.

[0096] The aforementioned systems, circuits, modules, and so on have been described with respect to interact between several components and / or blocks. It can be appreciated that such systems, circuits, components, blocks, and so forth can include those components or specified sub-components, some of the specified components or sub-components, and / or additional components, and according to various permutations and combinations of the foregoing. Sub -components can also be implemented as components communicativelycoupled to other components rather than included within parent components (hierarchical). Additionally, it should be noted that one or more components can be combined into a single component providing aggregate functionality or divided into several separate subcomponents, and any one or more middle layers, such as a management layer, can be provided to communicatively couple to such sub-components in order to provide integrated functionality. Any components described herein can also interact with one or more other components not specifically described herein but known by those of skill in the art.

[0097] Moreover, the words “example” or “exemplary” are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the words “example” or “exemplary” is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or clear from context, “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then “X employs A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form.

[0098] Finally, implementations described herein include collection of data describing a user and / or activities of a user. In one implementation, such data is only collected upon the user providing consent to the collection of this data. In some implementations, a user is prompted to explicitly allow data collection. Further, the user can opt-in or opt-out of participating in such data collection activities. In one implementation, the collected data is anonymized prior to performing any analysis to obtain any statistical patterns so that the identity of the user cannot be determined from the collected data.

Claims

CLAIMSWhat is claimed is:1 . A method comprising: identifying a video content item comprising a video portion and an audio portion, the audio portion to be translated to a first language from a second language; and causing the audio portion in the first language to be visually correlated with the video portion, wherein causing the audio portion in the first language to be visually correlated with the video portion comprises: providing, as input to a machine learning model, translation content, wherein the translation content represents the audio portion in the first language; generating, using an output of the machine learning model, one or more variations of the audio portion in the first language; and selecting, using a comparison of a speech component of an audio portion in the second language to a speech component of each of the audio portion in the first language and the one or more variations of the audio portion in the first language, the audio portion in the first language for the video content item.

2. The method of claim 1, wherein the audio portion in the first language comprises a first speech segment in the first language, and the audio portion in the second language comprises a second speech segment in the second language.

3. The method of claim 2, wherein the translation content comprises the first speech segment and at least one of the second speech segment or a speech component of the second speech segment.

4. The method of claim 2, wherein the audio portion in the first language is selected for the video content item from the first speech segment and one or more variations of the first speech segment.

5. The method of claim 4, wherein selecting the audio portion in the first language for the video content item comprises:selecting, from the first speech segment and the one or more variations of the first speech segment, a target speech segment having a speech duration that most closely matches a speech duration of the second speech segment.

6. The method of claim 5, wherein the speech duration of the target speech segment is determined by applying a text-to-speech (TTS) model to the target speech segment, and wherein the speech duration of the second speech segment is determined by applying the TTS model to the second speech segment.

7. The method of claim 4, wherein selecting the audio portion in the first language for the video content item comprises: selecting, from the first speech segment and the one or more variations of the first speech segment, a target speech segment associated with a first sequence of lip shapes with a lowest dynamic time warping (DTW) distance to a second sequence of lip shapes associated with the second speech segment.

8. The method of claim 4, wherein generating the one or more variations of the audio portion in the first language comprises: translating a first variation of the one or more variations of the first speech segment into at least one of the second language or a third language to obtain a third speech segment, wherein the first variation is output by the machine learning model; and translating the third speech segment back into the first language to obtain a second variation of the one or more variations of the first speech segment.

9. The method of claim 2, wherein at least a subset of variations of the one or more variations comprise a combination of words that is different than a combination of words in the first speech segment.

10. The method of claim 1, wherein the speech component comprises one or more of a speech duration, a sequence of lip shapes, or a speech structure.11 . The method of claim 1, wherein at least some of the one or more variations of the audio portion in the first language are generated using one or more labels indicative of a predefined relationship between the speech component of the audio portion in the firstlanguage and a speech component of a respective variation of the audio portion in the first language.

12. A system comprising: a memory; and a processor communicatively coupled to the memory, the processor to perform operations comprising: obtaining translation content representing a first training audio portion of a video content item in a first language, wherein the first training audio portion is translated from a second language; obtaining one or more variations of the first training audio portion in the first language; obtaining, for each of the one or more variations of the first training audio portion in the first language, a label indicating a relationship between a speech component of the first training audio portion in the first language and a respective speech component of a corresponding variation of the one or more variations of the first training audio portion in the first language; and generating training data to train a machine learning model (MLM) to: generate one or more variations of a predicted audio portion of a second video content item in the first language, and indicate a relationship between a speech component of the predicted audio portion in the first language and a speech component of a respective variation of the predicted audio portion in the first language, wherein the training data comprises (i) training input data comprising the first training audio portion in the first language and the one or more variations of the first training audio portion in the first language, and (ii) training output data comprising one or more labels of the one or more variations of the first training audio portion in the first language.

13. The system of claim 12, wherein the training input data further comprises a second training audio portion of the video content item, wherein the second training audio portion is in the second language and corresponds to the first training audio portion in the first language.

14. The system of claim 12, further comprising selecting a translated audio portion for the second video content item from the predicted audio portion and the one or more variations of the predicted audio portion.

15. The system of claim 12, wherein the training input data comprises historical output data obtained from the MLM.

16. The system of claim 12, wherein the speech component comprises one or more of a speech duration, a sequence of lip shapes, or a speech structure.

17. The system of claim 12, wherein the machine learning model comprises at least one of a regression model, a neural network, a supervised model, or an unsupervised model.

18. A non-transitory computer readable storage medium comprising instructions for a server that, when executed by a processing device, cause the processing device to perform operations comprising: identifying a video content item comprising a video portion and an audio portion, the audio portion to be translated to a first language from a second language; and causing the audio portion in the first language to be visually correlated with the video portion, wherein causing the audio portion in the first language to be visually correlated with the video portion comprises: providing, as input to a machine learning model, translation content, wherein the translation content represents the audio portion in the first language; generating, using an output of the machine learning model, one or more variations of the audio portion in the first language; and selecting, using a comparison of a speech component of an audio portion in the second language to a speech component of each of the audio portion in the first language and the one or more variations of the audio portion in the first language, the audio portion in the first language for the video content item.

19. The non-transitory computer readable storage medium of claim 18, wherein the audio portion in the first language comprises a first speech segment in the first language, and the audio portion in the second language comprises a second speech segment in the second language.

20. The non-transitory computer readable storage medium of claim 18, wherein the speech component comprises one or more of a speech duration, a sequence of lip shapes, or a speech structure.

Images