Optimizing Lip Sync in Natural Language Translation Video

Abstract

A method for generating optimized videos of a speaker translated from a source language to a target language, balancing the optimization of the translation into the target language with the speaker's lips synchronized with the translated audio. The source video can be fed into a neural machine translation model. The model can synchronize multiple potential translations. The translations can be received by a generative adversarial network, which generates a video for each translation and classifies the translations as in-sync or out-of-sync. A lip-sync score can be for each generated video classified as in-sync.

Description

[Technical Field]

[0001] The present invention relates generally to natural language processing, and more particularly to translating a video of a speaker in a source language into a target language to generate a video in which the speaker's lip movements are synchronized to the target language. [Background technology]

[0002] Generative adversarial networks (GANs) are an innovation in machine learning that utilizes deep learning methods for generative modeling. GANs can consist of two sub-models: a generator model trained to generate new examples, and a classifier model that classifies the generated examples as true or false. The two models are trained together in an adversarial loop until the classifier model classifies roughly half of the generated examples as true.

[0003] Natural language processing is the automated manipulation of natural language by software. Natural languages ​​are spoken and written languages ​​understood by humans, such as English, Arabic, Spanish, or Japanese. Natural language processing can include operations such as translating speech and text from one natural language to another. Additionally, natural language processing can involve manipulating natural language into a format understood by a computer, providing semantic meaning by converting words and sentences into vectors or similar representations. Summary of the Invention

[0004] Embodiments of the present disclosure include computer-implemented methods, computer program products, and systems for optimizing generated lip-sync videos for neural machine translation. The embodiments may include receiving, by a processor, a source video of a speaker in a source language. The embodiments may also include synchronizing multiple speech translations of the source video in a target language based on a neural machine translation model, the neural machine translation model being trained to synchronize a variable number of translations based on a variable beamwidth. Furthermore, the embodiments may include generating a lip-sync video for each of the multiple synchronized speech translations based on a generative sub-model within a generative adversarial network architecture. Furthermore, the embodiments may include classifying each lip-sync video as synchronous or asynchronous based on a classification sub-model within the generative adversarial network. The embodiments may also include generating a lip-sync score for each lip-sync video classified as synchronous.

[0005] The above summary is not intended to describe every illustrated embodiment of every implementation of the present disclosure. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a functional block diagram illustrating generally a video translation and lip sync optimization environment 100 for optimizing natural language translation and generating video with lip sync with natural language translation, in accordance with an embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram of a video translation and lip sync optimization engine according to an embodiment of the present invention. [Figure 3] 1 is an exemplary flow for training a cycle generative adversarial network for optimizing natural language translation and generating videos with lip sync with natural language translation, according to an embodiment of the present invention. [Figure 4]1 is a flowchart illustrating the operational steps of a method for optimizing natural language translation and generating video with lip synchronization with the natural language translation, according to an embodiment of the present invention. [Figure 5] 1 is a functional block diagram of an exemplary computing system within a video translation and lip sync optimization environment 100, in accordance with an embodiment of the present invention. [Figure 6] FIG. 1 illustrates a cloud computing environment according to an embodiment of the present invention. [Figure 7] FIG. 2 is a functional block diagram illustrating abstraction model layers according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0007] While the embodiments described herein are susceptible to various modifications and alternative forms, specifics thereof have been shown by way of example in the drawings and will be described in detail. It is to be understood, however, that the particular embodiments described are not to be construed in a limiting sense. On the contrary, it is intended to cover all modifications, equivalents, and alternatives falling within the scope of the present disclosure.

[0008] The embodiments shown and described herein recognize the benefits of optimizing lip sync in videos produced using natural language translation. In many translated videos, the original source audio is simply removed and the target language is dubbed over the video itself. This leads to a mismatch between the speaker's mouth movements, or lips, in the video and the audio. Humans are sensitive to any mismatch in lip sync and speech. Realistic lip animation synchronized with audio speech is not only pleasing to the human eye, but can also prevent distractions, potentially causing a person to pause or turn off the video, due to missynchronization between the lips and audio.

[0009] Machine translation of speech from a source language to a target language can sometimes pose challenges regarding synchronization between the speaker's lips and the translated audio. This can be related to the length or number of words in the translation. It can also occur that the translation is inaccurate, or that the text-to-audio dubbing style is incompatible with the generated lip sync. The embodiments described herein understand that two or more machine translations of speech may be required to generate a video with proper synchronization between the speaker's lips and the generated audio translation. Nevertheless, the embodiments presented herein can provide an improved user experience by providing an optimal translation in the target language along with a matched video with optimal lip sync. Note that throughout this specification, the terms lip sync, mouth movements, and lip movements can be used interchangeably. The aforementioned terms are intended to refer to the synchronization or matching of the audio and visual data of a speaker's video, thereby providing a pleasant user experience.

[0010] In an embodiment of the present invention, a neural machine translation system can generate most likely candidate translations in a target language for a video in a source language. A generative adversarial network can receive the candidate translations and generate a video for each of the candidate translations. The generative adversarial network can classify each of the generated videos as either (i) lip-synchronized to the translated text-to-audio dubbing or (ii) lip-synchronized to the translated text-to-audio dubbing. Additionally, a lip-sync score can be generated for any generated video classified as having lip-synchronized text-to-audio.

[0011] Additionally, in some embodiments, the neural machine translation system can generate one or more translation outputs in a target language of a source video in a source language. The output can be the top-K results or the most likely output in the target language of likely outputs based on a predetermined or dynamically determined beamwidth selection. In some embodiments, the beamwidth can be a learnable parameter of the neural machine translation system that can be optimized during the training process.

[0012] In embodiments, the generative adversarial network can utilize a cycle architecture ("cycle GAN"), in which a cycle consistency loss is continuously fed back from the discriminator sub-model to the generator sub-model based on a comparison between the source video and the generated video (described in more detail below). This process can be performed iteratively to further refine the cycle GAN and generate more realistic generated translated lip-sync videos. For example, a source video can be fed into the cycle GAN, resulting in multiple output translations and generated videos for the translations. Lip-sync can be compared to the source video for all generated videos. Any calculated loss can be incorporated into the generator sub-model to refine the generator sub-model.

[0013] A generative adversarial network is a machine learning model with at least two sub-models, where one model generates one or more images (e.g., for a video) and attempts to trick the classification sub-model into classifying the generated images as true. In the present invention, an embodiment attempts to generate videos classified as synchronous with respect to lip movements and translated audio data. In training the generative adversarial network, multiple types of images are provided, some of which have matched or lip-synced video, while others have out-of-sync audio and lip movements. The generative adversarial network is then trained until the generated translated videos are classified as synchronous approximately 50% of the time.

[0014] In embodiments of the present invention, the decision sub-model can be a recurrent neural network that utilizes landmarks on the speaker's face (e.g., lips, mouth, cheeks, etc.) to generate highly realistic facial poses for the generated speech. For example, the decision sub-model can have the ability to learn facial landmarks associated with phoneme-viseme mappings in a self-supervised manner during training. In embodiments, the decision sub-model can be a recurrent neural network with additional fully connected layers that smooth out any irregularities. This allows the decision sub-model to be language-independent, resulting in a system that can generate lip-sync videos for multiple target languages.

[0015] In embodiments of the present invention, a lip-sync scoring function may be present. For example, a fully connected neural network may act as a softmax layer to receive translated lip-sync videos labeled or classified as lip-sync. The lip-sync score may be a measure of phoneme accuracy relative to predicted lip landmarks, vocal stylization relative to facial expressions, or timing of facial / lip landmarks compared to predicted phonemes, or a combination thereof. In situations where multiple translated lip-sync videos are labeled as lip-sync, the lip-sync score may enable a decision as to which generated video should be presented.

[0016] When describing the embodiments in detail with reference to the drawings, references in the specification to "embodiments," "other embodiments," etc., indicate that the described embodiment may include a particular feature, structure, or characteristic, but it should be noted that not all embodiments may necessarily include that particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, when describing a particular feature, structure, or characteristic with respect to an embodiment, one skilled in the art has the knowledge to affect such feature, structure, or characteristic with respect to other embodiments, whether or not explicitly described.

[0017] 1 is a functional block diagram illustrating a high-level video translation and lip sync optimization environment 100. Shown within the video translation and lip sync optimization environment 100 are a server 102 and a network 120. Also shown in FIG. 1 is a video translation and lip sync optimization engine 110.

[0018] The server 102 can be a standalone computing device, an administrative server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, transmitting, and processing data. In other embodiments, the server 102 can represent a server computing system utilizing multiple computers. While FIG. 1 shows one server and one client computer, it should be noted that the video translation and lip sync optimization environment 100 can have any number of servers and client computers (e.g., 1, 2, n...n+1). In another embodiment, the server 102 can be a laptop computer, a tablet computer, a netbook computer, a personal computer, a desktop computer, or any programmable electronic device capable of communicating with other computing devices (not shown) in the video translation and lip sync optimization environment 100 via the network 120.

[0019] In another embodiment, the server 102 represents a computing system utilizing clustered computers and components (e.g., database server computers, application server computers, etc.) that can act as a single pool of seamless resources when accessed within the video translation and lip sync optimization environment 100. The server 102 can include internal and external hardware components as shown and described in further detail with respect to FIG.

[0020] In embodiments, the video translation and lip-sync optimization engine 110 may be a computer program that generates, from a video of a speaker speaking in a source language, a translated video of the speaker in a target language with lip and mouth movements synchronized to the generated translation. The video translation and lip-sync optimization engine 110 may have multiple computer modules (described in more detail below) for receiving a source video of the speaker in the target language, generating multiple best translations in the target language, generating a video of each translation with lip-sync to the translation, classifying each translation as synchronous or asynchronous, and generating a lip-sync score for any generated video classified as synchronous.

[0021] The network 120 may be, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the two, and may include wired, wireless, or fiber optic connections. In general, the network 120 may be any combination of connections and protocols that support communication between the server 102 and other computing devices (not shown) in the video translation and lip sync optimization environment 100.

[0022] 2 is a block diagram of the video translation and lip sync optimization engine 110. Illustrated in the video translation and lip sync optimization engine 110 are an audio extraction module 202, an automatic speech recognition module 204, a neural machine translation module 206, a text-to-speech module 208, a generative adversarial network 210, a video extraction module 216, and lip sync scoring 218.

[0023] The audio extraction module 202 is a computer module that can extract audio data from a video of a speaker. Additionally, in some embodiments, the audio extraction module 202 can separate natural language speech from other noises or sounds in the audio data from the source video. For example, the video translation and lip sync optimization engine 110 can receive a video of a speaker. The video can have audio data of the speaker and ambient background noise (e.g., birds chirping, dogs barking, passing vehicles). The audio extraction module 202 can separate the natural language speech or ambient noise based on recognizable wave patterns associated with either data point.

[0024] The automatic speech recognition module 204 is a computer module capable of isolating natural language utterances and generating a text-based representation or transcript of the audio data. In an embodiment, the automatic speech recognition module 204 can isolate natural language utterances in the audio data. The automatic speech recognition module 204 can identify phonemes based on recognizable patterns in the audio data and generate words based on the most likely combinations of phonemes. For example, the audio data can be fed into a neural network (e.g., a recurrent neural network, a long short-term memory, etc.), which can generate words based on the most likely combinations of phonemes. Furthermore, in some neural networks, word combinations can be determined based on the most likely word combinations or word associations.

[0025] The neural machine translation module 206 is a computer program capable of translating text in a source language into text in a target language. In an embodiment, the neural machine translation module 206 can receive text in the source language and generate multiple translation scripts. The number of translation scripts can be based on a beam search algorithm. The beam search algorithm is a trainable parameter that allows the neural machine translation module to translate source words into target words based on the source sentence. Multiple words in the target language can be determined for the input sentence (i.e., the top "k" words or beam width). The number of potential target words can allow multiple alternatives for the input sentence at each timestamp based on conditional probabilities and a dynamically determined or user-selected beam width (e.g., "k").

[0026] In an embodiment, the neural machine translation module 206 can generate the top "k" best translated test scripts in the target language based on a beamwidth "k" parameter optimized during training. The source text can be converted into an input vector. The input vector can be fed to an encoder made up of multiple nodes (e.g., a recurrent neural network, a translator, etc.) to create an encoding vector for each term, entity, and / or concept. The encoding vector can be fed to a decoder made up of multiple nodes (e.g., a recurrent neural network, a translator, etc.). The output vector can be fed to a fully connected neural network trained to predict the most likely word or words based on the output vector. Furthermore, in another embodiment, the fully connected layer can be optimized to predict the best words or the top "k" scripts based on the output vector.

[0027] The text-to-speech module 208 is a computer module that can receive text and / or a transcript in a target language and generate audio. In embodiments, the text-to-speech module 208 can generate phonemes associated with the text in the translated text or transcript to generate audio data in the target language. In embodiments, the text-to-speech module 208 can be trained with a corpus of audio data associated with speakers in the source language. The text-to-speech module 208 can be trained to accurately replicate the speaker's intonation, pace, and voice stylization based on the phoneme combinations of the source language to replicate the source video being translated.

[0028] The generative adversarial network 210 is a computer module that can have two separate sub-models operating on it. The generator sub-model 212 is a neural network that can receive video data of a speaker and synthesize a new image of the speaker with mouth and lip positions that correspond to the positions of the speaker and lip positions when speaking the transcript. In an embodiment, the generator sub-model 212 can generate still images associated with timestamps of the audio from the text-to-speech module, in which the speaker's mouth movements match the phonemes associated with the translated audio. Multiple still images are created and combined to create a seamless motion picture in which the speaker's mouth / lips are synchronized with the audio.

[0029] In an embodiment, the generator sub-model 212 may consist of multiple layers of a neural network. For example, mouth landmarks from a speaker may be identified and fed into a fully connected neural network. The fully connected neural network may then be followed by multiple layers of a convolutional neural network to generate synthetic images with accurate mouth positioning.

[0030] The classification sub-model 214 is a neural network that can identify whether an image or video is synchronized with associated audio. In an embodiment, the classification sub-model 214 can receive one or more generated still images and associated audio timestamps. The classification sub-model 214 can identify whether the lip or mouth shape is correct based on the phonemes associated with the audio. For example, the classification sub-model 214 can have multiple neural network layers, such as a multi-layer convolutional neural network, for analyzing input images and words / phonemes. The multi-layer convolutional neural network can be followed by an output softmax layer that receives the input of the final convolutional neural network and is configured to classify the image or images as synchronized or out-of-sync (i.e., 1 or 0).

[0031] The video extraction module 216 may be a computer module capable of isolating visual data from a source video. The video extraction module 216 may extract still images and timestamps associated with each image. Furthermore, the video extraction module 216 may provide the extracted visual data to the generative adversarial network 210.

[0032] The lip sync scoring module 218 is a computer module that can generate a lip sync score for a translated video generated by the generative adversarial network 110 that is classified as synchronous. In an embodiment, the lip sync scoring module 218 can have a fully connected neural network that can receive a translation of the generated video that is classified as synchronous. The neural network can have an output softmax layer configured to generate a lip sync score (e.g., the probability that the corresponding translation is correct or the best translation). The translation can be fed into the fully connected neural network.

[0033] FIG. 3 illustrates an exemplary flow for training a cyclic generative adversarial network within the lip sync optimization environment 100, generally designated 300, in accordance with an embodiment of the present invention. A source video 302 may be provided to the audio extraction module 202 and the video extraction module 216. The source video 302 may be video from a corpus of videos characterizing speakers in a source language. The source video 302 may have a known best translation or multiple known translations. The extracted audio data may be received by the automatic speech recognition module 204. The automatic speech recognition module 204 may generate text corresponding to the audio data. The neural machine translation module 206 may translate the generated text into multiple target language transcripts based on beamwidth. The translated text transcripts may be converted by the text-to-speech module 208 into a corresponding number of audio recordings in the target language. The video extraction module 216 may send the visual data extracted from the source video 302. The generator sub-model 212A can generate a lip-sync video for each translated audio recording. The classification sub-model 214A can classify each generated lip-sync video as in-sync or out-of-sync. A GAN loss A can be calculated from the generated video classified as out-of-sync, and the generator sub-model 212A can be updated using the GAN loss A.

[0034] The generated video 304 can be fed into the same network architecture, but the modules are configured to reverse the process described in the previous paragraph. Here, the automatic speech recognition module 204 is configured to recognize a target language, the neural machine translation model 206 is configured to translate text in the target language into a configurable number of source language transcripts, and the text-to-speech module 208 is configured to convert text in the source language into speech in the source language. However, the generative adversarial network B210B can generate lip-sync videos based on the generated speech, and the classification sub-model B214B can classify the generated videos as synchronized or out-of-sync. A GAN loss B can be calculated based on the generated videos classified as out-of-sync. Furthermore, a cycle consistency loss can be calculated based on a comparison of the videos generated by the generator sub-model B212B classified as synchronized by the classification sub-model B214B with the original source video 302.

[0035] FIG. 4 is a flowchart illustrating operational steps for lip sync optimization of natural language translation for a video 400. In step 402, a video in a source language may be received by the video translation and lip sync optimization engine 110. In step 404, the neural machine translation module 206 may generate one or more translations in a target language. In step 406, the generative adversarial network 210 may generate lip sync videos for each of the speech translations. In step 408, the generative adversarial network 210 may classify the generated lip sync videos as synchronous or out-of-sync. In decision step 410, the classification sub-model 214 may determine whether the generated lip sync videos are classified as synchronous (i.e., in sync). If the videos are classified as synchronous, the process proceeds to step 412. If the videos are classified as out-of-sync (i.e., out-of-sync), the process stops. In step 412, the lip sync scoring module 218 may generate lip sync scores for the generated lip sync videos classified as synchronous.

[0036] FIG. 5 illustrates an exemplary computer system 10, which represents a dynamically switching user interface computer 10. Computer system 10 includes a communications fabric 12 that provides communication between computer processors 14, memory 16, persistent storage 18, network adapters 28, and input / output (I / O) interfaces 26. Communications fabric 12 may be implemented using any architecture designed to pass data and / or control information between processors (such as microprocessors, communications and network processors), system memory, peripheral devices, and any other hardware components in the system. For example, communications fabric 12 may be implemented using one or more buses.

[0037] Computer system 10 includes processor 14, cache 22, memory 16, persistent storage 18, network adapter 28, input / output (I / O) interface 26, and communications fabric 12. Communications fabric 12 provides communication between cache 22, memory 16, persistent storage 18, network adapter 28, and input / output (I / O) interface 26. Communications fabric 12 may be implemented using any architecture designed to pass data and / or control information between processors (such as microprocessors, communications and network processors), system memory, peripheral devices, and any other hardware components in a system. For example, communications fabric 12 may be implemented using one or more buses or crossbar switches.

[0038] Memory 16 and persistent storage 18 are computer-readable storage media. In this embodiment, memory 16 includes random access memory (RAM) 20. Generally, memory 16 may include any suitable volatile or non-volatile computer-readable storage medium. Cache 22 is a high-speed memory that improves performance of processor 14 by retaining recently accessed data and data near recently accessed data from memory 16. As further shown and described below, memory 16 may include at least one of program modules 24 configured to perform the functions of embodiments of the present invention.

[0039] A program / utility having at least one program module 24 may be stored in memory 16, as may, for example and without limitation, an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networking environment. The program module 24 generally performs the functions and / or procedures of the embodiments of the present invention described herein.

[0040] Program instructions and data used to implement embodiments of the present invention may be stored in persistent storage 18 and memory 16 for execution by one or more of the respective processors 14 via cache 22. In an embodiment, persistent storage 18 includes a magnetic hard disk drive. Instead of, or in addition to, a magnetic hard disk drive, persistent storage 18 may include a solid-state hard drive, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0041] The media used by persistent storage 18 may be removable. For example, a removable hard drive may be used for persistent storage 18. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer-readable storage medium that is also part of persistent storage 18.

[0042] In these examples, network adapter 28 provides for communication with other data processing systems or devices. In these examples, network adapter 28 includes one or more network interface cards. Network adapter 28 may provide for communication through the use of either or both physical and wireless communication links. Program instructions and data used to implement embodiments of the present invention may be downloaded to persistent storage 18 through network adapter 28.

[0043] The I / O interface 26 allows for the input and output of data to and from other devices that may be connected to each computer system. For example, the I / O interface 26 may provide a connection to an external device 30, such as a keyboard, keypad, touch screen, or other suitable input device, or a combination thereof. The external device 30 may also include portable computer-readable storage media, such as thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to implement embodiments of the present invention may be stored on such portable computer-readable storage media and loaded into persistent storage 18 via the I / O interface 26. The I / O interface 26 also connects to a display 32.

[0044] Display 32 provides a mechanism for displaying data to a user and may be, for example, a computer monitor or a virtual graphic user interface.

[0045] Components described herein may be identified based on the application for which they are implemented in a particular embodiment of the invention. However, it should be understood that the naming of any particular component herein is for convenience only, and thus the invention should not be limited to use with only any particular application identified and / or implied by such naming.

[0046] The present invention may be a system, a method, or a computer program product, or a combination thereof, and may include a computer-readable storage medium having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0047] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or raised structures in grooves with instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as a transitory signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through fiber optic cable), or electrical signals transmitted over wires.

[0048] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to a computer-readable storage medium within the respective computing / processing device for storage.

[0049] Computer-readable program instructions for carrying out operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or conventional procedural programming languages, such as object-oriented programming languages ​​like Smalltalk® or C++, or conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the present invention.

[0050] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0051] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, causing the machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium and capable of instructing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that a computer-readable storage medium having instructions stored thereon includes a product containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0052] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0053] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. It should be noted that each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may possibly be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or a combination of dedicated hardware and computer instructions.

[0054] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0055] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort and service provider interaction. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0056] The characteristics are as follows:

[0057] On-demand self-service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed, without the need for human interaction with the service provider.

[0058] Broad Network Access: Capabilities are made available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0059] Resource Pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. Location independence is significant in that consumers typically have no control or knowledge over the exact location of the resources provided, but may be able to specify location at a higher level of abstraction (e.g., country, state, or data center).

[0060] Rapid Elasticity: Capacity can be rapidly and elastically provisioned, sometimes automatically, to instantly scale out, and rapidly released to instantly scale in. To the consumer, the capacity available for provisioning often appears unlimited and can be purchased in any quantity at any time.

[0061] Measured Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of utilized services.

[0062] The service model is as follows:

[0063] Software as a Service (SaaS): The capability offered to the consumer is the ability to use the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0064] Platform as a Service (PaaS): The ability offered to consumers is the ability to deploy applications they create or acquire, generated using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does exercise control over the deployed applications and, in some cases, the application hosting environment configuration.

[0065] Infrastructure as a Service (IaaS): The ability offered to consumers is the ability to provision processing, storage, networking, and other basic computing resources onto which they can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does exercise control over the operating systems, storage, deployed applications, and in some cases, limited control over select networking components (e.g., host firewalls).

[0066] The deployment model is as follows:

[0067] Private Cloud: The cloud infrastructure is operated exclusively for an organization. The cloud infrastructure may be managed by the organization or a third party and may be on-premise or off-premise.

[0068] Community Cloud: Cloud infrastructure is shared by multiple organizations to support a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). The cloud infrastructure may be managed by the organizations or a third party and may reside on-premise or off-premise.

[0069] Public Cloud: Cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0070] Hybrid Cloud: A cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain unique entities but are joined by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0071] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure comprising a network of interconnected nodes.

[0072] FIG. 6 is a block diagram illustrating a cloud computing environment 50 in accordance with at least one embodiment of the present invention. The cloud computing environment 50 includes one or more cloud computing nodes 40 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, or an automotive computer system 54N, or combinations thereof, may communicate. The nodes 40 can communicate with each other. They may be physically or virtually grouped in one or more networks (not shown), such as private, community, public, or hybrid clouds, or combinations thereof, as described above. This enables the cloud computing environment 50 to offer infrastructure, platform, or software, or combinations thereof, as a service without the cloud consumer having to maintain resources on their local computing device. The types of computing devices 54A-N shown in FIG. 6 are intended to be exemplary only, and it is understood that computing node 40 and cloud computing environment 50 may communicate with any type of computerized device over any type of network and / or network-addressable connection (e.g., using a web browser).

[0073] Figure 7 is a block diagram illustrating a set of functional abstraction model layers provided by the cloud computing environment 50 shown in Figure 6, in accordance with at least one embodiment of the present invention. It should be understood in advance that the components, layers, and functions illustrated in Figure 7 are intended to be merely exemplary, and embodiments of the present invention are not limited thereto. As illustrated, the following layers and corresponding functions are provided:

[0074] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, reduced instruction set computer (RISC) architecture-based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0075] The virtualization layer 70 provides an abstraction layer at which the following examples of virtual entities may be provided: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0076] In one example, the management layer 80 can provide the following functions: Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and provides billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides protection for data and other resources as well as identity verification for cloud consumers and tasks. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides cloud computing resource allocation and management to ensure requested service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides pre-allocation and procurement of cloud computing resources in anticipation of future requirements according to SLAs.

[0077] The workload layer 90 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instruction delivery 93, data analytics processing 94, transaction processing 95, and natural language translation and lip-sync optimization 96.

[0078] The description of various embodiments of the present invention has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the present invention. The terminology used herein has been selected to best explain the principles of the embodiments, their practical applications, or improvements of the technology over that found in the market, and to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for optimizing generated lip-sync video for neural machine translation by computer information processing, comprising: receiving, by a processor, a source video of a speaker in a source language; synchronizing multiple speech translations of the source video in a target language based on a neural machine translation model, the neural machine translation model being trained to synchronize a variable number of translations based on a variable beamwidth; generating a lip-sync video for each of the plurality of synchronized speech translations based on a generative sub-model within a generative adversarial network architecture; classifying each lip sync video as synchronous or asynchronous based on a classification sub-model in the generative adversarial network; generating a lip sync score for each lip sync video classified as synchronous; A method comprising:

2. Synchronizing the plurality of audio translations of the video in the target language comprises: extracting audio data from the source video; generating a transcript in the source language of the extracted audio data; generating at least one translation script in the target language based on the transcript in the source language, the number of translation scripts being based on the variable beam width; converting each translation script into speech based on a text-to-speech system; The method of claim 1 further comprising:

3. The method of claim 1 , further comprising training the generative adversarial network architecture to generate lip-sync videos.

4. Training the generative adversarial network architecture to generate lip sync videos includes: applying a corpus of audio and video data of speakers speaking in a source language to the initialized generative sub-models to generate lip-sync video in the target language, the corpus consisting of a plurality of original examples; generating lip-sync video data for each of the provided audio data in the target language; classifying each generated video data as synchronous or asynchronous; Computing a generative adversarial network loss for each generated video for the original examples; updating the generative sub-model using the calculated generative adversarial network loss; The method of claim 3 further comprising:

5. applying each generated lip-sync video in the target language to the initialized generative sub-model to generate a lip-sync video in the source language; generating secondary lip sync video data for each generated lip sync video in the source language; classifying each generated secondary video data as synchronous or asynchronous; Computing a second generative adversarial network loss for each generated video for the original examples; updating the generative sub-model initialized for generating lip syncs in the target language using the second generative adversarial network loss; The method of claim 4 further comprising:

6. 2. The method of claim 1, wherein the neural machine translation model is based on an auto-encoder model consisting of at least two recurrent neural networks and a fully connected neural network for generating probabilities of most likely translations.

7. Scoring each lip sync video classified as synchronous includes:

3. The method of claim 2, further comprising applying each lip sync video classified as synchronous to a fully connected neural network having a softmax output, the softmax output being a probability that the translated script and lip sync video are the most likely outcome.

8. 1. A computer system for optimizing generated lip sync video for neural machine translation, comprising: receiving a source video of a speaker in a source language; synchronizing multiple speech translations of the source video in a target language based on a neural machine translation model, the neural machine translation model being trained to synchronize a variable number of translations based on a variable beamwidth; generating a lip-sync video for each of the plurality of synchronized speech translations based on a generative sub-model within a generative adversarial network architecture; classifying each lip sync video as synchronous or asynchronous based on a classification sub-model in the generative adversarial network; generating a lip sync score for each lip sync video classified as synchronous; A computer system that performs the above.

9. Synchronizing the plurality of audio translations of the video in the target language comprises: extracting audio data from the source video; generating a transcript in the source language of the extracted audio data; generating at least one translation script in the target language based on the transcript in the source language, the number of translation scripts being based on the variable beam width; converting each translation script into speech based on a text-to-speech system; 9. The computer system of claim 8, further comprising:

10. The computer system of claim 8 , further comprising training the generative adversarial network architecture to generate lip-sync videos.

11. Training the generative adversarial network architecture to generate lip sync videos includes: applying a corpus of audio and video data of speakers speaking in a source language to the initialized generative sub-models to generate lip-sync video in the target language, the corpus consisting of a plurality of original examples; generating lip-sync video data for each of the provided audio data in the target language; classifying each generated video data as synchronous or asynchronous; Computing a generative adversarial network loss for each generated video for the original examples; updating the generative sub-model using the calculated generative adversarial network loss; The computer system of claim 10 further comprising:

12. applying each generated lip-sync video in the target language to the initialized generative sub-model to generate a lip-sync video in the source language; generating secondary lip sync video data for each generated lip sync video in the source language; classifying each generated secondary video data as synchronous or asynchronous; Computing a second generative adversarial network loss for each generated video for the original examples; updating the generative sub-model initialized for generating lip syncs in the target language using the second generative adversarial network loss; The computer system of claim 11 , further comprising:

13. 9. The computer system of claim 8, wherein the neural machine translation model is based on an auto-encoder model consisting of at least two recurrent neural networks and a fully connected neural network for generating probabilities of most likely translations.

14. Scoring each lip sync video classified as synchronous includes:

10. The computer system of claim 9, further comprising applying each lip sync video classified as synchronous to a fully connected neural network having a softmax output, the softmax output being a probability that the translated script and lip sync video are the most likely outcome.

15. 1. A computer program for optimizing generated lip sync video for neural machine translation, comprising: receiving a source video of a speaker in a source language; synchronizing multiple speech translations of the source video in a target language based on a neural machine translation model, the neural machine translation model being trained to synchronize a variable number of translations based on a variable beamwidth; generating a lip-sync video for each of the plurality of synchronized speech translations based on a generative sub-model within a generative adversarial network architecture; classifying each lip sync video as synchronous or asynchronous based on a classification sub-model in the generative adversarial network; generating a lip sync score for each lip sync video classified as synchronous; A computer program containing instructions for performing the following:

16. A computer-readable storage medium having the computer program according to claim 15 recorded thereon.

Images