Multilingual multimedia translation with synchronized audio and body gestures

Abstract

Multilingual multimedia translation by generating synchronized multimedia assets based on multimedia assets translated into different languages. Multilingual multimedia translation segments multimedia assets into audio segments and video segments. Multilingual translation utilizes audio segments to generate translated audio segments. Multilingual translation may utilize video segments to identify individual body gestures in video segments, generate translated body gestures, and generate translated lip movements. Using translated audio segments, and any translated lip movements or translated body gestures, multilingual translation generates synchronized multimedia assets.

Description

Background Technology

[0001] In video content, language barriers pose significant challenges to global communication and understanding. Traditional video translation methods focus primarily on verbal content, often ignoring important modes of non-verbal communication such as body language and gestures.

[0002] Current methods do not capture the nuances and cultural contexts embedded in body language that are essential for effective communication. This can lead to misunderstandings or misinterpretations of the original content. Existing systems often provide translations in the form of subtitles or dubbed audio that are not synchronized with the speaker's body gestures and facial expressions, resulting in a disconnected and less engaging viewing experience.

[0003] Content creators face difficulties in making their video content accessible and relevant to a global audience because they must consider not only language barriers but also cultural differences in non-verbal communication.

[0004] The problem of multilingual video translation is typically solved by using separate technologies for audio and subtitle translation. For example, subtitles and dubbed audio tracks are used to translate spoken language, but these solutions often lack synchronization with body gestures and facial expressions, resulting in a disconnected viewer experience. Brief explanation of the drawing

[0005] In order to make the manner in which the embodiments cited above are achieved understandable in detail, a more specific description of the embodiments described in this specification, briefly summarized above, may be provided with reference to the accompanying drawings. However, the attached drawings illustrate typical embodiments and, therefore, should not be considered as a limitation; it should be noted that other equally effective embodiments are considered. FIG. 1 illustrates a translation system for generating synchronized multimedia assets according to one embodiment. FIG. 2 illustrates a flowchart for creating synchronized multimedia assets according to one embodiment. FIG. 3 illustrates a workflow for implementing the flowchart described in FIG. 2 according to one embodiment. FIG. 4 illustrates a flowchart for identifying the difference between a first gesture and a translated gesture according to one embodiment. FIG. 5 illustrates a workflow for implementing the flowchart described in FIG. 4 according to one embodiment. FIG. 6 illustrates a flowchart for generating translated lip movements according to one embodiment. Specific details for implementing the invention

[0006] By providing a translation system that not only translates spoken words within videos into multiple languages ​​but also ensures that non-verbal modalities, such as gestures and facial expressions, are accurately synchronized and culturally adapted, the present disclosure addresses the task of providing translated media synchronized with the speaker's body gestures and facial expressions. This approach aims to make video content universally accessible and relevant.

[0007] The present disclosure describes a translation system that segments multimedia assets into video segments and audio segments. The translation system synchronizes the translated audio with body gestures within the video segments to ensure that both verbal and non-verbal communication are accurately conveyed across different languages. For example, in a video segment, an individual might say "There is an eagle over the house," and the individual points to "the house" as they speak. When the phrase "There is an eagle over the house" is translated into a target language, the order of words may shift around it, so when the individual points, the words do not match the emphasis of the word "the house" in the target language. To maintain the desired emphasis, the translation system synchronizes the translated audio with the pointing gesture.

[0008] In another exemplary embodiment, the translation system generates a translated gesture of an initial gesture performed by an individual within a multimedia asset based on the culture associated with the target language. For example, an individual nods their head to express agreement with a question posed to them. When the translation system translates the multimedia asset, the translation system checks whether the initial gesture has the same meaning in the culture associated with the target language. If the initial gesture has a different meaning, the translation system generates a translated gesture to match the meaning of the initial gesture in the target language. The translation system uses the generated translated gesture to replace the initial gesture within the multimedia asset. In an exemplary embodiment of the present disclosure, the translation system generates translated lip movements and synchronizes them with the translated audio, which enhances the user experience by making the translated content feel more natural.

[0009] FIG. 1 illustrates a system (100) which is a translation system for generating synchronized multimedia assets. The system (100) includes a processor (102) and memory (104). The processor (102) represents any number of processing elements, each of which may include any number of processing cores. The memory (104) may include volatile memory elements, non-volatile memory elements, and any combination thereof. The memory (104) includes a video processing engine (106), a gesture recognition engine (108), an Automatic Speech Recognition (ASR) engine (110), a machine translation engine (112), a rule-based engine (114), a Text-to-Speech (TTS) engine (116), a gesture synthesis model (118), and a motion synchronization model (120). Different software applications or software modules may exist within the same software application.

[0010] The video processing engine (106) segments a multimedia asset into video segments and audio segments. The multimedia asset may be .mp4, .avi, or any other video file format. The multimedia asset may have metadata describing, but not limited to, the language, culture, actors, genre, or period associated with the multimedia asset. When the video processing engine (106) segments the multimedia asset, the video segments generated may be .mp4, .avi, or any other video file format. In an exemplary embodiment, the video processing engine (106) generates multiple video segments, each of which is a different part of the entire multimedia asset. The audio segments generated by the video processing engine (106) may be .mp3, .wav, or any other audio file format. In an exemplary embodiment, the video processing engine (106) generates multiple audio segments, each of which is a different part of the entire multimedia asset.

[0011] The gesture recognition engine (108) captures gestures performed by individuals depicted in video segments from multimedia assets. In one embodiment, the gesture recognition engine (108) uses computer vision techniques along with databases of gestures to determine the gestures of individuals depicted in multimedia assets. For example, a video segment may depict an individual nodding their head in affirmation in response to a question. The gesture recognition engine (108) can identify that the individual is nodding their head up and down. In another example, a video segment may depict an individual waving their hand to say goodbye. The gesture recognition engine (108) can identify that the individual is moving their hand left and right. In an exemplary embodiment, the gesture recognition engine (108) captures the gestures of individuals in each of multiple video segments. In an exemplary embodiment, the gesture recognition engine (108) captures the gestures of different individuals in each of multiple video segments.

[0012] The ASR engine (110) generates a text transcription of an audio segment through systems such as DeepSpeech. In an exemplary embodiment, the ASR engine (110) receives an audio segment generated by a video processing engine (106). The ASR engine (110) can generate a text transcript of the audio segment from the video processing engine (106). In one embodiment, the ASR engine (110) receives a language input describing the language of the audio segment. Using the language input, the ASR engine (110) can determine the language to be used to generate a transcript of the audio segment. In an exemplary embodiment, the ASR engine (110) receives an acoustic model indicating a specific dialect or intonation of the language. Then, the ASR engine (110) can use the specific dialect or intonation of the language to generate a text transcript of the audio segment. In an exemplary embodiment, the ASR engine (110) receives information about the individual speaking in the audio segment, such as the individual's age, gender, etc.

[0013] A machine translation engine (112) receives a text transcript in one language and translates the text transcript into another language. The machine translation engine (112) receives an untranslated text transcript in the first language and a target language for translating the text transcript as input. In an exemplary embodiment, the machine translation engine (112) receives a text transcript generated by an ASR engine (110) for translation. The machine translation engine (112) outputs a translated text transcript in the target language.

[0014] The rule-based engine (114) examines a text transcript of an audio segment or a body gesture within a video segment for linguistic accuracy, cultural accuracy, or to determine the meaning of the body gesture. The rule-based engine (114) receives the text transcript or body gesture as input along with a target language. In an exemplary embodiment, the target language used by the rule-based engine (114) is the same target language as for the machine translation engine (112). In an exemplary embodiment, the rule-based engine (114) receives the culture related to the text transcript or body gesture as input. In an exemplary embodiment, the rule-based engine (114) receives the culture related to the text transcript or body gesture and the culture related to the multimedia asset as input. In an exemplary embodiment, instead of receiving it as input, the rule-based engine (114) determines the culture related to the text transcript or body gesture. In one embodiment, the rule-based engine (114) accesses a repository of predefined rules, cultural norms, or gestures related to the target language to check language accuracy, cultural accuracy, or to determine the meaning of a body gesture.

[0015] In an exemplary embodiment, the rule-based engine (114) receives a transcribed text of the translated text and the target language from the machine translation engine (112). The rule-based engine (114) accesses predefined rules for the target language, such as diacritics used for characters in the target language or grammar for the target language. In an exemplary embodiment, the rule-based engine (114) receives a gesture from gestures detected by the gesture recognition engine (108) and the culture associated with the gesture in a specific country or region. The rule-based engine (114) accesses the gestures associated with the country or region to determine the meaning of the gesture detected by the gesture recognition engine (108). For example, if the gesture was an individual shaking their head from side to side, the rule-based engine (114) would examine the gestures associated with one country and determine that shaking the individual's head from side to side means "Yes," but in another country, the same gesture means "No."

[0016] The TTS engine (116) generates audio using a text transcript. In an exemplary embodiment, the TTS engine (116) receives a translated text transcript, which is examined by a rule-based engine (114), as input. The TTS engine (116) inputs the translated transcript into a generative machine learning (ML) model to generate a translated audio segment. In an exemplary embodiment, the TTS engine (116) inputs the translated transcript into a generative ML model trained with the individual voice from the original audio segment to generate a translated audio segment. In an exemplary embodiment, the TTS engine (116) receives a translated text transcript generated by a machine translation engine (112) as input.

[0017] The gesture synthesis model (118) generates a translated gesture based on the original gesture using methods such as generative adversarial networks (GANs) or variational autoencoders (VAEs). In an exemplary embodiment, the gesture synthesis model (118) receives the original gesture from the gesture recognition engine (108), the meaning of the original gesture in the original culture, and the meaning of the original gesture in the culture related to the target language from the rule-based engine (114). The gesture synthesis model (118) checks to determine whether the original gesture has the same meaning between the original culture and the culture related to the target language. If the meanings are the same, the gesture synthesis model (118) outputs the original gesture. If the meanings are different, the gesture synthesis model (118) generates a translated gesture. The translated gesture has a meaning equivalent to that of the original gesture but is depicted in the culture related to the target language. In an exemplary embodiment, the gesture synthesis model (118) uses a trained ML model to generate a translated gesture based on the gesture. In an exemplary embodiment, the individual speaking in the original audio segment is not the same as the person describing the gesture in the video segment. For example, the first person described in the video segment may say "farewell," and the second person described in the video segment may perform a wave gesture in response to the first person's expression. The TTS engine (116) will translate the audio of the first person, and the gesture synthesis model (118) can generate a translated gesture for the second person's gesture.

[0018] The motion synchronization model (120) generates synchronized multimedia asset translated content having video segments through systems such as a first-order motion model. In an exemplary embodiment, the motion synchronization model (120) generates synchronized multimedia assets using translated audio segments and video segments. For example, the motion synchronization model (120) may receive translated audio segments from a TTS engine (116), gestures identified by a gesture recognition engine (108), and video segments and original audio segments from a video processing engine (106). The motion synchronization model (120) may determine the temporal alignment between the original audio segments and the identified gestures from the gesture recognition engine (108). The motion synchronization model (120) uses the temporal alignment to generate synchronized multimedia assets by synchronizing the gestures within the video segments with the translated audio segments. In another exemplary embodiment, the motion synchronization model (120) may receive a translated gesture from the gesture synthesis model (118) and a video segment from the video processing engine (106). The motion synchronization model (120) may replace a gesture within a portion of the video segment with a translated gesture.

[0019] The user interface (150) receives and displays multimedia assets. In an exemplary embodiment, when a user wishes to view multimedia assets, the user interface (150) may receive multimedia assets that depict a first gesture (152) and an output phrase (154). For example, the multimedia assets may depict a first gesture (152) of an individual raising their hand, and the individual saying the output phrase (154), "please stop the car here" in English. In an exemplary embodiment, the user wishes to view multimedia assets in a target language. As part of translating the multimedia assets into the target language, the original audio segment of the multimedia assets is translated into the target language. The translated audio segment of the multimedia assets may not be synchronized with the gestures within the video segment of the multimedia assets. For example, using the output phrase (154), "please stop the car here," from the previous version, the phrase may be spoken in a different order in the target language, such as "stop the car here, please." By changing the order of the words in the output phrase (154), the gestures of the individual speaking the output phrase (154) in the video segment will not emphasize the exact part of the output phrase (154) (i.e., the translated audio and gestures are no longer aligned in time). In an exemplary embodiment, the motion synchronization model (120) determines the temporal alignment between the original audio segment and the individual's gesture in the video segment by linking the individual's gesture in the video segment to a syllable, word, or phrase in the original audio segment. The motion synchronization model (120) determines the temporal alignment by determining the time at which the gesture occurs in the video segment and the word or phrase spoken at that time in the audio segment.By obtaining a temporal alignment between the original audio segment and the gesture, the motion synchronization model (120) can generate a synchronized multimedia asset having a gesture and a translated audio segment that match a linked syllable, word, or phrase. Next, the synchronized multimedia asset is transmitted to a user interface (150) for the user to view in the target language. Using the example above, the first gesture (152) of an individual raising his hand would be the same gesture as the gesture within the video segment from the multimedia asset, but the gesture would be depicted at a different time in the video to be temporally aligned with the word "stop". Additionally, the phrase output (154) is changed from "please stop the car here" in English to "stop the car here, please" in the target language.

[0020] Depending on the target language, an individual's gestures in a video segment may not match the meaning of the translated audio segment. For example, if an individual is described as nodding his head to say "Yes," in the target language, the gesture for saying "Yes" might be shaking his head from side to side. To correct this issue, the individual's gesture in the video segment will be replaced. In an exemplary embodiment, the multimedia asset is translated into the target language, and using a gesture synthesis model (118), a translated gesture is generated that matches the meaning of the first gesture (152) in the target language. A motion synchronization model (120) synchronizes the translated gesture with the translated audio and the rest of the video segment depicting the first gesture (152) to generate the synchronized multimedia asset. Next, the synchronized multimedia asset is transmitted to a user interface (150) for the user to view in the target language. This results in the first gesture (152) being replaced with the translated gesture and the output phrase (154) being replaced with the translated audio.

[0021] In an exemplary embodiment, instead of replacing the first gesture (152) with the translated gesture, the multimedia asset may depict the avatar (170) performing the translated gestures. For example, the avatar (170) may depict the translated audio in the target language or sign language for the translated gestures. In an exemplary embodiment, the avatar (170) is created using a generative ML model to depict the translated audio in the target language or sign language corresponding to the translated gestures. In an exemplary embodiment, the motion synchronization model (120) synchronizes the translated audio with the avatar (170) and the translated gestures to create the synchronized multimedia asset.

[0022] For clarity, the flowchart in FIG. 2 is discussed together with the workflow exemplified in FIG. 3. FIG. 2 illustrates a flowchart for generating synchronized multimedia assets. In block (202) of method (200), the video processing engine (106) receives the multimedia assets to be translated as input. In block (204), the video processing engine (106) segments the multimedia assets into video segments and audio segments. As illustrated in method (300), the video segments are output from the video processing engine (106) to the gesture recognition engine (108). The audio segments are output from the video processing engine (106) to the ASR engine (110).

[0023] In block (206), the gesture recognition engine (108) identifies a first gesture (152) performed by an individual as depicted in a video segment. As described above, the gesture recognition engine (108) uses computer vision techniques to identify the first gesture (152) performed by an individual. In an exemplary embodiment, the gesture recognition engine (108) outputs the identified gesture to a motion synchronization model (120). For example, the gesture recognition engine (108) may receive a video segment depicting an individual moving his hand from side to side. The gesture recognition engine (108) identifies that the individual is performing a shaking gesture.

[0024] In block (208), the TTS engine (116) generates a translated audio segment, such as a text transcription of an audio segment generated by the video processing engine (106). In an exemplary embodiment, the ASR engine (110) outputs the generated text transcription to a machine translation engine (112). The machine translation engine (112) translates the generated text transcription into a target language provided as input to the machine translation engine (112). In an exemplary embodiment, the machine translation engine (112) outputs the translated text transcription to a rule-based engine (114). The rule-based engine (114) examines the translated text transcription with predefined rules related to the target language, such as grammar, and predefined rules related to the cultural input related to the translated text transcription, such as dialect or intonation related to the culture. The rule-based engine (114) outputs the translated text transcription, having any modifications based on the predefined rules, to the TTS engine (116). The TTS engine (116) uses the translated text transcription to generate the translated audio segment.

[0025] In block (210), the system (100) determines the temporal alignment between a portion of an audio segment and an identified gesture. In an exemplary embodiment, the motion synchronization model (120) receives an audio segment from a video processing engine (106) and a gesture identified by a gesture recognition engine (108). For example, the motion synchronization model (120) receives an audio segment in which an individual says "I would like that slice of pizza" and an identified gesture that the individual is pointing to. In another example, the motion synchronization model (120) receives an audio segment in which an individual says "here is your change" and an identified gesture in which the individual holds money in his palm and extends his arm. The motion synchronization model (120) determines the temporal alignment between the timing of the audio segment and the timing of the identified gesture. In an exemplary embodiment, the motion synchronization model (120) determines the temporal alignment between the identified gesture and a syllable, word, or phrase within the audio segment. Using an example of a person pointing, the motion synchronization model (120) determines the temporal alignment between when the person says "I would like that slice of pizza" and when the person performs the pointing gesture. In this example, there is a temporal alignment between when the person starts pointing and when the person says the word "that." Using an example of a person extending his arm, the motion synchronization model (120) determines the temporal alignment between when the person says "here is your change" and when the person extends his arm. In this example, there is a temporal alignment between when the person starts extending his arm and when the person says the word "here."

[0026] In block (212), the system (100) generates synchronized multimedia assets. In an exemplary embodiment, the motion synchronization model (120) receives a video segment from the video processing engine (106), a gesture identified by the gesture recognition engine (108), and a translated audio segment generated by the TTS engine (116). When the audio segment is translated into a target language, the order of words may be rearranged so that the intended emphasis from the identified gesture does not match the timing of the translated audio segment. To correct the desync between the identified gesture and the translated audio segment, the motion synchronization model (120) uses the timing determined in block (210) to synchronize the timing of the translated audio segment with the gesture depicted in the video segment by aligning the translated audio segment temporally with the gesture depicted in the video segment so that the gesture emphasizes the same word in the translated audio segment as the gesture performed in the original audio segment. By aligning the translated audio segment temporally with the gesture described in the video segment, the motion synchronization model (120) can generate a synchronized multimedia asset. If an individual uses a pointing example, the pointing gesture will emphasize "would" within the translated audio segment, so the motion synchronization model (120) determines that the translated audio segment for the phrase "I would like that slice of pizza" is desynchronized with the pointing gesture. The motion synchronization model (120) uses temporal alignment from the block (210) to synchronize the gesture pointing to the translated audio segment so that the pointing gesture emphasizes the word "that" within the translated audio segment. In the synchronized multimedia asset, the pointing gesture is temporally aligned with the word "that" within the translated audio segment of the phrase "I would like that slice of pizza".The motion synchronization model (120) bases the word within the phrase for when the gesture occurs in the original audio segment on the basis of synchronization. In this example, since the motion synchronization model (120) has a temporal alignment that correlates with the gesture pointing to the word "that," the motion synchronization model (120) synchronizes the translated audio segment to align the gesture pointing to the translated word for "that." Using the example of an individual extending his arm, since the arm-extending gesture will emphasize "your" within the translated audio segment, the motion synchronization model (120) determines that the translated audio segment for the phrase "here is your change" is desynchronized with the arm-extending gesture. The motion synchronization model (120) uses the temporal alignment from the block (210) to synchronize the translated audio segment with the arm-extending gesture so that the arm-extending gesture emphasizes the word "here" within the translated audio segment. In the synchronized multimedia asset, the arm-reaching gesture is temporally aligned with the word "your" within the translated audio segment of the phrase "here is your change". In an exemplary embodiment, the synchronized multimedia asset has the same video as the multimedia asset input to the video processing engine (106) along with the translated audio from the target language.

[0027] For clarity, the flowchart in FIG. 4 is discussed together with the workflow exemplified in FIG. 5. FIG. 4 illustrates a flowchart for identifying differences between the meanings of a first gesture in two different cultures. In one embodiment, the method starts at block (402), where the video processing engine (106) accesses a multimedia asset. In one embodiment, the method (400) is performed after block (206) of the method (200). That is, the method (400) may be performed with the gesture / translated audio synchronization techniques described in FIG. 2, and instead of the method (400) starting at block (402), block (404) will be performed after block (206). However, in other embodiments, the method (400) may be a standalone method performed when converting a media presentation between two different cultures having different gestures for the same meaning. This may involve performing audio translation (where cultures have different languages) or two cultures may share the same language, in which case the audio is not translated but the gestures are translated.

[0028] In block (404), the gesture recognition engine (108) identifies a first gesture performed by an individual. As illustrated in FIG. 5, the video processing engine (106) receives multimedia assets as input. The video processing engine (106) segments the multimedia assets into video segments and audio segments. In an exemplary embodiment, the video processing engine (106) outputs the video segments to the gesture recognition engine (108). The gesture recognition engine (108) uses computer vision techniques, such as pose estimation, face tracking, hand gesture detection, or a combination thereof, to determine the first gesture (152) depicted by an individual in the video segments. The method (400) identifies the meaning of the first gesture (152) in the original culture corresponding to the original language of the audio by comparing it with the meaning of the first gesture (152) in the culture associated with the target language. In an exemplary embodiment, the gesture recognition engine (108) outputs the first gesture (152) to the rule-based engine (114) to determine the meaning of the first gesture (152) in the culture corresponding to the original language. The rule-based engine (114) determines the meaning of the first gesture (152) in the original culture by identifying predefined cultural norms based on the original culture. For example, the rule-based engine (114) may receive a gesture of an individual shaking his head from the gesture recognition engine (108) along with the original language. The rule-based engine (114) identifies the culture associated with the original language and determines that when the individual shakes his head, the gesture signifies that the individual is not agreeing.

[0029] In block (406), the system (100) determines a second gesture in a second culture that is different from a first gesture (152) in the original culture through a rule-based engine (114). The rule-based engine (114) may use language databases, expert-defined rules, ML models, context-adaptive techniques, or a combination thereof to enumerate cultural norms and identify the meaning of the gesture. For example, the rule-based engine (114) may use gesture databases, Ekman’s emotion and gesture studies, or any other language-specific corpora defining universal body language patterns. The rule-based engine (114) may be further refined by crowdsourced user feedback by allowing users to annotate and validate gesture-voice pairs. In an exemplary embodiment, the rule-based engine (114) receives a first gesture (152) and a culture associated with the target language as input. The rule-based engine (114) determines the meaning of the first gesture (152) in the culture associated with the target language by identifying predefined cultural norms based on the culture associated with the target language.

[0030] The system (100) determines whether the first gesture (152) has a different meaning between the original culture and the culture related to the target language. The system (100) determines the culture related to the target language based on geographic regions, user preferences, content metadata, and linguistic variations. When a language such as English is spoken across multiple cultures, the system (100) may use a hierarchical decision process to select the most relevant cultural adaptation. For example, the system (100) may use geographic metadata embedded within the source video, such as location tags or user-defined settings, to help determine the intended audience. Where explicit geographic metadata is not available, the system (100) uses linguistic markers, such as regional dialects, pronunciation styles, or lexical differences in multimedia assets, to infer the cultural context. Additionally, since the user or content creator may specify a preferred cultural adaptation, the system will prioritize the selected culture to determine the meaning of the gestures. In cases where the cultural context remains ambiguous, the system (100) may adopt default settings for determining the culture associated with the target language based on the content type of the multimedia asset. For example, if the multimedia asset is shared globally, the system (100) may select the most neutral or widely accepted cultural adaptation to ensure that gestures and voice patterns remain understandable to a broad audience. However, for region-specific content, the system (100) may prioritize the dominant cultural norms of that region.Referring to the example where an individual shakes his head, the rule-based engine (114) identifies the culture associated with the target language and determines that when the individual shakes his head, the gesture signifies that the individual agrees in the culture associated with the target language. In an exemplary embodiment, the rule-based engine (114) outputs the meaning of the first gesture (152) in the original culture and the meaning of the first gesture (152) in the culture associated with the target language to the gesture synthesis model (118). The gesture recognition engine (108) outputs the first gesture (152) to the gesture synthesis model (118). In an exemplary embodiment, the gesture synthesis model (118) compares the meanings of the first gesture (152) in the original culture with the meaning of the first gesture (152) in the culture associated with the target language. In one embodiment, if there is no difference between the meanings of the gestures, the method (400) proceeds back to the block (212) in FIG. 2 having the first gesture (152). That is, the method (400) does not change the gesture.

[0031] If there is a difference between the meaning of the first gesture (152) in the original culture and the culture related to the target language, the method (400) proceeds to block (408). In an exemplary embodiment, the gesture synthesis model (118) generates a translated gesture based on the first gesture (152) using a generative ML model, such as GANs or VAEs. To generate the translated gesture, the gesture synthesis model (118) inputs the gesture in the culture related to the target language and the first gesture (152), which correspond to the meaning of the original culture, into the generative ML model. In an exemplary embodiment, the gesture synthesis model (118) outputs the generated translated gesture to the motion synchronization model (120). The motion synchronization model (120) synchronizes the translated gesture with the video segment from the video processing engine (106) by replacing the first gesture (152) in the video segment with the translated gesture. The original gesture is replaced in the video segment by replacing the individual in the original video segment with an individual depicting the translated gesture. In an exemplary embodiment, the motion synchronization model (120) synchronizes the translated gesture with the video segment from the video processing engine (106) by adding an avatar in the video segment depicting the translated gesture as described above.

[0032] FIG. 6 illustrates a flowchart for generating translated lip movements. In block (602), the system (100) receives a translated audio segment. In an exemplary embodiment, the translated audio segment is generated by a TTS engine (116) based on an audio segment from a video processing engine (106). In an exemplary embodiment, block (602) is performed after block (208) of the method (200).

[0033] In block (604), the system (100) generates translated lip movements based on the translated audio segment. In an exemplary embodiment, the TTS engine (116) outputs the translated audio segment to a generative ML model, such as a primary motion model, to generate lip movements that match the translated audio segment. After generating lip movements that match the translated audio segment, the original lip movements can be cut from the video segment and replaced with the translated lip movements. In an exemplary embodiment, the generative ML model receives the translated audio segment and the translated language as input. In an exemplary embodiment, the generative ML model may receive the lip movements from the original video segment as input. By receiving the lip movements from the original video segment, the generative ML model can maintain an attitude between the original lip movements and the translated lip movements. In an exemplary embodiment, the method (600) is performed simultaneously with at least one of method (200) or method (400). For example, the system (100) can generate translated lip movements and determine the temporal alignment between the translated audio segment and the first gesture. In another example, the system (100) can generate translated lip movements and generate a translated gesture.

[0034] In block (606), the system (100) inserts translated lip movements into a multimedia asset. In an exemplary embodiment, the generative ML model outputs the translated lip movements to a motion synchronization model (120) along with the multimedia asset input to the video processing engine (106). The motion synchronization model (120) replaces the lip movements in the multimedia asset with the translated lip movements.

[0035] In the present disclosure, reference is made to various embodiments. However, it should be understood that the present disclosure is not limited to the specific embodiments described. Instead, any combination of the following features and elements is considered to implement and practice the teachings provided herein, whether or not they relate to different embodiments. Additionally, when elements of the embodiments are described in the form of “at least one of A and B,” it will be understood that embodiments including element A exclusively, element B exclusively, and elements A and B are considered, respectively. Furthermore, while some embodiments may achieve advantages over other possible solutions or prior art, whether or not a particular advantage is achieved by a given embodiment is not a limitation of the present disclosure. Accordingly, the aspects, features, embodiments, and advantages disclosed herein are merely illustrative and, except as expressly cited in the claim(s), are not the elements or limitations considered in the appended claims. Likewise, references to "invention" shall not be interpreted as a generalization of any inventive essence disclosed herein, and shall not be considered as elements or limitations of the appended claims, except as expressly cited in the claim(s).

[0036] As recognized by those skilled in the art, the embodiments described herein may be implemented as systems, methods, or computer program products. Accordingly, the embodiments may take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects that may all generally be referred to herein as “circuits,” “modules,” or “systems.” Furthermore, the embodiments described herein may take the form of computer program products implemented on one or more computer-readable media(s) on which computer-readable program code is implemented.

[0037] Program code implemented on a computer-readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, fiber optic cable, RF, etc., or any suitable combination of the above.

[0038] Computer program code for performing operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or a connection to an external computer may be made (e.g., via the Internet using an Internet service provider).

[0039] Aspects of the present disclosure are described herein with reference to flowchart examples or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart examples or block diagrams, and combinations of blocks in the flowchart examples or block diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a general-purpose computer, a special-purpose computer, or a processor of another programmable data processing device to create a machine such that instructions executed through a processor of a computer or another programmable data processing device create means for implementing functions / acts specified in the block(s) of the flowchart examples or block diagrams.

[0040] These computer program instructions, which can instruct a computer, another programmable data processing device, or other device to function in a specific manner to produce a manufactured article containing instructions that implement a specific function / act in the block(s) of the flowchart examples or block diagrams, may also be stored in a computer-readable medium.

[0041] Computer program instructions may also be loaded onto a computer, another programmable data processing device, or another device so that a series of operational steps are performed on the computer, another programmable data processing device, or another device to create a computer-implemented process, so that instructions executed on the computer, other programmable data processing device, or other device provide processes for implementing functions / acts specified in the block(s) of the flowchart examples or block diagrams.

[0042] Examples of flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible embodiments of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the examples of flowcharts or block diagrams may represent a module, segment, or part of code containing one or more executable instructions for implementing a specified logical function(s). It should also be noted that in some alternative embodiments, the functions mentioned in the blocks may occur regardless of the order mentioned in the drawings. For example, two blocks shown in succession may actually, substantially simultaneously, be executed, or the blocks may be executed in reverse order or out of order, depending on the functionality involved. It will also be noted that each block in the block diagrams or examples of flowcharts, and combinations of blocks in the block diagrams or examples of flowcharts, may be implemented by special-purpose hardware-based systems that perform the specified functions or acts, or by combinations of special-purpose hardware and computer instructions.

[0043] The foregoing relates to embodiments of the present disclosure, but other and additional embodiments of the disclosure may be devised without departing from the basic scope thereof, the scope thereof is determined by the following claims.

Claims

Claim 1 A method comprising: accessing a multimedia asset describing an individual speaking in a first language among a plurality of languages; extracting an audio segment of the individual speaking in the multimedia asset; identifying a first gesture described in the multimedia asset during a first part of the audio segment; determining a first temporal alignment between an utterance made in the first part of the audio segment and the first gesture; and generating a synchronized multimedia asset based on at least one of modifying the temporal position of the first gesture in the multimedia asset, or modifying the timing of the translated audio segment to establish a second temporal alignment between the first gesture and the translation of the utterance in the translated audio segment. Claim 2 The method of claim 1 further comprises: a step of identifying the meaning of the first gesture for the first language based on a first language ontology; a step of identifying a translated gesture having a meaning consistent with the first gesture for the target language based on a target language ontology; and a step of modifying the multimedia asset to depict the translated gesture while maintaining the first temporal alignment. Claim 3 In paragraph 2, the step of modifying the multimedia asset to describe the translated gesture comprises the step of processing the multimedia asset using a generative machine learning model to generate the synchronized multimedia asset, wherein the synchronized multimedia asset describes the individual performing the translated gesture. Claim 4 In paragraph 2, the step of modifying the multimedia asset to depict the translated gesture comprises: the step of creating an avatar that performs the translated gesture; and the step of adding the avatar to the synchronized multimedia asset. Claim 5 A method according to claim 1, wherein the step of generating the synchronized multimedia asset based on modifying the timing of the translated audio segment comprises the step of aligning the first gesture with the translated audio segment such that the first gesture performed during a first part of the audio segment is performed during a corresponding part of the translated audio segment. Claim 6 A method according to claim 1, further comprising the step of segmenting the multimedia assets into a plurality of video segments. Claim 7 In claim 1, the first part of the audio segment is a syllable, a word, or a phrase. Claim 8 A method according to claim 1, further comprising the steps of generating the individual’s translated lip movements to align with the translated audio segment, and adding the individual’s translated lip movements to the synchronized multimedia asset. Claim 9 In claim 8, the method wherein the translated lip movement is based on the original lip movement in the first part of the audio segment. Claim 10 A method comprising: accessing a multimedia asset depicting an individual from a first culture; identifying a first gesture performed by the individual in the multimedia asset; determining a second gesture in a second culture having a meaning consistent with the meaning of the first gesture in the first culture, wherein the first gesture and the second gesture are different; and adding the second gesture to the multimedia asset to modify the multimedia asset to reflect the second culture. Claim 11 In claim 10, the step of adding the second gesture to the multimedia asset comprises the step of replacing the first gesture performed by the individual with the second gesture in the multimedia asset. Claim 12 In claim 10, the step of adding the second gesture to the multimedia asset comprises: creating an avatar that performs the second gesture; and adding the avatar to the modified multimedia asset. Claim 13 A method according to claim 10, wherein the step of adding the second gesture to the multimedia asset comprises: identifying a second language associated with the second culture that is different from the first language of the first culture; and replacing the audio of the multimedia asset in the first language with the audio in the second language. Claim 14 In claim 13, the step of determining the second gesture comprises identifying a translated gesture having a meaning consistent with the first gesture for the second language based on a second language ontology. Claim 15 In claim 13, the step of replacing the audio of the multimedia asset comprises: generating a transcription of the audio of the multimedia asset; translating the transcription of the audio from the first language to the second language; and generating the translated audio of the multimedia asset using a generative machine learning model. Claim 16 A system comprising: one or more computer processors; and memory comprising a program that performs an operation when executed by said computer processors, wherein the operation is, The act of accessing multimedia assets that depict an individual speaking in a first language among multiple languages; The operation of extracting audio segments spoken by the individual from the above multimedia assets; An action of identifying a first gesture depicted in the multimedia asset during a first part of the above audio segment; An operation to determine a first temporal alignment between a speech performed in the first part of the above audio segment and the first gesture; and A system comprising the operation of generating a synchronized multimedia asset based on at least one of modifying the temporal position of the first gesture in the multimedia asset, or modifying the timing of the translated audio segment to establish a second temporal alignment between the first gesture and the translation of the utterance in the translated audio segment. Claim 17 A system according to claim 16, further comprising: an action of identifying the meaning of the first gesture for the first language based on the first language ontology; an action of identifying a translated gesture having a meaning consistent with the first gesture for the target language based on the target language ontology; and an action of modifying the multimedia asset to describe the translated gesture while maintaining the first temporal alignment. Claim 18 In claim 17, the operation of modifying the multimedia asset to describe the translated gesture includes the operation of processing the multimedia asset using a generative machine learning model to generate the synchronized multimedia asset, and the synchronized multimedia asset describes the individual performing the translated gesture, a system. Claim 19 In claim 17, the operation of modifying the multimedia asset to depict the translated gesture comprises: the operation of creating an avatar that performs the translated gesture; and the operation of adding the avatar to the synchronized multimedia asset, the system. Claim 20 A system according to claim 16, further comprising the operation of generating the individual’s translated lip movements to align with the translated audio segment, and the operation of adding the individual’s translated lip movements to the synchronized multimedia assets.

Images