Multi-language intelligent dubbing generation system based on sound cloning and emotion migration

Abstract

The invention discloses a multi-language intelligent dubbing generation system based on sound cloning and emotion migration. The multi-language intelligent dubbing generation system comprises a sound cloning module, a cross-language synthesis module, an emotion migration module and a lip shape synchronization module. By constructing a few-sample speaker encoder, a cross-language rhythm migration module, a fine-grained emotion control module and a video lip shape synchronization module, end-to-end automatic generation from original dubbing audio to multi-language target dubbing is realized, and tone consistency, emotion authenticity and picture synchronism are kept.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and speech synthesis, specifically to a multilingual intelligent dubbing generation system based on voice cloning and emotion transfer, applicable to various application scenarios such as multilingual dubbing of film and television, dubbing of animation, game localization, audiobook production, and virtual anchor dubbing. Background Technology

[0002] With the rapid development of global content dissemination, the demand for multilingual dubbing in film, television, games, animation, and other multimedia products is increasing daily. Traditional dubbing production processes rely on professional voice actors and involve multiple stages such as dialogue translation, dubbing recording, and post-production mixing. This process is time-consuming, costly, and suffers from a scarcity of excellent voice actors, making it difficult to meet the rapidly growing market demand.

[0003] In recent years, speech synthesis technology has made significant progress, and deep learning-based text-to-speech (TTS) systems can now generate speech that closely resembles natural human speech. However, existing technologies still face the following challenges when applied to professional dubbing scenarios:

[0004] 1. High demand for voice cloning samples: Traditional voice cloning techniques typically require hours or even tens of hours of high-quality recording samples from the target speaker to accurately reproduce their voice characteristics. This is difficult to achieve for deceased voice actors, historical figures, or scenarios where samples are scarce. Although few-sample voice cloning techniques have made progress, they still have shortcomings in terms of voice similarity, stability, and naturalness.

[0005] 2. Difficulty in transferring prosodic features in cross-language dubbing: Simple text translation followed by speech synthesis can cause the target language dubbing to lose the prosodic features (speech rate, pauses, stress, and intonation) of the original dubbing, resulting in significant differences in emotional expression and artistic effect between different language versions. Existing cross-language TTS models mainly focus on pronunciation accuracy and lack the ability to transfer prosodic features across languages.

[0006] 3. Insufficient precision in controlling emotional expression: Dubbed works usually contain a rich range of emotional changes (joy, sadness, anger, surprise, etc.). Although existing TTS systems support basic emotion classification, the granularity of emotion recognition is coarse and the control of emotion intensity is imprecise, making it difficult to accurately reproduce the subtle emotional expression of the original dubbing, especially in highly dramatic film and television works.

[0007] 4. Limitations of video lip-sync technology: Different languages ​​have significant differences in speech rate and syllable structure (e.g., English is faster, Chinese is slower), and direct translation and dubbing will lead to serious lip-sync problems. Existing lip-sync technologies either require modification of the video footage (high cost and technical complexity) or adjust the speech rate through simple time scaling (which is prone to distortion and unnaturalness).

[0008] 5. Poor coordination of voice acting in multi-character scenes: In multi-character dialogue scenes, the existing system has difficulty in accurately identifying the lines of different characters, lacks the ability to assign an independent voice model to each character, and cannot add acoustic effects that match the scene (such as indoor reverberation and stereo positioning), resulting in a lack of realism and immersion in multi-character voice acting.

[0009] 6. Low end-to-end integration: From original dubbing analysis, voice cloning, cross-language synthesis, emotion transfer to lip-sync, each step is usually completed by an independent system. The lack of a unified technical framework and end-to-end optimization mechanism leads to information loss, cumbersome processes, and difficulty in guaranteeing quality.

[0010] Therefore, there is an urgent need for an end-to-end intelligent dubbing generation system capable of cloning target timbres with a small number of samples, accurately transferring cross-lingual prosody and emotional features, achieving precise lip-syncing, and supporting dubbing for multiple roles in various scenarios. This invention solves the above-mentioned technical challenges by constructing a joint modeling framework of voice cloning, cross-lingual synthesis, emotion transfer, and lip-syncing, providing an efficient and reliable technical solution for the automation of multilingual dubbing. Summary of the Invention

[0011] To address the aforementioned issues, this invention provides a multilingual intelligent dubbing generation system based on voice cloning and emotion transfer. By constructing a few-sample speaker encoder, a cross-language prosody transfer module, a fine-grained emotion control module, and a video lip-syncing module, it achieves end-to-end automated generation from the original dubbing audio to the multilingual target dubbing, maintaining timbre consistency, emotional authenticity, and visual synchronization.

[0012] The specific plan is as follows:

[0013] A multilingual intelligent dubbing generation system based on voice cloning and emotion transfer includes:

[0014] Voice cloning module: Employs a few-sample speaker encoder based on a deep neural network to extract the timbre feature vector of the target speaker from a very small number of samples;

[0015] Cross-language synthesis module: Transfers the prosodic features (speech rate, pauses, stress, tone) of the original dubbing to the target language dubbing, maintaining the consistency of emotional expression across different language versions; the original dubbing audio and the target language text are used as inputs, and a cross-language prosodic mapping is established through an attention alignment mechanism to obtain a text-prosodic fusion feature vector, which is then combined with the extracted speaker timbre feature vector to generate a target language dubbing with both the target timbre and the prosodic features of the source language;

[0016] Emotion transfer module: Identify the type and intensity of emotions from the original dubbing, construct an emotion embedding vector, and input it together with the previously extracted speaker timbre feature vector and text-prosody fusion feature vector into the acoustic decoder to achieve joint control of timbre, prosody and emotion;

[0017] The lip-sync module analyzes the lip movement trajectory of the input video, dynamically aligns it with the energy envelope of the generated dubbing, constrains the acoustic decoder to generate dubbing that matches the lip rhythm through a synchronization loss function, and fine-tunes the speech rate through a time-scaling algorithm when necessary to achieve a dubbing effect that is consistent with both audio and visuals.

[0018] Furthermore, the sound cloning module specifically includes:

[0019] Let the target speaker's reference audio be... ,in Indicates the first An audio clip, The reference number of audio segments (usually 3-5 segments, total duration 1-5 minutes);

[0020] First, Mel spectral features are extracted for each audio segment; then, the audio signal is converted to a Mel scale after undergoing a short-time Fourier transform (STFT) to obtain the Mel spectral sequence. ,in Indicates the first Mel spectrum eigenvectors of the frame For audio clips Frame count, The dimension of the Mel spectrum (usually 80 dimensions);

[0021] The speaker encoder extracts the timbre embedding vector; the speaker encoder employs a multi-layer GRU (Gated Recurrent Unit) network structure to perform time-series modeling of the Mel spectrum sequence.

[0022]

[0023] in This indicates that the last layer of the GRU network is at the [missing information - likely a specific layer or level]. The hidden state of a frame. For audio clips Frame count, For audio clips The timbre embedding vector, The embedding vector dimension (usually 256 dimensions);

[0024] To improve the robustness and representativeness of timbre features, the embedding vectors of multiple reference audio segments are weighted and aggregated to obtain the final speaker timbre feature vector:

[0025]

[0026] in For the first The weight coefficients of each embedded vector. The L2 norm of the embedding vector is represented. This is a temperature parameter (usually taken as 0.5). This is the final speaker timbre feature vector. By exponentially weighting the embedding vector magnitude, the system can automatically increase the weight of high-quality audio segments and reduce the influence of noise or abnormal segments, thereby obtaining a more stable and accurate timbre representation.

[0027] Timbre Feature Vector It includes the unique acoustic features of the target speaker (pitch range, formant distribution, vocal cord vibration pattern, etc.), which will serve as global conditional information in the subsequent speech synthesis stage to ensure that the generated multilingual dubbing maintains the consistency of the target speaker's timbre.

[0028] Furthermore, the cross-language synthesis module specifically includes:

[0029] 1) Extraction of prosodic features of source language

[0030] Prosodic analysis was performed on the original dubbing audio to extract prosodic parameters such as fundamental frequency (F0), energy envelope, speech rate, and pauses; let the Mel spectrum of the original dubbing audio be... ,in The number of audio frames in the source language;

[0031] Extracting prosodic feature sequences using a prosodic encoder:

[0032]

[0033] in For the source language audio The prosodic feature vector of the frame, , The prosodic feature dimension is typically 128. The prosodic encoder uses a convolutional neural network (CNN) combined with a bidirectional LSTM structure to capture the local patterns and global temporal dependencies of the prosodic features.

[0034] 2) Target language text encoding

[0035] The target language text is converted into a phoneme sequence, and semantic and speech features are extracted using a text encoder. The target language text, after phoneme conversion, yields a sequence... ,in For the first One phoneme, The total number of phonemes;

[0036] The semantic representation at the phoneme level is obtained through a text encoder:

[0037]

[0038] in For the first A context representation vector for each phoneme. The hidden layer dimension is typically 512; the text encoder uses a Transformer architecture and models long-distance dependencies between phonemes through a multi-head self-attention mechanism.

[0039] 3) Cross-linguistic prosodic alignment and transfer

[0040] Because the source and target languages ​​differ in the number of phonemes and their temporal structure, an alignment relationship needs to be established between them. This invention employs an additive attention mechanism to achieve soft alignment and transfer of cross-linguistic prosody; for each phoneme of the target language... The attention mechanism is used to extract relevant prosodic information from the source language prosodic sequence.

[0041]

[0042] Attention score Calculations are performed using bilinear transformation and nonlinear activation:

[0043]

[0044] This process involves the phonemic features of the target language. Prosodic features of the source language respectively through projection matrix and Mapped to a unified attention space (dimension: Then, after adding element by element, it goes through... Nonlinear activation, finally through the weight vector Compress into scalar scores; The activation function constrains the feature values ​​to Range, avoid gradient vanishing, and make attention scores smoother and more stable; The attention hidden dimension (usually 256); Let be the attention weights after Softmax normalization, representing the th . The phoneme corresponds to the first phoneme in the source language. Correlation of frame prosody; This is the weighted aggregated prosodic context vector;

[0045] By fusing textual semantic features with transferred prosodic features, a comprehensive representation encompassing both target language semantics and source language prosody is obtained:

[0046]

[0047] in The prosodic feature projection matrix, This is a text-prosodic fusion feature vector;

[0048] 4) Acoustic Feature Generation

[0049] The text-prosodic fusion feature vector and the speaker timbre feature vector are input into the acoustic model to generate the Mel spectrum of the target language dubbing; the acoustic model adopts an autoregressive generative architecture, which is conditional on the text-prosodic fusion features, speaker timbre and the generated historical frames when generating each frame.

[0050] The Mel spectrum is generated frame by frame using an acoustic decoder:

[0051]

[0052] in The first time to dub the target language Frame Mel spectrum, , The number of dubbing frames for the generated target language; the acoustic decoder adopts an attention-based sequence-to-sequence (Seq2Seq) architecture, which dynamically controls the fusion ratio of text content, prosodic style and speaker timbre through a gating mechanism to generate a speech representation that retains the target speaker timbre and transfers the prosodic features of the source language;

[0053] Finally, the Mel spectrum is converted into a time-domain waveform signal using a neural vocoder (such as HiFi-GAN) to generate high-quality dubbing audio. Although the dubbing generated by the above process possesses the target timbre and rhythmic characteristics, it has not yet considered the emotional expression of the original dubbing. Therefore, it is necessary to further introduce an emotion transfer mechanism.

[0054] Furthermore, the emotion transfer module specifically includes:

[0055] 1) Source Dubbing Emotion Recognition

[0056] Sentiment analysis is performed on the original dubbing audio to identify the type and intensity of emotions; a pre-trained multimodal emotion recognition model is used to make emotion judgments by combining audio acoustic features and text semantic information.

[0057] Let the emotional feature vector of the original dubbing be... The probability distribution of sentiment categories is obtained through the sentiment classifier:

[0058]

[0059] in For sentiment classification weight matrix, For bias vectors, The number of emotion categories (usually including 6-8 categories such as happiness, sadness, anger, surprise, fear, and neutrality). For the emotional feature dimension, This is the vector representing the probability distribution of emotions.

[0060] Simultaneously, the emotional intensity value is predicted using an emotional intensity regressor:

[0061]

[0062] in For the intensity regression weight vector, For bias scalars, It is the Sigmoid activation function. The normalized emotion intensity value, where 0 represents weak emotion and 1 represents extremely strong emotion;

[0063] 2) Construction of Emotion Embedding Vectors

[0064] Based on the identified emotion category and intensity, an emotion embedding vector is constructed to guide the emotional expression of the target dubbing; the emotion embedding vector is obtained by jointly encoding discrete emotion categories and continuous emotion intensities:

[0065]

[0066] in For the first Learnable embedding vectors for sentiment-like data. , The probability of this type of emotion (from the emotion probability distribution vector output by the aforementioned emotion classifier). The (each component) A comprehensive sentiment embedding vector; the dimension of the sentiment embedding vector. The dimension is kept consistent with the speaker's timbre vector to facilitate feature fusion in the acoustic decoder; through intensity values... Scaling can be applied to achieve continuous control over the intensity of emotions;

[0067] 3) Emotional control synthesis

[0068] During the acoustic feature generation process, the emotion embedding vector is input into the acoustic decoder as additional conditional information to guide the generation of emotional speech. Specifically, at each time step of the decoder, the emotion embedding vector is concatenated or gatedly fused with the text-prosodic fusion features and the speaker timbre feature vector, so that the generated Mel spectrum contains both the timbre of the target speaker and the prosodic of the source language, and accurately expresses the recognized emotional state.

[0069] Through end-to-end training, the acoustic decoder learns to adjust the acoustic parameters of the generated speech (such as fundamental frequency response, energy envelope, and formant positions) based on the emotion embedding vector, achieving fine-grained emotion control. For user-specified emotion adjustment needs (such as enhancing sadness or reducing anger), flexible emotion editing can be achieved by manually modifying the emotion category probability or intensity value.

[0070] At this point, the system has achieved three core functions: timbre cloning, rhythm transfer, and emotion control. However, for dubbing video applications, it is still necessary to ensure that the generated dubbing is accurately synchronized with the lip movements of the characters in the video to avoid the problem of "not matching the lip movements".

[0071] Furthermore, the lip-sync module specifically includes:

[0072] 1) Video lip movement analysis

[0073] Face detection and lip landmark extraction are performed on the input video, and the opening and closing motion trajectory of the lips is analyzed; assuming the video frame sequence is... ,in The video's first Frame image, , This represents the total number of frames in the video.

[0074] First, facial landmark detection algorithms (such as Dlib's 68-point model or MediaPipe's 478-point model) are used to locate the key points in the lip region; for the lips, the coordinates of the midpoint of the upper lip, the midpoint of the lower lip, the left corner of the mouth, and the right corner of the mouth are extracted; let the first... The coordinates of the upper lip midpoint detected by the frame are: The coordinates of the midpoint of the lower lip are ;

[0075] Calculate lip opening characteristics:

[0076]

[0077] in For the first Vertical distance (in pixels) between the upper and lower lips of a frame. The maximum vertical distance when the speaker's lips are fully open throughout the entire video sequence (obtained by taking the maximum value across all frames or by pre-calibration). The normalized lip opening is represented by 0 indicating that the lips are completely closed and 1 indicating that the lips are completely open. The normalization process eliminates the influence of differences in facial size among different speakers, making the lip opening comparable.

[0078] Lip opening curve This reflects the rhythm and range of lip movements of the speaker in the video; note this. This indicates the video frame index, to distinguish it from the audio frame index used in previous sections. ;

[0079] 2) Voice-lip timing alignment

[0080] A temporal correspondence between the generated dubbing and the video lip movements is established using a Dynamic Time Warping (DTW) algorithm or an attention-based alignment model; the matching loss between the speech energy envelope and the lip opening curve is defined.

[0081]

[0082] in The aligned voice-over energy envelope in the video is shown in section [number]. The value corresponding to the frame; the specific calculation process is as follows: first, the Mel spectrum sequence of the target language generated by the acoustic decoder is processed. Calculate the energy of each frame (right The energy of each Mel frequency band is summed to obtain the energy sequence at the audio frame rate; then, it is converted from the audio frame rate (e.g., 100fps) to the video frame rate (24fps) through linear interpolation to obtain an energy sequence consistent with the number of video frames. ; For lip opening, the first item The mean square error of lip-energy matching, the second term To smooth the regularization term (penalize drastic changes in energy between adjacent frames to avoid unnatural abrupt changes). This is the smoothing weighting coefficient (usually taken as 0.1).

[0083] By minimizing the synchronization loss function, the system automatically adjusts the duration and rhythm of the generated dubbing to precisely align it with the lip movements in the video. For cases where there is a significant difference in speech rate between the source and target languages, the system considers the duration constraints of the target video during the prosodic transfer stage, adjusting phoneme duration allocation and speech rate factors to achieve video synchronization while maintaining naturalness.

[0084] Furthermore, in multi-role dialogue scenarios, realistic multi-role voice acting is generated through character recognition, voice allocation, and contextual processing; specifically including:

[0085] 1) Automatic segmentation of character dialogue

[0086] Speaker divination technology is used to automatically identify dialogue segments from different characters in the original dubbing. By combining character detection from video footage with the dialogue structure of the text, an independent voice model is constructed for each character.

[0087] 2) Personalized character voice synthesis

[0088] Assign different speaker timbre vectors to each character , Assign character numbers and use corresponding timbre vectors when synthesizing the lines of each character to ensure that different characters have distinguishable voice features;

[0089] 3) Adding scene acoustic effects

[0090] Based on the characteristics of the video scene (indoor / outdoor, space size, ambient noise) and the position of the characters in the frame, corresponding acoustic effects are added to the synthesized voice-over; including: reverberation effect: adding early reflections and reverberant tails according to the size of the scene space; distance simulation: simulating distance through energy attenuation and high-frequency filtering; stereo positioning: adjusting the energy distribution of the left and right channels according to the left and right positions of the characters in the frame; through scene-based processing, the spatial sense and realism of multi-character voice-over are significantly improved, and the audience can perceive the position of the characters and environmental characteristics through sound.

[0091] Furthermore, it also includes a post-processing module for building an end-to-end joint training framework to uniformly optimize each module. Unlike traditional pipeline processing, this invention achieves collaborative learning of each module through a joint loss function, enabling the speaker encoder, prosodic encoder, emotion recognizer, and acoustic decoder to cooperate with each other and automatically balance various optimization objectives during training.

[0092] The total loss function is defined as:

[0093]

[0094] in The Mel spectrum reconstruction loss, L1 loss, or L2 loss, is used to measure the consistency between the generated target language dubbing Mel spectrum and the reference Mel spectrum. Prosodic transfer loss is used to constrain the similarity between the prosodic features of the generated dubbing and the prosodic features of the source language. To preserve emotional loss, this is used to ensure that the emotional expression of the generated voice-over is consistent with the emotion of the original voice-over; The matching loss between the speech energy envelope and the lip opening curve is used to constrain the temporal alignment of dubbing and video lip movements. The loss of speaker timbre is used to ensure the timbre similarity between the generated dubbing and the reference audio; The weighting coefficients for each loss term can be flexibly adjusted according to the application scenario requirements.

[0095] Through multi-task joint optimization, the system ensures the accuracy of timbre cloning while taking into account prosodic transfer, emotional expression, and visual synchronization, generating high-quality multilingual dubbing. The information flow between modules is as follows: the reference audio is processed by a speaker encoder to extract timbre vectors → the original dubbing is processed by a prosodic encoder to extract prosodic features, and the emotion recognition module identifies emotions → the target text is encoded by a text encoder → cross-language prosodic alignment and fusion → the acoustic decoder combines timbre, prosodic, and emotion to generate a Mel spectrum → lip-sync constraints adjust the duration → the neural vocoder generates the final audio waveform.

[0096] The generated dubbing audio undergoes noise reduction, equalization, compression, and amplitude limiting operations through an intelligent post-processing module to match the sound quality and dynamic range of the original audio track. It is then intelligently mixed with background music and sound effects to output a dubbing product that meets broadcast-grade standards.

[0097] Through the above-mentioned technical solution, this invention achieves end-to-end automated generation from original dubbing to multilingual target dubbing. It reaches professional dubbing standards in key aspects such as timbre cloning, rhythm transfer, emotion control, and lip-syncing, providing an efficient and low-cost technical solution for multilingual dubbing production in film, animation, games and other fields.

[0098] The beneficial effects of this invention are as follows:

[0099] 1. This invention achieves multilingual dubbing generation through automation technology, eliminating the need for professional voice actors and reducing production costs by 70-80%. The generation time from the original dubbing to the target dubbing is shortened to 5-10% of the original length (approximately 15-30 minutes of processing is required for a 3-minute dubbing), and the production cycle is reduced from several weeks to several hours, significantly improving content production efficiency.

[0100] 2. Only 1-5 minutes of audio samples of the target speaker are needed to accurately clone their vocal characteristics, reducing the sample size required by more than 90% compared to traditional methods. This makes it possible to dub deceased voice actors, historical figures, or special voices with scarce samples, possessing significant cultural heritage and commercial value.

[0101] 3. The innovative prosodic attention transfer mechanism can accurately maintain the prosodic features of the original dubbing, such as speech rate, pauses, emphasis, and tone. The emotion recognition and control module achieves fine-grained emotion reproduction (emotion classification accuracy of 87.5%, intensity regression error <0.1), so that the dubbing of different language versions can maintain consistent artistic expression and appeal.

[0102] 3. Through dynamic alignment of voice and lip shape and time scaling optimization, accurate synchronization between dubbing and video footage is achieved (synchronization correlation coefficient 0.89), effectively solving the problem of lip shape mismatch caused by differences in speech speed across languages, and significantly improving the audience's viewing experience and immersion.

[0103] 4. A unified end-to-end system architecture was constructed, avoiding the information loss and cumbersome processes of traditional pipeline processing. Multi-task joint optimization enables modules to work collaboratively, and the overall performance is better than that of independent module combinations. The subjective evaluation of dubbing quality (MOS score) reaches 4.2 / 5.0, which is close to the level of professional voice actors.

[0104] 5. Supports automatic recognition and personalized dubbing for multi-character dialogue scenarios. Combined with scene-based acoustic effects processing (reverb, distance perception, stereo positioning), the generated multi-character dubbing has a significantly better sense of space and realism than traditional TTS systems, making it particularly suitable for applications such as film and television and games that require an immersive experience.

[0105] 6. It integrates professional audio post-processing technology, automatically completing noise reduction, compression, equalization, limiting and mixing, and outputting dubbing products that meet international broadcasting standards (EBUR128), which can be directly used for commercial distribution and broadcasting without additional manual post-processing.

[0106] 7. The technical framework of this invention has good scalability and can be adapted to multiple language pairs (Chinese-English, Chinese-Japanese, English-French, etc.), multiple dubbing styles (documentaries, dramas, animations, etc.), and multiple voice types (male voices, female voices, children's voices, etc.). It supports user-defined emotional adjustments and rhythm editing to meet personalized dubbing needs.

[0107] 8. This invention breaks through the dependence of traditional dubbing on professional actors and long production cycles, promotes the paradigm shift of dubbing from "craft" to "industrialization" and from "single language" to "multilingual", provides strong technical support for global content dissemination, cultural exchange and barrier-free access, and has broad market prospects and social value. Attached Figure Description

[0108] Figure 1 This is a flowchart of the present invention.

[0109] Figure 2 This is the system architecture diagram of the present invention. Detailed Implementation

[0110] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0111] As shown in the figure, this embodiment provides a multilingual intelligent dubbing generation system based on voice cloning and emotion transfer, taking the conversion of Chinese dubbing to English dubbing of film and television works as the application scenario. The specific solution is as follows:

[0112] I. System Architecture and Data Preparation

[0113] An end-to-end multilingual dubbing generation system was constructed, including a data preprocessing module, a voice cloning module, a cross-language synthesis module, an emotion transfer module, a lip-syncing module, and a post-processing module.

[0114] Input data: Original Chinese dubbing audio: 24kHz sampling rate, 16-bit quantization, 3 minutes duration; Target speaker reference audio: 3 English recordings, 1 minute each, total 3 minutes duration; English translation text: English translation of the Chinese dubbing content; Original video: 1920×1080 resolution, 24fps frame rate, 3 minutes duration.

[0115] The system preprocesses the input data, including audio resampling, noise reduction, normalization, video face detection and lip key point extraction, and text phoneme conversion.

[0116] II. Implementation of Small Sample Sound Cloning

[0117] 1) Mel spectrum extraction

[0118] 80-dimensional Mel-spectral features were extracted from three reference audio segments of the target speaker. A short-time Fourier transform was applied with a window length of 25 ms and a frame shift of 10 ms to obtain the Mel-spectral sequence. .

[0119] For the first audio segment (60 seconds long), the number of frames... (60 seconds ÷ 0.01 seconds / frame), Mel spectrum ,in .

[0120] 2) Speaker timbre embedding vector extraction

[0121] Phonological embedding vectors are extracted using a pre-trained speaker encoder (3-layer GRU network, 256 hidden layers):

[0122]

[0123] in For the last layer (layer 3) of the GRU network in the... The hidden state of a frame. For audio clips The number of frames. The following were obtained from the three audio segments: .

[0124] Calculate the final speaker timbre vector using weighted aggregation. Assume a temperature parameter. Calculate the weights:

[0125]

[0126] Assumption , , ,but: ; ;

[0127] After normalization, we get , , Final timbre vector:

[0128]

[0129] This timbre vector The vector accurately captures the vocal characteristics of the target speaker, and will be used in the subsequent synthesis process to generate English dubbing of the target speaker's voice.

[0130] After timbre cloning is completed, the system enters the cross-language prosodic transfer and synthesis stage. This stage combines the aforementioned timbre vectors, the prosodic features of the original Chinese dubbing, and the English translation text to generate an English dubbing Mel spectrum with the target timbre and the prosodic features of the source language.

[0131] III. Cross-linguistic prosodic transfer and synthesis implementation

[0132] 1) Source Language Prosody Extraction

[0133] Mel spectrum extraction from the original Chinese dubbing audio (80 dimensions, number of frames) (Corresponding to a 3-minute audio file).

[0134] Prosodic feature sequences are extracted using a prosodic encoder (2-layer CNN + bidirectional LSTM, 128 hidden dimensions):

[0135]

[0136] in It includes rhythmic features such as fundamental frequency, energy, and spectral tilt for each frame.

[0137] 2) Target language text encoding

[0138] Convert the English translation line (assumed to be "I never expected this would happen so soon") into a sequence of phonemes. Represented using the International Phonetic Alphabet (IPA), we get , a total of phonemes.

[0139] Obtain the phoneme representation through a Transformer text encoder (6 layers, hidden dimension 512, 8 attention heads):

[0140]

[0141] where .

[0142] 3) Cross-lingual prosodic alignment

[0143] For each phoneme position in English (a total of 42), extract relevant prosodic information from the Chinese prosodic sequence (18000 frames) through an additive attention mechanism.

[0144] Calculate the attention scores (attention hidden dimension ):

[0145]

[0146] where Project the English phoneme features from 512 dimensions to 256 dimensions, Project the Chinese prosodic features from 128 dimensions to 256 dimensions, add them in a unified 256-dimensional attention space, and after non-linear activation, compress them into scalar scores by the weight vector This design enables the attention mechanism to capture the semantic correlation between English phonemes and Chinese prosody.

[0147] Normalize to obtain the attention weights:

[0148]

[0149] Aligned prosodic context vectors:

[0150]

[0151] Assume that for the 1st phoneme "aɪ", the attention mainly focuses on the 120 - 180 frames of the Chinese prosodic sequence (corresponding to the Chinese syllable "我"), which contains the prosodic features of this syllable (such as the pitch contour).

[0152] Fuse the text and prosodic features (projection matrix ):

[0153]

[0154] get It contains English semantics and Chinese prosody information.

[0155] 4) Acoustic Feature Generation

[0156] This will fuse feature sequences and speaker timbre vectors. Input acoustic decoder (based on Tacotron 2 architecture, decoder LSTM hidden dimension 1024).

[0157] The acoustic decoder uses an autoregressive approach to generate the Mel spectrum of the English dubbing frame by frame. In generating the first... Frame time:

[0158]

[0159] The decoder internally uses an attention mechanism to automatically align the phoneme sequence with the acoustic frame sequence, and predicts the network based on phoneme duration, generating an average of 10-15 frames of Mel spectrum per phoneme. The final output is approximately... The English dubbing of the frame is Mel Spectrum (approximately 5.5 seconds, with English speaking at a faster pace than Chinese, a sentence from the original 3-minute segment).

[0160] The Mel spectrum is converted into 24kHz waveform audio using a HiFi-GAN vocoder.

[0161] The English dubbing generated by the above process already possesses the target speaker's timbre and the prosodic features of the source language, but it has not yet considered the emotional expression of the original dubbing. Next, the system uses emotion recognition and transfer mechanisms to ensure that the generated dubbing accurately conveys the emotional type and intensity of the original dubbing.

[0162] IV. Fine-grained emotion recognition and transfer implementation

[0163] 1) Source Dubbing Emotion Recognition

[0164] Extract emotional feature vectors from the corresponding segments of the original Chinese dubbing (assuming they express the emotion of "surprise"). (Through a pre-trained multimodal emotion recognition model).

[0165] Emotional categories (8 categories: happiness, sadness, anger, surprise, fear, disgust, neutrality, excitement):

[0166]

[0167] in Assuming the output probability distribution is The highest probability corresponds to "surprise" (Category 4, probability 0.78).

[0168] Emotional intensity regression:

[0169]

[0170] Assuming the calculation yields This indicates a high level of surprise.

[0171] 2) Construction of Emotion Embedding Vectors

[0172] Embedding vectors for each type of emotion (With the same dimensionality as the speaker's timbre vector) it is learned through training.

[0173] Comprehensive sentiment embedding vector:

[0174]

[0175] Because "surprise" has the highest probability (0.78). Mainly composed of The dominant strength factor of 0.85 ensures that the generated voice-over expresses a strong sense of surprise.

[0176] 3) Emotional control synthesis

[0177] When the acoustic decoder generates the Mel spectrum, the emotion embedding vector is... With speaker timbre vector splicing together to form , which serves as the global conditional input for the decoder.

[0178] The decoder adjusts the generated Mel spectral characteristics based on the emotional embedding, specifically by: increasing the amplitude of the fundamental frequency curve (surprise is often accompanied by a pitch rise); increasing the peak value of the energy envelope (to express strong emotions); and adjusting the formant position (affecting the emotional color of the timbre).

[0179] The generated English dubbing accurately conveyed the "surprise" emotion and its intensity from the original Chinese dubbing, achieving cross-language emotional transfer.

[0180] At this point, the generated voiceover possesses three core features: the target timbre, the original rhythm, and the original emotion. However, for video dubbing applications, it is still necessary to ensure that the duration and rhythm of the generated voiceover are precisely aligned with the lip movements of the characters in the video. The system solves this problem through lip-sync analysis and synchronization optimization mechanisms.

[0181] V. Video Lip Shape Analysis and Synchronous Implementation

[0182] 1) Lip movement extraction

[0183] For a 3-minute segment of the original video, lip opening features were extracted for each frame at 24fps. The Dlib 68-point facial landmark detection model was used to locate key points in the lip region. The Dlib model labeled the upper lip as points 48-54 and the lower lip as points 54-60. The Y-coordinates of the midpoints of the upper lip (point 51) and lower lip (point 57) were extracted and denoted as follows: and .

[0184] Calculate lip opening:

[0185]

[0186] in For the video number Vertical distance between the upper and lower lips of a frame (unit: pixels). This represents the maximum vertical distance when the speaker's lips are fully open. Specifically, this is calculated by iterating through all frames of the entire video sequence. Assuming the calculation yields... pixels, then for a given frame, if pixels, then This indicates that the lips are half-open.

[0187] Obtain the lip opening curve (3 minutes × 60 seconds × 24fps = 4320 frames), the peak value of the curve corresponds to the open state when the vowel is pronounced, and the valley value corresponds to the closed state of the consonant (such as p, b, m).

[0188] 2) Voice-lip timing alignment

[0189] Extract the energy envelope of the generated English dubbing. For the approximately 550 frames of the English dubbing Mel spectrum (80 dimensions per frame) generated by the aforementioned acoustic decoder, calculate the energy of each frame:

[0190]

[0191] in For the first Frame number The values ​​of each Mel band are obtained. The energy sequence is then derived. This corresponds to a 5.5-second voiceover clip.

[0192] The energy sequence is interpolated from the Mel-spectral frame rate (100fps, i.e., 10ms / frame) to the video frame rate (24fps). A linear interpolation method is used: for the video... Frame (moment) (seconds), find the indexes of the two most recent audio frames. and ,calculate ,in These are used as interpolation weights. This yields an energy sequence consistent with the video frame number. .

[0193] Calculate synchronization loss:

[0194]

[0195] During the training phase, by minimizing The phoneme duration prediction module of the acoustic decoder has been optimized to ensure that the rhythm of the generated dubbing is accurately matched with the lip movements in the video.

[0196] During the inference phase, if the initially generated English dubbing duration does not perfectly match the video (the English is too short), adjustments are made using the following strategies: 1. In the prosody transfer phase, appropriately extend the phoneme duration based on the target video duration constraint. 2. Use the WSOLA (Time Scaling Algorithm) to fine-tune the speech rate, stretching the 5.5-second dubbing to 5.8 seconds to align with the video clip. 3. Maintain the pitch unchanged during the stretching process to avoid distortion.

[0197] The correlation coefficient between the adjusted voice-over and the lip-sync curve reached 0.89 (1.0 for perfect synchronization), achieving high-quality video synchronization.

[0198] VI. Implementation of Multi-Character Scene Dubbing

[0199] Suppose a 3-minute clip contains a dialogue between two characters (Character A: protagonist, Character B: supporting character).

[0200] 1) Character Dialogue Segmentation

[0201] Using speaker separation technology, combined with character detection in video footage and dialogue annotation in the script, the system automatically identifies: Character A's dialogue: 12 lines, total duration 1 minute 50 seconds; Character B's dialogue: 8 lines, total duration 1 minute 10 seconds.

[0202] 2) Personalized character voices

[0203] Prepare reference audio for the target speaker for both Character A and Character B: Character A: Adult male voice, steady tone; Character B: Young female voice, bright tone.

[0204] Extract timbre vectors separately and When synthesizing the lines of each character, the corresponding timbre vector is used.

[0205] 3) Scene acoustic effects

[0206] The video scene analyzed is an "indoor conference room," a medium-sized space. The following were added to the synthesized voice-over: early reflection reverb (15ms delay, -6dB attenuation); reverb tail (RT60 = 0.4 seconds).

[0207] Based on the character's position in the screen, stereo positioning is added. If character A is on the left side of the screen, the left channel energy is increased by 3dB; if character B is on the right side of the screen, the right channel energy is increased by 3dB.

[0208] VII. End-to-end joint optimization implementation

[0209] End-to-end training is performed on a multilingual dubbing dataset, with the following weight settings for the overall loss function:

[0210]

[0211] in .

[0212] Training configuration: Optimizer: AdamW, learning rate Batch size: 16; Number of training epochs: 50; Gradient clipping: Maximum norm 1.0;

[0213] Training dataset: 1000 speakers, each with 5-10 minutes of recordings; Video-dubbing synchronization dataset: 2000 dubbed video clips;

[0214] Training results: After 50 epochs of training, on the validation set: Mel spectrum reconstruction loss (L1): 0.23; prosodic transfer correlation coefficient: 0.82; sentiment classification accuracy: 87.5%; lip-sync correlation coefficient: 0.89; speaker timbre similarity (cosine similarity): 0.91;

[0215] VIII. Post-processing and Output

[0216] The generated English dubbing audio has undergone intelligent post-processing:

[0217] 1. Noise Reduction: Background noise is removed using spectral subtraction, improving the signal-to-noise ratio to 45dB.

[0218] 2. Dynamic Range Compression: Compression ratio 4:1, ensuring consistent volume.

[0219] 3. Equalization processing: Boost the 200Hz-500Hz frequency band (to increase richness), and attenuate more than 8kHz (to remove harshness).

[0220] 4. Amplitude limiting: Peak value is limited to -1dB to avoid clipping distortion.

[0221] Intelligent mixing with the original video's background music and sound effects: Dubbing track: 3dB; Background music: 15dB; Sound effects: -6dB;

[0222] The output conforms to broadcast-grade standards (EBU R128, loudness -23 LUFS) and is a 48kHz / 24bit stereo WAV file, which can be used directly for film and television distribution.

[0223] Through the detailed description of this embodiment, those skilled in the art can clearly understand the technical solution of the present invention, and can implement the present invention in actual application scenarios according to the described method and system configuration to achieve high-quality multilingual intelligent dubbing generation.

[0224] In summary, this invention employs a deep learning-based speaker encoder, which can extract stable timbre embedding vectors from only 1-5 minutes of target speaker audio samples. By performing modulus-weighted aggregation on multiple audio segments, the robustness and representativeness of timbre features are improved, effectively solving the problems of insufficient samples and uneven quality.

[0225] This invention designs an attention-based cross-lingual prosodic alignment module, which establishes a temporal correspondence between the source language prosodic sequence and the target language phoneme sequence through a soft alignment mechanism, thereby achieving cross-lingual transfer of prosodic features. This mechanism can automatically learn the prosodic mapping patterns of different languages, avoiding the distortion problems caused by simple time scaling.

[0226] This invention constructs a dual-branch emotion recognition module based on emotion classification and intensity regression to accurately capture the emotion type and intensity of the original voice-over. By jointly encoding discrete emotion categories and continuous intensity values ​​into an emotion embedding vector, fine-grained emotion control of speech synthesis is achieved, ensuring that the target voice-over accurately conveys the emotional expression of the original voice-over.

[0227] This invention extracts the lip-opening curve and dubbing energy envelope from the video, defines a synchronization loss function, and guides the acoustic model to generate dubbing that is precisely aligned with lip movements. Combined with a time-scaling algorithm, it achieves speech rate adjustment and video synchronization while maintaining naturalness.

[0228] This invention constructs a unified end-to-end training framework that incorporates multiple objectives, such as voice cloning, prosody transfer, emotion control, and lip-syncing, into a joint loss function. Through multi-task learning, it achieves collaborative optimization of each module. This framework avoids the information loss problems associated with pipelined processing and significantly improves overall system performance.

[0229] This invention automatically identifies multi-character dialogue using speaker separation technology, assigns an independent timbre model to each character, and adds acoustic effects such as reverberation, distance perception, and stereo positioning based on video scene characteristics to generate multi-character dubbing with excellent spatial sense and realism.

[0230] This invention integrates professional audio processing technologies such as noise reduction, compression, equalization, and amplitude limiting to automatically optimize dubbing quality and match broadcast-grade standards. It supports intelligent mixing with background music and sound effects, and outputs dubbing products that can be directly used for commercial release.

[0231] The above embodiments are merely typical illustrative methods of the present invention, and the scope of protection of the present invention is not limited thereto. All equivalent substitutions and improvements made under the concept of the present invention should fall within the scope of protection. It should be emphasized that any modifications or minor adjustments made by those skilled in the art without departing from the basic principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multilingual intelligent dubbing generation system based on voice cloning and emotion transfer, characterized in that, include: Voice cloning module: Employs a few-sample speaker encoder based on a deep neural network to extract the timbre feature vector of the target speaker from a very small number of samples; Cross-language synthesis module: Taking the original dubbing audio and the target language text as input, it establishes a cross-language prosodic mapping through an attention alignment mechanism to obtain a text-prosodic fusion feature vector, and combines it with the extracted speaker timbre feature vector to generate a target language dubbing with target timbre and source language prosodic features; Emotion transfer module: Identify the type and intensity of emotions from the original dubbing, construct an emotion embedding vector, and input it together with the previously extracted speaker timbre feature vector and text-prosody fusion feature vector into the acoustic decoder to achieve joint control of timbre, prosody and emotion; The lip-sync module analyzes the lip movement trajectory of the input video, dynamically aligns it with the energy envelope of the generated dubbing, constrains the acoustic decoder to generate dubbing that matches the lip rhythm through a synchronization loss function, and fine-tunes the speech rate through a time-scaling algorithm when necessary to achieve a dubbing effect that is consistent with both audio and visuals.

2. The multilingual intelligent dubbing generation system based on voice cloning and emotion transfer according to claim 1, characterized in that, The sound cloning module specifically includes: Let the target speaker's reference audio be... ,in Indicates the first An audio clip, For reference audio quantity; First, Mel-spectral features are extracted for each audio segment; then, the audio signal is converted to a Mel-scale using a short-time Fourier transform to obtain the Mel-spectral sequence. ,in Indicates the first Mel spectrum eigenvectors of the frame For audio clips Frame count, For the Mel spectrum dimension; The speaker encoder extracts the timbre embedding vector; the speaker encoder employs a multi-layer GRU network structure to perform time-series modeling of the Mel spectrum sequence. in This indicates that the last layer of the GRU network is at the [missing information - likely a number]. The hidden state of a frame. For audio clips Frame count, For audio clips The timbre embedding vector, For the embedded vector dimension; The embedding vectors of multiple reference audio segments are weighted and aggregated to obtain the final speaker timbre feature vector: in For the first The weight coefficients of each embedded vector. Denotes the L2 norm of the embedding vector. For temperature parameters, This is the final speaker timbre feature vector.

3. The multilingual intelligent dubbing generation system based on voice cloning and emotion transfer according to claim 1, characterized in that, The cross-language synthesis module specifically includes: 1) Extraction of prosodic features of source language Prosodic analysis was performed on the original dubbing audio to extract prosodic parameters such as fundamental frequency, energy envelope, speech rate, and pauses; let the Mel spectrum of the original dubbing audio be... ,in The number of audio frames in the source language; Extracting prosodic feature sequences using a prosodic encoder: in For the source language audio The prosodic feature vector of the frame, , For prosodic features; the prosodic encoder uses a convolutional neural network (CNN) combined with a bidirectional LSTM structure to capture local patterns and global temporal dependencies of prosodic features. 2) Target language text encoding The target language text is converted into a phoneme sequence, and semantic and speech features are extracted using a text encoder. The target language text, after phoneme conversion, yields a sequence... ,in For the first One phoneme, The total number of phonemes; The semantic representation at the phoneme level is obtained through a text encoder: in For the first A context representation vector for each phoneme. The hidden layer dimension is used; the text encoder adopts the Transformer architecture and models the long-distance dependencies between phonemes through a multi-head self-attention mechanism. 3) Cross-linguistic prosodic alignment and transfer A soft alignment and transfer of cross-linguistic prosody is achieved using an additive attention mechanism; for each phoneme of the target language... The attention mechanism is used to extract relevant prosodic information from the source language prosodic sequence. Attention score Calculations are performed using bilinear transformation and nonlinear activation: This process involves the phonemic features of the target language. Prosodic features of the source language respectively through projection matrix and Mapped to a unified attention space, then element-wise added and processed... Nonlinear activation, finally through the weight vector Compress into scalar scores; The activation function constrains the feature values ​​to Range, avoid gradient vanishing, and make attention scores smoother and more stable; Hiding dimensions for attention; Let be the attention weights after Softmax normalization, representing the th . The phoneme corresponds to the first phoneme in the source language. Correlation of frame prosody; This is the weighted aggregated prosodic context vector; By fusing textual semantic features with transferred prosodic features, a comprehensive representation encompassing both target language semantics and source language prosody is obtained: in The prosodic feature projection matrix, This is a text-prosodic fusion feature vector; 4) Acoustic Feature Generation The text-prosodic fusion feature vector and the speaker timbre feature vector are input into the acoustic model to generate the Mel spectrum of the target language dubbing; the acoustic model adopts an autoregressive generative architecture, which is conditional on the text-prosodic fusion features, speaker timbre and the generated historical frames when generating each frame. The Mel spectrum is generated frame by frame using an acoustic decoder: in The first time to dub the target language Frame Mel spectrum, , The number of dubbing frames for the generated target language; the acoustic decoder adopts an attention-based sequence-to-sequence architecture, which dynamically controls the fusion ratio of text content, prosodic style and speaker timbre through a gating mechanism to generate a speech representation that retains the target speaker timbre and transfers the prosodic features of the source language. Finally, the Mel spectrum is converted into a time-domain waveform signal using a neural vocoder to generate high-quality dubbing audio.

4. The multilingual intelligent dubbing generation system based on voice cloning and emotion transfer according to claim 1, characterized in that, The emotion transfer module specifically includes: 1) Source Dubbing Emotion Recognition Sentiment analysis is performed on the original dubbing audio to identify the type and intensity of emotions; a pre-trained multimodal emotion recognition model is used to make emotion judgments by combining audio acoustic features and text semantic information. Let the emotional feature vector of the original dubbing be... The probability distribution of sentiment categories is obtained through the sentiment classifier: in For sentiment classification weight matrix, For bias vectors, For the number of emotion categories, For the emotional feature dimension, This is the vector representing the probability distribution of emotions. Simultaneously, the emotional intensity value is predicted using an emotional intensity regressor: in For the intensity regression weight vector, For bias scalars, It is the Sigmoid activation function. The normalized emotion intensity value, where 0 represents weak emotion and 1 represents extremely strong emotion; 2) Construction of Emotion Embedding Vectors Based on the identified emotion category and intensity, an emotion embedding vector is constructed to guide the emotional expression of the target dubbing; the emotion embedding vector is obtained by jointly encoding discrete emotion categories and continuous emotion intensities: in For the first Learnable embedding vectors for sentiment-like data. , This represents the probability of this type of emotion. A comprehensive sentiment embedding vector; the dimension of the sentiment embedding vector. Maintain consistency with the dimension of the speaker's timbre vector; through intensity values. Scaling can be applied to achieve continuous control over the intensity of emotions; 3) Emotional control synthesis During the acoustic feature generation process, the emotion embedding vector is input into the acoustic decoder as additional conditional information to guide the generation of emotional speech. Specifically, at each time step of the decoder, the emotion embedding vector is concatenated or gatedly fused with the text-prosodic fusion features and the speaker timbre feature vector, so that the generated Mel spectrum contains both the timbre of the target speaker and the prosodic of the source language, and accurately expresses the recognized emotional state.

5. A multilingual intelligent dubbing generation system based on voice cloning and emotion transfer according to claim 1, characterized in that, The lip-sync module specifically includes: 1) Video lip movement analysis Face detection and lip landmark extraction are performed on the input video, and the opening and closing motion trajectory of the lips is analyzed; assuming the video frame sequence is... ,in The video's first Frame image, , This represents the total number of frames in the video. First, a facial landmark detection algorithm is used to locate the key points in the lip region; for the lips, the coordinates of the midpoint of the upper lip, the midpoint of the lower lip, the left corner of the mouth, and the right corner of the mouth are extracted; let the first... The coordinates of the upper lip midpoint detected by the frame are The coordinates of the midpoint of the lower lip are ; Calculate lip opening characteristics: in For the first The vertical distance between the upper and lower lips of the frame. This represents the maximum vertical distance when the speaker's lips are fully open throughout the entire video sequence. For normalized lip opening, 0 represents lips completely closed and 1 represents lips completely open; Lip opening curve It reflects the rhythm and range of lip movements of the speaker in the video; 2) Voice-lip timing alignment A temporal correspondence between the generated dubbing and the video lip shape is established using a dynamic temporal warping algorithm or an attention-based alignment model; the matching loss between the speech energy envelope and the lip opening curve is defined. in The aligned voice-over energy envelope in the video is shown in section [number]. The value corresponding to the frame; the specific calculation process is as follows: first, the Mel spectrum sequence of the target language generated by the acoustic decoder is processed. Calculate the energy of each frame The energy sequence at the audio frame rate is obtained; then, it is converted from the audio frame rate to the video frame rate through linear interpolation to obtain an energy sequence with the same number of video frames. ; For lip opening, the first item The mean square error of lip-energy matching, the second term To smooth the regularization term, These are the smoothing weighting coefficients.

6. The multilingual intelligent dubbing generation system based on voice cloning and emotion transfer according to claim 1, characterized in that, In multi-role dialogue scenarios, realistic multi-role voice acting is generated through character recognition, voice allocation, and contextual processing; specifically including: 1) Automatic segmentation of character dialogue By using speaker separation technology, the system automatically identifies dialogue segments of different characters in the original dubbing; and by combining character detection from video footage with the dialogue structure of the dialogue text, it constructs an independent voice model for each character. 2) Personalized character voice synthesis Assign different speaker timbre vectors to each character , Assign character numbers and use corresponding timbre vectors when synthesizing the lines of each character to ensure that different characters have distinguishable voice features; 3) Add scene acoustic effects Based on the characteristics of the video scene and the position of the character in the frame, add corresponding acoustic effects to the synthesized voice-over; including: reverberation effect: add early reflections and reverberation tails according to the size of the scene space; distance simulation: simulate distance through energy attenuation and high-frequency filtering; stereo positioning: adjust the energy distribution of the left and right channels according to the left and right position of the character in the frame.

7. The multilingual intelligent dubbing generation system based on voice cloning and emotion transfer according to claim 1, characterized in that, It also includes a post-processing module for building an end-to-end joint training framework, which optimizes all modules in a unified manner; including: The total loss function is defined as: in The Mel spectrum reconstruction loss, L1 loss, or L2 loss, is used to measure the consistency between the generated target language dubbing Mel spectrum and the reference Mel spectrum. Prosodic transfer loss is used to constrain the similarity between the prosodic features of the generated dubbing and the prosodic features of the source language. To preserve emotional loss, this is used to ensure that the emotional expression of the generated voice-over is consistent with the emotion of the original voice-over; The matching loss between the speech energy envelope and the lip opening curve is used to constrain the temporal alignment of dubbing and video lip movements. The loss of speaker timbre is used to ensure the timbre similarity between the generated dubbing and the reference audio; These are the weighting coefficients for each loss term.

Images