Video dubbing generation method and device based on multi-scale style learning

Abstract

The application relates to the technical field of video dubbing generation, and discloses a video dubbing generation method and device based on multi-scale style learning. The method acquires a video to be dubbed, dubbing text and reference audio, performs phoneme conversion on the dubbing text; extracts phoneme-level acoustic style features and sentence-level global style features from the reference audio, extracts visual emotion features and lip movement information from the video to be dubbed; fuses acoustic style and visual emotion at a phoneme granularity, predicts and recalibrates phoneme duration based on lip-phoneme monotone alignment; modulates a mel-frequency spectrum decoding process by using the sentence-level global style, generates target dubbing speech and synthesizes the target dubbing speech with the video. The scheme can improve pronunciation clarity, emotion consistency, speaker similarity and audio-video synchronization of generated speech.

Description

Technical Field

[0001] This invention relates to the fields of speech synthesis and video dubbing technology, specifically to a method and apparatus for generating video dubbing based on multi-scale style learning. Background Technology

[0002] With the increasing demand for film and television localization, short video production, animation character dubbing, virtual digital human narration, and cross-language content dissemination, automatic video dubbing technology has gradually evolved from traditional manual recording and segment splicing to intelligent generation methods based on speech synthesis and multimodal learning. Video dubbing typically requires generating target speech that is accurate in content, similar in timbre, emotionally matched, and synchronized with the video's lip movements, based on the video to be dubbed, the dubbing text, and reference audio. Compared to ordinary text-to-speech, video dubbing not only requires natural and clear speech but also demands that the pronunciation duration, local rhythm, speaker style, and emotional state remain consistent with the video visuals.

[0003] In existing technologies, solutions have been developed to improve the generation effect from the perspective of lip movement and speech synchronization. CN113852851A discloses a fast lip movement-speech alignment method based on a parallel stream model. This method extracts video and audio features and performs length calibration, then combines a state module and a stream-based decoder to generate speech, ensuring time synchronization between video and audio frames. This solution can improve the efficiency and naturalness of lip movement to speech generation, but its focus is on constraining speech generation by lip movements. It does not fully incorporate speaker style information from the specified dubbing text and reference audio, making it difficult to stably control the target text content, local pronunciation habits, and overall speaking style.

[0004] Another automatic dubbing scheme mainly focuses on voice replacement and voiceprint preservation. CN108780643B discloses an automatic dubbing method and apparatus, which extracts speech from the audio portion of media content, obtains a voiceprint model, and uses the voiceprint model to generate alternative speech to replace the original speech. This type of scheme can achieve a certain degree of automatic dubbing and timbre similarity, but it usually focuses more on global voiceprint or speaker identity, and does not make sufficient use of fine-grained acoustic styles such as phoneme-level pronunciation habits, local prosody, stress changes, and pause patterns contained in the reference audio. Therefore, the generated speech may have a similar timbre, but it is insufficient in terms of local pronunciation clarity, prosodic naturalness, and speech style stability.

[0005] Furthermore, existing video dubbing or lip-sync solutions typically use video information primarily for lip-reading detection, duration calibration, or audio-video synchronization judgment, while rarely associating facial expressions, visual emotional states, and vocal emotional expression in a fine-grained manner. In real-world dubbing scenarios, the same text should correspond to different intonations, energy levels, rhythms, and cadences under different visual emotions such as anger, sadness, tension, or calmness. If the generated speech is based solely on the text and reference timbre, lacking the constraint of video emotional information, it is prone to inconsistencies between vocal emotion and facial expressions, reducing the realism of the dubbing result.

[0006] Meanwhile, while some existing solutions can make the overall duration of the generated speech close to the duration of the video clip, they lack fine-grained duration constraints between the target phoneme sequence and the lip movement sequence. Since the phoneme pronunciation order should be consistent with the lip movement change order, if only frame-level or overall duration matching is performed, problems such as local phoneme advances, lags, over-compression, or prolongation may still occur, resulting in similar overall durations but asynchronous local lip movements.

[0007] In summary, existing technologies have at least the following shortcomings: First, it is difficult to simultaneously utilize dubbing text, reference audio, and video information for unified generation; second, there is a lack of hierarchical modeling of phoneme-level local pronunciation styles and sentence-level global speaking styles in reference audio; third, there is insufficient modeling of the matching relationship between video visual emotion and speech emotion expression; fourth, there is insufficient constraint on the duration correspondence between target phonemes and lip movements; and fifth, timbre transfer, emotion expression, duration control, and lip-sync are often isolated from each other, lacking a unified and collaborative optimization mechanism.

[0008] Therefore, there is an urgent need for a video dubbing generation method and device that can learn phoneme-level local acoustic style and sentence-level global style in a hierarchical manner under the common constraints of the video to be dubbed, the dubbing text, and the reference audio, integrate video visual emotional features, and determine phoneme duration based on lip movement information, thereby generating target dubbing speech that is superior in terms of pronunciation clarity, emotional consistency, speaker similarity, audio-video synchronization, and overall naturalness. Summary of the Invention

[0009] To address the above problems, this invention proposes a video dubbing generation method based on multi-scale style learning, the method comprising the following steps: S1. Acquire and preprocess the video to be dubbed, the dubbing text, and the reference audio; S2. Perform phoneme conversion on the dubbing text to obtain the target phoneme sequence; S3. Extract phoneme-level acoustic style features based on the reference audio, and extract visual emotion features based on the video to be dubbed; S4. Based on the target phoneme sequence, phoneme-level acoustic style features, and visual emotion features, generate a phoneme-level intermediate speech representation; S5. Based on the lip movement information and target phoneme sequence of the video to be dubbed, determine the duration information corresponding to each phoneme so that the generated speech is aligned with the lip movements of the video; S6. Extract sentence-level global style features based on the reference audio, and generate a target Mel spectrum based on phoneme-level intermediate speech representation, duration information, and the sentence-level global style features; S7. Generate target dubbing speech based on the target Mel spectrum, and synthesize it with the video to be dubbed to obtain the target dubbing video.

[0010] Furthermore, the video to be dubbed, the dubbing text, and the reference audio are acquired and preprocessed, including: extracting video frames, detecting facial regions, and cropping lip regions from the video to be dubbed; performing text normalization on the dubbing text; and performing noise reduction, resampling, and spectral transformation on the reference audio to obtain video frame sequences, normalized text, and reference audio features that meet the model input requirements.

[0011] Furthermore, the dubbing text is converted into phonemes to obtain a target phoneme sequence, including: using a character-to-phoneme conversion module to convert the dubbing text into a phoneme symbol sequence corresponding to the target language, and encoding the phoneme symbol sequence to obtain a target phoneme embedding representation.

[0012] Furthermore, phoneme-level acoustic style features are extracted based on the reference audio, including: performing a short-time Fourier transform on the reference audio to obtain a reference Mel spectrum, combining the phoneme segmentation results, and extracting phoneme-level acoustic style features that reflect local pronunciation habits, prosodic details, and local acoustic style changes through a phoneme-level style coding network.

[0013] Furthermore, visual emotional features are extracted from the video to be dubbed, including: detecting facial regions from each frame of the video to be dubbed, extracting facial expression region features, and obtaining time-varying visual emotional features through an emotional feature encoding network to describe the character's emotional state in the dubbing segment.

[0014] Furthermore, based on the target phoneme sequence, phoneme-level acoustic style features, and visual emotional features, a phoneme-level intermediate speech representation is generated, including: using a multimodal fusion network to establish the correspondence between the target phoneme sequence and the phoneme-level acoustic style features, and between the target phoneme sequence and the visual emotional features, respectively, and fusing the two types of modal information to obtain a phoneme-level intermediate speech representation that integrates local acoustic style information and visual emotional information.

[0015] Furthermore, based on the lip movement information and target phoneme sequence of the video to be dubbed, the duration information corresponding to each phoneme is determined, including: extracting features from the lip region in the video to be dubbed to obtain lip movement sequence features; establishing a correspondence between the target phoneme sequence and the lip movement sequence using a monotonic attention mechanism; based on the correspondence, outputting the duration information corresponding to each phoneme through a duration prediction network, and constraining the duration of each phoneme according to the total duration of the video, so that the generated speech is consistent with the lip movement of the video in both the overall duration and the local pronunciation boundary.

[0016] Furthermore, sentence-level global style features are extracted based on the reference audio, including: extracting sentence-level global style vectors from the reference audio using a speaker representation extraction network. The sentence-level global style vectors are used to represent the overall timbre features, speaking habits, and overall sentence-level style consistency of the reference speaker.

[0017] Furthermore, based on phoneme-level intermediate speech representation, duration information, and sentence-level global style features, a target Mel spectrum is generated, including: length adjustment of the phoneme-level intermediate speech representation according to duration information to obtain temporal features that match the length of the target speech frame; inputting the length-adjusted temporal features and sentence-level global style features into a Mel spectrum decoding network to generate an initial Mel spectrum; and then refining the initial Mel spectrum through a spectrum post-processing network to obtain the target Mel spectrum.

[0018] Furthermore, the target dubbing speech is generated based on the target Mel spectrum and synthesized with the video to be dubbed to obtain the target dubbing video. This includes: using a vocoder to convert the target Mel spectrum into a target dubbing speech waveform, and synchronously synthesizing the target dubbing speech and the video to be dubbed according to the timeline of the video to be dubbed to output the target dubbing video.

[0019] Furthermore, the method also includes a training step: constructing a joint optimization objective that includes style consistency loss, sentiment classification loss, spectral reconstruction loss, monotonic alignment loss, and duration prediction loss, and performing end-to-end joint training on the phoneme-level acoustic style modeling module, visual sentiment modeling module, lip alignment module, and sentence-level global style modeling module to improve the timbre similarity, sentiment matching, pronunciation clarity, and audio-video synchronization effect of the generated speech.

[0020] Based on the above-mentioned video dubbing generation method based on multi-scale style learning, a video dubbing generation device based on multi-scale style learning is provided, including: a data preprocessing unit, a feature information extraction unit, a multimodal phoneme-level adaptation unit, a lip alignment unit, a sentence-level style learning unit, and a dubbing generation unit; each module is used to execute the corresponding steps in the above-mentioned video dubbing generation method based on multi-scale style learning.

[0021] Compared with the prior art, the beneficial effects of the present invention include: (1) This invention acquires the video to be dubbed, the dubbing text and the reference audio at the same time, and performs joint modeling of the text content, video screen and reference acoustic information, so that the generated speech can not only accurately express the target text content, but also take into account the video screen state and the style of the reference speaker, thereby improving the overall consistency of video dubbing generation.

[0022] (2) This invention extracts phoneme-level acoustic style features and sentence-level global style features from the reference audio, respectively, to achieve hierarchical learning of local pronunciation style and overall speaking style. Compared with the method of using only voiceprint or global speaker vector, this application can better preserve the local pronunciation habits, prosodic changes and overall timbre features of the reference speaker, thereby improving the pronunciation clarity, naturalness and speaker similarity of the generated speech.

[0023] (3) The present invention extracts visual emotional features from the video to be dubbed and associates the visual emotional information with the target phoneme sequence, so that the generated speech can be adjusted accordingly with the facial expressions and emotional state of the characters in the video, reducing the problem of inconsistency between the voice emotion and the screen expression, and improving the realism and immersion of the dubbing result.

[0024] (4) The present invention determines the duration information corresponding to each phoneme based on lip movement information and target phoneme sequence, and introduces monotonic alignment constraint to make the pronunciation order and local duration of the generated speech consistent with the changes in video lip shape, thereby improving the problem of local lip shape asynchrony that is easy to occur when only overall duration matching is performed.

[0025] (5) This invention optimizes the timbre transfer, emotional expression, pronunciation details and audio-visual synchronization in the same framework by jointly optimizing the goals of style consistency, emotional constraints, spectrum reconstruction, lip alignment and duration prediction, thereby improving the naturalness, stability and application adaptability of the target dubbing video. Attached Figure Description

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] Figure 1 A flowchart illustrating the video dubbing generation method based on multi-scale style learning provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a video dubbing generation device based on multi-scale style learning provided in an embodiment of the present invention; Figure 3 A schematic diagram illustrating the architectural principle of a video dubbing generation model based on multi-scale style learning provided in an embodiment of the present invention; Figure 4The Mel spectrum comparison diagram of the video dubbing generation method based on multi-scale style learning provided in the embodiments of the present invention and the audio generated by existing technologies. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the technical flow of the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] It should be noted that, in the absence of conflict, the various embodiments and technical features of the present invention can be combined with each other; in addition, the model structure, number of network layers, feature dimension, loss function form and processing order listed in the following embodiments are all preferred examples and do not constitute a limitation on the scope of protection of the present invention.

[0030] Example 1 This embodiment provides a video dubbing generation method based on multi-scale style learning. This method can be applied to film and television clip dubbing, animation character dubbing, short video speech generation, virtual digital human driving, and other scenarios requiring coordinated matching of speech with video lip movements, emotions, and speaking styles. For easier understanding, please refer to... Figure 1 The method includes the following steps: Step S101: Acquire and preprocess the video to be dubbed, the dubbing text, and the reference audio. In practice, the first step is to obtain the video to be dubbed. Dubbing text and reference audio The video to be dubbed is a video segment without target dubbing or where the original voice needs to be replaced; the dubbing text is the target text content for which voice is to be generated; and the reference audio is used to provide acoustic style and timbre information of the target speaker.

[0031] When preprocessing a video to be dubbed, you can first extract frames from the video according to a preset frame rate to obtain a video frame sequence: ; in, Indicates the number of video frames.

[0032] When preprocessing the dubbing text, text cleaning, punctuation standardization, number transcription, abbreviation restoration, and language unification can be performed to obtain standardized text. ; Preprocessing of the reference audio may include noise reduction, silence clipping, resampling, and short-time Fourier transform. Let the original reference audio be... After preprocessing, a standardized audio signal is obtained: ; Furthermore, the Mel spectrum can be extracted from the reference audio: ; in, The reference audio corresponds to the Mel spectrum characteristics.

[0033] In one embodiment, the video frame rate is set to 25fps and the audio sampling rate is set to 22.05kHz to allow for subsequent audio and video duration constraints and feature synchronization processing.

[0034] Step S102: Perform phoneme conversion on the dubbing text to obtain the target phoneme sequence. After text preprocessing, a character-to-phoneme conversion is performed on the normalized text T_norm to obtain the target phoneme sequence: ; in, Indicates the length of the phoneme sequence.

[0035] Furthermore, the target phoneme sequence can be mapped to a continuous embedding space using a phoneme encoder to obtain a phoneme embedding representation: ; in, For phoneme encoders, For target phoneme representation, For feature dimensions.

[0036] In one embodiment, the character-to-phoneme conversion module can be a rule-based G2P tool or a neural network-based end-to-end phoneme conversion model.

[0037] Step S103: Extract phoneme-level acoustic style features based on the reference audio, and extract visual emotion features based on the video to be dubbed. S103.1 Phoneme-level Acoustic Style Feature Extraction In this embodiment, the Mel spectrum of the reference audio. Local style extraction is performed to obtain phoneme-level acoustic style features that reflect pronunciation habits, local prosodic variations, stress variations, and local timbre preferences. This can be represented as: ; in, For phoneme-level style coding networks, These are the local style features after downsampling.

[0038] Furthermore, a correspondence can be established between text phonemes and local style features through cross-attention to obtain acoustic style context features aligned with the target phoneme sequence: ; in, As a query item As keys and values, this allows each target phoneme to select the local articulation style information most relevant to it from the reference audio.

[0039] S103.2 Visual Emotion Feature Extraction Simultaneously, face region detection and expression region cropping are performed on each frame of the dubbed video to extract facial expressions and visual emotional features. Let the video frame sequence be... Then, visual emotional characteristics can be represented as: ; Furthermore, visual emotion representations are obtained by temporal modeling and downsampling of visual emotion features using an emotion encoding network: ; Then, through cross-attention calculation between the target phoneme embedding and visual emotional features, the emotional context representation associated with the target phoneme sequence is obtained: ; That is, each target phoneme can obtain corresponding emotional information from changes in video facial expressions, which can be used for subsequent adjustment of voice emotional expression.

[0040] Step S104: Generate phoneme-level intermediate speech representation In obtaining local acoustic style context features and visual emotional context features Then, the two can be concatenated along the channel dimension and jointly modeled using a self-attention fusion module: ; Then, the fused multimodal context representation is fused element-wise or residually superimposed with the target phoneme embedding representation to obtain the phoneme-level intermediate speech representation: ; The purpose of this step is to simultaneously encode the local pronunciation style of the reference speaker and the visual emotional state of the video character at the phoneme granularity, so that the subsequently generated speech can not only retain the fine-grained pronunciation features of the target speaker, but also reflect the emotional changes consistent with the video at the local phoneme level.

[0041] Step S105: Determine the duration information corresponding to each phoneme To ensure that the generated speech is synchronized with the lip movements of the video to be dubbed, it is necessary to further combine lip movement information to predict the duration of each target phoneme. First, the lip region is extracted from the video frame sequence, and lip movement features are obtained through a lip-coding network: ; in, This represents the timing of lip movements.

[0042] Then, target phoneme embedding representation is used. Lip movement characteristics The monotonous attention mechanism between them establishes the sequential alignment relationship between phonemes and lip movements: ; In obtaining phoneme-lip movement context features Then, the duration prediction network outputs the duration corresponding to each phoneme: ; Furthermore, to ensure that the total duration of the generated speech matches the total duration of the video to be dubbed, a normalization constraint can be applied to the predicted duration: ; in, This represents the total frame length of the target audio, calculated from the video frame count. Indicates the duration of the constrained phoneme.

[0043] Step S106: Generate the target Mel spectrum After completing phoneme-level local modeling and duration prediction, sentence-level global style features need to be extracted from the reference audio to represent the target speaker's overall timbre, sentence-level speaking habits, and overall style consistency. Specifically, sentence-level style vectors can be extracted from the reference audio using a speaker representation extraction network: ; Then, the length adjuster is used according to the constrained phoneme duration. Phoneme-level intermediate speech representation Extending to the speech frame length, we obtain the length-adjusted timing representation: ; Then, the length-adjusted temporal representation and the sentence-level global style vector are input together into the Mel spectrum decoding network to generate the initial spectral latent representation: ; Furthermore, a style modulation mechanism can be introduced into the normalization layer of the decoding network, allowing sentence-level global style vectors to participate in feature transformations at each layer: ; in, and These represent the characteristic mean and standard deviation, respectively. and These represent the scaling and bias parameters generated from the sentence-level style vectors, respectively.

[0044] After the initial spectral implicit representation is generated, it is further refined by a spectral post-processing network to obtain the target Mel spectrum: ; Step S107: Generate the target voice-over and synthesize it with the video to be dubbed. In generating the target Mel spectrum Then, a vocoder can be used to convert it into the time-domain target voiceover speech: ; Furthermore, based on the timeline and frame sequence of the video to be dubbed, the target dubbing voice is... Synchronously synthesize the audio with the video to be dubbed to obtain the target dubbed video: ; In one embodiment, the vocoder can be a high-fidelity vocoder based on a generative adversarial network.

[0045] During the training phase, an end-to-end joint training approach can be adopted to jointly optimize the local style modeling module, visual emotion modeling module, lip alignment module, and global style decoding module. The total loss function can be expressed as: ; in, Style consistency loss is used to constrain the consistency of generated speech with the reference speaker in timbre and style; Emotional constraint loss is used to improve the matching degree between voice emotion and video facial expression state; This is the spectrum reconstruction loss, used to constrain the difference between the predicted spectrum and the true spectrum; This is a monotonic alignment loss used to ensure that the phoneme-lip movement alignment relationship satisfies temporal monotonicity; This is the duration prediction loss, used to improve the accuracy of duration prediction for each phoneme.

[0046] In one embodiment, the reference audio, the dubbing text, and the video to be dubbed come from the same training sample; wherein, the reference audio is used to provide speaker style information, the dubbing text is used to provide target semantic content, and the video is used to provide lip movement constraints and visual emotional information. Finally, through joint training, the model learns the mapping relationship between local pronunciation style, visual emotion, and video synchronization information.

[0047] To verify the effectiveness and advancement of this patented method, it is compared with current mainstream existing technologies in the same field. (See attached document.) Figure 4 Observing the red area of ​​the Mel spectrogram, it can be found that compared with other models, this method has less noise in the non-speech range of dubbing and has better sound quality. At the same time, the spectral lines in the white area are clearer, and the prosodic features reflected by the spectral contour are very close to those of the real audio, which indicates that the pronunciation quality and prosodic alignment effect of this method are also better.

[0048] Example 2 For ease of understanding, please refer to Figure 2 and Figure 3 This application also provides embodiments of a video dubbing generation apparatus based on multi-scale style learning, including: The data preprocessing unit 201 is used to acquire and preprocess the video to be dubbed, the dubbing text, and the reference audio. The feature information extraction unit 202 is used to perform phoneme conversion on the dubbing text to obtain a target phoneme sequence, extract phoneme-level acoustic style features based on the reference audio, and extract visual emotion features based on the video to be dubbed. The multimodal phoneme-level adaptation unit 203 is used to generate a phoneme-level intermediate speech representation based on the target phoneme sequence, the phoneme-level acoustic style features, and the visual emotion features. The lip alignment unit 204 is used to determine the duration information corresponding to each phoneme based on the lip movement information of the video to be dubbed and the target phoneme sequence, so as to align the generated speech with the lip movement of the video. The sentence-level style learning unit 205 is used to extract sentence-level global style features based on the reference audio, and generate a target Mel spectrum based on the phoneme-level intermediate speech representation, the duration information, and the sentence-level global style features. The dubbing generation unit 206 is used to generate target dubbing speech based on the target Mel spectrum, and synthesize it with the video to be dubbed to obtain the target dubbing video.

[0049] Furthermore, the data preprocessing unit 201 is specifically used for: The process involves frame extraction, face region detection, and lip region cropping for dubbed videos. The voice-over text was cleaned and standardized. The reference audio is subjected to noise reduction, resampling, and spectral transformation.

[0050] Furthermore, the feature information extraction unit 202 is specifically used for: The dubbing text is converted into a target phoneme sequence and then phoneme encoded. Extract phoneme-level acoustic style features from reference audio; Extract visual emotional features from videos to be dubbed.

[0051] Furthermore, the multimodal phoneme-level adaptation unit 203 is specifically used for: Establish the correspondence between the target phoneme sequence and the phoneme-level acoustic style features; Establish a correspondence between target phoneme sequences and visual emotional features; By integrating acoustic style information, visual emotional information, and target phoneme representation, a phoneme-level intermediate speech representation is generated.

[0052] Furthermore, the lip alignment unit 204 is specifically used for: Extract lip movement features from the video to be dubbed; Establish the alignment relationship between the target phoneme sequence and lip movement features; Predict the duration information corresponding to each phoneme and constrain it based on the total video duration.

[0053] Furthermore, Unit 205, which focuses on sentence-level style learning, is specifically used for: Extract sentence-level global style features from reference audio; The length of the intermediate speech representation at the phoneme level is adjusted based on the duration information; The target Mel spectrum is generated by combining sentence-level global style features.

[0054] Furthermore, the voice-over generation unit 206 is specifically used for: Convert the target Mel spectrum into the target dubbing speech; The target voiceover is synchronously synthesized with the video to be dubbed, and the target voiceover video is output.

[0055] It should be noted that although the above embodiments are mainly illustrated using video dubbing scenarios as examples, the technical solutions of this application are not limited to specific video types such as movies, animations, and short videos. They can also be applied to application scenarios that require consideration of timbre transfer, emotional expression, and audio-visual synchronization, such as digital human voice-driven, character localization dubbing, cross-language film and television re-dubbing, and human-computer interactive broadcasting.

[0056] The embodiments described above are merely illustrative of implementation methods of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A video dubbing generation method based on multi-scale style learning, characterized in that, include: S1. Obtain the video to be dubbed, the dubbing text, and the reference audio, and preprocess the video to be dubbed, the dubbing text, and the reference audio; S2. Perform phoneme conversion on the dubbing text to obtain the target phoneme sequence; S3. Extract phoneme-level acoustic style features based on reference audio, and extract visual emotional features based on the video to be dubbed; S4. Based on the correlation between the target phoneme sequence and phoneme-level acoustic style features and visual emotional features, multimodal fusion is performed at the phoneme granularity to obtain a phoneme-level intermediate speech representation. S5. Extract lip movement information from the video to be dubbed, perform monotonic attention alignment based on the target phoneme sequence and lip movement information to obtain lip-phoneme alignment features, predict the relative duration of each phoneme based on the lip-phoneme alignment features, and recalibrate the relative duration according to the total duration of the video to be dubbed to obtain the duration information of each phoneme. S6. Extract sentence-level global style features based on reference audio, adjust the length of phoneme-level intermediate speech representation according to duration information to obtain extended speech representation, normalize the spectral hidden representation during Mel spectrum decoding, generate gain and bias parameters based on sentence-level global style features, perform affine transformation on the normalized spectral hidden representation to generate target Mel spectrum; S7. Generate the target dubbing speech based on the target Mel spectrum, and synthesize it with the video to be dubbed to obtain the target dubbing video.

2. The video dubbing generation method based on multi-scale style learning according to claim 1, characterized in that, S1 includes: The process involves frame extraction, face region detection, and lip region cropping for dubbed videos. The voiceover text is cleaned, punctuation is standardized, digitally transcribed, and language is unified. The reference audio is denoised, silent segments are clipped, resampled, and spectral transformation is performed to obtain video frame sequences, standardized text, and reference audio features.

3. The video dubbing generation method based on multi-scale style learning according to claim 1, characterized in that, S2 includes: using a character-to-phoneme conversion module to convert the preprocessed dubbing text into a phoneme symbol sequence corresponding to the target language, and using a phoneme encoder to encode the phoneme symbol sequence to obtain the target phoneme embedding representation.

4. The video dubbing generation method based on multi-scale style learning according to claim 1, characterized in that, The step S3, which involves extracting phoneme-level acoustic style features based on reference audio, includes: performing time-frequency analysis on the reference audio to obtain a reference Mel spectrum; segmenting the reference Mel spectrum at the phoneme level based on the phoneme boundary information of the reference audio; and encoding the segmented reference Mel spectrum using a phoneme-level style encoder to obtain phoneme-level acoustic style features that reflect local pronunciation habits, prosodic details, and local acoustic style changes.

5. The video dubbing generation method based on multi-scale style learning according to claim 1, characterized in that, The step S3, which involves extracting visual emotional features from the video to be dubbed, includes: detecting and extracting face regions from the video to be dubbed; extracting frame-by-frame emotional features based on the face regions; performing temporal downsampling and encoding on the frame-by-frame emotional features to obtain visual emotional features; and establishing a correspondence between target phonemes and visual emotional expressions based on the correlation between the target phoneme sequence and the visual emotional features.

6. The video dubbing generation method based on multi-scale style learning according to claim 1, characterized in that, S4 includes: using the target phoneme embedding representation corresponding to the target phoneme sequence as the query term, performing cross-attention calculation with the phoneme-level acoustic style features and visual emotion features respectively to obtain acoustic style context features and visual emotion context features; and concatenating, residual superimposing or self-attention fusion of the acoustic style context features, visual emotion context features and target phoneme embedding representation to obtain the phoneme-level intermediate speech representation.

7. The video dubbing generation method based on multi-scale style learning according to claim 1, characterized in that, S5 includes: extracting lip region motion features from the video to be dubbed; performing monotonic attention calculation based on the target phoneme embedding representation corresponding to the target phoneme sequence and the lip region motion features to obtain lip-phoneme alignment features; inputting the lip-phoneme alignment features into the duration prediction module and outputting the relative duration corresponding to each phoneme; and normalizing or scaling the relative duration according to the number of video frames of the video to be dubbed or the total frame length of the target speech to obtain the duration information corresponding to each phoneme.

8. The video dubbing generation method based on multi-scale style learning according to claim 7, characterized in that, In the monotonic attention calculation process, a monotonic constraint loss is introduced to constrain the alignment path between the target phoneme sequence and the lip movement information to meet the order consistency, so that the generated speech is consistent with the video lip movement in terms of pronunciation order, overall duration and local pronunciation boundaries.

9. The video dubbing generation method based on multi-scale style learning according to claim 1, characterized in that, It also includes a model training step, in which speaker style consistency loss, emotion classification loss, spectrum reconstruction loss, lip shape monotonic alignment loss, and duration prediction loss are jointly optimized to improve the pronunciation clarity, emotion matching, speaker consistency, and audio-video synchronization of the generated speech.

10. A video dubbing generation device based on multi-scale style learning, characterized in that, include: The data preprocessing unit is used to acquire and preprocess the video to be dubbed, the dubbing text, and the reference audio. The feature information extraction unit is used to convert the dubbing text into phonemes to obtain the target phoneme sequence, extract phoneme-level acoustic style features based on the reference audio, and extract visual emotion features based on the video to be dubbed. The multimodal phoneme-level adaptation unit is used to perform multimodal fusion at the phoneme granularity based on the correlation between the target phoneme sequence and phoneme-level acoustic style features and visual emotional features, respectively, to generate a phoneme-level intermediate speech representation. The lip alignment unit is used to extract lip movement information from the video to be dubbed. Based on the target phoneme sequence and the lip movement information, monotonic attention alignment is performed to obtain lip-phoneme alignment features. Based on the lip-phoneme alignment features, the relative duration of each phoneme is predicted. The relative duration is recalibrated according to the total duration of the video to be dubbed to obtain the duration information of each phoneme. The sentence-level style learning unit is used to extract sentence-level global style features from the reference audio, adjust the length of the phoneme-level intermediate speech representation according to the duration information to obtain the extended speech representation, normalize the spectral hidden representation during the Mel spectrum decoding process, generate gain parameters and bias parameters based on the sentence-level global style features, and perform affine transformation on the normalized spectral hidden representation to generate the target Mel spectrum. The dubbing generation unit is used to generate target dubbing speech based on the target Mel spectrum, and synthesize it with the video to be dubbed to obtain the target dubbing video.

Images