A cross-language dubbing method based on acoustic feature anchoring and label reconstruction

Abstract

The application discloses a cross-language dubbing method based on acoustic feature anchoring and label reconstruction, comprising: collecting source language audio data; extracting features, constructing a judgment model according to the features, the judgment model identifying emotional intent and generating target language translation text; converting the emotional intent into a target language label sequence, inserting the target language label sequence into a semantic node to form a label-enhanced synthetic script, generating target language dubbing voice waveform through a synthesis rule, calculating a deviation value, identifying a non-key adjustable area based on a recognition rule, comparing the deviation value with a preset deviation threshold value, calling a stretching algorithm to adjust the time length when the deviation value exceeds the preset threshold value, generating dubbing voice waveform synchronized with the original video, and establishing the priority decision weight of acoustic features at the model training level, so that the emotional judgment takes the real physical performance evidence of the actor as the core basis, and only when the acoustic evidence is ambiguous, the text semantics is referenced for auxiliary calibration.

Description

Technical Field

[0001] This invention relates to the fields of speech signal processing and artificial intelligence, and in particular to a cross-language dubbing method based on acoustic feature anchoring and tag reconstruction. Background Technology

[0002] With the continued growth in demand for cross-border distribution of global film and television content, cross-language dubbing technology, which replaces the original audio with the target language while preserving the integrity of the original performance, has become a core component of media localization production. Traditional dubbing relies on human translators and actors recording, which is costly and time-consuming, making it difficult to meet the needs of large-scale production.

[0003] In recent years, deep learning-based automated dubbing technology has gradually become a mainstream solution in the industry. Its core technical path can be summarized as the fusion of voice conversion (VC) based on the original audio waveform and text-to-speech (TTS), also known in the industry as "voice cloning + cross-language TTS." This approach extracts the speaker's timbre features from the source speech and injects them into the target language speech synthesis process, thereby generating a target language dubbing while preserving the original actor's voice characteristics. However, due to the inherent coupling between the cross-language adaptation and voice conversion stages, this mainstream technical architecture has long suffered from the following two interrelated technical defects: First, there is the problem of accent remnants caused by prosodic feature contamination. During the voice cloning process, the latent space of existing VC models struggles to effectively separate features related to speaker identity from those related to the prosodic habits of the source language. When the model maps the source speech to the target timbre space and simultaneously completes language conversion, prosodic attributes such as the tone curve, rhythmic pattern, and stress distribution of the source language are forcibly transferred to the synthesized speech as unintended "accent" remnants. For example, when converting original Chinese speech to English, the synthesized speech often exhibits noticeable Chinese tone variations (i.e., a "Chinglish" accent), significantly reducing the naturalness and authenticity of the sound, making it difficult to meet the quality requirements of professional film and television translation. Second, there is the problem of lost performance intent due to the break in the emotional feature transfer path. Under the current "VC+TTS" splicing architecture, in order to achieve cross-language adaptation, the emotional expression in the original voice must undergo a multi-level transformation of "source language acoustic features → text semantics → target language acoustic features". This cascaded processing brings two dilemmas: if the VC module is relied upon to directly convey emotions, then because emotional features and prosodic features are highly entangled in the latent space of the VC model, when the prosodic features of the source language are forcibly stripped to adapt to the pronunciation habits of the target language, the emotional features are also indiscriminately attenuated or distorted; if a text sentiment analysis module is used for emotional calibration, then it falls into the dilemma of emotional ambiguity of a single text modality - the same text can correspond to completely different performance intentions, and text analysis cannot restore subtle performance details such as breathing, trembling, and accent.

[0004] In this splicing architecture, emotional features lack a direct and stable anchoring path from source speech performance to target speech synthesis. This results in a significant loss of rich performance details from the original video, insufficient emotional expressiveness in the synthesized dubbing, and a perceptible deviation between the emotional tone and the visual atmosphere—a problem of "consistency between audio and visual emotional expression." In summary, the fundamental shortcomings of the existing "voice cloning + cross-language TTS" technical approach lie in: insufficient decoupling ability of acoustic features (inability to separate identity, rhythm, and emotion) and a broken cross-language emotional reconstruction path (lack of a direct anchoring mechanism from performance to synthesis). These two defects are mutually reinforcing, making it difficult for this approach to simultaneously achieve the dual goals of "no accent residue" and "high-fidelity emotional transfer." Summary of the Invention

[0005] This application provides a cross-language dubbing method based on acoustic feature anchoring and label reconstruction. By establishing the priority decision weight of acoustic features at the model training level, the emotion judgment is based on the actor's real physical performance evidence (fundamental frequency tremor, energetic breathing, non-verbal voice) as the core basis, and the semantics of the text are referenced only for auxiliary calibration when the acoustic evidence is ambiguous.

[0006] This application provides a cross-language dubbing method based on acoustic feature anchoring and tag reconstruction, including: S101, Collect source language audio data, the source language audio data includes source language audio signal and source language text, and align the source language audio signal with the source language text; S102, extract the features of the source language audio signal and the features of the source language text, construct a judgment model based on the features of the source language audio signal and the features of the source language text, the judgment model recognizes the emotional intent based on the source language audio signal, and generates the target language translation text based on the source language text; S103, convert the emotional intent into a target language tag sequence, insert the target language tag sequence into the semantic node corresponding to the target language translated text, and form a tag-enhanced synthesis script; S104, the tag-enhanced synthesis script generates a target language dubbing voice waveform through synthesis rules, calculates the deviation value between the total duration of the target language dubbing voice waveform and the reference duration of the original video, identifies non-critical adjustable areas based on recognition rules, compares the deviation value with a preset deviation threshold, and when the deviation value exceeds the preset threshold, calls the scaling algorithm to adjust the duration to generate a dubbing voice waveform synchronized with the original video; otherwise, the duration is not adjusted. S105 encapsulates the voiceover waveform synchronized with the original video and the original video.

[0007] Preferably, the specific steps for converting emotional intent into a target language tag sequence are as follows: decompose the composite emotion in the emotional metadata, decompose the composite emotion into multiple sub-emotions, search for the tag corresponding to each sub-emotion in the cross-language emotion-prosody mapping knowledge base, if there are multiple candidate tags, select the one with the highest emotional intensity as the target tag, and convert each emotion category into a target tag to form a target language tag sequence.

[0008] Preferably, the method for identifying non-critical adjustable regions according to the recognition rules is as follows: the recognition rules include a first-level signal layer, a second-level syntactic layer, and a third-level instruction layer: the first-level signal layer identifies silent segments in the target language dubbed speech waveform based on energy threshold speech activity detection; the second-level syntactic layer identifies grammatical pauses corresponding to punctuation marks based on dependency parsing of the target language text in the tag-enhanced synthesis script; the third-level instruction layer parses the tag-enhanced synthesis script and identifies speech segments corresponding to non-ideographic acoustic control tags; the weighted union of the three-level recognition results is taken as the non-critical adjustable region.

[0009] Preferably, the calculated deviation value is compared with a preset deviation threshold. If the calculated deviation value is greater than the preset deviation threshold, the scaling algorithm is called to adjust the duration. Otherwise, the original speech is directly output, and the phase vocoder scaling algorithm is selected. Different scaling weights are assigned to the adjustable regions according to different categories. Among them, the weight of the silent segment is greater than that of the grammatical pause point, and the weight of the grammatical pause point is greater than that of the speech segment corresponding to the non-ideographic acoustic control tag.

[0010] Preferably, it also includes a cross-language dubbing system, the cross-language dubbing system comprising: The sentiment intent anchoring processor is used to extract features from the source language audio signal and the source language text to identify sentiment intent; A tag reconstruction and script generator is used to convert sentiment intent into a sequence of target language tags, forming a tag-enhanced synthetic script; A speech synthesis driver is used to generate target language dubbing speech waveforms from tag-enhanced synthesis scripts using synthesis rules; The time-domain calibration processor is used to call the scaling algorithm to adjust the duration; Audio and text input interfaces are used to receive source language audio, source language text, and target language translated text; The system bus and memory, the memory storing computer programs, drive the above-mentioned components to work together when the programs are loaded.

[0011] One or more technical solutions provided in this application have at least the following technical effects or advantages: By establishing the priority decision weight of acoustic features at the model training level, emotion judgment is based primarily on the actor's real physical performance evidence (fundamental frequency tremor, energetic breathing, non-verbal sounds), with textual semantics only used for auxiliary calibration when acoustic evidence is ambiguous. When acoustic performance conflicts with textual semantics, the system always prioritizes acoustic features, thereby eliminating the "consistency between audio-visual emotional expression" defect at its source, simulating the cognitive logic of human directors in "recognizing emotions through sound," and fundamentally solving the problem of modal unreliability in emotion judgment. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a cross-language dubbing method based on acoustic feature anchoring and tag reconstruction according to the present invention. Figure 2 This is a schematic diagram illustrating the reconstruction process of the emotional intent-based tag-enhanced synthetic script according to the present invention; Figure 3 This is a schematic diagram illustrating the process of adjusting the duration by calling the scaling algorithm in this invention. Detailed Implementation

[0013] To facilitate understanding of the present invention, a more complete description of this application will be given below with reference to the accompanying drawings, which illustrate preferred embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to enable a more thorough and complete understanding of the disclosure of the present invention.

[0014] It should be noted that the terms "vertical," "horizontal," "up," "down," "left," "right," and similar expressions used in this article are for illustrative purposes only and do not represent the only possible implementation.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention; the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0016] Example 1: Figure 1 This is a flowchart illustrating a cross-language dubbing method based on acoustic feature anchoring and tag reconstruction according to an embodiment of the present invention, including: S101, Collect source language audio data, the source language audio data includes source language audio signal and source language text, and align the source language audio signal with the source language text; Specifically, if audio is embedded in video, the audio track is separated from the video file. This involves using video editing software like Adobe Premiere to separate the audio track from the video. The video is dragged into the timeline, and the software automatically separates the video and audio tracks. The video track is deleted, leaving only the audio track. The audio track is then exported to generate source language audio data, and its timestamp is recorded. The source language audio data is then played back to check for any dropped notes, background noise, or silent segments. It is compared with the original video to check the integrity of the audio content. An Automatic Speech Recognition (ASR) tool is used to convert the source language audio data into source language text, and a timestamp is recorded. The source language text is then aligned with the source language audio signal using the timestamp. The source language audio signal is then preprocessed: it is converted to mono to reduce the complexity of multi-channel synchronization; the sampling frequency and bit depth are set; the overall audio volume is adjusted to a standard level to avoid errors caused by volume fluctuations; non-dialogue markers are removed from the source language text; speaker identifiers are cleaned up; punctuation marks are standardized; Chinese full-width characters are converted to half-width characters; and redundant spaces or line breaks are deleted.

[0017] S102, extract the features of the source language audio signal and the features of the source language text, construct a judgment model based on the features of the source language audio signal and the features of the source language text, the judgment model identifies the emotional intent based on the source language audio signal, and generates the target language translation text based on the source language text; wherein, the emotional intent includes emotional category, intensity value, and non-verbal event temporal information; Furthermore, source language audio signal features are extracted, including fundamental frequency trajectory, energy envelope, and non-verbal sound events. For fundamental frequency trajectory extraction, a deep learning-based fundamental frequency estimation method (CREPE model) is used to extract fundamental frequency value sequences of continuous speech segments. The fundamental frequency value sequence is used to identify fundamental frequency change patterns, which reflect emotional states. For example, a rising fundamental frequency with drastic fluctuations indicates anger or excitement; a falling fundamental frequency with a tendency to stabilize indicates sadness or fatigue; and a brief surge followed by a drop indicates surprise. These fundamental frequency change patterns constitute the fundamental frequency trajectory. For energy envelope extraction, the energy envelope (RMS) is quantified by calculating the root mean square of the signal samples within a short time frame of speech. Logarithmic transformation is often used to compress the dynamic range to reflect dynamic sound change patterns (such as sudden increases, sustained low frequencies, or drastic fluctuations), thereby associating different emotional states. The formula for calculating the energy envelope is: ,in, The energy envelope reflects the average power of the signal within a short frame and is directly related to the loudness or energy level of the sound. x(n) is the sampled value (amplitude) of the speech signal at discrete time point n, where n is the discrete-time index, and N is the total number of sampling points (frame length) in the current analysis frame. The speech is framed using a distance window, divided into 25ms windows with a 10ms step size. A time series is generated for each frame using the above formula. To compress the dynamic range, the calculated... Taking the logarithm, the formula is: ,in, Decibels (dB) are used to compress the dynamic range of signals, making them more suitable for human perception and digital processing. For energy envelope, The value is a very small positive number to avoid numerical overflow. For non-verbal sound events, which refer to sounds in speech that do not carry semantic content, such as sighs, laughter, coughs, and inhalations, they are often used to convey emotions or physiological states. Rules are set to calculate the frequency band energy of each frame and detect the energy decline trend. If the low-frequency energy continues to decline and the duration is greater than a preset threshold, it is identified as a sigh. If the high-frequency energy suddenly increases and the zero-crossing rate is greater than a preset frequency threshold, it is detected as laughter. A lightweight CNN model is used. The non-verbal sound events and the set rules are input into the CNN model. The model outputs the non-verbal event type and its start and end timestamps according to the set rules. Features of the source language text are extracted. A 768-dimensional semantic feature vector is extracted using the BERT encoder. The source language text ID sequence is input into the BERT encoder. Long-distance dependencies are captured through the self-attention mechanism. The mean pooling result of the last hidden state of the BERT model is taken as the 768-dimensional semantic representation of the text. The architecture of the decision model consists of an acoustic encoder, a text encoder, a fusion layer, and a classification layer. The acoustic encoder is a 6-layer one-dimensional convolutional neural network (CNN). Each layer uses small convolutional kernels to progressively extract local acoustic patterns and reduces temporal resolution through pooling layers. The text encoder uses a 12-layer Transformer structure of a pre-trained language model, freezing the bottom-level parameters and fine-tuning the top-level parameters to adapt to the sentiment classification task. The fusion layer concatenates acoustic and text feature vectors and maps them to the joint feature space through a fully connected layer. During the training phase, the decision model assigns higher optimization weights to the sentiment classification loss of the acoustic feature path than to the sentiment classification loss of the text feature path, thereby solidifying the priority decision-making power of acoustic features at the model parameter level. The classification layer of the decision model outputs sentiment metadata such as sentiment category, intensity value, and non-verbal events, mapping non-verbal event timestamps to the text word level to mark the key positions of sentiment expression, i.e., identifying sentiment intent. Sentiment vectors are injected into the Transformer architecture. The sentiment vectors are formed by encoding the sentiment category into an embedding vector. The embedding vectors are concatenated with word embeddings to generate the translated text. Specific implementation: The model architecture adopts a dual-encoder structure. The acoustic encoder consists of 6 layers of CNN, with an 80-dimensional Mel-frequency spectrogram as input and a 256-dimensional acoustic feature vector as output. The text encoder is a 12-layer Transformer, outputting a 768-dimensional semantic feature vector. The fusion layer concatenates the two and then reduces the dimensionality through a fully connected layer. The classification layer outputs the sentiment category probability. The loss function is designed as: L_total = L_cls + α×L_audio_cls + β×L_text_cls + γ×L_consistency Wherein: L_cls is the final classification loss (cross-entropy) based on fused features, L_audio_cls is the auxiliary classification loss based solely on acoustic features, L_text_cls is the auxiliary classification loss based solely on text features, and L_consistency is the modal consistency loss, used to encourage acoustic features and text features to align in the sentiment space. α>β is set, meaning the weight of the acoustic auxiliary classification loss is significantly higher than the weight of the text auxiliary classification loss. For example, α=0.7, β=0.3, and γ=0.2 can be selected. Through this asymmetric weight configuration, the model is forced to learn during training that when the sentiment indications of acoustic features and text features conflict, the prediction error of the acoustic path will result in a greater penalty, thus causing the model parameters to converge towards "more trust in acoustic evidence." The above loss function and asymmetric weight configuration are only one specific implementation method for prioritizing acoustic features in this invention, and not the only method.

[0018] S103, convert the emotional intent into a target language tag sequence, insert the target language tag sequence into the semantic node corresponding to the target language translated text, and form a tag-enhanced synthesis script; Specifically, such as Figure 2As shown, a cross-linguistic emotion-prosodic mapping knowledge base is set up. The pre-set cross-linguistic emotion-prosodic mapping knowledge base is queried. This knowledge base includes emotion-acoustic feature mapping and a language-independent label library. The cross-linguistic emotion-prosodic mapping knowledge base stores the typical acoustic differences in the same emotional intention across different language cultures, as well as the conversion rules from source language acoustic feature descriptions to target language structured acoustic control label sequences. It is used to convert source language emotion metadata into target language structured acoustic control label sequences, achieving cross-linguistic emotion transfer. The specific steps for converting emotion intention into target language label sequences are: decomposing the complex emotion in the emotion metadata, for example, decomposing depression / sadness + trembling into... The two sub-emotions, sadness and trembling, are identified by searching for corresponding tags in a cross-linguistic emotion-prosodic mapping knowledge base. If multiple candidate tags exist, the tag with the highest emotion intensity is selected as the target tag. Each emotion category is then converted into a target tag, forming a target language tag sequence. This target language tag sequence is not a simple mapping of the acoustic features of the source language, but a cross-linguistic reconstruction of the source performance intention. It deconstructs the emotional features in the source language that are highly coupled with prosody into abstract emotional instructions, and then reorganizes them into an ordered acoustic control flow according to the pronunciation habits of the target language. This establishes a direct anchoring path from the source performance to the target synthesis for emotional expression. Examples of conversion: Typical realization of "anger" in Chinese: high fundamental frequency, fast speech rate, high-frequency energy → Mapped to American English: enhanced stress, low-frequency threat, and falling intonation at the end of the sentence; Typical realization of "sadness" in Chinese: low fundamental frequency, slow speech rate, breathy voice → Mapped to American English: breathy voice, trill at the end of the sentence, and long pauses. The generated target language translation text is input into a Natural Language Processing (NLP) tool. The tool performs word segmentation, part-of-speech tagging (POS), and dependency parsing to identify key semantic components, such as the sentence "I feel". The tag "soheavy..." has the following parts of speech: I (pronoun), feel (verb), so (adverb), heavy (adjective), and (punctuation). The dependency relationship is that feel is the core verb, and heavy is the complement of feel. Semantic nodes related to emotion expression (adjective "heavy" and verb "feel") are marked. Based on the prosodic characteristics of the target language (English stress is on the second syllable and the intonation falls at the end of the sentence), candidate positions for tag insertion are determined. For example, "heavy" is the core word expressing a feeling of depression, so "suppressed sadness" is suitable for insertion. Based on the above steps, a list of semantic nodes and candidate insertion positions are output. Tag insertion rules are set, with compound tags taking precedence over single tags to avoid tag conflicts. Tags are inserted after adjectives, adverbs, and verbs. The tag insertion positions are adjusted according to the set tag insertion rules and context. If multiple tags compete for the same position, they are merged according to the tag priority in the cross-language emotion-prosodic mapping knowledge base. Based on the semantic node list, candidate insertion positions, and tag insertion rules, the target language tag sequence is inserted into the semantic nodes corresponding to the target language translated text to form a tag-enhanced synthesis script.

[0019] A specific example is as follows: Input emotional metadata: Chinese “suppressed sadness + trembling”; Target language translated text: English “I feel so heavy…”; Tag mapping: “suppressed sadness + trembling” → English tags [voice_tremor_fade], [breath_shaky]; Semantic node analysis: Text: “I feel so heavy…”; Key nodes: ("heavy", "ADJ") (expressing a feeling of suppression), ("feel", "VERB") (action initiation); Insert [voice_tremor_fade] after “heavy” and [breath_shaky] after “feel”; Dynamic adjustment: To avoid the conflict between the “feel” and “heavy” tags, set the intensity of [breath_shaky] to “light”, and the output is: “I feel [breath_shaky_light] so heavy[voice_tremor_fade]…”.

[0020] S104: The tag-enhanced synthesis script generates a target language dubbing voice waveform using synthesis rules. The deviation between the total duration of the target language dubbing voice waveform and the reference duration of the original video is calculated. Non-critical adjustable areas are identified based on recognition rules. The deviation value is compared with a preset deviation threshold. When the deviation value exceeds the preset threshold, a scaling algorithm is called to adjust the duration, generating a dubbing voice waveform synchronized with the original video. Otherwise, the duration is not adjusted. The original video refers to a complete video file that has not undergone dubbing processing and retains the original image and initial timeline. Furthermore, such as Figure 3 As shown, a pre-trained speech synthesis engine for the target language (based on the VITS architecture) is selected. The speech synthesis engine reads the text characters and label parameters of the label-enhanced synthesis script, converts the labels in the script into acoustic control parameters that the engine can understand, and generates the target language dubbing speech waveform through synthesis rules. The synthesis rules are long-range context caches. The long-range context cache method is as follows: during the speech synthesis process, the engine generates speech sentence by sentence in segments. When each segment is generated, the key matrix and value matrix of each attention layer of the Transformer decoder corresponding to the segment are stored in the cache. When the next sentence is generated, the synthesis driver loads the complete KV cache of the previous sentence as the initial context input for attention calculation. This makes it so that when the model generates the beginning of a new sentence, its attention distribution already contains the timbre, rhythm and emotional state information of the previous text. A sliding window management strategy is adopted to retain only the complete KV cache corresponding to the most recent 2000 characters. For earlier historical context, a weighted compression method is used for dimensionality reduction and preservation. The model based on the VITS architecture outputs the target language dubbing speech waveform. The duration deviation is calculated based on the total duration of the target language dubbed speech waveform and the reference duration of the original video. The formula is as follows: ,in, This is the duration deviation value. Total duration of the voice waveform for dubbing the target language. The original video reference duration is used, and a deviation threshold is set according to the application scenario. The recognition rules include a first-level signal layer, a second-level syntax layer, and a third-level instruction layer. The first-level signal layer is based on speech activity detection, identifying silent segments in the target language dubbing speech waveform with energy below -40dB and duration >50ms. The second-level syntax layer is based on dependency parsing tag-enhanced synthesis scripts, identifying grammatical pauses corresponding to punctuation marks such as commas, semicolons, and periods. The third-level instruction layer directly parses the speech segments corresponding to non-ideographic acoustic control tags in the tag-enhanced synthesis scripts, assigns weights to the above three levels of results, takes the weighted union of the three, and removes regions that are too close to the core content words to obtain the final list of scalable regions, i.e., a list of timestamps for key adjustable regions. The calculated deviation value is compared with the preset deviation threshold. If the calculated deviation value is greater than the preset deviation threshold, the scaling algorithm is called to adjust the duration; otherwise, the original speech is directly output, and a phase vocoder is selected. The Vocoder scaling algorithm allocates scaling ratios based on the priority of adjustable regions. Silent segments can be scaled up to 150% of their original length, punctuation pauses can be scaled up to 120% of their original length, and speech segments corresponding to non-representational acoustic control tags can be scaled up to 110% of their original length. The scaling algorithm is applied independently to each adjustable region, and the speech segments are re-seminated after adjustment to generate a dubbing speech waveform synchronized with the original video.

[0021] S105 encapsulates the voiceover waveform synchronized with the original video and the original video.

[0022] Specifically, the audio and video streams of the synchronized voiceover waveform and the original video are separated and reassembled. The video stream is extracted from the original video using an audio and video processing tool (FFmpeg) (only the video data is retained, and the original audio track is removed). Simultaneously, the synchronized voiceover waveform is introduced as a new audio stream. During this process, it is ensured that the timeline of the voiceover is perfectly aligned with the video frames to avoid audio-visual misalignment. The reassembled audio and video stream is then post-processed and optimized, adjusting the loudness of the voiceover audio to international standards and using frequency response adjustments (such as improving high-frequency clarity and enhancing low-frequency depth). This process makes the voiceover tone more in line with the video content. The FFmpeg encapsulation tool is used to merge the processed video stream with the optimized audio stream, and the output format is specified as MP4. Key parameters are set during encapsulation (encoding format H.264 / AVC, audio encoding AAC, bitrate adaptive, etc.) to balance file size and image / sound quality. The final generated file is a ready-to-publish product with strict synchronization of audio and video streams, clear voiceover, compliant loudness, and preservation of the original video's high-resolution image, meeting the publishing requirements of multiple platforms (YouTube, social media, streaming media).

[0023] The cross-language dubbing method based on acoustic feature anchoring and tag reconstruction also includes a cross-language dubbing system. The cross-language dubbing system comprises: an emotional intent anchoring processor, a tag reconstruction and script generator, a speech synthesis driver, a temporal calibration processor, audio and text input interfaces, and a system bus and memory. The emotional intent anchoring processor extracts features from the source language audio signal and source language text to identify emotional intent. The tag reconstruction and script generator converts the emotional intent into a target language tag sequence, forming a tag-enhanced synthesis script. The speech synthesis driver generates the target language dubbing speech waveform from the tag-enhanced synthesis script using synthesis rules. The temporal calibration processor calls a scaling algorithm to adjust the duration. The audio and text input interfaces receive source language audio, source language text, and target language translated text. The system bus and memory store a computer program that drives the above components to work together when loaded. The emotional intent anchoring processor is connected to the tag reconstruction and script generator, which is connected to the speech synthesis driver, and the speech synthesis driver is connected to the temporal calibration processor.

[0024] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages: By establishing the priority decision weight of acoustic features at the model training level, emotion judgment is based on the actor's real physical performance evidence (fundamental frequency tremor, energetic breathing, non-verbal sounds) as the core basis, and only references text semantics for auxiliary calibration when the acoustic evidence is ambiguous. When the acoustic performance conflicts with the text semantics, the system always takes the acoustic features as the standard, thereby eliminating the defect of "consistency between sound and image emotional expression" from the source, realizing the simulation of the human director's "hearing and recognizing emotions" cognitive logic, and fundamentally solving the problem of unreliable modalities in emotion judgment; "cross-language label reconstruction" is introduced as an intermediate conversion layer: the system does not directly transfer the acoustic features of the source language, but deconstructs them into abstract emotional instructions (such as [breath], [tremor]), and then reconstructs them into a structured acoustic control label sequence of the target language through a cross-language emotion-prosodic mapping knowledge base, which is finally executed by a pure target speech synthesis engine. This mechanism physically isolates the input of prosodic features of the source language at the architectural level, while establishing a direct anchoring path from the source performance to the target synthesis for emotional expression. It achieves the dual goals of "no accent residue" and "high-fidelity emotional transfer," fundamentally solving the problems of prosodic pollution and the break in the emotional transfer path. The generated structured acoustic control tag sequence provides a fine and programmable control interface for dubbing synthesis, making the control of performance details such as emotional intensity, local accents, rhythm changes, and non-verbal sounds (breathing, sighing, pauses) precise and repeatable. This significantly improves the artistic expressiveness of automated dubbing, enabling AI-synthesized speech to approach the performance level of professional voice actors.

[0025] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cross-language dubbing method based on acoustic feature anchoring and tag reconstruction, characterized in that, include: S101, Collect source language audio data, the source language audio data includes source language audio signal and source language text, and align the source language audio signal with the source language text; S102, extract the features of the source language audio signal and the features of the source language text, construct a judgment model based on the features of the source language audio signal and the features of the source language text, the judgment model recognizes the emotional intent based on the source language audio signal, and generates the target language translation text based on the source language text; S103, convert the emotional intent into a target language tag sequence, insert the target language tag sequence into the semantic node corresponding to the target language translated text, and form a tag-enhanced synthesis script; S104, the tag-enhanced synthesis script generates a target language dubbing voice waveform through synthesis rules, calculates the deviation value between the total duration of the target language dubbing voice waveform and the reference duration of the original video, identifies non-critical adjustable areas based on recognition rules, compares the deviation value with a preset deviation threshold, and when the deviation value exceeds the preset threshold, calls the scaling algorithm to adjust the duration to generate a dubbing voice waveform synchronized with the original video; otherwise, the duration is not adjusted. S105 encapsulates the voiceover waveform synchronized with the original video and the original video.

2. The cross-language dubbing method based on acoustic feature anchoring and tag reconstruction as described in claim 1, characterized in that, The source language audio signal features include fundamental frequency trajectory, energy envelope, and non-verbal sound events; The formula for calculating the energy envelope is: ,in, The energy envelope reflects the average power of the signal within a short frame. x(n) is the sampled value of the speech signal at discrete time point n, where n is the discrete time index and N is the total number of sampled points in the current analysis frame. The non-verbal sound event refers to the sound in the speech that does not carry semantic content.

3. The cross-language dubbing method based on acoustic feature anchoring and tag reconstruction as described in claim 2, characterized in that, The specific steps for converting emotional intent into a target language label sequence are as follows: decompose the complex emotion in the emotional metadata, decompose the complex emotion into multiple sub-emotions, search for the label corresponding to each sub-emotion in the cross-language emotion-prosody mapping knowledge base, if there are multiple candidate labels, select the one with the highest emotional intensity as the target label, and convert each emotion category into a target label to form a target language label sequence.

4. The cross-language dubbing method based on acoustic feature anchoring and tag reconstruction as described in claim 3, characterized in that, The duration deviation is calculated based on the total duration of the target language dubbed speech waveform and the reference duration of the original video. The formula is as follows: ,in, This is the duration deviation value. Total duration of the voice waveform for dubbing the target language. The original video has a reference duration.

5. The cross-language dubbing method based on acoustic feature anchoring and tag reconstruction as described in claim 4, characterized in that, The method for identifying non-critical adjustable regions based on recognition rules is as follows: The recognition rules include a first-level signal layer, a second-level syntax layer, and a third-level instruction layer: The first-level signal layer is based on energy threshold-based speech activity detection to identify silent segments in the target language dubbing speech waveform; The second-level syntax layer is based on the dependency parsing tag-enhanced synthesis script of the target language text to identify grammatical pauses corresponding to punctuation marks. The third-level instruction layer parses the label-enhanced synthetic script and identifies the speech segments corresponding to non-ideographic acoustic control labels; the weighted union of the three-level recognition results is taken as the non-critical adjustable region.

6. The cross-language dubbing method based on acoustic feature anchoring and tag reconstruction as described in claim 5, characterized in that, The calculated deviation value is compared with the preset deviation threshold. If the calculated deviation value is greater than the preset deviation threshold, the scaling algorithm is called to adjust the duration. Otherwise, the original speech is output directly, and the phase vocoder scaling algorithm is selected. Different scaling weights are assigned to the regions that can be adjusted according to different categories. Among them, the weight of the silent segment is greater than that of the grammatical pause point, and the weight of the grammatical pause point is greater than that of the speech segment corresponding to the non-representational acoustic control tag.

7. The cross-language dubbing method based on acoustic feature anchoring and tag reconstruction as described in claim 6, characterized in that, It also includes a cross-language dubbing system, which comprises: The sentiment intent anchoring processor is used to extract features from the source language audio signal and the source language text to identify sentiment intent; A tag reconstruction and script generator is used to convert sentiment intent into a sequence of target language tags, forming a tag-enhanced synthetic script; A speech synthesis driver is used to generate target language dubbing speech waveforms from tag-enhanced synthesis scripts using synthesis rules; The time-domain calibration processor is used to call the scaling algorithm to adjust the duration; Audio and text input interfaces are used to receive source language audio, source language text, and target language translated text; The system bus and memory, the memory storing computer programs, drive the above-mentioned components to work together when the programs are loaded.

Images