A multi-modal context-based video translation and dubbing fitting method and system

Abstract

The application provides a video translation and dubbing fitting method and system based on multi-modal context, and relates to the technical field of audio and video processing. The method comprises the following steps: obtaining a pure vocal track, constructing a visual semantic context atlas; generating a dynamic hot word list, generating a double-track subtitle file; assigning a unique role identifier, using a three-level priority multi-modal fusion decision model to determine the speaker attribution, obtaining a role-level accurate subtitle file; performing three-level ladder translation processing to generate a target language translation text; performing progressive acoustic feature compression to generate a target language dubbing audio; calculating a sensory tolerance threshold and performing three-level adaptive fitting; obtaining an audio-visual synchronous video; performing independent lip rendering processing on multi-person lip synchronization to obtain a final translation dubbing video. According to the application, the matching degree of target language dubbing and video timeline can be improved through the three-level priority multi-modal fusion decision model, three-level ladder translation processing and three-level adaptive fitting.

Description

Technical Field

[0001] This invention relates to the field of audio and video processing technology, and in particular to a method and system for video translation and dubbing fitting based on multimodal context. Background Technology

[0002] Current technologies often struggle to identify different speakers based on voiceprints, lip movements, and semantic context, lacking a stable conflict resolution mechanism. The duration of the target language translation often exceeds the available time of the original video clip; simply accelerating the audio or slowing down the video can lead to audio quality distortion or unnatural visuals. When large models translate long subtitles in batches, inconsistencies in character tone, pronoun referencing errors, and truncation of structured output are common. Ordinary TTS (text-to-speech) emotional tags struggle to dynamically adjust the dubbing rhythm based on plot location, historical emotions, and facial expressions. In scenes with multiple characters on screen or arguing, the target audio track is difficult to stably bind to facial trajectories, easily driving incorrect lip movements. In short, it is impossible to improve the matching degree between the target language dubbing and the video timeline using a three-level priority multimodal fusion decision model, a three-level tiered translation processing, and a three-level adaptive fitting.

[0003] The information disclosed in the background section of this application is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0004] This invention provides a video translation and dubbing fitting method and system based on multimodal context, which can solve the technical problem that related technologies cannot improve the matching degree between the target language dubbing and the video timeline based on a three-level priority multimodal fusion decision model, a three-level ladder-style translation processing and a three-level adaptive fitting.

[0005] According to the present invention, a video translation and dubbing fitting method based on multimodal context is provided, comprising: The original video is subjected to sound source separation to obtain a clean human voice track, and optical character recognition is performed on the frame to construct a visual semantic context map; A dynamic hot word list is generated based on the visual semantic context graph, and the decoding bias of candidate words is adjusted when performing automatic speech recognition on a clean human voice track to generate a dual-track subtitle file. By constructing a global role archive and assigning unique role identifiers, and by adopting a three-level priority multimodal fusion decision model, the speaker attribution of dual-track subtitle files is determined, and a role-level accurate subtitle file is obtained through a multi-source cross-validation mechanism; The character-level precise subtitle files are input in batches, and a three-level step-by-step translation process with time constraints is performed to generate the target language translated text. Progressive acoustic feature compression is performed on the pure human voice track, and emotional transfer and target language translation text are performed through a context-aware dynamic emotion computing network to generate dubbing audio in the target language; The original video is divided into multiple independent audio segments on the timeline, the sensory tolerance threshold of each audio segment is dynamically calculated, and the dubbed audio and the original video are subjected to three-level adaptive fitting according to priority. When the fitting fails, a timeout-marked problem segment is generated. The problem segment is then subjected to multi-level entropy reduction processing and a closed-loop feedback is formed to obtain audiovisual synchronized video. Independent lip-sync rendering is performed on the audiovisual synchronized video to obtain the final translated dubbing video.

[0006] Furthermore, sound source separation is performed on the original video to obtain a clean human voice track, and optical character recognition is performed on the video frames to construct a visual semantic context map, including: By employing a deep learning architecture based on a frequency band segmentation attention mechanism, a sound source separation model is used to separate the original mixed audio sources in the frequency domain space, thereby obtaining a clean vocal track. Extract all frame sequences of the original video frame by frame and calculate the similarity between adjacent frame pairs; If the similarity of multiple consecutive adjacent frame pairs is greater than the similarity threshold, then the multiple consecutive adjacent frame pairs are determined to belong to the same frame group. Within each frame group, the frame with the highest image quality is selected as the keyframe; The keyframes are used to locate the text region using an object detection model, and then the text content is extracted using an optical character recognition model to obtain a visual text sequence containing timestamps, text content, and bounding boxes. An entity dictionary is constructed based on the visual text sequence, wherein the entity dictionary includes character names, place names, proper nouns and their frequency of occurrence; By analyzing the role relationships and scene semantics in the entity dictionary through multimodal large model analysis, entity relationship and background story attributes are generated; Based on the entity dictionary, the entity relationships, and the background story attributes, a visual semantic context graph of the original video is constructed.

[0007] Furthermore, a dynamic hot word list is generated based on the visual semantic context graph, and the decoding bias of candidate words is adjusted when performing automatic speech recognition on a clean human voice track to generate a dual-track subtitle file, including: A dynamic hot word list is generated from the visual semantic context graph using a keyword extraction algorithm; The pure human voice track and the candidate words are input into the acoustic model and the general language model to obtain the first probability score and the second probability score. The decoding bias of candidate words is adjusted using the dynamic hot word list to obtain the context bias function; The first probability score, the second probability score, and the context bias function are weighted and summed to determine the enhanced decoding score of the candidate word in the speech recognition process. Based on the enhanced decoding score, speech recognition is enhanced, and a dual-track subtitle file is generated, wherein the dual-track subtitle file includes a plain text track and a speaker tag track.

[0008] Furthermore, by constructing a global role archive and assigning unique role identifiers, and employing a three-level priority multimodal fusion decision model, speaker attribution is determined for dual-track subtitle files. Through a multi-source cross-validation mechanism, precise role-level subtitle files are obtained, including: Based on the clean human voice track, dual-track subtitle file and original video, obtain the voiceprint feature vector, semantic feature vector and visual feature vector for each character; A global character archive is constructed based on the voiceprint feature vector, semantic feature vector, and visual feature vector of each character, and a unique character identifier is assigned to each character. The dual-track subtitle file is divided into multiple subtitles to be proofread. The voiceprint feature vectors of the pure human voice track segments corresponding to the subtitles to be proofread are extracted. The cosine similarity between the voiceprint feature vectors to be proofread and the voiceprint feature vectors of each character in the global character archive is calculated to obtain the first probability of physical voiceprint feature matching. Extract the visual feature vector of the original video segment corresponding to the subtitle to be proofread, calculate the cosine similarity between the visual feature vector to be proofread and the visual feature vector of each character in the global character archive, and obtain the second probability of synchronization between lip movements and speech rhythm. The third probability of the subtitles to be proofread being spoken by each character is inferred using a multimodal large model; A three-level priority multimodal fusion decision model is adopted, with voiceprint matching as the first priority. If the maximum value of the first probability is greater than the first high confidence threshold, the speaker is determined to belong to the role that makes the first probability the maximum value. With face detection in the image as the second priority, if the maximum value of the first probability is less than or equal to the first high confidence threshold, and the maximum value of the second probability is greater than the second high confidence threshold, then the speaker is determined to belong to the role that makes the weighted sum of the first probability and the second probability the maximum value. Using semantic context reasoning as the third priority, if the maximum value of the first probability is less than or equal to the first high confidence threshold, and the maximum value of the second probability is less than or equal to the second high confidence threshold, then the speaker is determined to belong to the role that maximizes the weighted sum of the first, second, and third probabilities. A multi-source cross-validation mechanism is used to assess and correct the confidence level of the speaker tag corresponding to each subtitle to be proofread, thereby obtaining a role-level accurate subtitle file carrying the unique role identifier. The multi-source cross-validation mechanism involves performing multiple independent inference samplings on the same set of inputs: clean audio track, dual-track subtitle file, and original video. Based on the sampling results, speaker confidence and timestamp confidence are statistically analyzed. When both exceed the corresponding threshold, the result is accepted; otherwise, downgrade processing is triggered.

[0009] Furthermore, the character-level precise subtitle files are input in batches, and a three-level stepped translation process with time constraints is performed to generate target language translated text, including: The character-level precise subtitle files are input in batches using an overlapping sliding window batching algorithm. Each batch of input contains the current core section to be translated and its adjacent context sections before and after it. The three-tiered translation processing includes the first tier injecting character settings and cultural adaptation rules to perform semantically equivalent translation; the second tier performing synonym substitution compression of the translation under time constraints based on the syllable space mapping relationship between the source language and the target language; and the third tier performing semantic skeleton extraction and minimal expression reconstruction in extreme over-limit scenarios. By using truncation repair and breakpoint resumption, the target language translation text is generated stably.

[0010] Furthermore, the duration constraint processing for the second-level ladder includes: The source language text is converted into a pinyin sequence and the number of syllables is counted; the target language translation is converted into a phoneme sequence and the pronunciation duration is estimated. The duration matching score, which measures the degree of translation duration exceeding the limit, is determined by the syllable space mapping ratio between the source language and the target language. When the translation time exceeds the available time, the synonym replacement compression strategy is activated.

[0011] Furthermore, progressive acoustic feature compression is performed on the purified human voice track, and emotion transfer and target language translation text are performed through a context-aware dynamic emotion computing network to generate dubbing audio in the target language, including: From the pure human voice track, extract the original audio segments of each character based on the character-level precise subtitle file; The original audio segment is subjected to progressive acoustic feature compression to obtain a purified audio segment; The purified audio segments are used as reference samples and input into the speech cloning engine for synthesis. The speech cloning engine routes between persistent cloning mode and real-time reference mode based on video task metadata, role cross-set reuse requirements, cumulative duration of reference samples, and persistent timbre quota status. The sentiment weight is calculated by using local acoustic sentiment intensity, sentiment attenuation accumulation term of historical context, and contextual position weight; Based on the emotional weight and the target language translated text, the synthesized speech rate, pitch, energy, and pause parameters are controlled to generate the target language dubbing audio.

[0012] Furthermore, the original video is divided into multiple independent speech segments along the timeline, and the sensory tolerance threshold of each speech segment is dynamically calculated. The dubbed audio and the original video are then subjected to a three-level adaptive fitting process based on priority, including: The sensory resilience coefficient is obtained through contextual information, wherein the sensory resilience coefficient includes auditory acceleration tolerance and visual deceleration tolerance; Obtain the duration of the dubbing audio, the duration of the target video, and the variance of emotional fluctuations; Based on the sensory elasticity coefficient, the duration of the dubbing audio, the duration of the target video, and the variance of emotional fluctuation, the sensory tolerance threshold for each speech segment is dynamically calculated, wherein the sensory tolerance threshold includes the maximum audio acceleration factor and the minimum video deceleration factor. The dubbing audio and the original video are subjected to a three-level adaptive fitting according to the priority of lossless replacement, one-way audio acceleration, and two-way audio-video compression; The first level of lossless replacement involves aligning the dubbing audio to the beginning of the video segment when the dubbing audio duration is less than or equal to the target video duration, and filling the end with silence to make up for the duration difference. The second level of unidirectional audio acceleration is to perform high-speed processing on synthesized speech while preserving the sound when the duration of the dubbing audio is greater than the duration of the target video and the required audio acceleration factor is less than or equal to the maximum audio acceleration factor. The third level of bidirectional audio and video compression involves locking the required audio acceleration factor to the maximum audio acceleration factor when the required audio acceleration factor is greater than the maximum audio acceleration factor, while simultaneously performing deceleration processing on the video segment. This deceleration processing of the video segment includes designing a cascaded speed-changing pipeline and a flexible BGM re-attachment mechanism.

[0013] Furthermore, when fitting fails, a timeout-marked problem segment is generated. This problem segment undergoes multi-level entropy reduction processing in sequence, forming a closed-loop feedback loop to obtain synchronized audiovisual video, including: The problematic segments marked with timeouts are sequentially subjected to multi-level entropy reduction processing, including output text extreme rewriting, input text skeleton extraction, iteration termination judgment, and image-generated video ultimate defense processing, to obtain the repaired segments. The repaired segment is sent back to the translation step for reprocessing, forming a closed-loop feedback loop to obtain synchronized audiovisual video.

[0014] Furthermore, the audiovisual synchronized video undergoes independent lip-sync rendering processing involving multiple lip movements to obtain the final translated dubbing video, including: Multi-channel sound source separation is performed on the synthesized multi-character target language dubbing audio in the audiovisual synchronization video to obtain independent audio tracks for each character, wherein each independent audio track carries a corresponding character identifier; Face detection and tracking are performed on the video frames of the audiovisual synchronized video. Continuous face trajectories are constructed through Kalman filtering and cross-union matching to generate pixel-level binary mask sequences covering the lower half of the face and lip region for each character. Based on the character identifiers and visual feature matching of the global character archive, the separated independent audio tracks and facial trajectories are deterministically bound through the character identifiers; For each character, the corresponding independent audio track and dynamic lip mask are extracted and input into the lip shape generation network along with the original video frames. Pixel deformation is calculated within the mask area defined by the pixel-level binary mask sequence, lip shape rendering is performed independently, and local lip shape update frames are generated. The local lip shape update frames of all characters are stitched into the original video frames according to their corresponding masking areas. Gaussian blur and Poisson fusion algorithms are applied at the pixel-level dynamic lip masking boundaries to eliminate stitching artifacts, and the final translated dubbing video is output.

[0015] Technical Effects: According to this invention, the accuracy of source language understanding is significantly improved through dual-track subtitle files. A three-level priority multimodal fusion decision model achieves the uniqueness of character voices and high-fidelity emotional transmission. A three-level tiered translation processing method optimizes the translation process through multi-objective joint optimization. A three-level adaptive fitting method (audio limiting acceleration and video limiting deceleration) improves the matching degree between the target language dubbing and the video timeline. Based on sound separation and video frame masking, the problem of lip-sync confusion in complex multi-person scenes is solved. The three-level priority multimodal fusion decision model, using first, second, and third probabilities, determines the speaker attribution for each subtitle in the dual-track subtitle file, breaking the ambiguity of weight allocation in traditional multimodal processing. A three-tiered priority arbitration mechanism was established to reduce the problem of speaker misjudgment in complex plots. Through a three-tiered tiered translation process, the duration of the translated audio is kept less than or equal to the available video time. Character design and cultural adaptation rules are injected into the translation process, allowing audiences in the target country to seamlessly immerse themselves in the story. The sliding window batching algorithm reduces contextual gaps, thereby minimizing issues such as unclear pronoun references and abrupt changes in tone caused by segmented translation. By introducing a multimodal large-scale model for multimodal speech rate perception, dynamic speed calculation is achieved, varying according to the speaker, context, and language. Based on sensory tolerance thresholds, a three-tiered degradation strategy prioritizes lossless audio, then lossless video, and finally bidirectional compression, maximizing the preservation of the original material's artistic expression.

[0016] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Other features and aspects of the invention will become clearer from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort. Figure 1 An exemplary flowchart of a video translation and dubbing fitting method based on multimodal context according to an embodiment of the present invention is shown. Figure 2 An exemplary flowchart of constructing a visual semantic context graph according to an embodiment of the present invention is shown; Figure 3 An exemplary flowchart for generating a dual-track subtitle file according to an embodiment of the present invention is shown; Figure 4 An exemplary flowchart illustrating the process of obtaining character-level accurate subtitle files according to an embodiment of the present invention is shown; Figure 5 An exemplary flowchart for generating target language translated text is shown according to an embodiment of the present invention; Figure 6 An exemplary flowchart illustrating the generation of dubbing audio in a target language according to an embodiment of the present invention is shown; Figure 7 A flowchart of a three-level adaptive fitting method according to an embodiment of the present invention is shown as an example; Figure 8 An exemplary flowchart for obtaining synchronized audiovisual video according to an embodiment of the present invention is shown; Figure 9 An exemplary flowchart illustrating the process of obtaining the final translated dubbing video according to an embodiment of the present invention is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0020] Figure 1 An exemplary flowchart of a video translation and dubbing fitting method based on multimodal context according to an embodiment of the present invention is shown, the method comprising: Step S1: Separate the sound sources from the original video to obtain a clean human voice track, and perform optical character recognition on the frame to construct a visual semantic context map. Step S2: Generate a dynamic hot word list based on the visual semantic context graph, and adjust the decoding bias of candidate words when performing automatic speech recognition on the clean human voice track to generate a dual-track subtitle file; Step S3: By constructing a global role archive and assigning a unique role identifier, and using a three-level priority multimodal fusion decision model, the speaker attribution of the dual-track subtitle file is determined, and a role-level accurate subtitle file is obtained through a multi-source cross-validation mechanism. Step S4: Input the character-level precise subtitle files in batches, perform a three-level step-by-step translation process with time constraints, and generate the target language translated text; Step S5: Perform progressive acoustic feature compression on the clean human voice track, and perform emotion transfer and target language translation text through a context-aware dynamic emotion computing network to generate dubbing audio in the target language; Step S6: Divide the original video into multiple independent speech segments on the timeline, dynamically calculate the sensory tolerance threshold of each speech segment, and perform three-level adaptive fitting on the dubbing audio and the original video according to priority. Step S7: When fitting fails, a timeout-marked problem segment is generated. The problem segment is then subjected to multi-level entropy reduction processing and a closed-loop feedback is formed to obtain audiovisual synchronized video. Step S8: Perform independent lip-sync rendering on the audiovisual synchronized video to obtain the final translated dubbing video.

[0021] According to embodiments of the present invention, the video translation and dubbing fitting method based on multimodal context significantly improves the accuracy of source language understanding through dual-track subtitle files. A three-level priority multimodal fusion decision model achieves the uniqueness of character voices and high-fidelity emotional transmission. A three-level tiered translation processing method optimizes the translation process through multi-objective joint optimization. A three-level adaptive fitting method with audio limiting acceleration and video limiting deceleration improves the matching degree between the target language dubbing and the video timeline. Based on sound separation and video frame masking, the method solves the problem of lip-sync confusion in complex multi-person scenes.

[0022] According to one embodiment of the present invention, in step S1, the original video is subjected to sound source separation to obtain a pure human voice track, and optical character recognition is performed on the screen frames to construct a visual semantic context map.

[0023] Figure 2 An exemplary flowchart for constructing a visual semantic context graph according to an embodiment of the present invention is shown.

[0024] According to an embodiment of the present invention, step S1 includes: step S11, using a deep learning architecture based on a frequency band segmentation attention mechanism to separate the original mixed audio sources in the frequency domain space through a sound source separation model to obtain a clean human voice track; step S12, extracting all frame sequences of the original video frame by frame and calculating the similarity of adjacent frame pairs; step S13, if the similarity of multiple consecutive adjacent frame pairs is greater than a similarity threshold, then determining that multiple consecutive adjacent frame pairs belong to the same frame group; step S14, in each frame group, selecting the frame with the highest image quality as a keyframe; step S15, setting the keyframe as a keyframe. Keyframes are used to locate text regions using an object detection model, and then the text content is extracted using an optical character recognition model to obtain a visual text sequence containing timestamps, text content, and bounding boxes; Step S16: An entity dictionary is constructed based on the visual text sequence, wherein the entity dictionary contains character names, place names, proper nouns, and their frequency of occurrence; Step S17: The character relationships and scene semantics in the entity dictionary are analyzed using a multimodal large model to generate entity relationships and background story attributes; Step S18: A visual semantic context graph of the original video is constructed based on the entity dictionary, the entity relationships, and the background story attributes.

[0025] According to one embodiment of the present invention, a sound source separation model employs a deep learning architecture based on a frequency band segmentation attention mechanism (e.g., a Transformer network based on rotational position coding) to perform sound source separation in the frequency domain space, thereby obtaining a clean vocal track. Let the original mixed audio be represented in the time-frequency domain as the original mixed audio in the time-frequency domain. Where t is the time frame and f is the frequency bin, the original mixed audio can be regarded as the time-frequency domain representation of pure human voice as time-frequency domain pure human voice. The time-frequency domain representation of background noise is called time-frequency domain background noise. The linear superposition of the two. The high-purity human voice spectrum can be determined based on the original time-frequency domain mixed audio, the time-frequency domain pure human voice, and the time-frequency domain background noise, according to the formula: , and Determine the high purity of the human voice spectrum ,in, It is the ideal proportional masking matrix. This represents the linear superposition of pure human voice and background noise in the time-frequency domain, indicating the original mixed audio in the time-frequency domain. The ideally scaled masking matrix has each element value between 0 and 1. This indicates the probability that the corresponding time-frequency point belongs to human voice. By element-wise multiplying the ideal-scale masking matrix with the original mixed audio in the time-frequency domain, a high-purity human voice spectrum can be reconstructed. This spectrum can then be converted back to a time-domain waveform using an inverse short-time Fourier transform, thus obtaining a clean human voice track. This separation-before-recognition strategy physically eliminates background noise interference to the speech recognition decoder, significantly improving the signal-to-noise ratio of subsequent speech recognition and voiceprint clustering. The video decoder extracts all frame sequences from the original video frame by frame. A lightweight convolutional neural network extracts the deep visual feature vector of each frame to calculate the similarity between adjacent frame pairs. Let the continuous frame sequence of the original video be F = F_1, F_2, ..., F_n. The lightweight convolutional neural network extracts the deep visual feature vector of the u-th frame. According to the formula Calculate the similarity between adjacent frame pairs. ,in, Let u+1 be the deep visual feature vector of the frame. If the similarity of multiple consecutive adjacent frame pairs is greater than a similarity threshold (e.g., 0.7), then the multiple consecutive adjacent frame pairs are determined to belong to the same image group. For example, frames 1 to 20 are determined to belong to the same image group. In each image group, the frame with the highest image quality (e.g., sharpness, contrast) is selected as the keyframe. The keyframe is used to locate text regions using an object detection model (e.g., a convolutional neural network model), and then the text content is extracted using an optical character recognition model (e.g., a convolutional neural network model). This yields a visual text sequence containing timestamps, text content, and bounding boxes, compressing the computational workload that originally required processing tens of thousands of frames to hundreds of keyframes. This achieves an exponential improvement in computational efficiency while ensuring zero omission of visual text information. Using the Named Entity Recognition (NER) algorithm in Natural Language Processing, key entities are extracted from the visual text sequence to construct an entity dictionary (a set of key entities). The entity dictionary includes role names, place names, proper nouns, and their frequencies. The entity dictionary, along with its temporal context, is analyzed using a multimodal large-scale model (e.g., Transformer series models) to examine the character relationships and scene semantics within the entity dictionary. Leveraging the powerful semantic understanding and logical reasoning capabilities of the multimodal large-scale model, it outputs entity relationships (e.g., who is the protagonist, who is the supporting character, and who is the rival) and background story attributes (containing attribute information such as character relationships, background stories, and character traits summarized by the multimodal large-scale model). The entity dictionary, entity relationships, and background story attributes are then used to construct a visual semantic context graph of the original video.

[0026] According to one embodiment of the present invention, in step S2, a dynamic hot word list is generated based on the visual semantic context graph, and when performing automatic speech recognition on a clean human voice track, the decoding bias of the candidate words is adjusted to generate a dual-track subtitle file.

[0027] Figure 3 A flowchart for generating a dual-track subtitle file according to an embodiment of the present invention is shown as an example.

[0028] According to an embodiment of the present invention, step S2 includes: step S21, generating a dynamic hot word list from the visual semantic context map using a keyword extraction algorithm; step S22, inputting the clean human voice track and the candidate words into an acoustic model and a general language model to obtain a first probability score and a second probability score; step S23, adjusting the decoding bias of the candidate words with the dynamic hot word list to obtain a context bias function; step S24, performing a weighted summation of the first probability score, the second probability score, and the context bias function to determine the enhanced decoding score of the candidate words during speech recognition; step S25, enhancing speech recognition based on the enhanced decoding score and generating a dual-track subtitle file, wherein the dual-track subtitle file includes a plain text track and a speaker tag track.

[0029] According to one embodiment of the present invention, a dynamic hot word list is generated from the visual semantic context graph by using a keyword extraction algorithm (e.g., TF-IDF weighted or graph node importance ranking). A clean human voice track and a dynamic hot word list are simultaneously fed into the speech recognition service. The dynamic hot word list is formatted into a parameter format acceptable to the speech recognition model (e.g., GPT series) (e.g., a JSON array). During the beam search decoding stage of speech recognition, candidate words (with ambiguous pronunciation or homophones) and the clean human voice track are input into the acoustic model (e.g., CTC, RNN-T, or attention model) to obtain a first probability score, which measures the degree of matching between the candidate word and the clean human voice track. The higher the first probability score, the more similar the word sounds to the user's spoken words from a pronunciation perspective. A second probability score is given by a general language model (e.g., a neural network language model), which measures the reasonableness of the candidate word's appearance in the current context. The higher the second probability score, the more fluent and grammatically correct the candidate word is in the sentence. When a candidate word matches or partially matches a word in the dynamic hot word list (e.g., contains a substring, has similar pinyin, etc.), the context bias function returns a positive reward score (e.g., 0.2); otherwise, it returns 0. The enhanced decoding score of candidate words in the speech recognition process can be determined based on the first probability score, the second probability score, and the context bias function, according to the formula: Determine candidate words Enhanced decoding score ,in, The first probability score, The second probability score, Here, H is the context bias function, and H is the dynamic hotword list. , and For weight hyperparameters, To achieve the weighted sum of the first probability score, the second probability score, and the cumulative reward from the context bias function—that is, the enhanced decoding score—a higher enhanced decoding score means the speech recognition model will tend to select words that match the current video's background story and character relationships (i.e., the dynamic hot word list H) when encountering ambiguous pronunciations or homophones. After enhancing speech recognition, the output is structurally decomposed and persistently stored, constructing a dual-track data persistence layer, namely a dual-track subtitle file, including a plain text SRT track and a speaker tag SRT track. The plain text SRT track only contains timestamps and the recognized text content, removing all speaker tags. The speaker tag SRT track, based on the text content, uses a voiceprint clustering algorithm to assign strictly normalized speaker tags (e.g., "male lead," "female supporting character 1," etc.) to each line of dialogue. This dual-track design decouples data flow from control flow, meets the differentiated data needs of different modules in a complex pipeline, and provides a solid data persistence guarantee for the stable operation of the entire system.

[0030] According to an embodiment of the present invention, in step S3, a unique role identifier is assigned by constructing a global role archive, and a three-level priority multimodal fusion decision model is used to determine the speaker affiliation of the dual-track subtitle file. The role-level accurate subtitle file is obtained through a multi-source cross-validation mechanism.

[0031] Figure 4 An exemplary flowchart illustrating the process of obtaining character-level accurate subtitle files according to an embodiment of the present invention is shown.

[0032] According to an embodiment of the present invention, step S3 includes: step S31, obtaining the voiceprint feature vector, semantic feature vector, and visual feature vector of each character based on the clean human voice track, the dual-track subtitle file, and the original video; step S32, constructing a global character archive based on the voiceprint feature vector, semantic feature vector, and visual feature vector of each character, and assigning a unique character identifier to each character; step S33, dividing the dual-track subtitle file into multiple subtitles to be proofread, extracting the voiceprint feature vector to be proofread from the clean human voice track segment corresponding to the subtitle to be proofread, and calculating the voiceprint feature vector to be proofread and the global character archive. Step S34: Extract the visual feature vector of the original video segment corresponding to the subtitle to be proofread from the cosine similarity between the visual feature vector to be proofread and the visual feature vector of each character in the global character archive to obtain the second probability of synchronization between lip movements and speech rhythm; Step S35: Infer the third probability of the subtitle to be proofread being spoken by each character through a multimodal large model; Step S36: Use a three-level priority multimodal fusion decision model, with voiceprint matching as the first priority, if the first probability... If the maximum value of the first probability is greater than the first high confidence threshold, then the speaker is determined to belong to the role that maximizes the first probability; Step S37, with face detection in the image as the second priority, if the maximum value of the first probability is less than or equal to the first high confidence threshold, and the maximum value of the second probability is greater than the second high confidence threshold, then the speaker is determined to belong to the role that maximizes the weighted sum of the first probability and the second probability; Step S38, with semantic context reasoning as the third priority, if the maximum value of the first probability is less than or equal to the first high confidence threshold, and the maximum value of the second probability is less than or equal to the second high confidence threshold... If the threshold is set, the speaker is determined to belong to the role that maximizes the weighted sum of the first probability, the second probability, and the third probability. In step S39, the confidence level of the speaker tag corresponding to each subtitle to be proofread is evaluated and corrected through a multi-source cross-validation mechanism to obtain a role-level accurate subtitle file carrying the unique role identifier. The multi-source cross-validation mechanism performs multiple independent inference samplings on the same set of inputs: clean audio track, dual-track subtitle file, and original video. Based on the sampling results, the speaker confidence and timestamp confidence are statistically analyzed. When both exceed the corresponding threshold, the result is accepted; otherwise, downgrade processing is triggered.

[0033] According to one embodiment of the present invention, based on the pure human voice track MP3, the dual-track subtitle file SRT, and the original video MP4, the system obtains the voiceprint feature vector (e.g., highly stable physical features such as deep timbre, slow speech rate, and breathy endings), semantic feature vector (e.g., domineering CEO personality, specific catchphrases, and word preferences) and visual feature vector (e.g., gender, age group, facial features, and signature accessories; the system actively reduces the dimensionality or removes variable clothing color features when extracting this vector). Each character also includes status metadata, containing time-series tracking information such as the first episode appearance number, the last episode appearance number, the cumulative number of episodes appearing, and the current active status. When processing the original video of episode 1, a full historical character database is constructed, that is, the voiceprint feature vector, semantic feature vector, and visual feature vector of each character in the original video of episode 1 are combined into a high-dimensional feature vector, i.e. ,in, Let i be the high-dimensional feature vector of the i-th character. Let i be the voiceprint feature vector of the i-th character. Let i be the semantic feature vector of the i-th character. Let i be the visual feature vector of the i-th character. Then, the high-dimensional feature vectors of all characters from the original video of episode 1 are grouped into a set, which is the complete historical character database. , These are the high-dimensional feature vectors of the 1st, 2nd, ..., mth characters, respectively. For the union of the original video sets N, when the system processes the original video set N, for a newly detected unknown speaker entity (new character) in the Nth set, the system calculates its multidimensional weighted similarity with all characters in the historical database, according to the formula: Calculate the multidimensional weighted similarity between the new character and the i-th character in the full historical character database. ,in, For the high-dimensional feature vector of the new character, For the voiceprint feature vector of the new character, For the semantic feature vector of the new character, For the visual feature vectors of the new character, Let the similarity between the voiceprint feature vector of the new character and the i-th character in the full historical character database be calculated. Let the semantic feature vector similarity between the new character and the i-th character in the full historical character database be denoted as . Let the visual feature vector of the new character be similar to that of the i-th character in the full historical character database. , and For the weights, and Much larger and , where i is a positive integer. , and The values ​​can be 0.7, 0.2, and 0.1 respectively. Calculate the multidimensional weighted similarity between the new character and every character in the full historical character database, taking the maximum value. If the maximum value is greater than 0.8, reuse the ID (character identifier) ​​of that historical character; otherwise, the match fails, and a completely new topological ID is assigned to the new character to reduce cross-set character consistency issues. Let the time period of the x-th subtitle to be proofread be t_x. Extract the voiceprint feature vector of the clean audio track corresponding to the subtitle to be proofread. Calculate the cosine similarity between the voiceprint feature vector of the subtitle to be proofread and the voiceprint feature vector of each character in the global character archive to obtain the first probability of physical voiceprint feature matching. Perform face detection and lip movement analysis to identify the speaking face region. Extract the visual feature vector of the original video segment corresponding to the subtitle to be proofread. Calculate the cosine similarity between the visual feature vector of the subtitle to be proofread and the voiceprint feature vector of each character in the global character archive to obtain the second probability of synchronization between lip movements and speech rhythm. Based on the contextual dialogue, character traits, and plot logic, a multimodal large model is used to infer the third probability that the subtitles to be proofread are spoken by each character. Based on the first, second, and third probabilities, a three-level priority multimodal fusion decision model (including voiceprint matching as the first priority, face detection as the second priority, and semantic context reasoning as the third priority) is employed to determine the speaker attribution for each subtitle in the dual-track subtitle file, obtaining the speaker label corresponding to each subtitle to be proofread. This can be calculated using the formula: First priority: and Second priority: and Third priority: A judgment is made, in which, The probability of matching the xth subtitle to be proofread with the physical voiceprint features of character s in the global character archive is the first probability. The second probability is the synchronization of the xth subtitle to be proofread with the lip movements and voice rhythm of character s in the global character archive. To infer the probability that the xth subtitle to be proofread is spoken by character s from the global character archive, To assign the speaker to the subtitle in the xth section that needs to be proofread. To find the maximum value function, To select the independent variable that makes the function reach its maximum value, The first high confidence threshold, The second highest confidence threshold. , , , and is the weighting coefficient, and x is a positive integer. To obtain the maximum first probability of matching the physical voiceprint features of all characters in the global character archive, the first priority formula means that if the maximum first probability of matching the physical voiceprint features of all characters in the global character archive is greater than the first high confidence threshold (e.g., 0.8), then the speaker attribution of the x-th subtitle to be proofread is determined to be the character s that makes the first probability reach the maximum value, and the speaker label of character s is the speaker label of the x-th subtitle to be proofread. To obtain the maximum value of the second probability of synchronization between the x-th subtitle to be proofread and the lip movements and voice rhythm of all characters in the global character archive. Let be the weighted sum of the first and second probabilities, representing the first mixture confidence level, where , The values ​​can be 0.6 and 0.4 respectively. The second priority formula means that if the maximum value of the first probability of the x-th subtitle to be proofreading matching the physical voiceprint features of all characters in the global character archive is less than or equal to the first high confidence threshold, and the maximum value of the second probability of the x-th subtitle to be proofreading synchronizing with the lip movements and speech rhythm of all characters in the global character archive is greater than the second high confidence threshold (e.g., 0.7), then the speaker attribution of the x-th subtitle to be proofreading is determined to be the character s that makes the first mixed confidence reach the maximum value, and the speaker label of the character s is the speaker label of the x-th subtitle to be proofreading. The second mixed confidence level is represented by a weighted average of the first, second, and third probabilities, where, , and The values ​​can be 0.5, 0.3, and 0.2 respectively. The third priority formula means that if the maximum value of the first probability of the x-th subtitle to be proofreading matching the physical voiceprint features of all characters in the global character archive is less than or equal to the first high confidence threshold, and the maximum value of the second probability of the x-th subtitle to be proofreading synchronizing with the lip movements and speech rhythm of all characters in the global character archive is less than or equal to the second high confidence threshold, then the speaker of the x-th subtitle to be proofreading is determined to be the character s that makes the second mixed confidence reach the maximum value. The speaker label of character s is the speaker label of the x-th subtitle to be proofreading. This breaks the ambiguity of weight allocation in traditional multimodal processing, establishes a three-level priority arbitration mechanism, and reduces the problem of speaker misjudgment under complex plots. The program employs a multi-source cross-validation mechanism. Specifically, for the same set of input data (MP3+MP4+SRT), the program calls the large model to perform K independent and parallel inference samplings (K=3 or 5). During each sampling, the program appropriately adjusts the inference temperature parameter to improve the independence of each inference. The frequency of each candidate speaker's appearance in the K samplings is counted, and the percentage of the highest frequency is taken as the speaker's confidence level. Where d is the d-th sampling, d≤K, Let be the number of times character s appears in all K samples. To select the character *s* that appears most frequently, for example, if in 5 samples, it is identified as the male lead 4 times and as the male supporting character 1 once, then the speaker confidence score is 4 / 5 = 0.8. Calculate the variance of the starting timestamps in the K samples, then calculate the ratio of this variance to the normalized time constant (e.g., 1). Subtract this ratio from 1 to obtain the timestamp confidence score. , This is the start timestamp of the d-th sampling. Let be the mean of the starting timestamps from the K samples. Let V be the variance of the starting timestamps in the K samplings. To normalize the time constant, if the speaker confidence score is greater than or equal to the speaker confidence score threshold (e.g., 0.8) and the timestamp confidence score is greater than or equal to the timestamp confidence score threshold (e.g., 0.9), the result with the most occurrences in K inferences (the mode) is taken as the final output. If either the speaker confidence score or the timestamp confidence score fails to reach the corresponding threshold, a downgrade process is triggered, including readjusting the temperature parameter of the prompt word for secondary inference, increasing the number of samplings K, or marking the subtitle as requiring manual review. The program has a built-in regular expression interceptor that forces all identified character names to be mapped to standardized topological labels (e.g., [Male Lead], [Female Lead], [Male Supporting Actor 1], [Extra Male Actor 1]), abandoning the fragile identification method that relies on specific names and adopting abstract topological labels, thus achieving perfect decoupling between data flow and specific business logic. The rule engine repairs homophone errors left over from the speech recognition stage, outputting the final high-precision Chinese SRT file, i.e., the character-level accurate subtitle file.

[0034] In this way, a three-level priority multimodal fusion decision model can be used to determine the speaker attribution of each subtitle in the dual-track subtitle file by using the first probability, the second probability, and the third probability. This breaks the ambiguity of weight allocation in traditional multimodal processing, establishes a three-level priority arbitration mechanism, and reduces the problem of speaker misjudgment in complex plots.

[0035] According to one embodiment of the present invention, in step S4, the character-level precise subtitle files are input in batches, and a three-level step-by-step translation process with time constraints is performed to generate target language translation text with accurate timestamps, clear speaker identities, and adapted to the target language culture.

[0036] Figure 5 An exemplary flowchart for generating target language translated text is shown according to an embodiment of the present invention.

[0037] According to an embodiment of the present invention, step S4 includes: step S41, using an overlapping sliding window batching algorithm to input the character-level precise subtitle file in batches, each batch containing the current core interval to be translated and its adjacent context intervals; step S42, the three-level laddered translation processing includes the first level injecting character settings and cultural adaptation rules for semantic equivalence translation, the second level performing synonym substitution compression of the translation under time constraints based on the syllable space mapping relationship between the source language and the target language, and the third level performing semantic skeleton extraction and minimum expression reconstruction in extreme over-limit scenarios; step S43, through truncation repair and breakpoint resumption, stably generating the target language translated text.

[0038] According to one embodiment of the present invention, in order to reduce the attention decay problem when processing long texts and the context breakage problem caused by batch processing, an overlapping sliding window batching algorithm is designed. For example, assuming that there are X subtitles in the entire video, the base size of each batch is set to B (e.g., B=30), and the overlapping window size is W (e.g., W=5). This can be achieved using the formula: , , and Where k is the kth batch, For the input subtitle range, For the actual range of outputs adopted, To obtain and The minimum value, the adoption intervals of each batch have no overlap or gaps, and each subtitle is permanently saved only once. and This means that the preceding W and following W subtitles are input into the multimodal large model. When translating the core content of the kth batch, the multimodal large model extracts the contextual information of the preceding W and following W subtitles. The translation results of the actual adopted output range are persisted to reduce problems such as unclear pronoun references and abrupt changes in tone caused by segmented translation. Character characteristics from the global character archive and the cultural background of the target country are obtained. The first-level ladder injects character design and cultural adaptation rules for semantic equivalence translation. For example, for the male lead, a domineering CEO, the tone is cold and concise; when translating into English, American street slang is injected. The second-level ladder performs synonym substitution compression of the translation under time constraints based on the syllable space mapping relationship between the source and target languages, which can be calculated using the formula: Calculate the estimated pronunciation duration ,in, For language types, For English, In Vietnamese In Thai Y represents the baseline speech rate parameter for the target language, and Y represents the translation. The number of natural syllables in the translation. The word count of the translation. This represents the number of characters in the translated text. This is the baseline speech rate parameter for the target language, measured in syllables per second. For example, 5.0 for English. If the language is English, the mapping is calculated based on the number of natural syllables, and the estimated pronunciation duration is... If the language is Vietnamese, the mapping is based on word count. Vietnamese is a monosyllabic language, with one word approximately equal to one syllable. The estimated pronunciation time is... If the language is Thai, the mapping is calculated as the number of characters / 3.5. Thai has complex vowel and consonant combinations; on average, 3.5 characters ≈ 1 pronunciation unit. The estimated pronunciation duration is... The third level performs semantic skeleton extraction and minimal expression reconstruction in extreme over-limit scenarios. Specifically, it prioritizes removing honorifics, modal particles, non-critical degree adverbs (e.g., "very"), and time adverbs from the target language. When the context is clear, it omits subjects or objects, reconstructing complex sentences into simpler ones. If timeouts still occur, it abandons the complete grammatical structure, directly extracting the core action intent and replacing it with a minimalist imperative sentence or short, colloquial phrase. When the program receives the returned text from the large model, it first performs regular expression matching and JSON parsing. If it finds that the JSON structure is truncated (e.g., ending with} or { without a closing bracket ]), the program will not crash directly but will perform intelligent completion: finding the last complete JSON object closing bracket}, truncating the remaining content, and forcibly completing ] to make it a valid JSON array. Subsequently, the program performs integrity verification. Let the expected number of subtitles be N_expected, and the actual number of successfully parsed subtitles be N_actual: ΔN = N_expected - N_actual, where ΔN is the difference in the number of remaining subtitles. If ΔN > 0, the program arrangement layer will automatically trigger the Continuation logic: extract the ID of the last successfully parsed subtitle (let's call it ID_last). Construct a special Continuation prompt: "You have output N_actual subtitles, and you still need to output ΔN more. The last processed ID is ID_last. Please start from the next ID and continue outputting the complete JSON array of the remaining subtitles. Do not add any explanatory text." Add this prompt as a user role to the dialogue history and call the large model again. Execute this process repeatedly until actual = N_expected, or the maximum number of retries is reached. This solves the instability of the large model in long text generation tasks, enabling the system to stably handle video translation tasks lasting several hours without human intervention.

[0039] In this way, a three-tiered translation process can be used to ensure that the duration of the translated text is less than or equal to the available video length. Character design and cultural adaptation rules are injected into the translation process, allowing audiences in the target country to seamlessly immerse themselves in the story. The sliding window batching algorithm reduces contextual gaps, thereby reducing problems such as unclear pronoun references and abrupt changes in tone caused by segmented translation.

[0040] According to one embodiment of the present invention, the duration constraint processing of the second-level ladder includes: converting the source language text into a pinyin sequence and counting the number of syllables; converting the target language translation into a phoneme sequence and estimating the pronunciation duration; determining a duration matching score that measures the degree of duration exceeding the limit of the translation by using the syllable space mapping ratio between the source language and the target language; and activating a synonym substitution compression strategy when the translation duration exceeds the available duration.

[0041] According to one embodiment of the present invention, it can be determined according to the formula: Calculation of time-matching score ,in, The available duration of the video clip. To estimate the duration of pronunciation, if If the duration is deemed safe, then the duration is considered safe. If the timeout is severe, a synonym substitution compression strategy is initiated. For example, while keeping the core semantics unchanged, long words are replaced with short words and clause structures are simplified into phrase structures, so that the estimated pronunciation time of the translation is close to the available time.

[0042] According to one embodiment of the present invention, let the amount of core dramatic information contained in the original Chinese subtitles be... The edge modification information content is Total information content , target language translation The actual amount of core information conveyed is The edge information content is Define semantic fidelity in, and These are the weighting coefficients, and To maximize When the channel is limited, priority should be given to using Even if sacrifice (Right now Even if the term approaches 0, it is still worthwhile. Define channel utilization. The ultimate optimization goal is to ensure that the channel is not overloaded. Maximizing semantic fidelity under the hard constraints of ) and cultural resonance enhancement : and .

[0043] According to one embodiment of the present invention, in step S5, progressive acoustic feature compression is performed on the pure human voice track, and emotional transfer and target language translation text are performed through a context-aware dynamic emotion computing network to generate dubbing audio in the target language.

[0044] Figure 6 An exemplary flowchart illustrating the generation of dubbed audio in a target language according to an embodiment of the present invention is shown.

[0045] According to an embodiment of the present invention, step S5 includes: step S51, extracting the original audio segments of each character from the purified human voice track based on the character-level precise subtitle file; step S52, performing progressive acoustic feature compression on the original audio segments to obtain purified audio segments; step S53, inputting the purified audio segments as reference samples into the speech cloning engine for synthesis, wherein the speech cloning engine routes between persistent cloning mode and real-time reference mode based on video task metadata, character cross-set reuse requirements, cumulative duration of reference samples, and persistent timbre quota status; step S54, calculating emotional weights through local acoustic emotional intensity, accumulated emotional attenuation terms of historical context, and contextual position weights; step S55, controlling the synthesized speech rate, pitch, energy, and pause parameters based on the emotional weights and the target language translated text to generate dubbing audio in the target language.

[0046] According to one embodiment of the present invention, original audio segments of each character are extracted from a clean vocal track based on a character-level precise subtitle file. Progressive acoustic feature compression is performed on the original audio segments, for example, intelligent energy density truncation, retaining the segments with the highest energy density and the most intense vocal activity. The formula is expressed as finding the optimal starting point: ,in, For a moment The short-term energy, the truncated audio frequency is denoted as The maximum allowed duration is , Indicates from Beginning, Continue The integral of energy over a given time period. For all possible starting points In the given information, find the one that maximizes the above integral value. Nyquist safe downsampling gradually reduces the sampling rate from the original high frequency (e.g., 44.1kHz or 48kHz) to 24kHz and 16kHz. Spectral threshold silence removal prioritizes the removal of masking information in frequency bands insensitive to the human ear. The bit rate is gradually reduced from 192kbps to 128kbps, 64kbps, and 32kbps, safely compressing the excessive original audio segments to within the threshold range to obtain purified audio segments. These purified audio segments are then used as reference samples input into a speech cloning engine for synthesis. The speech cloning engine uses video task metadata (e.g., total video duration, episode ID, expected processing time) to perform synthesis. The system considers factors such as the number of episodes, the need for cross-episode reuse of characters, the cumulative duration of reference samples, and the persistent timbre quota status. It routes between persistent cloning mode and real-time reference mode. If the cumulative duration of the reference samples is the length of a long TV series (e.g., 40 hours), the synthesis mode type is determined to be persistent cloning mode. The reference samples are uploaded to the underlying large model (e.g., a neural network language model), generating a globally unique, permanent sound tag. In subsequent episodes, when encountering the same character, the synthesis interface is directly called through this sound tag, eliminating the need to repeatedly upload reference samples, improving the consistency of timbre features, and reducing the slight feature drift caused by each re-extraction of samples. If the cumulative duration of the reference samples is a single episode of short video (e.g., 1 minute), the synthesis mode type is determined to be real-time reference mode. The system does not generate a permanent sound tag; instead, each time a single line of dialogue is requested for synthesis, the original audio segment corresponding to that line is sent as reference audio along with the text to the underlying large model, without consuming the system's persistent timbre quota. In persistent clone mode, as the number of episodes processed increases, the number of Voice IDs stored by the system will inevitably reach the physical limit of the underlying model (e.g., 30). To prevent the pipeline from crashing due to quota exhaustion, the program orchestration layer introduces a smart eviction and normalized recycling mechanism based on least-used frequency. The system maintains a global timbre mapping table locally, recording the Voice ID of each character and its usage_count in the current batch of tasks. When the underlying model throws an exception indicating that the number of timbres has reached its limit, the scheduler triggers the following blocking and recycling logic: Protection period determination: Lock all characters appearing in the currently processed video and add them to the exemption list to prevent the synthesis of the current video from being interrupted; Victim selection: In the non-exempt list, traverse the timbre mapping table and find the character with the smallest usage_count as the victim; API-level destruction and local normalization: Call the underlying model's Delete interface to completely destroy the victim's Voice ID, release the cloud quota, and erase its record in the local mapping table to complete the state normalization; Breakpoint retry: After the quota is released, the system automatically retrys previously failed clone requests.The local acoustic sentiment intensity of the reference sample is analyzed (calculated through energy variance and fundamental frequency fluctuation), and the sentiment attenuation accumulation term of the historical context and the contextual position weight output by the multimodal large model are obtained. If the current sentence is determined to be an "emotional outburst point," this term will provide a significant gain. The three dimensions of local acoustic sentiment intensity, historical context sentiment attenuation accumulation term, and contextual position weight are weighted and fused to calculate the sentiment weight, i.e., according to the formula: ,in, The local acoustic emotional intensity of the reference audio for the current sentence, r represents the current sentence. Forgetting factor, Let q be the contextual sentiment intensity of sentence q. The contextual position weights are output by the inference layer of the large model. This is the normalized adjustment coefficient. For the cumulative term of emotional decay in historical context, This is the interval between the current sentence and the previous sentence. The forgetting factor (e.g., 0.3) represents the inertial influence of preceding emotions on the current sentence (e.g., after a heated argument, even if the next sentence is calm, the voice should still carry a hint of anger and resentment). The emotion weights, emotion category labels inferred from the multimodal large model (e.g., emo_text="angry"), and high-purity reference audio compressed using the PAFC algorithm are packaged together to form an API request payload. This payload is sent to the underlying TTS (text-to-speech) engine, which can more accurately reproduce the complex emotions of a character in the current specific plot context. This involves controlling the speech rate, pitch, energy, and pause parameters of the speech synthesis (e.g., mapped to speech rate adjustment coefficients, pitch offsets, energy scaling factors, and pause duration parameters). Then, by translating the text into the target language, the target language dubbing audio can be obtained, achieving truly atmospheric dubbing.

[0047] According to one embodiment of the present invention, in step S6, the original video is divided into multiple independent speech segments on the time axis, the sensory tolerance threshold of each speech segment is dynamically calculated, and the dubbing audio and the original video are subjected to three-level adaptive fitting according to priority.

[0048] According to one embodiment of the present invention, the original video is divided into multiple independent atomic segments along the timeline, and the total duration of the original video is given by... Its timeline can be represented as a continuous interval. Based on the timestamp set in the dual-track subtitle file. The system divides the timeline into two mutually exclusive sub-intervals: speech segments, () refers to the time period in the corresponding dual-track subtitle file that contains text and speaker tags. Ambient Segment () represents the blank time period between two adjacent lines of dialogue. Through this segmentation, the original video is transformed into an ordered sequence of segments: Where ⊕ represents a splicing operation on the timeline, and subsequent speed-changing operations (acceleration or deceleration) only strictly apply to the timeline. Fragments, and all The segments retain their original duration and frame rate, preserving the natural rhythm of non-dialogue parts of the video to the greatest extent possible (e.g., people walking, landscape shots, etc.). After segmentation, the system... Create a separate, isolated workspace. The system will then store the corresponding audio generated by TTS. The data is allocated to this workspace. Local audio acceleration or video deceleration is applied to each segment. Since the fitting of each segment is independent, the program orchestration layer can utilize multithreading or asynchronous coroutines to process multiple segments in parallel, greatly improving system throughput. After all speech segments have been processed, a new set of segments is obtained. Because some segments may have undergone video slowdown, their duration is [not specified]. The program orchestration layer calls the video processing engine (e.g., a low-level wrapper based on FFmpeg) to reassemble all processed segments with the unchanged ambient sound segments. At this point, the total duration of the reconstructed video is... .

[0049] Figure 7 A flowchart of a three-level adaptive fitting method according to an embodiment of the present invention is shown as an example.

[0050] According to an embodiment of the present invention, step S6 includes: step S61, obtaining a sensory resilience coefficient through contextual information, wherein the sensory resilience coefficient includes auditory acceleration tolerance and visual deceleration tolerance; step S62, obtaining the duration of the dubbing audio, the duration of the target video, and the variance of emotional fluctuation; step S63, dynamically calculating a sensory tolerance threshold for each speech segment based on the sensory resilience coefficient, the duration of the dubbing audio, the duration of the target video, and the variance of emotional fluctuation, wherein the sensory tolerance threshold includes a maximum audio acceleration factor and a minimum video deceleration factor; step S64, performing a three-level adaptive fitting on the dubbing audio and the original video according to the priority of lossless replacement, unidirectional audio acceleration, and bidirectional audio-video compression; step S65, the first-level lossless replacement involves aligning the dubbing audio to the beginning of the video segment and adding silence at the end to compensate for the duration difference when the dubbing audio duration is less than or equal to the target video duration; S66, the second-level unidirectional audio acceleration involves performing high-speed processing on the synthesized speech while preserving the voiceover when the dubbing audio duration is greater than the target video duration and the required audio acceleration factor is less than or equal to the maximum audio acceleration factor; S67, the third-level bidirectional audio-video compression involves locking the required audio acceleration factor to the maximum audio acceleration factor when the required audio acceleration factor is greater than the maximum audio acceleration factor, while simultaneously performing deceleration processing on the video segment. This deceleration processing includes designing a cascaded speed-changing pipeline and a flexible BGM re-attachment mechanism.

[0051] According to one embodiment of the present invention, a multimodal large model performs inference through contextual information to obtain sensory resilience coefficients and auditory acceleration tolerance. (1.0 represents the baseline tolerance), visual deceleration tolerance Let the upper limit of the system's baseline audio acceleration be... (For example, 1.2), the lower limit for baseline video deceleration is (For example, 0.55), a sensory tolerance threshold is determined based on the sensory elasticity coefficient, the duration of the dubbing audio, the duration of the target video, and the variance of emotional fluctuation. The sensory tolerance threshold includes the maximum audio acceleration factor and the minimum video deceleration factor, according to the formula: and Calculate the maximum audio speedup factor for the j-th speech segment. The minimum video deceleration factor for the j-th audio segment , The auditory acceleration tolerance of the j-th speech segment, Let be the visual deceleration tolerance of the j-th speech segment. The duration of the dubbing for the j-th speech segment. Let j be the target video duration for the j-th audio segment. For the variance of mood fluctuation, The penalty coefficient is... This is the adjustment coefficient. This is the penalty coefficient (e.g., 0.2). This is an adjustment factor (e.g., 0.3). This is a duration difference penalty. The larger the duration difference penalty, the higher the degree of compression is required. Conversely, the speedup limit needs to be tightened to prevent audio distortion. This is the reciprocal of the visual deceleration tolerance for the j-th speech segment. The higher the visual deceleration tolerance, the lower the minimum video deceleration factor. The larger the target video length, the more intense the emotion, and the lower the visual tolerance for slow motion (because slowing down intense movements can appear comical). The first level of lossless replacement aligns the voice-over audio to the beginning of the video segment and adds silence at the end to compensate for the duration difference when the voice-over audio length is less than or equal to the target video length. The second level of unidirectional audio acceleration performs only voice-preserving high-speed processing on the synthesized speech when the voice-over audio length exceeds the target video length and the required audio acceleration factor is less than or equal to the maximum audio acceleration factor, calculating the required audio acceleration factor. : The third level of bidirectional audio-video compression locks the required audio acceleration factor to the maximum when the required acceleration factor exceeds the maximum. Simultaneously, it slows down the video clip to lengthen the frame, ensuring precise alignment of audio and video on the timeline, and locks the audio acceleration factor to a dynamic upper limit. At this point, the duration of the accelerated audio is: To match the new audio duration, the video needs to be stretched, requiring the calculation of the necessary video slowdown factor. (The smaller the value, the slower): At this point, the judgment Is it below the lower limit of dynamic video deceleration calculated by the multimodal large model? .if This indicates that the deceleration is within a visually acceptable range. The system will accelerate the audio. Multiplier, while slowing down the video. Double the alignment to complete. If This indicates that even with audio accelerated to its limit and video slowed down to its limit, alignment is still impossible. Audio acceleration (e.g., simple resampling) causes a rise in sound frequencies, severely cloning the character's timbre. This is addressed using a high-speed, frequency-preserving technique based on the WSOLA (Waveform Similarity Based Overlap-Add) algorithm (e.g., FFmpeg's atemppo filter). The WSOLA algorithm maintains the fundamental frequency while changing the duration by finding similar waveform segments in the time domain and overlapping them. However, the effective processing range of a single atemppo filter is usually hard-coded to... In extreme cases (e.g., a very short interjection is translated into a long sentence, resulting in a speedup exceeding 2.0), directly calling the underlying tools will cause errors and crashes. Therefore, the program orchestration layer is designed with a cascaded speed-up pipeline. When the target speedup... At that time, it will automatically perform prime factorization or logarithmic splitting to construct multiple cascaded filters. For example, if the target speedup is... It automatically breaks it down into two filters connected in series. and . ,in , To accommodate the effective parameter range allowed by a single filter, this cascaded design overcomes the physical limitations of the underlying tools, ensuring perfect preservation of the character's timbre and formants under any extreme speedup ratio, preventing system crashes due to extreme parameters. The existence of the third-level audio-visual bidirectional compression means the total duration of the reconstructed video is greater than or equal to the original video's total duration. Directly pasting the original background music (BGM) back would result in a vacuum period without BGM at the end of the video, or a misalignment between the BGM's climax and the visual climax. Therefore, the system employs a flexible BGM pasting mechanism: anchor point locking: using a beat detection algorithm to identify strong beat points and structural transition points in the BGM (e.g., the transition from verse to chorus), and using these strong beat points and structural transition points as time anchor points; non-destructive stretching: during the corresponding time period when the video slows down, it searches for... The system identifies non-melodic dense areas in the background music (BGM) (e.g., continuous chords, ambient noise, or repetitive drum beats) and uses minute loop extensions or phase vocoder temporal stretching to fill audio gaps caused by video stretching. Global dynamic mixing: The stretched BGM is dynamically equalized with the fitted TTS vocal track. This is achieved through sidechain compression technology, where the BGM gain is automatically reduced (e.g., reduced by -6dB) when the TTS vocals appear and smoothly restored when the vocals end, ensuring that the target language dialogue can be clearly heard even in complex acoustic environments.

[0052] In this way, multimodal speech rate perception can be achieved by introducing a multimodal large model, realizing dynamic speed calculation that varies from person to person, from context to language. Based on the sensory tolerance threshold, a three-level degradation strategy is used to prioritize ensuring audio losslessness, then ensuring video losslessness, and finally performing bidirectional compression to protect the artistic expression of the original material to the greatest extent.

[0053] According to one embodiment of the present invention, in step S7, when the fitting fails, a problem segment with a timeout mark is generated, and multi-level entropy reduction processing is performed on the problem segment in sequence to form a closed-loop feedback to obtain audiovisual synchronized video.

[0054] Figure 8 An exemplary flowchart for obtaining synchronized audiovisual video according to an embodiment of the present invention is shown.

[0055] According to an embodiment of the present invention, step S7 includes: step S71, performing multi-level entropy reduction processing on the timeout-marked problem segment sequentially, including output text limit rewriting, input text skeleton extraction, iteration termination determination, and image-generated video ultimate defense processing, to obtain the repaired segment; step S72, sending the repaired segment back to the translation step for reprocessing to form a closed-loop feedback and obtain audiovisual synchronized video.

[0056] According to one embodiment of the present invention, when physical fitting fails, the target language translation is first rewritten to eliminate semantic redundancy. Secondly, the target language translation undergoes skeleton extraction to reduce information content at the source. A large model is then invoked to extract the skeleton from the original source language text. If the extraction returns "success," the simplified original text is re-entered into the complete translation workflow; if it returns "failure," the final defense is triggered. To prevent the system from entering an infinite loop, a strict termination function is defined in the program orchestration layer. : (When the fit is successful, or the number of iterations reaches the limit, or the second-level degradation fails), otherwise The ultimate defense against image-based video translation involves extracting the last frame of the current video segment as a seed image when all text-level optimization methods fail to fit the problematic segment to the audio-visual format. This seed image is then input into an image-based video model (e.g., using a Transformer architecture) to generate supplementary video frames that conform to physical laws and visual coherence. These supplementary video frames are then stitched to the original video segment to extend the usable video duration, ensuring that the synthesized speech is aligned within a threshold of sensory tolerance. The repaired segment is then fed back to the translation step for reprocessing, forming a closed-loop feedback loop to obtain synchronized audiovisual video. This reduces the uncertainty of cross-language translation and improves the stability of industrial-grade mass production.

[0057] According to one embodiment of the present invention, in step S8, the audiovisual synchronized video is subjected to independent lip-sync rendering processing for multiple people lip-syncing to obtain the final translated dubbing video.

[0058] Figure 9 An exemplary flowchart illustrating the process of obtaining the final translated dubbing video according to an embodiment of the present invention is shown.

[0059] According to an embodiment of the present invention, step S8 includes: step S81, performing multi-channel sound source separation on the synthesized multi-character target language dubbing audio in the audiovisual synchronization video to obtain independent audio tracks for each character, wherein each independent audio track carries a corresponding character identifier; step S82, performing face detection and tracking on the video frames of the audiovisual synchronization video, constructing continuous face trajectories through Kalman filtering and cross-union matching, and generating pixel-level binary mask sequences covering the lower half of the face and lip region of each character; step S83, based on the character identifier and visual feature matching of the global character archive, separating... The independent audio tracks and facial trajectories are deterministically bound through character identifiers; in step S84, for each character, the corresponding independent audio track and dynamic lip mask are extracted and input into the lip shape generation network along with the original video frames. Pixel deformation is calculated within the mask area defined by the pixel-level binary mask sequence, and lip shape rendering is performed independently to generate local lip shape update frames; in step S85, the local lip shape update frames of all characters are spliced ​​into the original video frames according to their corresponding mask areas, and Gaussian blur and Poisson fusion algorithms are applied at the boundaries of the pixel-level dynamic lip mask to eliminate splicing traces, and the final translated dubbing video is output.

[0060] According to one embodiment of the present invention, the deterministic binding is implemented as follows: Multi-channel sound source separation based on frequency band segmentation and time-frequency masking is performed on the synthesized multi-character target language dubbing audio in the audiovisual synchronization video to obtain independent audio tracks for each character, with each independent audio track carrying a corresponding character identifier. Multi-target face detection and tracking are performed on the video frames of the audiovisual synchronization video to establish the spatiotemporal trajectory of each face, and Kalman filtering is used for trajectory smoothing and prediction. For the input original video frame sequence, a high-precision dense face detection and tracking algorithm is first deployed. In each frame, the algorithm not only locates the bounding boxes of all faces but also extracts key feature points and constructs a continuous set of face tracks on the time axis using Kalman filtering and Intersection over Union (IoU) matching techniques. For each facial trajectory T j The system utilizes an image segmentation model to generate a binary mask sequence accurate to the pixel level. The generation function of the mask matrix is ​​defined as follows: Where t is time, For pixel coordinates, Let be the lip region on the target character's trajectory at time t. This mask only covers the lower half of the target character's face and lip region, while making the rest of the image (including the faces of other characters and the background) invisible. Visual feature vectors of each face trajectory are extracted, and their cosine similarity to the visual feature vectors of each character in the global character archive is calculated. ,in, facial trajectory Visual feature vectors, For the visual feature vector of the e-th character in the global character archive, a strict similarity threshold (e.g., 0.9) is set to map the facial trajectory to the corresponding character identifier. Using this unique character identifier as a key, independent audio tracks are deterministically bound to facial trajectories, achieving a one-to-one correspondence between audio and visual data. Logical binding based on the global context reduces the uncertainty in traditional cross-modal matching and lowers the matching error rate. For each character e in the scene, the system extracts its corresponding independent audio track. and dynamic masking sequences The original video frames, along with the generated lip shape, are fed into the lip-shape generation network. The generation network only generates lip shapes under the mask. Calculate pixel deformation within the defined effective area and output the local lip shape update frame for the character. ,in, These are the original video frames. This is a pixel-by-pixel multiplication. Because the rendering process for each character is completely physically isolated, even in scenes with multiple characters arguing intensely and highly overlapping voices, the lip-syncing of each character does not interfere with each other, and each follows its own specific audio instructions for precise movement. A multi-character frame fusion operation is performed, updating the local frames of all characters. According to the corresponding masked area, it is seamlessly stitched into the original video frame V. To eliminate the harshness of the stitching edges, the system applies Gaussian blur and Poisson fusion algorithms at the mask boundaries to remove stitching artifacts, resulting in natural transitions in skin tone, lighting, and texture. The final output frame after fusion... ,in, The set of masks for all characters is defined as the union of the masked sets, where E is the number of characters, e ≤ E, and both e and E are positive integers. This involves fusing the local lip-sync update frames of all characters into the original video frames to output the final translated dubbing video. The constructed multi-person lip-syncing engineering system based on sound separation and video frame masking successfully overcomes the industry challenge of chaotic multi-person lip-syncing in complex film and television scenes through a rigorous chain of auditory decoupling, visual isolation, deterministic logic binding, and independent rendering fusion. This system not only demonstrates extremely high robustness and accuracy in its technical implementation but also achieves extreme compression of system entropy at the information theory level, providing industrial-grade high-quality visual presentation assurance for cross-language video translation.

[0061] The video translation and dubbing fitting method based on multimodal context according to embodiments of the present invention significantly improves the accuracy of source language understanding through dual-track subtitle files. A three-level priority multimodal fusion decision model achieves the uniqueness of character voices and high-fidelity emotional transmission. A three-level tiered translation processing method jointly optimizes the translation process across multiple objectives. A three-level adaptive fitting method, employing audio limiting acceleration and video limiting deceleration, improves the matching degree between the target language dubbing and the video timeline. Based on sound separation and video frame masking, it solves the problem of lip-sync confusion in complex multi-person scenes. By using a three-level priority multimodal fusion decision model with first, second, and third probabilities, the method determines the speaker attribution for each subtitle in the dual-track subtitle file, breaking the ambiguity of weight allocation in traditional multimodal processing, establishing a three-level priority arbitration mechanism, and reducing the problem of speaker misjudgment in complex plots. A three-tiered translation process ensures that the duration of the translated audio is less than or equal to the available video length. Character design and cultural adaptation rules are integrated into the translation process, allowing audiences in the target country to seamlessly immerse themselves in the story. A sliding window batching algorithm reduces contextual gaps, thus mitigating issues such as ambiguous pronoun references and abrupt changes in tone caused by segmented translation. Furthermore, a multimodal large-scale model is introduced for multimodal speech rate perception, enabling dynamic speed calculations that vary depending on the person, context, and language. Based on sensory tolerance thresholds, a three-tiered degradation strategy prioritizes lossless audio, followed by lossless video, and finally bidirectional compression, maximizing the preservation of the original material's artistic expression.

[0062] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0063] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and explained in the embodiments, and any variations or modifications may be made to the implementation of the present invention without departing from the stated principles.

Claims

1. A video translation and dubbing fitting method based on multimodal context, characterized in that, include: The original video is subjected to sound source separation to obtain a clean human voice track, and optical character recognition is performed on the frame to construct a visual semantic context map; A dynamic hot word list is generated based on the visual semantic context graph, and the decoding bias of candidate words is adjusted when performing automatic speech recognition on a clean human voice track to generate a dual-track subtitle file. By constructing a global role archive and assigning unique role identifiers, and by adopting a three-level priority multimodal fusion decision model, the speaker attribution of dual-track subtitle files is determined, and a role-level accurate subtitle file is obtained through a multi-source cross-validation mechanism; The character-level precise subtitle files are input in batches, and a three-level step-by-step translation process with time constraints is performed to generate the target language translated text. Progressive acoustic feature compression is performed on the pure human voice track, and emotional transfer and target language translation text are performed through a context-aware dynamic emotion computing network to generate dubbing audio in the target language; The original video is divided into multiple independent audio segments on the timeline, the sensory tolerance threshold of each audio segment is dynamically calculated, and the dubbed audio and the original video are subjected to three-level adaptive fitting according to priority. When the fitting fails, a timeout-marked problem segment is generated. The problem segment is then subjected to multi-level entropy reduction processing and a closed-loop feedback is formed to obtain audiovisual synchronized video. Independent lip-sync rendering is performed on the audiovisual synchronized video to obtain the final translated dubbing video.

2. The video translation and dubbing fitting method based on multimodal context according to claim 1, characterized in that, The original video undergoes sound source separation to obtain a clean human voice track, and optical character recognition is performed on the video frames to construct a visual semantic context map, including: By employing a deep learning architecture based on a frequency band segmentation attention mechanism, a sound source separation model is used to separate the original mixed audio sources in the frequency domain space, thereby obtaining a clean vocal track. Extract all frame sequences of the original video frame by frame and calculate the similarity between adjacent frame pairs; If the similarity of multiple consecutive adjacent frame pairs is greater than the similarity threshold, then the multiple consecutive adjacent frame pairs are determined to belong to the same frame group. Within each frame group, the frame with the highest image quality is selected as the keyframe; The keyframes are used to locate the text region using an object detection model, and then the text content is extracted using an optical character recognition model to obtain a visual text sequence containing timestamps, text content, and bounding boxes. An entity dictionary is constructed based on the visual text sequence, wherein the entity dictionary includes character names, place names, proper nouns and their frequency of occurrence; By analyzing the role relationships and scene semantics in the entity dictionary through multimodal large model analysis, entity relationship and background story attributes are generated; Based on the entity dictionary, the entity relationships, and the background story attributes, a visual semantic context graph of the original video is constructed.

3. The video translation and dubbing fitting method based on multimodal context according to claim 1, characterized in that, A dynamic hotword list is generated based on the visual semantic context graph, and the decoding bias of candidate words is adjusted when performing automatic speech recognition on a clean human voice track to generate a dual-track subtitle file, including: A dynamic hot word list is generated from the visual semantic context graph using a keyword extraction algorithm; The pure human voice track and the candidate words are input into the acoustic model and the general language model to obtain the first probability score and the second probability score. The decoding bias of candidate words is adjusted using the dynamic hot word list to obtain the context bias function; The first probability score, the second probability score, and the context bias function are weighted and summed to determine the enhanced decoding score of the candidate word in the speech recognition process. Based on the enhanced decoding score, speech recognition is enhanced, and a dual-track subtitle file is generated, wherein the dual-track subtitle file includes a plain text track and a speaker tag track.

4. The video translation and dubbing fitting method based on multimodal context according to claim 1, characterized in that, By constructing a global role archive and assigning unique role identifiers, and employing a three-level priority multimodal fusion decision model, speaker attribution is determined for dual-track subtitle files. Furthermore, a multi-source cross-validation mechanism is used to obtain precise role-level subtitle files, including: Based on the clean human voice track, dual-track subtitle file and original video, obtain the voiceprint feature vector, semantic feature vector and visual feature vector for each character; A global character archive is constructed based on the voiceprint feature vector, semantic feature vector, and visual feature vector of each character, and a unique character identifier is assigned to each character. The dual-track subtitle file is divided into multiple subtitles to be proofread. The voiceprint feature vectors of the pure human voice track segments corresponding to the subtitles to be proofread are extracted. The cosine similarity between the voiceprint feature vectors to be proofread and the voiceprint feature vectors of each character in the global character archive is calculated to obtain the first probability of physical voiceprint feature matching. Extract the visual feature vector of the original video segment corresponding to the subtitle to be proofread, calculate the cosine similarity between the visual feature vector to be proofread and the visual feature vector of each character in the global character archive, and obtain the second probability of synchronization between lip movements and speech rhythm. The third probability of the subtitles to be proofread being spoken by each character is inferred using a multimodal large model; A three-level priority multimodal fusion decision model is adopted, with voiceprint matching as the first priority. If the maximum value of the first probability is greater than the first high confidence threshold, the speaker is determined to belong to the role that makes the first probability the maximum value. With face detection in the image as the second priority, if the maximum value of the first probability is less than or equal to the first high confidence threshold, and the maximum value of the second probability is greater than the second high confidence threshold, then the speaker is determined to belong to the role that makes the weighted sum of the first probability and the second probability the maximum value. Using semantic context reasoning as the third priority, if the maximum value of the first probability is less than or equal to the first high confidence threshold, and the maximum value of the second probability is less than or equal to the second high confidence threshold, then the speaker is determined to belong to the role that maximizes the weighted sum of the first, second, and third probabilities. A multi-source cross-validation mechanism is used to assess and correct the confidence level of the speaker tag corresponding to each subtitle to be proofread, thereby obtaining a role-level accurate subtitle file carrying the unique role identifier. The multi-source cross-validation mechanism involves performing multiple independent inference samplings on the same set of inputs: clean audio track, dual-track subtitle file, and original video. Based on the sampling results, speaker confidence and timestamp confidence are statistically analyzed. When both exceed the corresponding threshold, the result is accepted; otherwise, downgrade processing is triggered.

5. The video translation and dubbing fitting method based on multimodal context according to claim 1, characterized in that, The character-level precise subtitle files are input in batches, and a three-level stepped translation process with time constraints is performed to generate target language translated text, including: The character-level precise subtitle files are input in batches using an overlapping sliding window batching algorithm. Each batch of input contains the current core section to be translated and its adjacent context sections before and after it. The three-tiered translation processing includes the first tier injecting character settings and cultural adaptation rules to perform semantically equivalent translation; the second tier performing synonym substitution compression of the translation under time constraints based on the syllable space mapping relationship between the source language and the target language; and the third tier performing semantic skeleton extraction and minimal expression reconstruction in extreme over-limit scenarios. By using truncation repair and breakpoint resumption, the target language translation text is generated stably.

6. The video translation and dubbing fitting method based on multimodal context according to claim 5, characterized in that, The duration constraint processing for the second-level ladder includes: The source language text is converted into a pinyin sequence and the number of syllables is counted; the target language translation is converted into a phoneme sequence and the pronunciation duration is estimated. The duration matching score, which measures the degree of translation duration exceeding the limit, is determined by the syllable space mapping ratio between the source language and the target language. When the translation time exceeds the available time, the synonym replacement compression strategy is activated.

7. The video translation and dubbing fitting method based on multimodal context according to claim 1, characterized in that, Progressive acoustic feature compression is performed on the clean human voice track, and emotion transfer and target language translation text are performed through a context-aware dynamic emotion computing network to generate dubbing audio in the target language, including: From the pure human voice track, extract the original audio segments of each character based on the character-level precise subtitle file; The original audio segment is subjected to progressive acoustic feature compression to obtain a purified audio segment; The purified audio segments are used as reference samples and input into the speech cloning engine for synthesis. The speech cloning engine routes between persistent cloning mode and real-time reference mode based on video task metadata, role cross-set reuse requirements, cumulative duration of reference samples, and persistent timbre quota status. The sentiment weight is calculated by using local acoustic sentiment intensity, sentiment attenuation accumulation term of historical context, and contextual position weight; Based on the emotional weight and the target language translated text, the synthesized speech rate, pitch, energy, and pause parameters are controlled to generate the target language dubbing audio.

8. The video translation and dubbing fitting method based on multimodal context according to claim 1, characterized in that, The original video is segmented into multiple independent audio segments along the timeline. The sensory tolerance threshold for each audio segment is dynamically calculated. A three-level adaptive fitting process is then performed on the dubbed audio and the original video according to priority, including: The sensory resilience coefficient is obtained through contextual information, wherein the sensory resilience coefficient includes auditory acceleration tolerance and visual deceleration tolerance; Obtain the duration of the dubbing audio, the duration of the target video, and the variance of emotional fluctuations; Based on the sensory elasticity coefficient, the duration of the dubbing audio, the duration of the target video, and the variance of emotional fluctuation, the sensory tolerance threshold for each speech segment is dynamically calculated, wherein the sensory tolerance threshold includes the maximum audio acceleration factor and the minimum video deceleration factor. The dubbing audio and the original video are subjected to a three-level adaptive fitting according to the priority of lossless replacement, one-way audio acceleration, and two-way audio-video compression; The first level of lossless replacement involves aligning the dubbing audio to the beginning of the video segment when the dubbing audio duration is less than or equal to the target video duration, and filling the end with silence to make up for the duration difference. The second level of unidirectional audio acceleration is to perform high-speed processing on synthesized speech while preserving the sound when the duration of the dubbing audio is greater than the duration of the target video and the required audio acceleration factor is less than or equal to the maximum audio acceleration factor. The third level of bidirectional audio and video compression involves locking the required audio acceleration factor to the maximum audio acceleration factor when the required audio acceleration factor is greater than the maximum audio acceleration factor, while simultaneously performing deceleration processing on the video segment. This deceleration processing of the video segment includes designing a cascaded speed-changing pipeline and a flexible BGM re-attachment mechanism.

9. The video translation and dubbing fitting method based on multimodal context according to claim 1, characterized in that, When fitting fails, a timeout-marked problem segment is generated. This problem segment undergoes multi-level entropy reduction processing to form a closed-loop feedback loop, resulting in synchronized audiovisual video, including: The problematic segments marked with timeouts are sequentially subjected to multi-level entropy reduction processing, including output text extreme rewriting, input text skeleton extraction, iteration termination judgment, and image-generated video ultimate defense processing, to obtain the repaired segments. The repaired segment is sent back to the translation step for reprocessing, forming a closed-loop feedback loop to obtain synchronized audiovisual video.

10. The video translation and dubbing fitting method based on multimodal context according to claim 1, characterized in that, Independent lip-sync rendering is performed on the synchronized audiovisual video to obtain the final translated dubbing video, including: Multi-channel sound source separation is performed on the synthesized multi-character target language dubbing audio in the audiovisual synchronization video to obtain independent audio tracks for each character, wherein each independent audio track carries a corresponding character identifier; Face detection and tracking are performed on the video frames of the audiovisual synchronized video. Continuous face trajectories are constructed through Kalman filtering and cross-union matching to generate pixel-level binary mask sequences covering the lower half of the face and lip region for each character. Based on the character identifiers and visual feature matching of the global character archive, the separated independent audio tracks and facial trajectories are deterministically bound through the character identifiers; For each character, the corresponding independent audio track and dynamic lip mask are extracted and input into the lip shape generation network along with the original video frames. Pixel deformation is calculated within the mask area defined by the pixel-level binary mask sequence, lip shape rendering is performed independently, and local lip shape update frames are generated. The local lip shape update frames of all characters are stitched into the original video frames according to their corresponding masking areas. Gaussian blur and Poisson fusion algorithms are applied at the pixel-level dynamic lip masking boundaries to eliminate stitching artifacts, and the final translated dubbing video is output.

Images