Voice dubbing method, apparatus and electronic device

Abstract

The application discloses a dubbing method and device and electronic equipment. The method comprises the following steps: obtaining a material to be dubbed; extracting a lip feature of a target object from the material to be dubbed, wherein the lip feature is used to represent at least a change of a lip movement state of the target object over time; determining a plurality of candidate texts corresponding to a subtitle text of the target object in the material to be dubbed, wherein the plurality of candidate texts comprise the subtitle text expressed in a plurality of expression manners by using a language of the same language type; determining a target text matched with the lip feature from the plurality of candidate texts, and dubbing the material to be dubbed by using the target text. The application solves the technical problem of dubbing distortion caused by the fact that, in the intelligent dubbing of a language, a related technology is used to fit the lip feature by using a post-processing method such as speed stretching after a fixed translation.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and more specifically, to a dubbing method, apparatus, and electronic device. Background Technology

[0002] In language-based intelligent dubbing, the relevant technology first generates a fixed translation of the target language by human or machine translation, then uses a TTS system to generate dubbing audio, and then forcibly matches the duration and rhythm of the character's lip movements in the original video by means of audio speed stretching, mute insertion, or end compensation. It relies on the uniqueness of the translation and the passive adaptation of the audio, resulting in a large number of dubbing audios having serious mismatches with the original mouth movements.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a dubbing method, apparatus, and electronic device to at least solve the technical problem in language intelligent dubbing where related technologies use post-processing methods such as speed stretching to match lip movements after fixing the translation, resulting in dubbing distortion.

[0005] According to one aspect of the embodiments of this application, a dubbing method is provided, comprising: acquiring dubbing material; extracting lip-shape features of a target object from the dubbing material, wherein the lip-shape features are at least used to characterize the change of the lip movement state of the target object over time; determining a plurality of candidate texts in the dubbing material corresponding to the subtitle text of the target object, wherein the plurality of candidate texts include subtitle texts expressed in multiple ways using the same language type; determining a target text that matches the lip-shape features from the plurality of candidate texts, and dubbing the dubbing material using the target text.

[0006] In this embodiment, a lip-shape feature-driven candidate text selection method is adopted. By extracting the dynamic motion features of the lip region of the target object in the dubbing material, and generating a multi-version candidate text pool based on the semantic equivalence principle, the target text that matches the lip-shape features in terms of temporal rhythm is then selected. This achieves the goal of selecting the expression text that best fits the lip-shape rhythm without changing the semantics, thereby realizing the technical effect of automatically adapting the dubbing text to the lip-shape movement. This solves the technical problem in language intelligent dubbing where related technologies use post-processing methods such as speed stretching to fit the lip-shape after fixing the translation, resulting in dubbing distortion. Attached Figure Description

[0007] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0008] Figure 1 This is a hardware structure block diagram of a computer terminal for a dubbing method according to an embodiment of this application;

[0009] Figure 2 This is a flowchart of a dubbing method according to an embodiment of this application;

[0010] Figure 3 This is a schematic diagram of the overall system architecture of a dubbing method according to an embodiment of this application;

[0011] Figure 4 This is a flowchart of the selection process for a feasible candidate set of a dubbing method according to an embodiment of this application;

[0012] Figure 5 This is a decision path audit flowchart of a dubbing method according to an embodiment of this application;

[0013] Figure 6 This is a flowchart illustrating the application of a language personality profile in a dubbing method according to an embodiment of this application;

[0014] Figure 7 This is a schematic diagram of the structure of a dubbing device according to an embodiment of this application. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0016] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0017] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0018] Mouth ROI (Region of Interest): In video image processing, a sub-region is a specific target area that is locally extracted and analyzed. In this embodiment, it refers to the local image region of the mouth of the target object extracted from the video frame to be dubbed, which is used to stably extract lip movement features.

[0019] Lip Movement Curve (LMC): A sequence of lip opening and closing intensity that is calculated from a sequence of mouth ROI images and changes over time. In this embodiment, it is used to quantify the amplitude and rhythm of the movement of the target object's lips in the time dimension.

[0020] Pause Candidate Interval: On the lip movement curve, the continuous time interval in which the amplitude of lip movement is consistently lower than a preset threshold is used to identify potential locations of natural pauses or breaths in speech, serving as a hard constraint for aligning candidate text pause points.

[0021] Prominent Peak Location: On the lip movement curve, the local maximum point where the lip opening amplitude is significantly higher than that of the adjacent area is also called the strong peak point. In the embodiments of this application, it is used to identify the moment when the lip opening is most intense, and serves as a key time anchor point for priority alignment of stressed syllables, plosives or open vowels in candidate text.

[0022] Available Duration Interval: The minimum and maximum duration range for allowed speech, determined by combining the subtitle display window and the boundaries of adjacent shots. In this embodiment, it is used to constrain the expected duration of candidate text and ensure that the dubbing does not cross shot transitions or exceed the subtitle display time window.

[0023] Semantic Codebook (SCB): A structured collection of multiple semantically equivalent expressions of the same semantic content in the target language. Each expression may also be accompanied by an interpretable attribute label. In the embodiments of this application, it is used to provide a candidate text pool and support the selection of a more fitting expression mode while maintaining the original meaning.

[0024] Filler Codebook (FCB): A set of lightweight language fragments used to adjust rhythm without changing key semantics. It includes rules for limiting the position and frequency of use. In this embodiment, it is used to compensate for rhythm when candidate text is too short, while controlling "machine generation traces" to maintain a natural feel.

[0025] Feasible Candidate Set (FCS): A set of candidate texts that meet the basic matching conditions of lip-sync rhythm after being filtered by hard constraints such as duration, pauses, and terminology compliance. In this embodiment, it serves as a prerequisite for candidate selection, realizing the decision logic of "first proving feasibility, then seeking the optimal one", reducing invalid TTS and post-processing overhead.

[0026] In multilingual dubbing production, different languages ​​have significant differences in the number of syllables, stress positions, and pause habits. The traditional workflow usually involves first fixing the translation, then calling TTS for synthesis, and finally attempting alignment through speed stretching, silent padding, or Lip-Sync post-processing. This type of approach often has the following shortcomings in film and television production:

[0027] (1) Fixed translation makes it difficult to match the lip movements: After the translation is fixed, the difference in lip rhythm can only be dealt with by strong stretching or end compensation, which is easy to be distorted or still does not match the lip movements.

[0028] (2) End compensation is unexplainable: When it fails, it is difficult to determine whether "the text is too long, the pause is wrong, or the lip movements are unreliable", and can only be done manually through repeated trial and error.

[0029] (3) Unstable production: A large number of invalid attempts consume computing power and time, and the results are difficult to audit and review.

[0030] To address the aforementioned technical problems, this application provides corresponding solutions, which are detailed below.

[0031] The dubbing method embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal for implementing a dubbing method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1The different configurations shown.

[0032] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0033] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the dubbing method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the dubbing method described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0034] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a radio frequency (RF) module, used for wireless communication with the Internet.

[0035] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0036] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware components (including circuitry), software components (including computer code stored on a computer-readable medium), or a combination of both hardware and software components. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0037] In the above operating environment, this application provides a dubbing method embodiment. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.

[0038] Figure 2 This is a flowchart of a dubbing method according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0039] Step S202: Obtain the material to be dubbed.

[0040] In step S202 above, the dubbing material refers to a digital media file containing original audio and video content, which includes, but is not limited to, video frame sequences and associated subtitle text. The video part must contain the facial dynamics of the target object (such as a film or television character, a broadcaster, etc.) when speaking, especially a clearly visible image of the lip area.

[0041] In some embodiments of this application, the material to be dubbed includes the lip region of the target object. The lip region specifically refers to a local image sub-region (i.e., mouth ROI) located from a video frame by facial key point detection and contour segmentation technology, which includes the upper and lower lips and part of the gum / nose region.

[0042] Related technologies, by directly using fixed translations and ignoring lip-sync data, necessitate reliance on non-semantic compensation methods such as post-production speed-up or silent insertion, leading to distortion issues such as speech trailing, word skipping, and rhythmic breaks. In some embodiments of this application, real-time lip region extraction based on the video stream can be employed. Specifically, after the system receives the original video stream, a lightweight face detection and tracking model (such as RetinaFace or MediaPipe) deployed on an edge server automatically locates and crops the mouth ROI in each frame, forming a sequence of lip images indexed by "frame ID + lens ID." This method is suitable for scenarios with high real-time requirements, such as live streaming and short videos, ensuring that the lip region dynamically updates with camera movement.

[0043] It should be noted that the lip area of ​​the target object in the dubbing material may be unreliable due to shooting angle, occlusion, or low lighting, leading to failure or severe distortion in subsequent lip feature extraction. To address this issue, a Lip Confidence Marker (LCM) can be introduced as a dynamic constraint weight adjuster. Specifically, the system can automatically calculate a lip confidence value in the [0,1] interval based on the pixel stability, edge sharpness, tracking confidence, and historical continuity of the lip ROI. When the LCM is lower than a preset threshold (e.g., 0.4), the system no longer uses the Peak Position (PPL) and Pause Candidate Interval (PCI) as hard constraints, but only as soft constraints in the selection and scoring. If the LCM is further lower than 0.2, a downgrade path is triggered: that is, the lip constraint is automatically abandoned, and the system reverts to the traditional candidate ranking mode based on duration and semantic consistency. Furthermore, the "lip unreliable, downgraded to no lip constraint mode" can be explicitly recorded in the decision path audit snapshot, providing a traceable basis for manual review.

[0044] The above mechanism ensures that the system remains robust when input quality fluctuates, avoiding global dubbing failure due to unreliable local data, thereby achieving the production-grade reliability goal of accurate dubbing when there is lip-syncing and stable dubbing when there is no lip-syncing.

[0045] Step S204: Extract the lip shape features of the target object from the dubbing material, wherein the lip shape features are used to characterize the change of the target object's lip movement state over time.

[0046] In step S204 above, lip shape features refer to the quantitative indicators of dynamic lip movement that change continuously over time, extracted from the mouth ROI sequence (lip region) in the material to be dubbed. Their function is to characterize the temporal characteristics of the target object's lip opening and closing amplitude, movement speed, and rhythmic pattern during speech. In some embodiments of this application, in addition to simple image visual description, these features can be structured into lip shape activity curves and further abstracted into a lip shape rhythm summary (also called a lip shape codebook) containing multiple dimensions such as strong peak positions, pause candidate intervals, available duration intervals, and lip shape credibility markers.

[0047] It should be noted that the change in lip movement over time refers to the continuous deformation process of the lips in the time dimension, which is reflected in the displacement of the coordinates of the lip edge points, the fluctuation of the opening area, and the rhythm of the closing frequency. This change pattern directly maps the phoneme sequence and intonation rhythm of speech, and is a bridge connecting visual lip shape and speech production. The system does not rely on the classification and recognition of specific viseme (lip shape unit), but directly uses the statistical characteristics of its movement trend as the basis for constraint, avoiding semantic mismatch caused by classification algorithm errors and improving the system's generalization ability.

[0048] In traditional dubbing processes, lip movements serve only as a visual reference for post-dubbing alignment and cannot participate in pre-dubbing decisions. This results in the system lacking the ability to quantitatively judge key issues such as "whether the sentence duration matches," "whether the pauses are reasonable," and "whether the stress is appropriate," relying solely on manual trial and error or inefficient audio waveform stretching compensation. Step S204, by extracting semantically related lip movement features, for the first time realizes lip rhythm as a pre-constraint condition for text optimization, fundamentally changing the passive mode of text shaping → audio generation → forced lip-syncing, and shifting to an active generation logic of lip-guided → text optimization → accurate dubbing.

[0049] In some embodiments of this application, a mouth movement curve based on optical flow and regional deformation can be constructed. Specifically, the system performs pixel-level optical flow calculations on the extracted mouth ROI sequence to obtain the displacement vector field of the lip edge between adjacent frames. Combined with the integral change of the opening area over time, a smooth mouth movement curve (LMC) is generated. After median filtering and dynamic threshold denoising, this curve can stably identify strong peak positions (e.g., peak height > mean + preset increment) and pause candidate intervals (e.g., duration ≥ T_pause and activity amplitude < threshold). This method is suitable for high-definition, frontal shooting footage and has strong robustness to changes in illumination.

[0050] Furthermore, a lightweight lip-sync encoding based on deep feature embedding can be adopted. Specifically, to improve processing efficiency, the system uses a pre-trained convolutional neural network (such as MobileNetV3) to encode the features of the lip ROI, outputting a low-dimensional temporal embedding vector sequence. Then, a self-attention mechanism is used to model the global rhythmic pattern of lip movements, directly generating the probability distribution of strong peak positions and pause candidate intervals. This method does not require explicit calculation of optical flow or area, has low computational overhead, and is suitable for deployment in high-concurrency services on mobile devices or in the cloud. It can be used in the mass production of micro-dramas, and its output results can complement and verify traditional methods, improving the stability of feature extraction.

[0051] In some specific embodiments of this application, the lip-sync rhythm summary (i.e., lip-sync features) can be constructed as follows: The mouth ROI is extracted from the video, and the sequence of changes in lip opening and closing intensity over time is extracted; the original sequence is smoothed and denoised to generate a stable lip-sync activity curve; significant mouth-opening moments are identified through peak detection, forming a set of strong peaks, and their confidence levels are labeled; potential natural pause intervals are extracted based on low activity duration thresholds; the minimum and maximum allowed pronunciation duration ranges for the sentence are determined by combining the subtitle time window and shot boundaries; and a lip-sync confidence marker is calculated by considering factors such as occlusion, side profile, and tracking stability, to dynamically adjust the strength of strong peaks and pause constraints, converting low confidence levels to soft constraints or downgrading processing, and retaining audit traces. Specifically:

[0052] (1) Mouth movement curve: Extract the opening and closing intensity sequence m(t) that changes with time on the mouth ROI and smooth / denoise to obtain m_smooth(t).

[0053] (2) Strong peak set: The set of strong peaks K={t_j} is obtained by peak detection on m_smooth(t), and each strong peak is attached with confidence features such as peak height / sharpness.

[0054] (3) Candidate pause interval: Extract pause interval P={(s_k,e_k)} on m_smooth(t) according to the rule of "low activity duration ≥ T_pause".

[0055] (4) Available time range: The available time range [D_min, D_max] is given by combining the subtitle window and the shot boundary (optional, allowing a small amount of budget to be borrowed from adjacent gaps, but setting an upper limit to avoid the overall rhythm being stretched).

[0056] (5) Lip shape reliability: Conf_mouth∈[0,1] is generated by combining occlusion / side face / distant view / tracking stability; when Conf_mouth is below the threshold, K / P is used as a soft constraint or a degradation path is directly triggered and a trace is left.

[0057] It should be noted that the lip rhythm summary can be constructed based on one or more of the above dimensions, and no limitation is made here.

[0058] Step S206: Determine multiple candidate texts in the dubbing material that correspond to the subtitle text of the target object. The multiple candidate texts include subtitle texts that use the same language type but are expressed in multiple ways.

[0059] In step S206 above, candidate text refers to a collection of multiple language versions in the target language (such as Chinese, English, etc.) that are semantically equivalent to the original subtitles but have different expressions. These candidate texts are characterized by the coexistence of semantic consistency and expression diversity. They are not simple paraphrases, but rather a structured pool of semantically equivalent expressions, covering various styles such as colloquial versions, compressed versions, emphasized versions, polite versions, and expanded versions with filler phrases. Each version is accompanied by quantifiable attributes (i.e., speech attributes, such as expected duration, pause positions, tone intensity, terminology hit rate, etc.).

[0060] In related technologies, once the translation result is determined, it cannot be adjusted. The system can only passively adapt to the lip movements, resulting in a large number of rhythm misalignment and lip movement mismatch problems caused by rigid semantic expression. Step S206 constructs a structured candidate text pool, uses semantic equivalence as the screening boundary, and uses expression diversity as the optimization space, to achieve a paradigm shift from translation determining lip movements to lip movements guiding translation.

[0061] In some embodiments of this application, candidate generation based on rule templates and linguistic knowledge can be employed. Specifically, the system utilizes predefined semantic equivalence transformation rules to structurally rewrite the original subtitle text, such as compressing long sentences, converting written language to spoken language, strengthening tone, and adjusting politeness. Furthermore, each candidate text can automatically generate attribute tags, such as "number of words," "number of syllables," "suggested pauses," "tone intensity," and "whether it contains filler phrases," providing a structured basis for subsequent feasibility filtering.

[0062] Furthermore, semantic equivalence generation and attribute annotation based on Large Language Models (LLM) can be employed. Specifically, the system explicitly injects constraints into the prompts, such as: "Keep the core semantics of the original sentence unchanged, only adjust the expression; require output of three styles: colloquial, concise, and emphatic; annotate the estimated duration, pause position, and whether it contains modal particles"; and calls lightweight inference models (such as Qwen-Turbo, ChatGLM3-6B) to generate candidate texts. The model output results are automatically processed by the post-processing module to extract attributes, such as estimating duration through syllable statistics, predicting pause points through punctuation and semantic boundaries, and marking terminology consistency through dictionary matching.

[0063] In some embodiments of this application, any one or more of the following attributes can be recorded for each candidate text c (which can be obtained through rules, lightweight models, or TTS trial synthesis, and are not limited here):

[0064] (1) Estimated duration D_hat(c): The actual duration can be obtained by small-scale TTS trial synthesis, or by a priori estimation of the number of syllables / words + speech rate.

[0065] (2) Expected pause set Pi_hat(c): The set of pause positions obtained from punctuation / sentence break candidates / language model suggestions (which can be represented by relative position or time proportion).

[0066] (3) Strong Peak Friendly Feature F_peak(c): Characterizes "whether there is a greater likelihood of open / plosive pronunciations that are more consistent with strong mouth shapes near the strong peak", used to quantify the physiological fit between the pronunciation of candidate text c in the temporal region of strong mouth shape and the peak of video mouth shape motion. Its calculation is based on the mouth opening and closing dynamics of the phonemes contained in the candidate text in the neighborhood of the strong peak, and its determination method includes but is not limited to:

[0067] If the TTS / aligner can output phonemes / stresses: Calculate the proportion of "open vowels / plosives / stressed syllables" near strong peaks in the candidate text. Specifically, perform speech synthesis on the candidate text to generate a time-stamped phoneme sequence; based on the position of each strong peak in the lip-sync codebook, define a fixed-duration neighborhood window centered on each strong peak; within each neighborhood window, identify and count phonemes that conform to the strong lip-sync characteristics, including high-open vowels, plosive consonants, and semantically stressed syllables, collectively referred to as "strong lip-sync units"; calculate the proportion of strong lip-sync units in each neighborhood to the total number of phonemes; weight the proportions of each neighborhood according to the motion confidence of the corresponding strong peak to obtain the global strong peak friendly feature value of the candidate text.

[0068] If only text is available: the proportion of syllables with "larger opening" is approximated using the target language's alphabetic / phonetic rules or dictionary, and filler words are restricted from being near strong peaks. Specifically, based on the target language's orthography and phonological rules, a language-level mouth shape mapping dictionary is constructed, labeling each syllable as "high opening," "medium opening," or "low opening." The candidate text is segmented into syllables, identifying the opening level of each syllable, and semantically emphasized syllables are determined by combining sentence stress patterns. Based on the position of strong peaks in the mouth shape codebook, a neighborhood interval is defined within a preset time range before and after them. The proportion of "high-opening syllables" within this neighborhood is statistically analyzed as an estimate of mouth shape fit. Furthermore, any lightweight filler morphemes—i.e., non-core semantic units used for rhythm compensation, semantic weakening, or pragmatic mitigation—can be prohibited from appearing in the neighborhood of any strong peak. If such morphemes are detected within a preset time window before or after the strong peak position, the strong peak-friendly feature value of the candidate text should be forcibly reset to zero, and it should be eliminated.

[0069] (4) Fill phrase usage: the number of fill fragments N_fill(c), their position distribution and whether they are stacked consecutively, etc., are used for risk penalty and explanation.

[0070] Step S208: Determine the target text that matches the lip-sync features from multiple candidate texts, and use the target text to dub the material to be dubbed.

[0071] In step S208 above, the target text refers to the selected text with the highest comprehensive score in the candidate text set after at least evaluation by lip-sync rhythm constraints, so as to achieve the highest degree of natural alignment with lip-sync rhythm while ensuring semantic integrity and consistency of tone.

[0072] In some embodiments of this application, a target text matching the lip-sync feature can be determined from multiple candidate texts in the following manner: obtaining a first lip-sync feature from the lip-sync feature, wherein the first lip-sync feature is used to characterize the hard constraints when dubbing the material to be dubbed in physical space and time; matching the first lip-sync feature with the speech attributes of multiple candidate texts respectively to obtain a first candidate text, wherein the speech attributes are used to characterize the acoustic rhythm features of multiple candidate texts during the dubbing process; and determining the target text from the first candidate text.

[0073] It should be noted that the first lip-sync feature refers to the lip-sync rhythm summary extracted from the video frames of the material to be dubbed, which has rigid physical and spatiotemporal constraints. Essentially, it is a set of hard boundary conditions for lip movements in the temporal and spatial dimensions (i.e., a set of lip-sync rhythm summaries with physical measurability and rule enforceability used in the feasibility screening stage). This includes, but is not limited to, sub-attributes such as the available duration interval [D_min, D_max], the set of strong peak positions K, the pause candidate interval P, and the lip-sync credibility marker Conf_mouth. The speech attributes refer to the set of structured attributes representing the acoustic rhythmic features of each candidate text, predicted by a lightweight model or language rules before text-to-speech (TTS) synthesis. These attributes include the predicted pronunciation duration D_hat, pause positions Pi_hat, stress distribution, proportion of open syllables, and usage of filler phrases.

[0074] Hard constraints are a set of uncompromising rules that must be strictly met during the feasibility screening stage, defined based on the first lip-shape feature. They are decision-making criteria formed after the system logically maps the first lip-shape feature; they are not the lip-shape feature itself, but rather judgment rules extracted from it. In some embodiments of this application, hard constraints include, but are not limited to:

[0075] (1) Duration hard constraint: For example, the estimated duration D_hat of the candidate text must satisfy D_min≤D_hat≤D_max, otherwise FAIL_LEN will be triggered;

[0076] (2) Pause alignment hard constraint: For example, the suggested pause point of the candidate text must fall within any pause candidate interval, with a tolerance of ±150ms, otherwise FAIL_PAUSE will be triggered;

[0077] (3) Compliance hard constraints: For example, if the lip-sync confidence Conf_mouth < 0.3, the system will not enforce the lip-sync related hard constraints, but if Conf_mouth ≥ 0.3, the above duration and pause constraints must be strictly met, otherwise the system will be eliminated;

[0078] (4)Hard constraints on the position of filler words: For example, if the filler phrase in the candidate text (such as "um", "actually") appears at a strong peak position, it directly triggers FAIL_FILL. This is a composite hard constraint defined by the联动 of the strong peak of the mouth shape and the characteristics of the speech syllables.

[0079] Specifically, through a high-precision mouth ROI tracking algorithm, combined with subtitle window and shot boundary information, the legal pronunciation time range [D_min, D_max] of the audio of this sentence can be automatically calculated (such as allowing borrowing adjacent gaps not exceeding ±0.2 seconds); at the same time, using peak detection and low-activity detection algorithms, the strong peak set K and pause intervals P are extracted; and Conf_mouth is generated based on face pose, occlusion ratio, image clarity, etc.

[0080] Furthermore, for each candidate text, a lightweight duration prediction model (based on syllable count + speech rate prior) or small-scale TTS trial synthesis is called to obtain D_hat and Pi_hat; then a triple hard match is performed with the first mouth shape feature: ① If D_hat < D_min or D_hat > D_max → mark FAIL_LEN; ② If the candidate pause point Pi_hat does not fall into any pause candidate interval P (such as allowing a tolerance of ±0.15 seconds) → mark FAIL_PAUSE; ③ If the candidate text contains a filler phrase but the position conflicts with the strong peak (such as inserting "um" at the strong peak) → mark FAIL_FILL. Only the candidates that pass all hard constraints enter the first candidate text set.

[0081] Furthermore, for the first candidate text set, the comprehensive scores of "strong peak friendliness" and "rhythm correlation" can be calculated. Among them, the strong peak friendliness can be evaluated, for example, by counting whether the candidate text is an open vowel or a plosive at the strong peak position t_j (such as "a", "o", "b", "k"); the rhythm correlation can be measured, for example, by the dynamic time warping (DTW) alignment score between the energy envelope e(t) of the audio generated by TTS and the mouth shape activity curve m_smooth(t); semantic consistency (such as back-translation similarity ≥ 0.9) and character persona portrait matching degree (such as for a serious executive role, rejecting colloquial expressions) can also be combined for weighted ranking, and the one with the highest score is output as the target text.

[0082] The traditional dubbing process ignores the physical space-time boundary, resulting in generated results often crossing shots, exceeding the duration, and violating the shot rhythm, requiring manual repeated editing and rework. The above embodiments for the first time elevate the mouth shape from an observation object to an insurmountable engineering boundary, realizing the industrial production logic of "first prove feasibility, then talk about optimality".

[0083] It should be noted that the first lip-sync feature may be falsely detected due to degraded shooting quality (such as side profile, low light, or occlusion), leading to a shift in the strong peak position or omission of pause intervals, and thus misjudging the candidate text as "infeasible" or "mismatched". To solve this problem, in some embodiments of this application, a lip-sync confidence-driven elastic constraint mechanism can be constructed. When acquiring the first lip-sync feature, Conf_mouth is output simultaneously, and a multi-level response strategy is set, for example (the following threshold values ​​are only examples):

[0084] (1) When Conf_mouth ≥ 0.7: strong peaks and pauses are hard constraints, and strict filtering is applied;

[0085] (2) When 0.4≤Conf_mouth<0.7: strong peaks and pauses are converted into soft constraints and are only used as scoring weights (e.g., strong peak alignment adds 0.3 points, misalignment deducts 0.2 points), and are not used for elimination;

[0086] (3) When Conf_mouth < 0.4: the downgrade path is triggered. The system automatically hides the lip-sync constraint and selects the best option based only on the three principles of “duration matching + semantic consistency + role consistency”. The system records in the decision path audit snapshot: “Because the lip-sync is unreliable (Conf_mouth = 0.3), the no-lip-sync mode has been enabled. The final selection criteria are: semantic integrity and role tone consistency.”

[0087] To address the three core issues of duration compliance, pause alignment, and content security in dubbing generation, the following method can be used to match the first lip-sync feature with the speech attributes of multiple candidate texts to obtain the first candidate text: The duration interval in the first lip-sync feature is compared with the expected duration in the speech attributes to obtain the first comparison result, where the duration interval represents the time occupied by the subtitle text in the dubbing material; the pause interval in the first lip-sync feature is compared with the expected pause point in the speech attributes to obtain the second comparison result, where the pause interval represents the time period during which the lip movement state of the target object meets a preset state; text detection is performed on multiple candidate texts to obtain detection results, where text detection is used to check whether the text content of multiple candidate texts meets preset rules; the first candidate text is determined based on the first comparison result, the second comparison result, and the detection result.

[0088] It should be noted that the duration interval refers to the physical time range within which the target text can be safely used in the video, determined by the subtitle display window and the boundaries of adjacent shots, and is denoted as [D_min, D_max]. The expected duration refers to the time length required for the candidate text to be actually pronounced in the target language after being synthesized by a lightweight model or a small-scale TTS trial, and is denoted as D_hat.

[0089] The pause interval refers to the set of natural pause time periods identified from the lip movement curve where the amplitude of lip movement is below a preset threshold and lasts for a sufficient duration (≥T_pause), denoted as {(s_k, e_k)}. It represents the physiological silence when a person breathes, thinks, or changes tone during natural speech and is the breathing point with the most semantic load in the lip rhythm. The expected pause point refers to the set of semantic pause moments that should appear in the speech stream after the candidate text is analyzed by a language model or parsed by punctuation structure, denoted as Pi_hat. This attribute is derived from the internal structure of the text (such as commas, periods, and interjections).

[0090] In some embodiments of this application, the expected pause point Pi_hat can be extracted from the punctuation and language model output of the candidate text, such as 0.3 seconds for a comma and 0.6 seconds for a period. At the same time, each (s_k, e_k) in the candidate pause interval P is used as an allowed pause window. If any pause point of the candidate does not fall within any pause interval (allowing a tolerance of ±0.15 seconds), then FAIL_PAUSE is marked.

[0091] Text detection refers to a series of content and style compliance checks performed on candidate texts, including but not limited to: prohibited word matching, terminology consistency checks, phrase abuse detection, culturally sensitive word filtering, and character personality profile conflict identification.

[0092] In some embodiments of this application, the target text can be determined from the first candidate text in the following way: determining the matching degree corresponding to each first candidate text, wherein the matching degree is used to quantify the degree of consistency between the first candidate text and the second lip shape feature in the lip shape feature in the visual and auditory dimensions, and the second lip shape feature is used to characterize the soft constraint condition when dubbing the dubbing material in terms of content adaptation; and determining the second candidate text with the highest matching degree among the first candidate texts as the target text.

[0093] It should be noted that the matching degree refers to the quantitative score of the degree of fit between each first candidate text and lip-shape features in both visual and auditory dimensions. It is a comprehensive evaluation index of lip-shape rhythm fit. The visual dimension reflects whether the pronunciation sequence of the candidate text is synchronized with the morphological changes of lip movements, while the auditory dimension measures whether the distribution of speech energy and the lip-shape activity curve co-evolve in the temporal domain. The matching degree is not a single index, but a composite value weighted by sub-scores such as strong peak alignment, rhythmic relevance, and semantic rhythm consistency. It is used to identify the optimal expression among the first candidate texts that have passed the hard constraints. For example, the matching degree can be a weighted composite value that integrates multiple cognitive and perceptual levels such as lip-shape rhythm fit, semantic conservation, and consistency between character language and personality. Its role is to provide a ranking basis for all first candidate texts that have passed the hard constraints, thereby determining the final output target text.

[0094] The second lip-sync feature refers to the flexible rhythmic constraint signal used to guide the final selection after the hard constraint screening is completed. This includes the smooth shape of the lip-sync curve, the confidence strength of strong peaks, the distribution density of pause intervals, and the dynamic trend of lip movements. The second lip-sync feature is a set of fine-grained rhythmic features that measure the visual and auditory consistency of candidate texts, providing quantifiable scoring criteria for the selection stage. It possesses continuous and relative characteristics in lip-sync information. Unlike the rigid boundary attributes of the first lip-sync feature, the second lip-sync feature is a plasticity perception criterion. Its role is to guide the system to judge "which candidate sounds more like a natural, human mouth." For example, a strong peak may match in time, but if its peak sharpness is low and accompanied by slight jitter, it indicates that the lip movement at that moment is unreliable, and the system should reduce its weight; while a continuous, full, and noise-free strong peak should be given a higher matching weight.

[0095] Soft constraints refer to a weighted set of preference rules constructed based on second lip-sync features to guide the candidate text selection process. These rules are used to determine which rule "best matches the aesthetic and rhythmic perception expectations of film and television dubbing" among all the first candidate texts that pass the hard screening. In some embodiments of this application, soft constraints include, but are not limited to:

[0096] (1) Strong Peak Friendly Scoring Rules: For example, if the phonemes located near the strong peak in the candidate text are open vowels or plosives, they will be given extra points; if they are weakly pronounced function words, they will be given extra points.

[0097] (2) Rhythm fit weighting function: For example, using the mouth movement curve as a reference, calculate the dynamic time regularity matching degree of the candidate audio energy envelope with it. The higher the score, the more synchronized the speech force and mouth opening and closing are.

[0098] (3) Semantic and style consistency preference: For example, if a candidate’s speaking style is “calm and reserved”, the system will prioritize candidates with slower speaking speed and more natural pauses, even if their duration is slightly shorter than the upper limit.

[0099] (4) Global consistency constraint: The verbal tics and accents of the same character should remain consistent throughout the play. If a candidate breaks the established rhythm pattern (such as the protagonist who is usually short suddenly having a long trailing tone), even if the local matching degree is high, it will be penalized.

[0100] It should be noted that the weight of soft constraints can be dynamically adjusted according to program type, character setting, or operational feedback to achieve personalized optimization. For example, in variety shows, when characters have intense emotions, the system can increase the weight of "strong peak matching degree"; while in documentaries, more emphasis is placed on semantic integrity and speech rate stability, and the priority of rhythm fit is reduced accordingly.

[0101] Specifically, the matching degree can be calculated in the following way:

[0102] (1) Visual dimension scoring: Calculate the alignment accuracy between the predicted pronunciation timing of the candidate text and the set K of strong peaks in the second mouth shape feature. Specifically, use the strong peak friendly index: Statistically determine whether the syllable of the candidate at the strong peak moment tj is an open vowel (such as a, o, e) or a plosive (such as b, k, t). If there is a match, add points; if it is a weak stress word (such as "de", "le", "ne"), subtract points. For example, if candidate A pronounces "kai" (open vowel) at the strong peak and candidate B pronounces "le" (weakened sound), then A scores 15% higher.

[0103] (2) Auditory dimension scoring: Perform lightweight TTS synthesis on the candidate text, extract its audio energy envelope e(t), and use dynamic time warping (DTW) to align it to the second mouth shape activity curve m_smooth(t). Calculate the time alignment error and correlation coefficient between the two. If the synchronization of the energy peaks and the opening peak of the mouth shape is high and the coincidence of the valleys and the pause intervals is high, the score will increase.

[0104] (3) Comprehensive matching degree: Weightedly fuse the visual and auditory scores. For example, S_match = 0.6×S_visual + 0.4×S_audio. Further, a semantic rhythm consistency correction term can also be introduced: If the semantic sentence segmentation of the candidate text does not match the intensity of the character's emotion (such as using a jumping fast reading in a sad scene), then an additional weight reduction is applied. Finally, the matching degree scores of each candidate are output to form a sortable list.

[0105] In some embodiments of the present application, the matching degree corresponding to each first candidate text can also be determined in the following way: Determine a first index between the first candidate text and the second mouth shape feature, where the first index is used to quantitatively represent the matching degree between the speech energy corresponding to the first candidate text and the mouth shape rhythm of the target object; Determine a second index between the first candidate text and the second mouth shape feature, where the second index is used to quantitatively represent the similarity between the semantics of the first candidate text and the subtitle text; Determine a third index between the first candidate text and the second mouth shape feature, where the third index is used to quantitatively represent the degree of fit between the expression mode of the first candidate text and the language personality portrait of the target object; Determine the matching degree based on at least one of the first index, the second index, and the third index.

[0106] It should be noted that the first index refers to a numerical evaluation parameter used to quantitatively represent the degree of co-evolution in the time domain between the speech energy corresponding to the first candidate text and the mouth shape rhythm characterized by the second mouth shape feature, and can be achieved by comparing the time alignment error, the correlation of energy peaks, or the dynamic time warping (DTW) score between the audio energy envelope (e(t)) of the candidate text after TTS synthesis and the mouth shape activity curve (m_smooth(t)) extracted from the mouth ROI.

[0107] Specifically, lightweight TTS synthesis can be performed on candidate texts to generate their audio waveforms, and short-time energy envelopes e(t) in 10ms units can be extracted. Simultaneously, the mouth movement curve m_smooth(t) (denoised and smoothed) from the second mouth shape feature is used as a visual rhythm template. Dynamic Time Warping (DTW) algorithm is used to align e(t) and m_smooth(t), calculating the minimum cumulative distance between them on the time axis to obtain a basic alignment score. Further, a strong peak energy alignment rate is introduced: at the location of a strong mouth shape peak, whether the corresponding speech energy is higher than the local mean (i.e., whether the sound is strong when the mouth is open) is statistically analyzed. If a match is found, a score is added; if the speech energy is at a trough at the strong peak (e.g., when pronouncing "de" or "le"), a score is deducted. Finally, the first indicator = α·DTW score + β·strong peak energy matching rate, with weights α and β dynamically adjusted according to language characteristics (e.g., Chinese stress distribution, English intonation patterns).

[0108] The second metric refers to a quantitative parameter used to measure the semantic consistency between the first candidate text and the original subtitle text at the semantic content level. This metric can be achieved through methods such as back-translation similarity (calculating BLEU or BERTScore with the original sentence after translating the candidate text back into the source language), key fact retention rate (whether names, numbers, and proper nouns remain unchanged), and sentiment polarity bias (e.g., if the original sentence is a negative evaluation, the candidate text must not be changed to a positive one).

[0109] Specifically, a pre-trained back-translation model (such as a bilingual translator based on BART or T5) is invoked to translate the candidate text back into the source language, and its semantic similarity with the original subtitle text is calculated (such as using cosine similarity using Sentence-BERT). A terminology validator scans the candidate text for changes in key entities (such as "Li Hua" becoming "Xiao Li"), numerical tampering ("three hours" becoming "two hours"), or negative semantic reversal ("cannot be used" becoming "can be used"). A sentiment analysis model (such as RoBERTa-sentiment classifier) ​​is used to determine whether the sentiment polarity of the candidate text is consistent with that of the original sentence (positive / neutral / negative). If the deviation exceeds a threshold (such as ±0.3), it is considered semantic distortion. The three scores are weighted and synthesized into a second index.

[0110] The third indicator refers to the evaluation parameters used to assess whether the expression style, sentence structure, and tone intensity of the first candidate text are consistent with the target audience's linguistic personality profile. The linguistic personality profile is a set of pre-defined verbal tic traits (such as frequent use of "um" or "you see"), politeness level (such as "you" or "you"), emotional expressiveness (such as repressed / extroverted), and sentence structure preference (such as short / long sentences) for each role. This indicator can be calculated using a role profile matching model (such as a classifier or semantic embedding similarity) to determine the degree of fit between the candidate text and the profile.

[0111] Specifically, the system reads the target character's profile tags, such as "sentence structure preference: short sentences," "tone characteristics: direct / rough," and "prohibited words: 'sorry,' 'I think,'" etc.; it performs style classification on candidate texts: using a lightweight semantic classifier (such as FastText or a finely tuned BERT) to determine whether it belongs to style categories such as "formal / spoken / angry / restrained," and calculates the category matching degree with the profile tags; at the same time, it detects whether it hits style taboos in the profile (such as the profile prohibiting the use of interjections, and deducting points if the candidate contains "ah" or "ne") or style enhancements (such as the profile requiring the frequent use of "zan" or "zanjia," and adding points if the candidate contains them); finally, the third indicator = 1 - |style classification deviation| + style enhancement score - style taboo penalty, to achieve fine-grained character consistency control.

[0112] The above embodiments solve the problems of mechanical, unnatural, and uncontrollable dubbing caused by relying on fixed translation → TTS → post-processing Lip-Sync in traditional multilingual dubbing. Through a three-dimensional index-driven intelligent optimization mechanism, the system achieves a triple unity of rhythm fit, semantic fidelity, and personality continuity at the generation layer, upgrading the dubbing system from a passive compensation tool to an active creative engine, significantly improving the naturalness of film and television dubbing.

[0113] In some embodiments of this application, the matching degree can also be determined in the following ways: determining the target phonemes in the phoneme sequence of the first candidate text, wherein the target phonemes include phonemes whose lip opening and closing degree meets the preset opening and closing conditions during pronunciation; taking multiple strong peaks in the second lip shape feature as time anchors respectively, judging whether the target phonemes are aligned with the strong peaks within a preset time window, and obtaining the judgment result, wherein the strong peaks include the time point when the lip opening and closing degree of the target object reaches the local maximum value in the dubbing material; determining the strong peak alignment degree corresponding to the first candidate text based on the judgment result, wherein the strong peak alignment degree is used to quantify the degree of consistency between the speech content of the first candidate text and the lip movement state of the target object in the visual dimension; and determining the matching degree based on the strong peak alignment degree.

[0114] Specifically, within the feasible candidate set C_feasible, the system first performs lightweight TTS synthesis on each candidate text to generate its speech audio and extracts the time-aligned energy envelope e(t). Simultaneously, it calls the constructed second lip-sync feature, namely the smoothed lip-sync activity curve m_smooth(t), and the set of strong peaks K. The system uses the strong peak points as key anchor points for visual rhythm and prioritizes calculating the strong peak alignment score S_peak(c): It statistically analyzes whether the target phonemes (such as open vowels and plosives) corresponding to the candidate text effectively cover the peak within a ±150ms window around each strong peak position, and compares the phoneme center with the strong peak. The time offset at the center is penalized by distance; the smaller the offset and the more complete the coverage, the higher the score. The system calculates the correlation score S_corr(c) between the energy envelope and the lip-sync curve. It aligns e(t) and m_smooth(t) through dynamic time warping (DTW) to measure the degree of waveform coordination between the two in the overall time domain and suppress global mismatch caused by local syllable misalignment. The system uses strong peak alignment as the main factor and overall energy coordination as a secondary factor, and performs weighted fusion with fixed weights (e.g., w1=0.4, w2=0.6) to generate a comprehensive lip-sync score S_lip(c). It also superimposes semantic consistency and role tone consistency scores to form the final ranking.

[0115] The above embodiments ensure that the selection process prioritizes the most significant visual-sound moment, so that the dubbing is naturally aligned at the moment when the audience is most sensitive to the mouth movement. Thus, without relying on a complex generative model, high-fidelity cross-language lip-sound synchronization is achieved in an interpretable and reproducible manner.

[0116] After identifying the target text that matches the lip-sync features from multiple candidate texts, the following steps can be performed: First, determine the candidate graph structure based on the multiple candidate texts corresponding to the multiple subtitle texts of the target object. Second, determine the linguistic personality profile of the target object and, based on the linguistic personality profile, determine the consistency constraints common to the multiple subtitle texts. The linguistic personality profile is used to characterize the target object's linguistic style in historical materials. Third, use the target matching degree between each candidate text and the lip-sync features as the node weight in the candidate graph structure and determine the edge weights of directed edges in the candidate graph structure based on the consistency constraints. Fourth, determine the target candidate path from the candidate graph structure based on the node weights and edge weights, and dub the material to be dubbed according to the candidate text corresponding to the target candidate path.

[0117] It should be noted that the candidate graph structure refers to a directed graph model constructed with the temporal relationship of the subtitle sequence as its skeleton. The nodes in the candidate graph structure represent candidate texts, and the directed edges in the candidate graph structure connect the candidate texts corresponding to two adjacent subtitle texts. That is, each node represents a candidate dubbing expression of a certain subtitle text in the target language. The nodes are connected to the candidate texts corresponding to adjacent subtitle sentences through directed edges, forming a candidate path network from the first sentence to the last sentence. This structure transforms the originally isolated sentence-by-sentence optimization problem into a global path optimization problem, enabling the system to find the optimal solution from the semantic coherence and stylistic consistency of the entire dialogue or monologue.

[0118] Language personality profiles refer to a set of structured language behavior features extracted from historical dubbing materials, original performances, or script settings of a target object (such as a character in a play). These features include sentence preference (such as short / long sentences), tone tendency (such as direct / euphemistic), verbal tic frequency (such as whether "um" or "you see" is used frequently), emotional expression (such as repressed / extroverted), consistency of address (such as always using "you" or "you"), and taboo expressions (such as prohibiting the use of profanity). These features are used to guide the construction of edge weights in the candidate graph structure to ensure that the character maintains personality consistency in dubbing across languages ​​and sentence segments.

[0119] Node weight refers to the score assigned to each candidate text based on its matching degree with the lip-sync features of the current subtitle sentence (such as S_lip(c) mentioned above), reflecting the local optimality of the candidate in visual alignment within this sentence. Edge weight refers to the consistency penalty cost carried by the directed edge connecting two adjacent candidate text sentences, which is jointly determined by the linguistic personality profile and the semantic coherence of the context, such as abrupt changes in address, jumps in tone and style, and inconsistencies in terminology.

[0120] Specifically, following the temporal order of video subtitles, a multi-layered node structure can be constructed, using the candidate text pool for each subtitle (generated from the preceding feasibility filtering stage) as a node set: the first layer corresponds to all candidates for the first subtitle, the second layer corresponds to all candidates for the second subtitle, and so on. Subsequently, fully connected directed edges are established between adjacent layers of nodes, meaning that any candidate for the i-th subtitle can be connected to any candidate for the (i+1)-th subtitle, forming a dense directed graph.

[0121] Furthermore, the system calls upon a pre-built character language personality profile library to obtain the target character's tags, such as: title preferences, emotional style, prohibited words, and habitual sentence patterns. Based on this, the system automatically generates cross-sentence consistency rules: for example, ① if the i-th sentence uses "you," then the i+1-th sentence must not use "you"; ② if the i-th sentence is a low-speed, low-energy expression, then if the i+1-th sentence suddenly changes to a high-speed, exclamatory sentence, a consistency penalty will be triggered; ③ if the i-th sentence contains the proper noun "xx technology," the i+1-th sentence must not be replaced with "xx company."

[0122] Furthermore, the node weight directly adopts the matching degree (i.e., the target matching degree) calculated in the previous step as the local performance value of the node, while the edge weight is calculated by the consistency constraint function: if candidate A (the i-th sentence) and candidate B (the (i+1)-th sentence) have conflicts in terms of title, tone intensity, and terminology, then the edge weight = the basic penalty value (e.g., 1.0) × the conflict severity coefficient (e.g., title mutation = 1.5, tone fault = 2.0); if there is no conflict, then the edge weight is 0.

[0123] Furthermore, with the goal of minimizing the total cost, an improved Dijkstra's algorithm or Viterbi's algorithm is used to search for the optimal path from the first sentence to the last sentence in the candidate graph. When expanding nodes at each level, the algorithm comprehensively evaluates the path cost to that node (cost of the predecessor node + edge cost + current node cost), retains the optimal predecessor, and finally outputs a complete candidate sequence.

[0124] The above embodiments solve the problem of fragmented character personalities caused by sentence-by-sentence optimization in the industrial production of multilingual dubbing. Through a three-in-one mechanism of candidate graph structure + language personality profile + global cost optimization, cross-sentence continuous modeling of language personality is achieved in dubbing, which significantly improves the narrative coherence and character credibility of film and television dubbing.

[0125] In some embodiments of this application, during the candidate generation and screening stage, the system performs full-element annotation on each candidate text to form a snapshot of candidate attributes, including but not limited to: the original candidate text, the generation source (such as rule templates, LLM generation, artificial dictionary), the estimated duration D_hat(c), the suggested pause set Pi_hat(c), the strong peak friendly feature F_peak(c), the number and position of fill phrases used N_fill(c), term hit status, style tags, etc. All data are stored in the candidate pool table in the form of structured fields. When performing hard constraint filtering (duration, pauses, compliance, fill restrictions, etc.) on each candidate text, the system checks each item and outputs a clear failure reason code (such as FAIL_LEN, FAIL_PAUSE, FAIL_COMPLIANCE, FAIL_FILL), while recording the relaxation amount (such as timeout 180ms, pause misalignment 220ms). For candidates that are rejected, the system records their rejection reason, the triggered constraint rule ID, and the specific conflict point. For candidates that pass, they are marked with a "constraint-satisfied list" to form a feasibility decision log.

[0126] Furthermore, when sorting within the feasible candidate set, the system not only outputs the final selected candidate, but also simultaneously generates a score decomposition report for the best candidate: clearly recording the scores and weight configurations of each dimension, such as S_lip (lip-syncing score), S_semantic (semantic consistency score), S_personality (role personality fit), and S_global (global consistency score).

[0127] When an unreliable lip movement pattern is encountered (e.g., Conf_mouth < 0.5) or the feasible candidate set is empty, the system automatically triggers a degradation process and records the operation trajectory at each level:

[0128] Level 1: Try to relax soft constraints (such as allowing 10% of the time limit to be exceeded, and enabling previously prohibited interjections).

[0129] Level 2: Expand the candidate pool (call for stronger LLM generation, and introduce translations of historically similar sentences);

[0130] Level 3: Revert to the original translation or mark as "requires manual intervention".

[0131] Each step of the operation records the type of degradation action, execution time, triggering conditions, and parameter change records, forming a clear "degradation tree" to ensure that system behavior is traceable, auditable, and accountable.

[0132] Furthermore, while outputting the final dubbing text and audio, the system automatically generates a standard audit snapshot, which includes: sentence ID, lip-sync credibility Conf_mouth, available duration range [D_min, D_max], summary of strong peaks and pauses, all candidate lists and their attributes, hard constraint screening conclusion, optimization scores for each dimension, final selected candidate ID, whether downgrade was triggered, downgrade path details, model version number used (e.g., TTS_v3.1, LLM_prompt_v2), parameter configuration version (e.g., weight_lip=0.6, weight_sem=0.3), etc. This snapshot is stored in the audit database in JSON or CSV format and supports multi-dimensional retrieval by series, character, language, time, and other conditions.

[0133] In some embodiments of this application, the following steps may also be performed: determining audit link data corresponding to the subtitle text, wherein the audit link data is used to record the decision basis for determining the target text from multiple candidate texts.

[0134] It should be noted that audit link data refers to the complete chain of decision-making evidence for each subtitle text during the intelligent dubbing generation process, recording the selection, sorting, and final selection of the target text from the candidate pool. For example, this data structure includes, but is not limited to: the source of the candidate text (such as rule templates, LLM generation, or manual annotation), the reasons for the elimination of each candidate (such as timeout, misaligned pauses, compliance conflicts), the reasons for the final selection of the candidate (such as the highest strong peak alignment, the best semantic consistency score), whether filler phrases are introduced and their usage, the version of key parameters (such as the TTS model version, lip-sync credibility threshold), and the degradation path triggered by the system (such as "relaxing soft constraints → reverting to the original translation").

[0135] Audit data is the core carrier for achieving interpretability and auditability. Its functions are as follows: First, it provides transparent decision-making basis for the production process, enabling editors, legal personnel, and internal auditors to trace "why this text was chosen instead of another text"; second, it supports gray release and version rollback. When dubbing results cause public opinion risks or abnormal viewing experience, the audit snapshot can quickly locate whether it is caused by model parameter deviation, missing candidate pool, or misjudgment of lip-sync credibility; third, it serves as input for the operational feedback loop. For example, when editors frequently correct certain line alignment methods, the system can automatically optimize the sorting weight based on "rejected candidates" and "manually reselected paths" in the audit data.

[0136] Specifically, the audit link data includes at least one of the following: candidate source identifier, used to record the generation mechanism or data source of each candidate text; hard constraint violation code, used to record the reason for not being selected for candidate texts other than the target text; selection criteria, used to record the reason for selection for the target text; parameter version data, used to record the runtime environment context on which the target text is relied upon; and rollback point, used to identify the previous state of the decision path corresponding to the target text that can be rolled back.

[0137] (1) Candidate source identifier refers to the metadata field used to record the generation mechanism or data source of each candidate text. For example, whether the candidate is generated from rule template generation, Large Language Model (LLM) rewriting, business dictionary mapping, or generated by feedback from historical manual revision samples. The candidate source identifier can realize the traceability and auditability of the candidate pool, so that the subsequent selection decision and the final output have source transparency characteristics. For example, when a candidate is selected because it "uses the filler phrase 'actually'", the auditing system can trace back whether the candidate was generated by LLM under the "colloquialization enhancement" task through the source identifier, thereby judging its rationality and preventing the system from unintentionally introducing excessive colloquialization or style drift expressions.

[0138] (2) Hard constraint violation coding refers to the standardized elimination reason code marked by the system for candidate texts that are not selected, such as FAIL_LEN (duration exceeded), FAIL_PAUSE (pause misalignment), FAIL_COMPLIANCE (compliance conflict), FAIL_FILL (filling exceeded), etc. This coding is used to transform subjective trial and error into structured failure diagnosis. For example, when the number of words in the candidate text is too large, resulting in an expected duration of 5.2 seconds, while the available duration range is [3.0, 4.5] seconds, the system automatically marks it as FAIL_LEN instead of just returning "inappropriate". This coding allows operators to count high-frequency failure patterns, optimize the candidate pool generation strategy, or adjust the duration tolerance threshold of the lip-sync codebook.

[0139] (3) Selection criteria refer to the composition and decision-making logic of the ranking score of the final selected target text, including but not limited to quantitative or logical scoring factors such as lip rhythm fit (e.g., strong peak alignment score), semantic consistency weight, character tone matching degree, and global consistency cost. Selection criteria are the support for achieving interpretable selection. Its role is to transform "which one is better" from a black box evaluation into a reviewable multi-dimensional decision-making path. For example, the selection criteria for the final selected candidate can be recorded as: "S_peak=0.92 (high proportion of open syllables at strong peaks), S_semantic=0.95 (back-translation similarity), S_role=0.88 (fits the character's 'tsundere' verbal tic)," so that the editor can verify why the system abandoned another shorter candidate - even though its length is compliant, it was judged by the system as rhythmic inconsistency because the light-tone word "de" was used at the strong peak.

[0140] (4) Parameter version data refers to the snapshot of the runtime environment context on which the target text is generated, including the model version used (e.g., TTS_v3.1, LLM_prompt_v2), the lip-sync codebook extraction algorithm version (e.g., MouthNet_v1.4), the filler codebook rule set version (FillerRules_2024Q2), and soft constraint weight coefficients (w1=0.6, w2=0.3), etc. The purpose of this data is to ensure that the decision-making process for each dubbing output is reproducible. For example, if users report that the dubbing tone is stiff after a certain episode is released, the operations and maintenance team can use the parameter version data to reconstruct the decision-making environment at that time, determine whether the prosody shift was caused by the TTS model upgrade, and thus decide whether to roll back to the previous version (e.g., v3.0).

[0141] (5) Rollback point refers to the previous state node that can be safely rolled back to in the decision-making path. It corresponds to a stable snapshot of a certain step or parameter version of the candidate selection, such as "original translation version", "candidate set before relaxing the filling constraints", and "selection result without enabling the role profile". Rollback points are used to build production fault tolerance and gray release mechanism. For example, when the system selects a candidate that should be more conversational but violates the "aloof setting" of the role after enabling the role profile (language personality profile), the system can automatically mark the rollback point of the path as "selection result without enabling the profile in the previous version" and trigger the manual review process to ensure that the modification is reversible and the risk is controllable.

[0142] In cross-language intelligent dubbing production, the above embodiments transform the fuzzy experience behavior of post-event compensation for lip-sync rhythm consistency into an industrialized production process driven by pre-constraints, with interpretable optimization and auditable rollback.

[0143] In some embodiments of this application, an adaptive update mechanism of must-align (must be aligned) / cannot-align (cannot be aligned) based on editor feedback can also be adopted. The operation of repeatedly "dragging the lines near a certain lip movement peak" by the editor is recorded as must-align samples, and the operation of repeatedly "moving the lines away from a certain segment of the screen / disabling a certain alignment point" by the editor is recorded as cannot-align samples. The constraint weights and thresholds are updated by program / shot type, so that the system gradually approaches the real aesthetic and production habits, and supports version rollback.

[0144] Through steps S202 to S208 above, the method of selecting candidate texts driven by lip shape features is adopted. By extracting the dynamic motion features of the lip region of the target object in the dubbing material, and generating a multi-version candidate text pool based on the principle of semantic equivalence, the target text that matches the lip shape features in terms of temporal rhythm is then selected. This achieves the goal of selecting the expression text that best fits the lip shape rhythm without changing the semantics, thereby realizing the technical effect of automatically adapting dubbing text to lip shape movement. This solves the technical problem in language intelligent dubbing where related technologies use post-processing methods such as speed stretching to fit the lip shape after fixing the translation, resulting in dubbing distortion.

[0145] Figure 3 This is a schematic diagram of the overall system architecture of a dubbing method according to an embodiment of this application, such as... Figure 3 As shown, in some embodiments of this application, the following steps are included:

[0146] S302: Input Data (Video / Subtitles / Target Language). Specifically, the system receives multi-source heterogeneous input data, including: the original video stream (including audio and video tracks), the source language subtitle timeline (SRT / ASS format), the target dubbing language identifier (such as English, Spanish), and optional shot breakdown information and character metadata. After standardized preprocessing, the input data undergoes temporal alignment and spatial positioning: video frames are sampled at the frame rate, and the mouth region (ROI) is extracted by a face detection and tracking algorithm; the subtitle time window and shot boundaries are parsed synchronously to form a three-dimensional spatiotemporal index of "sentence-shot-frame".

[0147] S304: Semantic Codebook Construction (Candidate Pool). Specifically, based on the source language subtitle text, the system calls a multi-source generation engine to construct a target semantically equivalent candidate pool. This process combines rule templates (such as compression, colloquialization, and polite rewriting), domain dictionaries (terminology database, prohibited word database), and a lightweight language generation model (LLM) to generate multiple semantically equivalent but structurally different candidate texts. Each candidate text is attached with interpretable attribute tags, including: estimated number of syllables, suggested pause positions, style tags (formal / colloquial / emphasis), terminology hit rate, whether filler phrases are introduced, and a rough estimate of the expected duration (D_hat), etc. All candidates are stored in a semantic codebook database in a structured index of "sentence ID – target language – version number".

[0148] S306: Lip Shape Codebook Construction (Rhythm Summary). Specifically, the system extracts the ROI sequence of the mouth region within the time window corresponding to each subtitle sentence in the input video, and generates a continuous lip shape activity curve m_smooth(t) based on visual feature extraction algorithms (such as Dlib, MediaPipe). Through peak detection and low-activity interval identification, a structured lip shape rhythm summary is extracted, including: a set of strong peaks K (significant mouth opening moments), a pause candidate interval P (mouth stillness periods), an available duration interval [D_min, D_max] (determined by subtitle window and shot boundary constraints), and a lip shape confidence marker Conf_mouth (a [0,1] confidence value based on comprehensive evaluation of occlusion, side profile, distant view, and tracking stability). This summary is stored in the lip shape codebook with "sentence ID – shot ID" as the index, forming a visual constraint object for dubbing alignment.

[0149] It should be noted that in some embodiments of this application, a codebook for filling (i.e., filling phrases, filling function words, filling morphemes) can also be constructed. Specifically, the codebook for filling is a codebook that pre-sets filling fragments such as "modal words / connective words / repetition and emphasis" according to the language habits of the target language, limits the number of times and positions of use, and is used to supplement rhythm and pauses without changing key facts or introducing compliance risks. For example, a small number of "um / actually / look" are used in Chinese, and a small number of "well / actually" are used in English, while avoiding continuous stacking that would cause machine generation traces.

[0150] S308: Feasibility Filtering (Hard Constraints). Specifically, the system takes the structured constraints provided by the lip-reading codebook as input and performs hard constraint filtering on each candidate text in the semantic codebook to select a candidate set C_feasible that meets physiological and engineering constraints. Filtering conditions include: duration constraints (D_min≤D_hat(c)≤D_max), pause alignment constraints (candidate pause points must fall within the lip-reading pause interval, tolerance δ≤200ms), compliance constraints (banned / sensitive words are not hit, term replacement is successful), and fill phrase usage constraints (e.g., N_fill≤2, continuous stacking is prohibited, and placement in strong peak positions is prohibited). For each candidate that does not meet the conditions, the system outputs a standardized failure reason code (e.g., FAIL_LEN, FAIL_PAUSE, FAIL_COMPLIANCE) and a relaxation amount (e.g., "timeout 140ms"). If the feasible set is empty, a degradation link is triggered (see S316), and the current filtering status is recorded to ensure that the decision-making process is auditable.

[0151] S310: Candidate Selection (lip-sync + semantic consistency + character tone). Specifically, within the feasible candidate set C_feasible, the system performs multi-objective ranking and selection, comprehensively evaluating the matching degree across three dimensions:

[0152] (1) Lip fit (S_lip): By weighted fusion of strong peak alignment score (S_peak) and energy envelope correlation (S_corr), open syllables are prioritized to align with strong peak points to avoid light syllables occupying the visual climax position;

[0153] (2) Semantic consistency (S_semantic): Back-translation similarity, terminology consistency verification and key fact comparison are used to ensure that proper nouns and core semantics have not been tampered with;

[0154] (3) Character tone consistency (S_personality): Based on the character's language personality profile, the intensity of tone, sentence preference, politeness level, and verbal tic are scored, and priority is given to matching the character's inherent expression pattern.

[0155] The final ranking score S_final = w1·S_lip + w2·S_semantic + w3·S_personality, with the weights dynamically adjusted according to the type of series. The system can also output a list of selected rankings and a breakdown of scores for each dimension, providing interpretable evidence for auditing.

[0156] S312: TTS Synthesis and Output. Specifically, the system uses a high-fidelity speech synthesis engine to generate target language dubbing audio for the selected final candidate text. It simultaneously outputs a time-aligned phoneme sequence and energy envelope, ensuring precise time alignment between the audio and the original video to guarantee a perfect match between the speech start and end points and the subtitle window, avoiding interruptions or trailing. The output includes: the final dubbing audio file, the synchronized subtitle file (including corrected text), and structured metadata corresponding to the S310 selection results.

[0157] S314: Decision Path Audit (Reason / Version / Rollback). Specifically, the system automatically generates a standardized audit snapshot, recording the complete decision path: sentence ID, lip-reading credibility, available duration range, peak / pause summary, candidate list and its attributes, hard constraint screening conclusion and failure reason code, optimization score for each dimension, final selected candidate ID, whether degradation was triggered, degradation path details, and the model and parameter version used. This snapshot is persistently stored in the audit database in the form of a structured log, supporting retrieval by time, role, language, issue type, and other dimensions.

[0158] S316: Gray-scale and Rollback. Specifically, the system supports gray-scale release and version rollback mechanisms in the production environment. When an audit finds abnormalities in the dubbing (such as semantic distortion, lip-sync inconsistencies, or character personality breaks) or abnormal user feedback, it can roll back to the previous stable version with one click based on the version snapshot in S314, or re-trigger the optimization process and apply updated constraint rules (such as updating character profiles and tightening the threshold for filler words). The system supports "parameter versioning" and "model versioning" management to ensure that every adjustment is traceable, verifiable, and reproducible.

[0159] Figure 4 This is a flowchart illustrating the selection process for a feasible candidate set of a dubbing method according to an embodiment of this application, such as... Figure 4 As shown, in some embodiments of this application, the feasible candidate set (first candidate text) can be filtered through the following steps:

[0160] S402: Generate a candidate pool. Specifically, the system constructs a structured candidate expression set based on the source language subtitle text and the semantic expression characteristics of the target language through a multi-source generation mechanism. The specific implementation of this step can be referred to the previous embodiment, and will not be repeated here.

[0161] S404: Determine candidate attributes (word count / pauses / tone / ...). Specifically, for each candidate text in the candidate pool, the system automatically extracts and labels a set of quantifiable and comparable structured attributes to support subsequent hard constraint judgments. Attributes include, but are not limited to: estimated duration D_hat(c) (obtained by syllable count × speech rate prior or lightweight TTS trial synthesis), suggested pause set Pi_hat(c) (pause positions predicted based on punctuation, semantic segmentation models, or language models), tone intensity labels (such as "high emotion", "neutral", "low tone"), terminology hit status (whether it contains proper nouns that need to be retained or sensitive words that need to be replaced), the number and position of filler phrases (such as whether "um", "actually", "well", etc. appear and whether they are consecutive), style labels (such as "formal", "colloquial", "child-oriented"), etc. All attributes are stored in the candidate attribute table in key-value pairs, forming a "behavioral profile" for each candidate.

[0162] S406: Generate a lip-sync rhythm summary. Specifically, the implementation of this step can be found in the previous embodiment, and will not be repeated here.

[0163] S408: Determine the set of hard constraints (duration / pauses / terminology / ...). Specifically, the system constructs a standardized set of hard constraints based on the lip-sync rhythm summary and business rules, serving as a rigid criterion for candidate selection. This set includes:

[0164] (1) Duration constraint: The candidate estimated duration D_hat(c) must satisfy D_min≤D_hat(c)≤D_max;

[0165] (2) Pause alignment constraint: Each pause position in the candidate suggested pause point Pi_hat(c) must fall within the lip-sync pause interval P, with a time tolerance of ±200ms.

[0166] (3) Terminology compliance constraints: Key terms in candidate texts must match the terminology database exactly, prohibited words must not appear, and sensitive content must trigger replacement rules;

[0167] (4) Fill phrase constraints: The number of times the fill word is used is N_fill(c)≤2, consecutive use is prohibited, and it is prohibited to appear in the strong peak position (within the ±150ms window of K).

[0168] (5) Role / Style Constraints (Partially Hard): If a role explicitly prohibits the use of "you" and requires the use of "you (formal)," then this constraint is considered a hard constraint that cannot be violated.

[0169] S410: Hard Constraint Filtering. Specifically, the system takes candidate attributes and lip-sync rhythm summaries as input and executes the discrimination logic in the hard constraint set one by one. For each candidate text, the system checks in sequence whether its attributes satisfy all hard constraint conditions. If any condition is not satisfied, it is immediately determined to be an infeasible candidate.

[0170] S412: Output the feasible candidate set. Specifically, the system integrates all candidate texts filtered through hard constraints into a feasible candidate set C_feasible, which serves as the input for the next stage, "candidate optimization." Each candidate in this set possesses the following characteristics: semantic validity, duration compliance, pause rationality, terminology accuracy, and padding safety. The set is output in a structured data format, containing fields such as candidate ID, text content, attribute labels, and constraint pass status. It maintains an indexed association with the lip-shape codebook and semantic codebook, ensuring that the subsequent optimization process is traceable and verifiable.

[0171] S414: Statistics on Reasons for Infeasibility. Specifically, the system performs aggregated analysis on the failure reasons of rejected candidate texts, generating a statistical report on reasons for infeasibility. This report includes: the frequency distribution of various failure codes (e.g., FAIL_LEN accounts for 32%, FAIL_PAUSE accounts for 21%), the average relaxation amount (e.g., average timeout of 147ms), high-frequency failure patterns (e.g., rejection due to pause misalignment in a certain language), and whether constraints fail due to low lip-sync credibility. These statistical results are used to: ① guide the optimization of candidate pool generation strategies (e.g., increase short sentence candidates); ② adjust constraint thresholds (e.g., relax pause tolerance); ③ provide priority for manual review (e.g., centrally handle high-frequency failure types); ④ support system self-learning and model iteration.

[0172] Figure 5 This is a decision path audit flowchart of a dubbing method according to an embodiment of this application, such as... Figure 5 As shown, in some embodiments of this application, the audit process includes:

[0173] S502: Generate a candidate pool. Specifically, based on the source language subtitles and the target language, a set of semantically equivalent candidate expressions is generated through rule templates, domain dictionaries, and language generation models. Each candidate is given an interpretable attribute label (such as expected duration, pause distribution, and style features), and a structured semantic codebook is constructed.

[0174] S504: Filtering (feasible candidate set). Specifically, based on the lip-sync rhythm summary and preset hard constraints (duration, pauses, terminology, padding restrictions), a structured filter is performed on the candidate pool to select a feasible candidate set C_feasible that meets all the rigid conditions.

[0175] S506: Record the reason for elimination (per candidate). Specifically, for each candidate text that fails the hard constraint filtering, the system automatically generates a structured record of the reason for elimination, including: failure code, specific constraint item violated, quantified relaxation amount, triggering rule version number and contextual basis (e.g., "Because the banned word 'you' hit compliance library v2.1").

[0176] S508: Generate the preferred set. Specifically, based on the feasible candidate set C_feasible, the system sorts the candidates according to a multi-objective scoring function (such as S_lip, S_semantic, S_personality, and S_global mentioned above) and outputs a list of preferred candidates arranged from high to low.

[0177] S510: Record the final reason (sentence by sentence). Specifically, for each final selected candidate, the system generates a summary of the decision reason, recorded in both natural language and structured tags: "Why was this candidate selected?", including:

[0178] (1) The main selection criteria (e.g., "S_lip=0.94 because the strong peak position is aligned with the open vowel 'a'").

[0179] (2) Priority sorting logic (e.g., "among candidates with similar S_lip, the one that wins is the one that prefers shorter sentences due to the character personality profile").

[0180] (3) Reasons for excluding other preferred candidates (e.g., "Candidate B has low semantic consistency, and 'company' is mistranslated as 'enterprise'");

[0181] (4) Whether to enable fill phrases and their positions (e.g., “Use 'um' once, in a non-strong peak area in the sentence”).

[0182] S512: Final dialogue + TTS synthesis. Specifically, using the final selected candidate text determined in S510 as input, a high-fidelity speech synthesis system (TTS) is invoked to generate dubbing audio, and phoneme alignment information and energy envelope are output simultaneously.

[0183] S514: Audit Snapshot (Version / Rollback Point). Specifically, the system automatically generates a standardized audit snapshot after the dubbing output is completed.

[0184] S516: Canary Release or Rollback. Specifically, the system supports versioned canary releases and precise rollbacks based on audit snapshots. In a production environment, new voiceovers can be pushed to only some users or scenarios for effect verification; if semantic deviations, style changes, or compliance risks are found, the system can restore to a historical stable version with one click based on the rollback point marker in the snapshot, or re-trigger the optimization process and apply updated constraint rules (such as correcting character profiles and tightening filler word strategies).

[0185] Figure 6 This is a flowchart illustrating the application of a language personality profile in a dubbing method according to an embodiment of this application, such as... Figure 6 As shown, in some embodiments of this application, the step of candidate text selection based on language personality profile includes:

[0186] S602: Obtain character history information. Specifically, the system extracts multimodal data of the target character from the media asset database in historical dramas, past dubbed versions, or content in the same series, including: historical dialogue text (source language and target language), dubbing audio, subtitle alignment records, manual editing modification logs (such as "unify 'you' to 'you'"), audience feedback tags (such as "speaking with a cold laugh"), etc.

[0187] S604: Constructing a Linguistic Personality Profile. Specifically, based on the historical corpus obtained in S602, the system employs statistical analysis and lightweight machine learning methods to automatically extract and quantify the linguistic features of the characters, constructing a structured linguistic personality profile. Profile dimensions include:

[0188] (1) Writing style (such as the proportion of written language and the tendency to use long sentences);

[0189] (2) Level of politeness (e.g., frequency of use of honorifics "you" / humble expressions, strength of negative sentences);

[0190] (3) Verbal tic features (such as the high frequency of specific interjections: "ma", "ne", "you know", and the repetitive structure "really really");

[0191] (4) The degree of emotional expression (such as the proportion of exclamatory sentences, the variance of speech rate, and the fluctuation of pitch);

[0192] (5) Sentence structure preferences (such as rhetorical questions, sentence segmentation patterns, and tendency to omit the subject).

[0193] Each dimension is represented by a numerical score (0–1) or a rule template (such as “must use 'you')”, with an added confidence level and update source (such as “based on 12 episodes of samples, confidence level 0.87”), forming a versionable, updatable, and interpretable digital fingerprint of the character language.

[0194] S606: Define the profile constraints (style / politeness / verbal tics). Specifically, the system transforms the language personality profile generated in S604 into executable hard and soft constraint rules, embedding them into the candidate screening and selection process:

[0195] Hard constraints: Set inviolable rules for the core language features of the character (such as "Character A is prohibited from using 'you' and must use 'you'"). Violators will be directly removed from the candidate pool.

[0196] Soft constraints: Weighted scoring of style preferences (e.g., "prefer short sentences" → deduct 0.2 points from the S_personality score of candidates with long sentences; "habitual use of 'um'" → add 0.15 points to candidates containing 'um').

[0197] The constraint rules are bound to the character ID and can be dynamically loaded by series, episode, and emotional scene to ensure that the language consistency of the same character in different contexts can still be recognized and maintained by the system.

[0198] S608: Candidate Generation and Optimization. Specifically, in the candidate pool generation stage, the LLM or rule engine explicitly injects language personality profile constraints into the generated prompts (such as "Please rewrite in the tone of character A, avoid using 'you,' and prefer to use short sentences and endings with 'ma'"). In the optimization stage, the system incorporates the profile matching degree as the core scoring dimension S_personality into the final ranking function.

[0199] S610: Profile Hit / Deviation. Specifically, the system automatically compares the final selected candidate with the profile constraints and outputs a profile consistency assessment report, including: hit items (e.g., "Successfully used 'you' 3 times, meeting hard constraints"); deviation items (e.g., "Not used 'ma,' a 67% decrease from the historical average"); style deviation intensity score (e.g., "Style consistency score 0.72, below the threshold of 0.8"); and comparison benchmark (e.g., "Compared to the historical average sentence length of the role, the current sentence length is 12% shorter"). This assessment result is recorded in the audit snapshot (see [link]). Figure 5 (S514) serves as auditable evidence of "whether the voice acting is more like the character," and is available for review by editors, directors, or legal professionals to ensure that the voice acting does not sacrifice the character's recognizability in pursuit of rhythmic fit.

[0200] S612: Manual review and feedback. Specifically, when the system detects that the deviation of the portrait is higher than the threshold, or when the editor clearly marks "the character's tone is wrong" during the review process, the system triggers a closed-loop mechanism for manual feedback.

[0201] Figure 7 This is a structural diagram of a dubbing device according to an embodiment of this application, such as... Figure 7 As shown, the device includes:

[0202] Module 702 is used to acquire the materials to be dubbed;

[0203] Extraction module 704 is used to extract the lip shape features of the target object from the dubbing material, wherein the lip shape features are used to characterize the change of the lip movement state of the target object over time;

[0204] The determining module 706 is used to determine multiple candidate texts in the dubbing material that correspond to the subtitle text of the target object, wherein the multiple candidate texts include subtitle texts that use the same language type and are expressed in multiple ways;

[0205] The dubbing module 708 is used to determine the target text that matches the lip-sync features from multiple candidate texts, and to dub the dubbing material using the target text.

[0206] It should be noted that, Figure 7 The dubbing device shown is used to perform Figure 2 The dubbing method shown, therefore Figure 2 The relevant explanations and instructions in the dubbing methods also apply to Figure 7 The dubbing device shown will not be described in detail here.

[0207] This application also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute steps that implement the dubbing methods in various embodiments of this application.

[0208] This application also provides a non-volatile storage medium including a stored computer program, wherein the device containing the non-volatile storage medium executes the steps of the dubbing method in various embodiments of this application by running the computer program.

[0209] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the dubbing method in various embodiments of this application.

[0210] This application also provides a computer program that, when executed by a processor, implements the steps of the dubbing method in various embodiments of this application.

[0211] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0212] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0213] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0214] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0215] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0216] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A dubbing method, characterized in that, include: Obtain the materials to be dubbed; Extract the lip-shape features of the target object from the dubbing material, wherein the lip-shape features are at least used to characterize the change of the target object's lip movement state over time; Determine multiple candidate texts in the dubbing material that correspond to the subtitle text of the target object, wherein the multiple candidate texts include subtitle texts that use the same language type but are expressed in multiple ways; The target text that matches the lip-sync feature is determined from the plurality of candidate texts, and the target text is used to dub the material to be dubbed.

2. The method according to claim 1, characterized in that, Determining the target text that matches the lip shape features from the plurality of candidate texts includes: A first lip-shape feature is obtained from the lip-shape features, wherein the first lip-shape feature is used to characterize the hard constraints when dubbing the material to be dubbed in physical space and time. The first lip-shape feature is matched with the speech attributes of the plurality of candidate texts respectively to obtain the first candidate text, wherein the speech attributes are used to characterize the acoustic rhythm features of the plurality of candidate texts during the dubbing process; The target text is determined from the first candidate text.

3. The method according to claim 2, characterized in that, The first lip-shape feature is matched with the speech attributes of the plurality of candidate texts to obtain the first candidate text, including: The duration interval in the first lip-sync feature is compared with the expected duration in the speech attribute to obtain a first comparison result, wherein the duration interval is used to characterize the time occupied by the subtitle text in the dubbing material; The pause interval in the first lip shape feature is compared with the expected pause point in the speech attribute to obtain a second comparison result, wherein the pause interval is used to characterize the time period during which the lip movement state of the target object meets the preset state; Text detection is performed on the multiple candidate texts to obtain detection results, wherein the text detection is used to detect whether the text content of the multiple candidate texts meets preset rules; The first candidate text is determined based on the first comparison result, the second comparison result, and the detection result.

4. The method according to claim 2, characterized in that, Determining the target text from the first candidate text includes: Determine the matching degree corresponding to each first candidate text, wherein the matching degree is used to quantify the degree of consistency between the first candidate text and the second lip-shape feature in the lip-shape feature in the visual and auditory dimensions, and the second lip-shape feature is used to characterize the soft constraint conditions when dubbing the material to be dubbed in terms of content adaptation; The second candidate text with the highest matching degree among the first candidate texts is determined as the target text.

5. The method according to claim 4, characterized in that, Determining the matching degree for each of the first candidate texts includes: A first index is determined between the first candidate text and the second lip shape feature, wherein the first index is used to quantify the degree of matching between the speech energy corresponding to the first candidate text and the lip shape rhythm of the target object; A second index is determined between the first candidate text and the second lip-sync feature, wherein the second index is used to quantify the semantic similarity between the first candidate text and the subtitle text; A third indicator is determined between the first candidate text and the second lip-shape feature, wherein the third indicator is used to quantify the degree to which the expression of the first candidate text matches the linguistic personality profile of the target object; The matching degree is determined based on at least one of the first indicator, the second indicator, and the third indicator.

6. The method according to claim 4, characterized in that, The method further includes: Determine the target phonemes in the phoneme sequence of the first candidate text, wherein the target phonemes include phonemes whose lip opening and closing degree meets a preset opening and closing condition during pronunciation; Using multiple strong peaks in the second lip shape feature as time anchors, it is determined whether the target phoneme is aligned with the strong peaks within a preset time window to obtain a judgment result. The strong peaks include the time point when the degree of lip opening and closing of the target object in the dubbing material reaches a local maximum value. Based on the judgment result, the strong peak alignment degree corresponding to the first candidate text is determined, wherein the strong peak alignment degree is used to quantify the degree of consistency between the speech content of the first candidate text and the lip movement state of the target object in the visual dimension. The matching degree is determined based on the strong peak alignment.

7. The method according to claim 1, characterized in that, After determining the target text that matches the lip shape feature from the plurality of candidate texts, the method further includes: The candidate graph structure is determined based on the multiple candidate texts corresponding to the multiple subtitle texts of the target object; A linguistic personality profile of the target object is determined, and consistency constraints corresponding to the multiple subtitle texts are determined based on the linguistic personality profile, wherein the linguistic personality profile is used to characterize the linguistic style of the target object in historical materials; The target matching degree between each candidate text and the lip shape feature is used as the node weight of the node in the candidate graph structure, and the edge weight of the directed edge in the candidate graph structure is determined based on the consistency constraint. Based on the node weights and edge weights, a target candidate path is determined from the candidate graph structure, and the dubbing material is dubbed according to the candidate text corresponding to the target candidate path.

8. The method according to claim 1, characterized in that, The method further includes: determining audit link data corresponding to the subtitle text, wherein the audit link data is used to record the decision basis for determining the target text from the plurality of candidate texts.

9. The method according to claim 8, characterized in that, The audit link data includes at least one of the following: Candidate source identifier, used to record the generation mechanism or data source of each candidate text; Hard constraint violation coding is used to record the reasons why candidate texts other than the target text were not selected; The selection criteria are used to record the reasons for selecting the target text. Parameter version data is used to record the runtime environment context on which the target text was generated; A rollback point is used to identify the preceding state of the decision path corresponding to the target text that can be rolled back.

10. A dubbing device, characterized in that, include: The acquisition module is used to acquire the materials to be dubbed. An extraction module is used to extract the lip-shape features of a target object from the dubbing material, wherein the lip-shape features are at least used to characterize the change in the lip movement state of the target object over time; The determining module is used to determine multiple candidate texts in the dubbing material that correspond to the subtitle text of the target object, wherein the multiple candidate texts include subtitle texts that use the same language type and are expressed in multiple ways; The dubbing module is used to determine the target text that matches the lip-sync features from the plurality of candidate texts, and to dub the material to be dubbed using the target text.

11. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the dubbing method according to any one of claims 1 to 9.

12. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the dubbing method according to any one of claims 1 to 9.

Images