A method, apparatus, device, and medium for dynamic speech segmentation and temporal self-healing alignment for long video translation.
By using dynamic speech segmentation and temporal self-healing alignment methods, the problems of hardware crashes and audio-visual asynchrony in long video translation are solved, achieving efficient audio-visual synchronization and semantically complete translation results.
Patent Information
- Application Number
- CN202611399775.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-10
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-09-10
AI Technical Summary
Existing technologies cannot effectively segment long audio segments when translating long videos, leading to problems such as hardware memory overflow, translation system crashes, semantic distortion, and audio-visual desynchronization.
A dynamic speech segmentation and temporal self-healing alignment method is adopted. Acoustic pre-segmentation is performed by setting an audio energy sliding window to remove meaningless noise and transient interference. Key parameters are monitored for truncation, and temporal cross-validation and self-healing compensation are performed to correct time axis deviation.
It achieves a dynamic balance between hardware security and semantic integrity, avoiding memory overflow and semantic distortion, and ensuring efficient translation with synchronized audio and video.
Smart Images

Figure CN122889014B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech processing technology, specifically to a method, apparatus, device, and medium for dynamic speech segmentation and temporal self-healing alignment for long video translation. Background Technology
[0002] With the deepening of multilingual communication and cooperation, the demand for cross-language audio and video content dissemination has surged. To improve the efficiency of cross-language communication, fully automated video processing systems are typically used for audio and video processing. Generally speaking, an automated video translation chain includes Automatic Speech Recognition (ASR), Machine Translation (MT), and Text-to-Speech (TTS). ASR extracts the speech from the original video and converts it into text in the source language. Machine Translation accurately translates the extracted source language text into the target language. TTS regenerates the translated target text into emotionally charged synthetic speech and aligns and merges it with the original video footage on the timeline.
[0003] However, in actual processing of long videos, fully automated video processing systems are limited by the physical constraints of computer hardware memory (i.e. video memory) and maximum context processing capacity, making it impossible to process ultra-long audio streams in one go. Therefore, before entering the core processing chain, the long audio stream needs to be divided into shorter, independent audio blocks.
[0004] In addition, the different lengths of expressions in different languages cause misalignment and disconnect between the final translated speech and the mouth movements, body language, and scene transitions in the original video, resulting in a poor video viewing experience.
[0005] Against this backdrop, how to segment long audio files and achieve audio-visual synchronization has become an urgent problem to be solved. Summary of the Invention
[0006] This invention provides a method, apparatus, device, and medium for dynamic speech segmentation and temporal self-healing alignment for long video translation, in order to solve the problem of how to segment long audio and achieve audio-visual synchronization.
[0007] In a first aspect, the present invention provides a dynamic speech segmentation and temporal self-healing alignment method for long video translation, the method comprising: An audio energy sliding window is set, and acoustic pre-segmentation is performed on the audio data to be processed based on the audio energy sliding window. The pre-segmented audio data to be processed is then glued, filtered, and extended to output candidate speech islands. Automatic speech recognition is performed on candidate speech islands, and key parameters are monitored, including the cumulative processing time of the current segment, the total number of currently decoded characters, and the semantic completion status indicator. Set priority weights, evaluate key parameters based on priority weights, and then truncate them; The truncated audio data is subjected to temporal cross-validation and temporal self-healing compensation to output synthesized speech.
[0008] This invention performs acoustic pre-segmentation, gluing, filtering, and extension on the audio data to be processed. While segmenting the audio data, meaningless environmental noise and transient interference are eliminated. During the speech recognition process, key parameters are monitored and evaluated, and truncation is performed to fundamentally avoid the problem of graphics card memory overflow caused by excessively long input sequences. The truncated audio data is subjected to temporal cross-validation and temporal self-healing compensation to correct the time axis deviation accumulated due to the difference in duration between languages, so that the final output synthesized speech is synchronized with the original video, improving the viewing experience of long video translation.
[0009] In one optional implementation, an audio energy sliding window is set, and acoustic pre-segmentation is performed on the audio data to be processed based on the audio energy sliding window, including: Set the basic environmental energy threshold and the number of frames to be merged for micro-pauses, and use the basic environmental energy threshold and the number of frames to be merged for micro-pauses as the initialization parameters for the audio energy sliding window; When the audio signal energy of the audio data to be processed is detected to fall below the basic environmental energy threshold, a frame counter is started. When the accumulated frame count of the frame counter is greater than the micro-pause merging frame count parameter, the current position is marked as a physical breakpoint, and the audio data to be processed is segmented into segments according to the marked physical breakpoint.
[0010] This invention uses the basic environmental energy threshold and the micro-pause merging frame number parameter as the initialization parameters for the audio energy sliding window to determine the boundaries of speech and silence. When the accumulated frame number of the frame counter is greater than the micro-pause merging frame number parameter, a physical breakpoint is marked, which effectively avoids erroneous segmentation caused by pauses such as normal breathing of the speaker.
[0011] In one optional implementation, the acoustically pre-segmented audio data to be processed is glued, filtered, and extended to obtain candidate speech islands, including: Set a tolerance threshold for minor pauses; When the duration of a silent segment between adjacent audio segments is less than the micro-pause tolerance threshold, the adjacent audio segments are glued together as a continuous whole. Set a minimum speech threshold and remove segments whose speech threshold is less than the minimum speech threshold; By extending outwards from the beginning and end boundaries of a continuous audio segment, candidate speech islands are obtained.
[0012] This invention eliminates speech breaks caused by normal breathing or pauses by bonding adjacent audio segments, ensuring acoustic continuity. It also removes segments with a speech threshold lower than the minimum speech threshold to eliminate transient background noise. Furthermore, it extends the continuous audio segments outward from the beginning and end boundaries to fully preserve the consonant airflow and coda of the sentence, providing high-quality input material for subsequent automatic speech recognition.
[0013] In one alternative implementation, monitoring key parameters includes: Record the physical time span from the start point of the current segment to the current processing frame, and determine the recorded result as the cumulative processing time of the current segment; The total number of text characters currently output by the automatic speech recognition engine is counted, and the result is determined as the total number of currently decoded characters. Based on natural speech processing rules, the system matches whether the end of the current output text is a strong semantic punctuation mark, and determines the matching result as a semantic completion status identifier.
[0014] This invention records the cumulative processing time of the current segment, counts the total number of currently decoded characters, quantifies the hardware resource consumption status from the dimensions of physical time span and text data volume, matches strong semantic punctuation marks, and obtains semantic completion status identifiers to accurately identify whether a sentence has completely ended, thus achieving multiple intelligent perceptions.
[0015] In one optional implementation, priority weights are set, and key parameters are evaluated based on these priority weights for truncation, including: Set perfect semantic truncation as the first priority, flexible forced truncation as the second priority, and acoustic gap truncation as the third priority; When the semantic completion status is true and the cumulative processing time of the current segment is greater than the minimum anti-fragmentation threshold, a perfect semantic truncation is performed at the punctuation mark. When the semantic completion status flag is not triggered, and the cumulative processing time of the current segment reaches the upper limit of the video memory safe time or the total number of currently decoded characters reaches the upper limit of the translation model, reverse backtracking is used to find the suboptimal silent gap and perform flexible forced truncation. If the first priority and second priority conditions are not met, and the strong silence signal reaches the preset strong silence duration, the acoustic gap will be cut off.
[0016] This invention covers truncation decisions across all scenarios by employing a three-tiered progressive priority system of perfect semantic truncation, flexible forced truncation, and acoustic gap truncation. It prioritizes semantic integrity, ensures hardware security, and achieves a dynamic balance between hardware preservation and maximum semantic retention.
[0017] In one optional implementation, temporal cross-validation is performed on the truncated audio data, including: The timing offset is calculated based on the timing crossover compensation formula, which is as follows:
[0018] in, This is the timing offset. This refers to the semantic end point of the last word recognized by the automatic speech recognition model. This represents the actual time point at which the underlying physical truncation is performed. The confidence factor is dynamically calculated based on the audio signal-to-noise ratio.
[0019] This invention calculates timing offsets and quantifies accumulated time deviations, providing a basis for correction in subsequent timing self-healing compensation.
[0020] In one optional implementation, temporal self-healing compensation is performed on the truncated audio data, including: The difference between the original visual time window and the temporal offset of the audio data to be processed in the original video is determined as the strict target physical duration of the newly generated translated speech; The ratio of the actual pronunciation duration obtained from actual measurement to the strict target physical duration of the newly generated translated speech is determined as the temporal scaling factor; By using the time-series scaling factor, extreme points and similar segments of the waveform are found in the time domain. The pitch period is overlapped, discarded, and smoothly spliced to compress or lengthen the duration of the audio to be processed to the strict target physical duration of the newly generated translated speech. Dynamically lengthed silence data packets are inserted into the audio data that reaches the strict target physical duration for generating the translated speech, and zeros are added to fix the true acoustic starting point of the translated audio at the absolute timestamp of the original video frame.
[0021] This invention achieves adaptive scaling of speech rate by performing time-series self-healing compensation on truncated audio data, compressing or stretching the audio duration, and inserting dynamic length silence data packets for zero padding, thereby achieving audio-visual synchronization between synthesized speech and the original video footage.
[0022] Secondly, the present invention provides a dynamic speech segmentation and temporal self-healing alignment device for long video translation, the device comprising: The segmentation module is used to set an audio energy sliding window and perform acoustic pre-segmentation on the audio data to be processed based on the audio energy sliding window. It then performs bonding, filtering, and extension on the pre-segmented audio data to be processed and outputs candidate speech islands. The monitoring module is used to perform automatic speech recognition on candidate speech islands and monitor key parameters, including the cumulative processing time of the current segment, the total number of currently decoded characters, and the semantic completion status indicator. The truncation module is used to set priority weights, evaluate key parameters based on the priority weights, and then truncate the data. The output module is used to perform temporal cross-validation and temporal self-healing compensation on the truncated audio data and output synthesized speech.
[0023] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the dynamic speech segmentation and temporal self-healing alignment method for long video translation described in the first aspect or any corresponding embodiment thereof.
[0024] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the dynamic speech segmentation and temporal self-healing alignment method for long video translation described in the first aspect or any corresponding embodiment thereof. Attached Figure Description
[0025] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0026] Figure 1 This is a flowchart illustrating a dynamic speech segmentation and temporal self-healing alignment method for long video translation according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the audio stream processing flow according to an embodiment of the present invention; Figure 3 This is a structural block diagram of a dynamic speech segmentation and temporal self-healing alignment device for long video translation according to an embodiment of the present invention. Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation
[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0028] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention in an appropriate manner in accordance with relevant laws and regulations, and their permission should be obtained.
[0029] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.
[0030] In related technologies, the following solutions are used when processing long audio segmentation: Option 1: Fixed-duration forced segmentation, relying on a timer to forcibly truncate the audio stream at fixed intervals (e.g., every 20 seconds); Option 2: Voice Activity Detection (VAD) relies on monitoring the acoustic energy (i.e. volume) of the audio signal. Once the volume is lower than a preset background noise threshold and the "silent" state lasts for a certain period of time (e.g., 500 milliseconds), it is determined that the speaker has finished speaking, and physical cut-off is performed at the silence point.
[0031] However, when processing long videos that last for tens of minutes or even several hours (such as large conferences, teaching courses, etc.), the relevant technologies face the following three shortcomings: Flaw 1: Hardware memory overflow leads to complete system crash. In many speech or report scenarios, speakers often speak rapidly and emotionally, without noticeable pauses for several minutes at a time. In this situation, existing "speech endpoint detection methods (segmenting by finding silence)" completely fail, unable to find any gaps for segmentation. The system can only forcefully feed this extremely long audio file, lasting several minutes, as a single unit to the large AI model. Because the amount of data in this audio file far exceeds the physical limits of the current top-tier computer graphics card (GPU), it causes an instantaneous overload and crash (Out of Memory, or OOM phenomenon), resulting in the complete interruption and failure of the entire translation task. Defect 2: Forced truncation of context leads to severe semantic distortion and "machine illusion" in the translation system. To avoid the aforementioned system crash, if a "fixed-duration forced segmentation method" (e.g., truncating every 20 seconds) is adopted as a compromise, it will likely split the speaker's long and complex sentences, or even an idiom, in half. When the backend translation model receives such a "disjointed" or "incomplete" half-sentence, it not only fails to provide an accurate translation based on the context, but also triggers the "illusion effect" of the large model due to the incomplete input information—that is, the AI will fabricate and translate randomly to complete the sentence, resulting in an inaccurate final translation and causing serious communication misunderstandings. Defect 3: Accumulated time errors lead to severe audio-visual desynchronization in the final video (i.e., timeline offset). The timestamps recorded by the machine during speech recognition often contain minute physical errors of tens of milliseconds. In long videos, audio is cut into dozens or hundreds of segments, and these tiny segmentation errors accumulate. Furthermore, different languages naturally have different expression lengths (for example, the same sentence might only take 3 seconds in Chinese, but could take 5 seconds in Thai). Current technology lacks a self-correcting mechanism for the timeline, which results in a severe misalignment and disconnect between the generated translated speech and the speaker's lip movements, body language, and scene transitions (such as turning pages in a PowerPoint presentation) in the original video, leading to a poor viewing experience.
[0032] To address the dilemma of related technologies either "rigidly segmenting and destroying semantics" or "blindly stacking and exhausting video memory," this invention provides a dynamic speech segmentation and temporal self-healing alignment method for long video translation. It abandons single-dimensional segmentation methods and combines a three-dimensional joint dynamic decision algorithm based on "physical time limit (ensuring hardware doesn't crash)," "character capacity (ensuring the translation model can process the data)," and "natural language semantic boundaries (such as punctuation marks, ensuring sentence integrity and preventing segmentation)." Ultimately, while ensuring the security of the underlying hardware computing power and eliminating memory overflow crashes, it intelligently finds complete logical pauses in sentences for segmentation, much like human translator, maximizing the preservation of translation context. Simultaneously, through a unique timestamp self-healing algorithm, it automatically repairs time errors caused by segmentation and language conversion, perfectly achieving efficient translation and high-precision audio-visual synchronization of multilingual long videos.
[0033] According to an embodiment of the present invention, a method for dynamic speech segmentation and temporal self-healing alignment for long video translation is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0034] This embodiment provides a dynamic speech segmentation and temporal self-healing alignment method for long video translation. Figure 1 This is a flowchart of a dynamic speech segmentation and temporal self-healing alignment method for long video translation according to an embodiment of the present invention, as shown below. Figure 1 As shown, the process includes the following steps: Step S101: Set an audio energy sliding window, and perform acoustic pre-segmentation on the audio data to be processed based on the audio energy sliding window. Then, glue, filter, and extend the acoustically pre-segmented audio data to be processed to output candidate speech islands.
[0035] In this embodiment of the invention, after receiving a continuous long audio data stream to be processed, a dynamically updated audio energy sliding window is initialized and maintained. The adaptive audio energy sliding window performs acoustic pre-segmentation on the audio data to be processed. After acoustic pre-segmentation, a three-step cleaning process of gluing, filtering, and extension is performed, and finally, an audio data block with complete acoustic continuity is output, which is called a "candidate speech island".
[0036] Step S102: Perform automatic speech recognition on the candidate speech islands and monitor key parameters.
[0037] In this embodiment of the invention, candidate speech islands, as clean material with high signal-to-noise ratio, are streamed into the ASR (Automatic Speech Recognition) engine for automatic speech recognition. During this process, a multi-dimensional status monitoring probe is used to poll and update key parameters in real time at millisecond-level frequencies. These key parameters include the current segment's cumulative processing time, the total number of currently decoded characters, and the semantic completion status indicator.
[0038] Step S103: Set priority weights, evaluate key parameters based on priority weights, and truncate accordingly.
[0039] In this embodiment of the invention, priority weights are preset, the above-mentioned key parameters are continuously evaluated, and dynamic audio truncation operations are performed based on the decision tree.
[0040] Step S104: Perform temporal cross-validation and temporal self-healing compensation on the truncated audio data, and output synthesized speech.
[0041] In this embodiment of the invention, temporal cross-validation is introduced to check for misalignment between text semantic timestamps and audio timestamps. Temporal self-healing compensation is introduced to perform speech rate adaptive scaling processing, adjust the start and stop rhythm of synthesized speech, and output the final synthesized speech.
[0042] The dynamic speech segmentation and temporal self-healing alignment method for long video translation provided in this embodiment performs acoustic pre-segmentation, gluing, filtering, and extension on the audio data to be processed. While segmenting the audio data, meaningless environmental noise and transient interference are eliminated. During the speech recognition process, key parameters are monitored and evaluated, and truncation is performed to fundamentally avoid the problem of graphics card memory overflow caused by excessively long input sequences. The truncated audio data is subjected to temporal cross-validation and temporal self-healing compensation to correct the time axis deviation accumulated due to cross-language duration differences, so that the final output synthesized speech is synchronized with the original video, improving the viewing experience of long video translation.
[0043] This embodiment provides a dynamic speech segmentation and temporal self-healing alignment method for long video translation, which includes the following steps: Step S201: Set an audio energy sliding window, and perform acoustic pre-segmentation on the audio data to be processed based on the audio energy sliding window. Then, glue, filter, and extend the acoustically pre-segmented audio data to be processed to output candidate speech islands.
[0044] Specifically, step S201 above, which involves setting an audio energy sliding window and performing acoustic pre-segmentation of the audio data to be processed based on the audio energy sliding window, includes: Step S2011: Set the basic environmental energy threshold and the number of frames for merging micro-pauses, and use the basic environmental energy threshold and the number of frames for merging micro-pauses as the initialization parameters for the audio energy sliding window. Step S2012: When the audio signal energy of the audio data to be processed is detected to fall below the basic environmental energy threshold, a frame counter is started. When the accumulated frame count of the frame counter is greater than the small pause merging frame count parameter, the current position is marked as a physical breakpoint, and the audio data to be processed is segmented into segments according to the marked physical breakpoint.
[0045] In this embodiment of the invention, a basic environmental energy threshold is set. Combine frame count parameters with minor pauses The audio energy sliding window is initialized using a base environmental energy threshold and a micro-pause merging frame count parameter. When the audio signal energy of the audio data to be processed is detected to fall below the base environmental energy threshold, it is not immediately cut off, but instead a frame counter is started. Only when the accumulated frame count of the frame counter is greater than the micro-pause merging frame count parameter, does the low-energy state continue for more than the micro-pause merging frame count parameter. The current position is marked as a preliminary physical breakpoint. Based on the marked physical breakpoint, the audio data to be processed is segmented into segments, completing the VAD (Voice Activity Detection) detection phase.
[0046] By introducing an audio energy sliding window, the detection criteria are adaptively adjusted in real time based on the ambient noise and speaker volume changes in the current long video. This mechanism can not only accurately filter out meaningless ambient noise and breathing sounds, but also intelligently "glue" tiny, brief pauses in speech, thereby extracting complete and pure "candidate speech segments" (i.e., speech islands) at the physical level before entering the speech recognition model, which greatly reduces the ineffective occupation of computing resources by useless data.
[0047] By using the basic environmental energy threshold and the micro-pause merging frame number parameter as the initialization parameters for the audio energy sliding window, the system can determine the boundaries of speech and silence. When the accumulated frame number of the frame counter is greater than the micro-pause merging frame number parameter, a physical breakpoint is marked, which effectively avoids erroneous segmentation caused by pauses such as normal breathing of the speaker.
[0048] Specifically, in step S201 above, the acoustically pre-segmented audio data to be processed is glued, filtered, and extended to obtain candidate speech islands, including: Step S2023: Set a small pause tolerance threshold; Step S2024: When the duration of a silent segment between adjacent audio segments is less than the micro-pause tolerance threshold, the adjacent audio segments are glued together into a continuous whole. Step S2025: Set a minimum speech threshold and remove segments whose speech threshold is less than the minimum speech threshold; Step S2026: Extend the continuous audio segments outward from the beginning and end boundaries to obtain candidate speech islands.
[0049] In this embodiment of the invention, after the VAD detection phase, a one-dimensional Boolean state sequence arranged in frames is obtained, where 1 represents the detection of human voice and 0 represents silence.
[0050] Preset small pause tolerance threshold The tolerance threshold for this tiny pause The time is set according to the pronunciation habits of the language, usually between 300ms and 500ms.
[0051] The Boolean state sequence is scanned along the time axis in memory. When a silent segment between two adjacent audio segments is detected, its duration is less than the micro-pause tolerance threshold, the sequence is considered complete. When the brief silence is determined to be a normal breath, a short pause in tone, or a natural closure of a voiceless consonant or plosive, rather than a true semantic sentence-ending boundary, the closing operation logic is triggered, forcibly flipping the state of the silence frame from 0 to 1, thereby bridging and smoothly gluing the two short spoken segments into a continuous whole.
[0052] After the bonding is completed, a set minimum speech threshold is introduced. For example, minimum speech threshold For 100ms, segments in the sequence with a speech threshold less than the minimum speech threshold, and isolated and extremely short pulses (such as pure bumping sounds or coughing sounds) are removed as transient background noise and their states are set to 0.
[0053] Finally, at the defined start and end boundaries of the continuous audio segment, i.e. the beginning and end, each segment is extended outwards by a fixed time margin, which is usually 50ms to 100ms. In this way, the weak consonant or coda of the beginning and end of the sentence is preserved, preventing the words from being abruptly cut off.
[0054] After the above three-step cleaning process of bonding, filtering, and extending, candidate speech islands are output. Meaningless environmental noise and brief breathing sounds are filtered out from the physical acoustic level, avoiding the generation of extremely fragmented invalid audio blocks and significantly reducing the pressure of concurrent requests from downstream large models.
[0055] By aggregating adjacent audio segments, speech breaks caused by normal breathing or pauses are eliminated, ensuring acoustic continuity. Segments with speech thresholds lower than the minimum speech threshold are removed to eliminate transient background noise. The continuous audio segments are extended outward from the beginning and end boundaries to fully preserve the consonant airflow and coda of the sentence, providing high-quality input material for subsequent automatic speech recognition.
[0056] Step S202: Perform automatic speech recognition on the candidate speech islands and monitor key parameters.
[0057] Specifically, the key parameters monitored in step S202 above include: Step S2021: Record the physical time span from the start point of the current segment to the current processing frame, and determine the recorded result as the cumulative processing time of the current segment; Step S2022: Calculate the total capacity of text characters currently output by the automatic speech recognition engine, and determine the statistical result as the total number of currently decoded characters; Step S2023: According to the natural speech processing rules, match whether the end of the current output text belongs to a strong semantic punctuation mark, and determine the matching result as a semantic completion status identifier.
[0058] In this embodiment of the invention, the multi-dimensional state monitoring probe updates the following three key water level counters in real time at a millisecond frequency: Dimension 1: Physical duration safety water level; Dimension 2: Character capacity optimal water level; Dimension 3: Natural language semantic boundary. The physical duration safety water level is used to monitor the cumulative physical playback duration of the current segment in real time to prevent excessive single input from overloading the video memory. The character capacity optimal water level is used to count the total number of identified characters in real time to ensure that the amount of data sent to the downstream translation model is within its optimal processing window and to avoid machine translation illusion. The natural language semantic boundary is used to perform semantic parsing on the currently identified text in real time to detect whether a period, question mark, exclamation mark, or other "strong semantic rest" representing the completeness of the sentence appears at the end.
[0059] Specifically, the key parameters corresponding to the physical duration safety level, the optimal character capacity level, and the natural language semantic boundary are the cumulative processing time for the current segment. Total number of characters currently decoded Semantic completion status identifier .
[0060] Record the physical time span from the start point of the current segment to the current processing frame, and determine the recorded result as the cumulative processing time of the current segment. Specifically, the cumulative processing time of the current segment is recorded by a monitoring probe, representing the physical time limit. The processing time is determined by recording the number of frames that the streaming audio enters the ASR and multiplying the number of processed frames by the duration of a single frame. Alternatively, the timer API (Application Programming Interface) can be directly called to perform the timing.
[0061] The total number of text characters successfully output by the automatic speech recognition engine is counted, and the result is determined as the total number of characters decoded so far. Specifically, the total number of currently decoded characters is counted using a monitoring probe, representing the character capacity, which is achieved by continuously accumulating the string length returned by the ASR engine via the WebSocket or gRPC interface.
[0062] Based on natural speech processing rules, the system matches the end of the current output text in real time to determine whether it contains strong semantic punctuation marks, such as periods, question marks, and exclamation marks, and uses the matching results as semantic completion status markers. If a match is found, it is marked as True; otherwise, it is marked as False. Specifically, in the probe pipeline, lightweight punctuation restoration models are cascaded, such as sequence labeling models based on BERT (Bidirectional Encoder Representations from Transformers). When the probe detects that the current character in the model's output is classified as... <period>(period),<question_mark> (Question mark) or<exclamation_mark> When (exclamation mark) is present, the inside of the probe... The status is immediately set to True.
[0063] By recording the cumulative processing time of the current segment and counting the total number of currently decoded characters, the hardware resource consumption status is quantified from the dimensions of physical time span and text data volume. Strong semantic punctuation is matched to obtain semantic completion status identifiers, so as to accurately identify whether the sentence has completely ended and realize multiple intelligent perceptions.
[0064] Step S203: Set priority weights, evaluate key parameters based on priority weights, and truncate accordingly.
[0065] Specifically, step S203 includes: Step S2031: Set perfect semantic truncation as the first priority, flexible forced truncation as the second priority, and acoustic gap truncation as the third priority. Step S2032: When the semantic completion status is true and the cumulative processing time of the current segment is greater than the minimum anti-fragmentation threshold, perform perfect semantic truncation at the punctuation mark. Step S2033: When the semantic completion status flag is not triggered, and the current segment's cumulative processing time reaches the upper limit of the video memory safe time or the current total number of decoded characters reaches the upper limit of the translation model, backtrack in reverse to find the suboptimal silent gap and perform flexible forced truncation. Step S2034: When the first priority and second priority conditions are not met, and the strong silence signal reaches the preset strong silence duration, the acoustic gap is truncated.
[0066] In this embodiment of the invention, a progressive decision-making network of "conditional triggering and priority concession" is constructed, and intelligent game is executed at the following three levels: The first layer of progression: Regular waiting (pursuing perfect semantics, Dimension 3 is dominant). When the physical duration (Dimension 1) and character size (Dimension 2) of the audio segment are both within safe limits, the highest priority is "protecting the context". At this time, continuous polling and patient waiting for the appearance of "Dimension 3" are performed. Once a strong semantic pause (such as a period or question mark) is detected, "perfect semantic truncation" is immediately performed, sending a complete sentence to the downstream translation model; The second layer of progression: alert triggering (hardware protection takes priority, Dimension 1 / Dimension 2 take over). If a continuous stream of long and complex sentences appears in the video, causing punctuation to be missed for a long time (i.e., missing Dimension 3), as audio data continues to accumulate, when either "Dimension 1 (duration)" or "Dimension 2 (number of characters)" is about to reach the preset alert limit, a risk of memory overflow (OOM) is determined. At this time, a priority switch will occur—the hardware protection mechanism instantly overrides semantic integrity, no longer waiting for a period, and immediately triggers a forced truncation command; The third layer of advancement: Flexible backoff (suboptimal fallback, acoustic feature compensation). Unlike existing technologies, when forced truncation is required, it never "hard-cuts" the audio stream from its current physical breakpoint (which would shred the words being spoken). Instead, a "flexible backoff protection mechanism" is activated, allowing the algorithm to backtrack along the timeline of historical audio frames to find the closest tiny pause or suboptimal silence point to the current truncation point, and then perform a lossless cut at that silent location.
[0067] Through this rigorous progressive logic of "prioritizing semantics (first layer) - exceeding the limit to protect hardware (second layer) - backtracking to find pauses (third layer)," absolute operational safety is achieved at the lowest possible cost in the face of unavoidable extremely long sentence scenarios, truly realizing a perfect dynamic balance between "absolute hardware preservation" and "maximum semantic retention."
[0068] Specifically, such as Figure 2 As shown, perfect semantic truncation is the first priority, flexible forced truncation is the second priority, and acoustic gap truncation is the third priority.
[0069] Based on preset priority weights, continuously evaluate the cumulative processing time of the current segment. Total number of characters currently decoded Semantic completion status identifier Based on the following decision tree, dynamic memory release and audio truncation operations are performed.
[0070] The first priority, namely perfect semantic truncation: when the semantic completion status flag is detected as true, i.e. If the cumulative processing time of the current segment exceeds the minimum anti-fragmentation threshold, a perfect semantic truncation will be immediately performed at that punctuation point.
[0071] This method maximizes the protection of the sentence's complete structure within a safe water level, providing an extremely coherent context for downstream machine translation and resulting in the highest translation quality.
[0072] The second priority, namely flexible forced truncation: when long and complex sentences appear in the video, a semantic completion status indicator is used. It has not been triggered for a long time, and the current segment has accumulated processing time. Imminently breaking the upper limit of video memory safe duration or the total number of characters currently decoded Reaching the upper limit of the translation model If the normal waiting is interrupted, the historical frames are traced back along the time axis to find the nearest suboptimal silent gap, such as a weak punctuation mark or a short pause, and then a flexible forced truncation is performed.
[0073] Specifically, the implementation of flexible forced truncation relies on a joint feature scoring algorithm based on a multimodal ring buffer, and includes the following steps: (1) Establish a multimodal sliding buffer: Allocate and continuously maintain a fixed-length (e.g., record data from the past 3 to 5 seconds) circular buffer in main memory. This buffer stores two real-time feature streams in parallel: 1) Acoustic feature stream: Record the short-time energy (STE) and VAD confidence of historical audio in frames (e.g., every 10ms or 20ms); 2) Semantic timestamp stream: Record the partial output (Partial Results) given by the ASR engine streaming decoding, accurately locate the start and end timestamps of each word or character boundary, and weak punctuation marks (e.g., commas, pause marks) accompanying the output. (2) Triggering backtracking and joint feature scoring: the cumulative processing time of the current segment Or the total number of characters currently decoded If the set high-water mark safety threshold is triggered but a strong semantic punctuation mark (such as a period) is not received, the algorithm suspends and waits. Starting from the current over-limit frame time t, it traverses all candidate frames in the buffer along the historical timeline tn. A suboptimal breakpoint evaluation function is introduced to calculate the truncation suitability score for each candidate frame. :
[0074] in, For acoustic quietness, the lower the average acoustic energy near the frame (indicating minor pauses or breaths), the higher the score. This is the weighted value for acoustic quietness; The time decay penalty is the time span between the candidate point and the current out-of-limit point. The closer the candidate point is to the current point, the lower the computational duplication rate caused by the backoff, and the smaller the penalty. This represents the weighting value for time decay loss; As a semantic boundary reward, if a candidate frame is exactly aligned with the word boundary or weak punctuation mark in the local output, it is given a high multiplier reward weight; if it is forcibly cut off in the middle of the word pronunciation, it is given an extremely high penalty score. The weight value for the semantic boundary reward.
[0075] (3) Perform flexible cutting and smooth state inheritance: Lock the highest-scoring candidate frame as the "second-best silent gap" in a very short time (milliseconds). Encapsulate the audio block before the gap and submit it to the downstream translation queue to release the video memory. Seamlessly connect and smoothly move the remaining audio segments after the gap and the hidden state of the ASR model to the beginning of the next processing cycle. This prevents OOM (Out of Memory) while ensuring that no "dirty cutting" of individual words is interrupted.
[0076] In this way, when faced with uninterrupted speech lasting several minutes, a fallback mechanism is used to force a circuit breaker, fundamentally preventing GPU memory overflow and system crashes. At the same time, the flexible backtracking mechanism avoids forcibly splitting the word being spoken in the middle.
[0077] The third priority, namely acoustic gap truncation: when the first priority and second priority conditions are not met, but the strong silence signal in the streaming process reaches the preset strong silence duration, that is, when a strong silence signal with a sufficiently long duration is encountered, the safety truncation is triggered and the acoustic gap is truncated.
[0078] By defining three progressive priorities—perfect semantic truncation, flexible forced truncation, and acoustic gap truncation—the truncation decision-making is covered across all scenarios. Prioritizing semantic integrity and ensuring hardware security, a dynamic balance is achieved between hardware preservation and maximum semantic retention.
[0079] Step S204: Perform temporal cross-validation and temporal self-healing compensation on the truncated audio data, and output synthesized speech.
[0080] Specifically, the temporal cross-validation of the truncated audio data in step S204 above includes: Step S2041: Calculate the timing offset according to the timing crossover compensation formula.
[0081] In embodiments of the present invention, such as Figure 2 As shown, due to the forced physical segmentation in extreme cases such as the second priority, the "text semantic timestamp" output by ASR and the "audio timestamp" of VAD physical segmentation are misaligned at the millisecond level. To address the accumulated timeline error caused by repeated cutting of long videos and the natural accumulation of timeline errors after multilingual conversion, a time-series cross-validation mechanism is introduced.
[0082] Specifically, a timing crossover compensation formula is introduced to calculate the timing offset.
[0083] The formula for timing crossover compensation is as follows:
[0084] in, This is the timing offset. This refers to the semantic end point of the last word recognized by the automatic speech recognition model. This represents the actual time point at which the underlying physical truncation is performed. The confidence factor is dynamically calculated based on the audio signal-to-noise ratio.
[0085] By calculating the timing offset and quantifying the accumulated time deviation, a correction basis is provided for subsequent timing self-healing compensation.
[0086] Specifically, the temporal self-healing compensation for the truncated audio data in step S204 above includes: Step S2042: The difference between the original visual time window and the temporal offset of the audio data to be processed in the original video is determined as the strict target physical duration of the newly generated translated speech. Step S2043: The ratio of the actual pronunciation duration obtained by actual measurement to the strict target physical duration of the newly generated translated speech is determined as the temporal scaling factor; Step S2044: Using the time-series scaling factor, find the extreme points and similar segments of the waveform in the time domain, overlap, discard, and smoothly splice the pitch period, and compress or lengthen the duration of the audio to be processed to the strict target physical duration of the generated translated speech. Step S2045: Insert a dynamic length silence data packet into the audio data whose duration reaches the strict target physical duration for generating the translated speech, and pad with zeros to fix the true acoustic starting point of the translated audio on the absolute timestamp of the original video frame.
[0087] In embodiments of the present invention, such as Figure 2 As shown, the timing offset is extracted and transmitted to the downstream text-to-speech (TTS) module. The text-to-speech module dynamically performs time-stretching (adaptive scaling of speech rate) processing to adjust the start and stop rhythm of subsequent synthesized speech.
[0088] Specifically, time-stretching (adaptive scaling of speech rate) processing and adjustment of start and stop rhythm mainly rely on closed-loop timing feedback calculation and pitch-synchronous stretching mechanism, including the following steps; First, when the downstream media processing module receives a TTS (text-to-speech) task, it first obtains the original visual time window of the audio segment in the original video. Using the temporal offset from the upstream ASR probe as a correction factor, the difference between the original visual time window and the temporal offset in the original video is calculated for the audio data to be processed. This difference is determined as the strict target physical duration of the newly generated translated speech.
[0089] in, The strict target physical duration for the newly generated translated speech, This represents the original visual time window of the audio data to be processed within the original video.
[0090] Then, the speech synthesis engine first generates initial translated audio at a natural baseline speech rate, and uses internal media probes (such as ffprobe) to precisely measure its actual pronunciation duration in memory. It then calculates the ratio of the measured actual pronunciation duration to the strict target physical duration of the newly generated translated speech, which is determined as the timing scaling factor.
[0091] in, For time-series scaling factor, This refers to the actual pronunciation duration obtained from actual measurements.
[0092] To ensure a good listening experience, a no-adjustment zone is set (e.g., 0.95 < Ratio < 1.05). If the Ratio falls outside this zone, a time-stretching operation is forcibly triggered.
[0093] Next, in order to change the speech rate without destroying the speaker's fundamental frequency and original timbre (after timbre cloning), a Waveform Similarity Overlap-Add (WSOLA) algorithm or an adaptive audio temporal filter in the media processing layer (such as ffmpeg's atempo algorithm) is used. This algorithm uses the temporal scaling factor Ratio to find extreme points and similar segments of the waveform in the time domain. By overlapping, discarding, or smoothly splicing the fundamental frequency periods, it losslessly compresses or stretches the physical audio to the target duration. This process involves a maximum of 1-3 rounds of closed-loop iteration until the error is less than 100 milliseconds.
[0094] Finally, after stretching the speech rate of the core pronunciation segments, the algorithm performs beat-level "start and stop" fine-tuning for the tiny time gaps before and after audio blocks. Dynamically long silence data packets are automatically inserted at the beginning or between sentences for zero-padding. By controlling the duration of this silence, the "real acoustic starting point" of the translated audio is precisely anchored to the absolute timestamp of the original video frame, ensuring that the pronunciation fluctuations in the final product are visually and psychologically synchronized with the on-screen actions (such as PPT page turning, lecturer gestures, and camera transitions).
[0095] This method completely eliminates the cumulative timing deviation caused by slicing, ensuring that the final translated audio track is precisely limited to the "visual time window" corresponding to the original video, and perfectly avoids cross-scene overlap caused by sound overflow (such as: accurately matching visual nodes such as PPT page turning and speaker screen switching).
[0096] By performing time-series self-healing compensation on the truncated audio data, compressing or stretching the audio duration, adaptive scaling of speech rate is achieved. By inserting dynamically long silence data packets for zero padding, audio-visual synchronization between the synthesized speech and the original video is achieved.
[0097] The dynamic speech segmentation and temporal self-healing alignment method for long video translation provided in this embodiment has the following advantages compared with the prior art: (1) Completely solves the problem of memory overflow (OOM) in large model inference and effectively improves robustness. Through the unique dual forced security monitoring of "physical duration and character level" and "flexible backoff truncation" mechanism, it can automatically find the suboptimal silent gap within the safe level to melt down when faced with extreme uninterrupted speech of several minutes, fundamentally eliminating the overload of graphics card computing power and memory crash caused by excessively long single input sequence, and greatly improving the success rate of fully automatic processing of long videos (such as large conference records of more than 1 hour); (2) Maximizing the preservation of contextual integrity significantly reduces the "semantic illusion" in machine translation. Abandoning the traditional fixed-duration mechanical segmentation, the highest priority truncation condition is "detecting the final punctuation of natural language sentences." This intelligent game strategy ensures that the downstream machine translation model always receives structurally complete and logically coherent sentences, greatly reducing mistranslations, omissions, and machine illusions caused by fragmented word groups. When facing multilingual translation tasks with complex word order, it significantly improves the fluency and accuracy of the target translation. (3) A pioneering time-series self-healing mechanism perfectly eliminates accumulated clock deviations and the phenomenon of "audio track crosstalk". In response to the timestamp misalignment caused by repeated cutting of long videos, the mechanism automatically calculates the compensation factor by cross-comparing the semantic timestamp and the physical truncation timestamp, and links the downstream to adjust the speech rate and interval of the text-to-speech (TTS). This mechanism completely eliminates the common audio-visual disconnect problem in the industry, ensuring that no matter how the physical pronunciation duration of the target language changes, the newly generated translated speech can be accurately limited to the "visual time window" corresponding to the original video, perfectly matching visual nodes such as PPT page turning and speaker camera switching; (4) Optimize underlying resource scheduling to significantly improve the efficiency of end-to-end engineering delivery. The "adaptive acoustic feature pre-filtering" mechanism cleans up meaningless background noise in advance, avoiding the waste of GPU resources for concurrent requests from massive invalid fragments. At the same time, combined with the end-to-end state machine intelligent scheduling mechanism and media reuse strategy, the overall processing, merging and rendering time of a single task is greatly shortened, significantly reducing the computing cost of multilingual video matrix production.
[0098] This embodiment also provides a dynamic speech segmentation and temporal self-healing alignment device for long video translation. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0099] This embodiment provides a dynamic speech segmentation and temporal self-healing alignment device for long video translation, such as... Figure 3 As shown, it includes: The segmentation module 301 is used to set an audio energy sliding window and perform acoustic pre-segmentation on the audio data to be processed based on the audio energy sliding window. It then performs bonding, filtering, and extension on the pre-segmented audio data to be processed and outputs candidate speech islands. The monitoring module 302 is used to perform automatic speech recognition on candidate speech islands and monitor key parameters, including the cumulative processing time of the current segment, the total number of currently decoded characters, and the semantic completion status indicator. The truncation module 303 is used to set priority weights and evaluate key parameters based on the priority weights to perform truncation. The output module 304 is used to perform temporal cross-validation and temporal self-healing compensation on the truncated audio data and output synthesized speech.
[0100] In some alternative implementations, the segmentation module 301 includes: The first setting unit is used to set the basic environmental energy threshold and the number of frames for merging micro-pauses, and uses the basic environmental energy threshold and the number of frames for merging micro-pauses as the initialization parameters for the audio energy sliding window. The segmentation unit is used to start a frame counter when the audio signal energy of the audio data to be processed falls below the basic environmental energy threshold. When the accumulated frame count of the frame counter is greater than the small pause merging frame count parameter, the current position is marked as a physical breakpoint, and the audio data to be processed is segmented into segments according to the marked physical breakpoint.
[0101] In some optional implementations, the segmentation module 301 further includes: The second setting unit is used to set the tolerance threshold for minor pauses; The bonding unit is used to bond adjacent audio segments into a continuous whole when the duration of a silent segment between adjacent audio segments is less than the micro-pause tolerance threshold. The elimination unit is used to set a minimum speech threshold and eliminate segments whose speech threshold is less than the minimum speech threshold. The extension unit is used to extend outward from the beginning and end boundaries of a continuous audio segment to obtain candidate speech islands.
[0102] In some alternative implementations, the monitoring module 302 includes: The recording unit is used to record the physical time span from the start point of the current segment to the current processing frame, and the recording result is determined as the cumulative processing time of the current segment; The statistics unit is used to count the total number of text characters currently output by the automatic speech recognition engine and determine the statistical result as the total number of currently decoded characters. The matching unit is used to match whether the end of the current output text belongs to a strong semantic punctuation mark according to natural speech processing rules, and to determine the matching result as a semantic completion status identifier.
[0103] In some alternative implementations, the cutoff module 303 includes: The priority setting unit is used to set perfect semantic truncation as the first priority, flexible forced truncation as the second priority, and acoustic gap truncation as the third priority. The first truncation unit is used to perform perfect semantic truncation at the punctuation mark when the semantic completion status is true and the cumulative processing time of the current segment is greater than the minimum anti-fragmentation threshold. The second truncation unit is used to backtrack and find the suboptimal silent gap when the semantic completion status flag is not triggered and the current segment's cumulative processing time reaches the upper limit of the video memory safe time or the current total number of decoded characters reaches the upper limit of the translation model, and to perform flexible forced truncation. The third cutoff unit is used to cut off the acoustic gap when the first priority and second priority conditions are not met and the strong silence signal reaches the preset strong silence duration.
[0104] In some alternative implementations, the output module 304 includes: The timing offset calculation unit is used to calculate the timing offset according to the timing crossover compensation formula, which is as follows:
[0105] in, This is the timing offset. This refers to the semantic end point of the last word recognized by the automatic speech recognition model. This represents the actual time point at which the underlying physical truncation is performed. The confidence factor is dynamically calculated based on the audio signal-to-noise ratio.
[0106] In some alternative implementations, the output module 304 further includes: The first determining unit is used to determine the difference between the original visual time window and the temporal offset of the audio data to be processed in the original video as the strict target physical duration of the newly generated translated speech; The second determining unit is used to determine the ratio of the actual pronunciation duration obtained from actual measurement to the strict target physical duration of the newly generated translated speech as the temporal scaling factor; The splicing unit is used to find extreme points and similar segments of waveforms in the time domain by using the time scaling factor, and to overlap, discard, and smoothly splice the pitch period to compress or lengthen the duration of the audio to be processed to the strict target physical duration of the newly generated translated speech. The zero-padding unit is used to insert dynamically lengthed silence data packets into audio data whose duration reaches the strict target physical duration for generating the translated speech, thereby padding the true acoustic starting point of the translated audio onto the absolute timestamp of the original video frame.
[0107] The dynamic speech segmentation and temporal self-healing alignment device for long video translation provided in this embodiment of the invention can execute the dynamic speech segmentation and temporal self-healing alignment method for long video translation provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method. Further functional descriptions of the above modules and units are the same as in the corresponding embodiments described above, and will not be repeated here.
[0108] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.
[0109] The following is a detailed reference. Figure 4 This diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from memory 408 into random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the electronic device. The processor 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0110] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.
[0111] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 409, or installed from a memory 408, or installed from a ROM 402. When the computer program is executed by the processor 401, it performs the functions defined in the dynamic speech segmentation and temporal self-healing alignment method for long video translation according to embodiments of the present invention.
[0112] Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.
[0113] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the dynamic speech segmentation and temporal self-healing alignment method for long video translation shown in the above embodiments is implemented.
[0114] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.
[0115] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and all such modifications and variations fall within the scope defined by the appended invention.< / period>
Claims
1. A dynamic speech segmentation and temporal self-healing alignment method for long video translation, characterized in that, The method includes: An audio energy sliding window is set, and acoustic pre-segmentation is performed on the audio data to be processed based on the audio energy sliding window. The pre-segmented audio data to be processed is then glued, filtered, and extended to output candidate speech islands. Automatic speech recognition is performed on the candidate speech islands, and key parameters are monitored, including the cumulative processing time of the current segment, the total number of currently decoded characters, and the semantic completion status indicator. Priority weights are set, and the key parameters are evaluated based on these priority weights for truncation. The process of setting priority weights and evaluating the key parameters for truncation includes: setting perfect semantic truncation as the first priority, flexible forced truncation as the second priority, and acoustic gap truncation as the third priority; when the semantic completion status is true and the current segment's cumulative processing time exceeds the minimum anti-fragmentation threshold, perfect semantic truncation is performed at the punctuation mark; when the semantic completion status is not triggered, and the current segment's cumulative processing time reaches the upper limit of the video memory safety time or the current total number of decoded characters reaches the upper limit of the translation model, a reverse backtracking process is used to find a suboptimal silence gap for flexible forced truncation; when the first and second priority conditions are not met, and the strong silence signal reaches the preset strong silence time, acoustic gap truncation is performed. The truncated audio data undergoes temporal cross-validation and temporal self-healing compensation to output synthesized speech. The temporal cross-validation of the truncated audio data includes calculating the temporal offset according to a temporal cross-compensation formula, which is as follows: in, This is the timing offset. This refers to the semantic end point of the last word recognized by the automatic speech recognition model. This represents the actual time point at which the underlying physical truncation is performed. The confidence factor is dynamically calculated based on the audio signal-to-noise ratio. The truncated audio data undergoes temporal self-healing compensation, including: determining the difference between the original visual time window and the temporal offset of the audio data to be processed in the original video as the strict target physical duration of the newly generated translated speech; determining the ratio of the actual measured pronunciation duration to the strict target physical duration of the newly generated translated speech as the temporal scaling factor; using the temporal scaling factor, finding the extreme points and similar segments of the waveform in the time domain, overlapping, discarding, and smoothly splicing the pitch period to compress or lengthen the duration of the audio to be processed to the strict target physical duration of the newly generated translated speech; inserting dynamically lengthed silence data packets into the audio data whose duration reaches the strict target physical duration of the generated translated speech, padding with zeros, and fixing the true acoustic starting point of the translated audio to the absolute timestamp of the original video frame.
2. The method according to claim 1, characterized in that, The process of setting an audio energy sliding window and performing acoustic pre-segmentation of the audio data to be processed based on the audio energy sliding window includes: Set a basic environmental energy threshold and a parameter for merging small pauses, and use the basic environmental energy threshold and the parameter for merging small pauses as the initialization parameters for the audio energy sliding window; When the audio signal energy of the audio data to be processed is detected to fall below the basic environmental energy threshold, a frame counter is started. When the accumulated frame count of the frame counter is greater than the micro-pause merging frame count parameter, the current position is marked as a physical breakpoint, and the audio data to be processed is segmented into segments according to the marked physical breakpoint.
3. The method according to claim 1, characterized in that, The process of joining, filtering, and extending the acoustically pre-segmented audio data to obtain candidate speech islands includes: Set a tolerance threshold for minor pauses; When the duration of a silent segment between adjacent audio segments is less than the micro-pause tolerance threshold, the adjacent audio segments are glued together as a continuous whole. Set a minimum speech threshold and remove segments whose speech threshold is less than the minimum speech threshold; By extending outwards from the beginning and end boundaries of a continuous audio segment, candidate speech islands are obtained.
4. The method according to claim 1, characterized in that, The key monitoring parameters include: Record the physical time span from the start point of the current segment to the current processing frame, and determine the recorded result as the cumulative processing time of the current segment; The total number of text characters currently output by the automatic speech recognition engine is counted, and the result is determined as the total number of currently decoded characters. Based on natural speech processing rules, the system matches whether the end of the current output text is a strong semantic punctuation mark, and determines the matching result as a semantic completion status identifier.
5. A dynamic speech segmentation and temporal self-healing alignment device for long video translation, characterized in that, The device includes: The segmentation module is used to set an audio energy sliding window and perform acoustic pre-segmentation on the audio data to be processed based on the audio energy sliding window. The pre-segmented audio data to be processed is then glued, filtered, and extended to output candidate speech islands. The monitoring module is used to perform automatic speech recognition on the candidate speech islands and monitor key parameters, including the current segment's cumulative processing time, the current total number of decoded characters, and the semantic completion status indicator. The truncation module is used to set priority weights, evaluate the key parameters according to the priority weights, and perform truncation. Specifically, the truncation module is used to: set perfect semantic truncation as the first priority, flexible forced truncation as the second priority, and acoustic gap truncation as the third priority; when the semantic completion status flag is true and the cumulative processing time of the current segment is greater than the minimum anti-fragmentation threshold, perform perfect semantic truncation at the punctuation mark; when the semantic completion status flag is not triggered and the cumulative processing time of the current segment reaches the upper limit of the video memory safe time or the total number of currently decoded characters reaches the upper limit of the translation model, backtrack to find the suboptimal silence gap and perform flexible forced truncation; when the first priority and second priority conditions are not met and the strong silence signal reaches the preset strong silence time, perform acoustic gap truncation. The output module is used to perform temporal cross-validation and temporal self-healing compensation on the truncated audio data and output synthesized speech. The output module is specifically used for: performing temporal cross-checking on the truncated audio data, including: calculating the timing offset according to the temporal cross-compensation formula, which is as follows: in, This is the timing offset. This refers to the semantic end point of the last word recognized by the automatic speech recognition model. This represents the actual time point at which the underlying physical truncation is performed. The confidence factor is dynamically calculated based on the audio signal-to-noise ratio. The output module is specifically used for: performing time-series self-healing compensation on the truncated audio data, including: The difference between the original visual time window and the temporal offset of the audio data to be processed in the original video is determined as the strict target physical duration of the newly generated translated speech; the ratio of the actual pronunciation duration obtained by actual measurement to the strict target physical duration of the newly generated translated speech is determined as the temporal scaling factor; using the temporal scaling factor, extreme points and similar segments of the waveform are found in the time domain, and the pitch period is overlapped, discarded, and smoothly spliced to compress or lengthen the duration of the audio to be processed to the strict target physical duration of the newly generated translated speech; dynamic length silence data packets are inserted into the audio data whose duration reaches the strict target physical duration of the generated translated speech, and zero padding is performed to fix the true acoustic starting point of the translated audio on the absolute timestamp of the original video frame.
6. An electronic device, characterized in that, include: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes the computer instructions to perform the dynamic speech segmentation and temporal self-healing alignment method for long video translation as described in any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the dynamic speech segmentation and temporal self-healing alignment method for long video translation as described in any one of claims 1 to 4.
Citation Information
Patent Citations
Intelligent collaborative translation method of translation device for multi-person session and translation device
CN122221883A
Joint Endpointing And Automatic Speech Recognition
US20200335091A1
