A method, system, and program product for tail lag optimization for speech translation
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN TIMEKETTLE TECH CO LTD
- Filing Date
- 2026-05-09
- Publication Date
- 2026-08-07
AI Technical Summary
然而，现有的语音翻译系统由于语言结构的差异，不同语言之间的词长比例差异大，导致目标语言合成时长不可控
[0014] Thirdly, embodiments of this application provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the method described above.
Smart Images
Figure CN122531355A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of speech synthesis and artificial intelligence, and more specifically, to a speech translation tail-end delay optimization method, system, and program product. Background Technology
[0002] Currently, real-time speech translation technology is widely used in various fields such as international conferences, cross-border customer service, and public speeches. Its core purpose is to achieve real-time cross-language communication through a pipelined process of speech recognition, machine translation, and speech synthesis. However, due to differences in language structure and significant variations in word length ratios between different languages, existing speech translation systems suffer from uncontrollable target language synthesis time. In sentence-by-sentence translation, the playback delay of the target language accumulates over time, ultimately resulting in a poor user experience. Furthermore, simply increasing the playback rate reduces intelligibility and lacks flexibility and scalability, making it difficult to meet current demands.
[0003] Therefore, there is an urgent need for a tail-cast delay optimization scheme that can effectively suppress delay accumulation while ensuring semantic integrity. Summary of the Invention
[0004] The purpose of this application is to provide a speech translation tail-end delay optimization method, system and program product, which is based on a collaborative strategy of dynamic buffering, on-demand synthesis, semantic compression and adaptive speed adjustment to effectively suppress delay accumulation and has the advantages of strong stability, high versatility and controllable delay.
[0005] Firstly, this application provides a method for optimizing the delay of voice translation tail-end playback, including the following steps: At least one text to be synthesized is stored in a buffer; wherein each text to be synthesized is a clause split from the audio to be translated; Iterate through each text to be synthesized in the buffer and estimate the end time of playback of the target language audio corresponding to the current text to be synthesized; If the time difference between the playback end time and the current time is less than a preset time difference threshold, the translation speech synthesis will continue for the subsequent text to be synthesized from the current text to be synthesized. If the time difference between the playback end time and the current time is greater than or equal to the preset time difference threshold, the traversal of the buffer ends.
[0006] Optionally, in the speech translation tail-end delay optimization method described in this application, the at least one text to be synthesized is obtained by performing the following operations: The audio to be translated is subjected to speech preprocessing to obtain effective audio; wherein, speech preprocessing includes one or more of speech activity detection, noise reduction and redundancy removal. The effective audio input is processed by the streaming speech recognition engine to obtain streaming text with sentence segmentation markers; Based on the semantic analysis results of the streaming text, the streaming text is split into at least one clause to obtain at least one text to be synthesized.
[0007] Optionally, in the speech translation tail-end delay optimization method described in this application, after storing at least one text to be synthesized in the buffer, it further includes: If the cumulative original audio duration of at least one text to be synthesized is greater than a first preset duration threshold, semantic compression is performed on each text to be synthesized in the at least one text to be synthesized; wherein, the cumulative original audio duration is determined by combining the start time and end time of the audio segment to be translated corresponding to each text to be synthesized; And / or, If the cumulative text length of at least one text to be synthesized is greater than a preset length threshold, semantic compression is performed on each text to be synthesized.
[0008] Optionally, in the speech translation tail-end delay optimization method described in this application, the semantic compression of each text to be synthesized includes: For each of the at least one texts to be synthesized, text preprocessing is performed on the texts to be synthesized to obtain valid text; wherein, the text preprocessing includes deduplication and / or redundancy removal. A large language model is used to extract semantic information from the effective text based on the constraints input by the user, and to generate a simplified text corresponding to the text to be synthesized. If the semantic similarity between the simplified text and the text to be synthesized is determined to be greater than a preset similarity threshold, the text to be synthesized is updated to the simplified text.
[0009] Optionally, in the speech translation end-of-play delay optimization method described in this application, the estimation of the end time of playback of the target language audio corresponding to the current text to be synthesized includes: The playback end time of the target language audio corresponding to the current text to be synthesized is determined based on the current time, the playback end time of the target language audio corresponding to the previous text to be synthesized, and the playback duration of the target language audio corresponding to the current text to be synthesized.
[0010] Optionally, in the speech translation tail-end delay optimization method described in this application, the step of performing translation speech synthesis on the subsequent text to be synthesized from the current text to be synthesized includes: If the remaining original audio duration of the subsequent text to be synthesized is greater than a second preset duration threshold, the summary information of the subsequent text to be synthesized is used for speech synthesis; wherein, the remaining original audio duration is determined by combining the start and end times of the audio segments to be translated corresponding to each text to be synthesized in the subsequent text to be synthesized; If the remaining original audio duration is less than or equal to the second preset duration threshold, the first text to be synthesized in the subsequent text to be synthesized is translated into speech.
[0011] Optionally, in the speech translation end-of-play delay optimization method described in this application, before estimating the end time of playback of the target language audio corresponding to the current text to be synthesized, the method further includes: If the current accumulated latency is greater than a preset latency threshold, the acceleration ratio is determined based on the current accumulated latency; wherein, the current accumulated latency is determined based on the current playable duration, the current time, and the start time of the audio segment to be translated corresponding to the first text to be synthesized in the at least one text to be synthesized; The current text to be synthesized is accelerated according to the acceleration ratio to obtain the target language audio corresponding to the current text to be synthesized.
[0012] Optionally, in the speech translation tail-end delay optimization method described in this application, the speedup factor is: ; Where S is the acceleration factor, L is the current cumulative delay, Z is the preset maximum tolerable delay, and k is the preset acceleration gain coefficient.
[0013] Secondly, this application provides a voice translation tail-end delay optimization system, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements the method described above.
[0014] Thirdly, embodiments of this application provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the method described above.
[0015] As described above, the speech translation end-of-stream latency optimization method, system, and program product provided in this application splits the audio to be translated into several clause texts after speech preprocessing, streaming recognition, and semantic segmentation, and stores them in a buffer. By traversing the text in the buffer, the playback end time of the target language audio corresponding to each text to be synthesized is estimated. Based on whether the time difference between the playback end time and the current time meets a preset threshold, the system controls whether to continue synthesizing subsequent translated speech. Simultaneously, semantic compression is triggered by combining the cumulative duration and text length, and the speech synthesis acceleration rate is dynamically adjusted according to the cumulative latency, thereby optimizing the end-of-stream latency of the streaming translation process. This application can significantly reduce the end-of-stream latency of speech translation, improve the real-time performance and smoothness of translation playback, and is suitable for real-time speech translation scenarios.
[0016] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. Attached Figure Description
[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 A flowchart illustrating a speech translation tail-end delay optimization method provided in this application embodiment; Figure 2 A flowchart illustrating a speech translation tail-end delay optimization method provided in this application embodiment; Figure 3 This is a schematic diagram of the target audio synthesis, delivery, and closed-loop update process of a speech translation tail-end delay optimization method provided in an embodiment of this application. Detailed Implementation
[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0020] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0021] Please refer to Figure 1 , Figure 1 This is a flowchart of a speech translation end-of-stream latency optimization method according to some embodiments of this application. This speech translation end-of-stream latency optimization method is used in terminal devices, such as computers and mobile phones. A speech translation end-of-stream latency optimization method includes the following steps: S11. Store at least one text to be synthesized in a buffer; wherein each text to be synthesized in the at least one text to be synthesized is a clause split from the audio to be translated; S12. Traverse each text to be synthesized in the buffer and estimate the end time of playback of the target language audio corresponding to the current text to be synthesized. S13. If the time difference between the playback end time and the current time is less than a preset time difference threshold, continue to perform translation speech synthesis on the subsequent text to be synthesized of the current text to be synthesized. S14. If the time difference between the playback end time and the current time is greater than or equal to the preset time difference threshold, the traversal of the buffer ends.
[0022] As an example, complete clauses are pre-splitting from the audio to be translated as text to be synthesized, resulting in at least one text to be synthesized.
[0023] In practical applications, the audio to be translated can be the complete audio generated by the user in a cross-language communication scenario, which can be obtained in real time through the microphone configured on the terminal device itself or through external audio acquisition devices.
[0024] It should be noted that complete audio refers to the original speech stream continuously generated by a user in a cross-language communication scenario, which still contains valid speech and short pauses without being segmented or processed. A complete clause refers to the smallest linguistic unit that is semantically consistent, grammatically complete, and capable of independently expressing a clear meaning.
[0025] A buffer data structure is constructed with the text to be synthesized as the smallest storage unit and the original audio timestamp associated with the text to be synthesized as the timing reference, to stably store all the text to be synthesized obtained from the audio to be translated.
[0026] It should be noted that the original audio timestamp may include the original audio start time and / or the original audio end time. The original audio start time is the start time when the system receives the audio segment to be translated corresponding to the text to be synthesized, and the original audio end time is the end time when the system receives the audio segment to be translated corresponding to the text to be synthesized.
[0027] In practical applications, the Voice Activity Detection (VAD) model can be used to determine the start time of the original voice (accurate to milliseconds) when valid voice is detected in the audio segment to be translated, and to determine the end time of the original voice (accurate to milliseconds) when a silent environment is detected later.
[0028] According to the buffer data structure, at least one text to be synthesized is stored in the buffer.
[0029] The system sequentially traverses each text to be synthesized in the buffer. For the current text to be synthesized, it estimates the end time of playback of the target language audio corresponding to the current text after translation and speech synthesis.
[0030] The estimated end time of playback is calculated by comparing it with the current system time, and the resulting time difference is compared with a preset time difference threshold.
[0031] In practical applications, the preset time difference threshold can be set to 4 seconds, or it can be dynamically adjusted according to network conditions and scenario requirements.
[0032] When the time difference is less than the preset time difference threshold, it is assumed that the target language audio corresponding to the current text to be synthesized will not be delayed and will not affect the playback of subsequent audio. At this time, the translation and speech synthesis of subsequent texts to be synthesized after the current text to be synthesized will continue to be performed in batches to avoid playback interruption.
[0033] When the time difference is greater than or equal to the preset time difference threshold, it is assumed that the target language audio corresponding to the text to be synthesized will be delayed, which may affect the subsequent audio playback. At this time, the buffer traversal is immediately terminated and no additional synthesis is performed to prevent excessive synthesis from causing audio accumulation and continuous accumulation of delay. In the end, an on-demand synthesis core framework is formed that is dynamically scheduled based on the real-time playback status of the client, thus eliminating the problem of linear growth of tail playback delay caused by the traditional real-time synthesis mode.
[0034] In an optional embodiment, the at least one text to be synthesized is obtained by performing the following operations: The audio to be translated is subjected to speech preprocessing to obtain effective audio; wherein, speech preprocessing includes one or more of speech activity detection, noise reduction and redundancy removal. The effective audio input is processed by the streaming speech recognition engine to obtain streaming text with sentence segmentation markers; Based on the semantic analysis results of the streaming text, the streaming text is split into at least one clause to obtain at least one text to be synthesized.
[0035] As an example, considering that the audio to be translated usually contains invalid audio such as short pauses, background noise and repeated speech, speech preprocessing can be performed on the audio to be translated to remove invalid audio and obtain valid audio. Speech preprocessing includes one or more of speech activity detection, noise reduction and redundancy removal.
[0036] In practical applications, speech activity detection can employ a deep learning-based VAD model, identifying valid speech by setting speech start and silence end thresholds. Noise reduction can utilize spectral subtraction, deep learning noise reduction models, or other noise reduction algorithms to remove background noise and prevent it from interfering with subsequent speech recognition. Redundancy removal can filter out "micro-pauses" (such as pauses ≤300ms, often breaths in spoken language) in the audio to be translated, retaining "valid pauses" (≥500ms, often sentence breaks), thus preventing fragmented text from being generated in subsequent speech recognition.
[0037] The valid audio is input into the Automatic Speech Recognition (ASR) engine, which then converts the valid audio into streaming text with punctuation marks, such as streaming text with punctuation.
[0038] It should be noted that the streaming ASR engine configuration uses the "streaming speech recognition" mode, which means that the effective audio is segmented according to the set segment length, such as 200ms / segment, and speech recognition is performed on each segment, rather than the entire effective audio. This allows the streaming ASR engine to output the recognized text segments in real time. At the same time, the sentence segmentation marking function of the streaming ASR engine is enabled, so that the streaming ASR engine automatically adds sentence segmentation marks when it recognizes obvious pauses and terminators, providing a preliminary basis for subsequent segmentation.
[0039] Semantic analysis is performed on the streaming text to obtain the semantic analysis results. Combining the sentence segmentation markers in the streaming text with the semantic analysis results, the streaming text is split into at least one complete clause, thereby obtaining at least one text to be synthesized.
[0040] In practical applications, text segments can be initially identified by matching punctuation marks representing semantic termination in each language using regular expressions. Then, natural language processing tools, such as the Language Technology Platform (LTP) which focuses on Chinese or the multilingual general-purpose library spaCy, can be used for dependency parsing. Lightweight artificial intelligence (AI) models such as MiniLM can be combined to predict the probability of semantic termination, and semantically and grammatically complete clauses can be selected. Finally, the complete clauses are used as the text to be synthesized, and the original sound timestamps determined in the previous VAD detection stage are bound to the text to be synthesized. At least one text to be synthesized is stored in the buffer in the order of the original sound timestamps from earliest to latest.
[0041] By adopting a dual semantic analysis mechanism of "rule engine + AI semantic verification", complete clauses can be accurately split, avoiding "overly fine splitting (fragmentation)" or "overly coarse splitting (long sentence accumulation)".
[0042] In an optional embodiment, after storing at least one text to be synthesized in the buffer, the method further includes: If the cumulative original audio duration of at least one text to be synthesized is greater than a first preset duration threshold, semantic compression is performed on each text to be synthesized in the at least one text to be synthesized; wherein, the cumulative original audio duration is determined by combining the start time and end time of the audio segment to be translated corresponding to each text to be synthesized; And / or, If the cumulative text length of at least one text to be synthesized is greater than a preset length threshold, semantic compression is performed on each text to be synthesized.
[0043] As an example, after storing the at least one text to be synthesized in the buffer, for each text to be synthesized in the buffer, the original duration of a single sentence can be calculated based on the start and end times of the audio segment to be translated corresponding to the text to be synthesized. The original duration of all single sentences can be accumulated to obtain the cumulative original duration. The cumulative original duration can be compared with a first preset duration threshold to determine whether semantic compression should be enabled based on the comparison result.
[0044] When the cumulative original audio duration exceeds the first preset duration threshold, it is considered that the text to be synthesized in the buffer is seriously piled up, and the synthesis playback duration is likely to be too long. At this time, the system automatically triggers semantic compression processing on each text to be synthesized in the buffer to generate simplified text that retains the core semantic information.
[0045] When the cumulative original audio duration is less than or equal to the first preset duration threshold, it is assumed that there is no accumulation of text to be synthesized in the buffer, and the synthesized playback duration is usually reasonable. At this time, the system maintains the current state and there is no need to trigger semantic compression processing.
[0046] Alternatively, for each text to be synthesized in the buffer, the length of a single sentence can be counted, the lengths of all single sentences can be summed to obtain the cumulative text length, and the cumulative text length can be compared with a preset length threshold to determine whether semantic compression should be enabled based on the comparison result.
[0047] When the cumulative text length exceeds the preset length threshold, it is considered that the text to be synthesized in the buffer is seriously piled up, and the synthesis playback time is likely to be too long. At this time, the system automatically triggers semantic compression processing on each text to be synthesized in the buffer to generate simplified text that retains the core semantic information.
[0048] When the cumulative text length is less than or equal to the preset length threshold, it is assumed that there is no accumulation of text to be synthesized in the buffer, and the synthesis playback duration is usually reasonable. At this time, the system maintains the current state and there is no need to trigger semantic compression processing.
[0049] By directly shortening the length of the text to be synthesized when there is a serious backlog of text in the buffer, the speech synthesis time and playback time can be reduced, forming a key technical means to alleviate end-play delay from the text dimension.
[0050] In an optional embodiment, the semantic compression of each text to be synthesized includes: For each of the at least one texts to be synthesized, text preprocessing is performed on the texts to be synthesized to obtain valid text; wherein, the text preprocessing includes deduplication and / or redundancy removal. A large language model is used to extract semantic information from the effective text based on the constraints input by the user, and to generate a simplified text corresponding to the text to be synthesized. If the semantic similarity between the simplified text and the text to be synthesized is determined to be greater than a preset similarity threshold, the text to be synthesized is updated to the simplified text.
[0051] As an example, after the system triggers semantic compression of each text to be synthesized in the buffer, for each text to be synthesized, considering that the text to be synthesized may contain repeated words and redundant words (such as words with similar meanings, modal particles, etc.), the text to be synthesized can first be preprocessed to remove these invalid texts and obtain valid text. The text preprocessing includes deduplication and / or deredundancy processing.
[0052] After obtaining the valid text, the user can determine the constraint instructions according to the actual business needs, input the valid text and constraint instructions into the large language model, so that the large language model can extract semantic information from the valid text according to the constraint instructions input by the user and generate the simplified text corresponding to the text to be synthesized.
[0053] After obtaining the simplified text, the semantic similarity between the simplified text and the text to be synthesized is evaluated, and the semantic similarity is compared with a preset similarity threshold.
[0054] When the semantic similarity is greater than the preset similarity threshold, it is considered that the semantic compression is appropriate and will not affect the understanding of the text. At this time, the text to be synthesized is updated to the simplified text.
[0055] When the semantic similarity is less than or equal to a preset similarity threshold, it is considered that the semantic compression is excessive and affects text understanding. In this case, the semantic compression process can be re-executed on the text to be synthesized until the semantic similarity between the newly obtained simplified text and the text to be synthesized meets the requirements. If a simplified text with a satisfactory semantic similarity is still not obtained after N retries, the semantic compression process will no longer be performed, and the original text to be synthesized will be used for subsequent translation and speech synthesis processing to ensure the stability and real-time performance of the system under extreme conditions. Here, N is the preset maximum number of retries, which can be flexibly configured according to the actual deployment scenario and latency tolerance.
[0056] For example, the specific execution process of semantic compression is as follows: First, text preprocessing identifies and removes repeated words, filler words, and redundant colloquial expressions from the text to be synthesized to obtain effective text; then, the effective text is input into a large language model, and refined in conjunction with user-input constraints, extracting key semantic information from the effective text to generate corresponding simplified text. A GPT-3.5 Turbo quantized version or Llama 2 is used as the semantic compression model, and constraints are issued: "As a real-time translation and refinement engine, only redundant content is removed; the subject, core verbs, object, and emotional intent of the original text must not be removed or changed; no new information outside the original text is introduced," and a low-temperature coefficient is configured to ensure output stability; finally, semantic vector similarity is checked to calculate the semantic similarity between the simplified text and the original text to be synthesized. Only when the semantic similarity is greater than a preset similarity threshold is the text to be synthesized updated to the simplified text, ensuring that the compressed text is highly consistent with the original text at the semantic level, avoiding the loss of key semantics due to compression. If the semantic similarity does not meet the requirements, the semantic compression process is repeated on the text to be synthesized until the semantic similarity between the latest simplified text and the text to be synthesized meets the requirements. Then the text to be synthesized is updated to the latest simplified text.
[0057] In an optional embodiment, estimating the end time of playback of the target language audio corresponding to the current text to be synthesized includes: The playback end time of the target language audio corresponding to the current text to be synthesized is determined based on the current time, the playback end time of the target language audio corresponding to the previous text to be synthesized, and the playback duration of the target language audio corresponding to the current text to be synthesized.
[0058] As an example, the system collects three types of core time-series data in real time: the current time of the system, the end time of the playback of the target language audio corresponding to the previous text to be synthesized, and the playback duration of the target language audio corresponding to the current text to be synthesized. In practical applications, factors such as the current time of the system, the playback status of the target language audio corresponding to the previous text to be synthesized, and the synthesis and playback duration of the target language audio corresponding to the current text to be synthesized can be comprehensively considered to accurately predict the end time of the playback of the target language audio corresponding to the current text to be synthesized after translation and speech synthesis.
[0059] In practical applications, the `max` function can be used to select the larger value between the current time and the playback end time of the target language audio corresponding to the previous text to be synthesized. This ensures that the playback timing is continuous without backtracking or overlap. Then, the playback duration of the target language audio corresponding to the current text to be synthesized is added to obtain an accurate and reliable estimated playback end time. This provides a unique, stable, and quantifiable timing judgment basis for subsequent on-demand synthesis traversal. For example, the playback end time of the target language audio corresponding to the current text to be synthesized can be estimated using the following formula: ; in, This represents the end time of playback of the target language audio corresponding to the current text to be synthesized. For the current moment, This is the end time of playback of the target language audio corresponding to the previous text to be synthesized. The playback duration of the target language audio corresponding to the current text to be synthesized is denoted as n, where n represents the sequence number of the current text to be synthesized and n-1 represents the sequence number of the previous text to be synthesized.
[0060] In an optional embodiment, the translation speech synthesis of subsequent texts to be synthesized from the current text to be synthesized includes: If the remaining original audio duration of the subsequent text to be synthesized is greater than a second preset duration threshold, the summary information of the subsequent text to be synthesized is used for speech synthesis; wherein, the remaining original audio duration is determined by combining the start and end times of the audio segments to be translated corresponding to each text to be synthesized in the subsequent text to be synthesized; If the remaining original audio duration is less than or equal to the second preset duration threshold, the first text to be synthesized in the subsequent text to be synthesized is translated into speech.
[0061] As an example, after determining to continue synthesizing subsequent text to be synthesized, the system first calculates the remaining original audio duration of the subsequent text to be synthesized by combining the start and end times of the audio segments to be translated corresponding to each text to be synthesized in the subsequent text to be synthesized, compares the remaining original audio duration with the second preset duration threshold, and determines the synthesis range and strategy based on the comparison result.
[0062] When the remaining original audio duration exceeds the second preset duration threshold, the load is considered normal. At this time, the summary information of the subsequent text to be synthesized is extracted for translation speech synthesis, so as to preserve the core semantics while significantly compressing the synthesis duration.
[0063] When the remaining original audio duration is less than or equal to the second preset duration threshold, it is considered that there is too much subsequent text to be synthesized and the risk of playback delay is high. Only the first text to be synthesized in the subsequent text to be synthesized is translated and synthesized normally without additional compression, so as to balance synthesis efficiency, system resource consumption, playback smoothness and semantic integrity.
[0064] In an optional embodiment, before estimating the end time of playback of the target language audio corresponding to the current text to be synthesized, the method further includes: If the current accumulated latency is greater than a preset latency threshold, the acceleration ratio is determined based on the current accumulated latency; wherein, the current accumulated latency is determined based on the current playable duration, the current time, and the start time of the audio segment to be translated corresponding to the first text to be synthesized in the at least one text to be synthesized; The current text to be synthesized is accelerated according to the acceleration ratio to obtain the target language audio corresponding to the current text to be synthesized.
[0065] As an example, before translating and synthesizing the current text to be synthesized to obtain the target language audio corresponding to the current text to be synthesized, the system can calculate the current cumulative latency in real time based on the current playable duration, the current time, and the start time of the audio segment to be translated corresponding to the first text to be synthesized in the buffer.
[0066] It should be noted that the current playable duration = the system's maximum playable duration - the actual playback duration of the target language audio corresponding to the text to be synthesized.
[0067] The current accumulated latency is compared with a preset latency threshold, and the voice acceleration is activated based on the comparison result.
[0068] When the current cumulative latency exceeds the preset latency threshold, it is considered that the playback delay has affected the user experience. At this time, the acceleration ratio is determined based on the current cumulative latency, and adaptive accelerated speech synthesis is immediately triggered. The current text to be synthesized is translated and accelerated speech synthesis is performed according to the acceleration ratio to obtain the target language audio corresponding to the current text to be synthesized.
[0069] When the current cumulative latency is less than or equal to the preset latency threshold, the translation and speech synthesis of the current text to be synthesized are performed at the standard 1.0x speed, achieving precise control of "no acceleration without latency, acceleration only when there is latency", ensuring low latency while maintaining the naturalness of the speech to the greatest extent.
[0070] In practical applications, the preset delay threshold can be 3 seconds.
[0071] In an optional embodiment, the acceleration factor is: ; Where S is the acceleration factor, L is the current cumulative delay, Z is the preset maximum tolerable delay, and k is the preset acceleration gain coefficient.
[0072] In practical applications, the value of k can be any value between 0.5 and 1.2, and can be dynamically adjusted according to the language. For example, Chinese and English can be set to 1.0, while agglutinative languages such as Japanese and Korean can be set to 1.2, which can adapt to the speech expression characteristics of different languages. The preset maximum tolerable delay is the maximum delay duration that the system can accept, so as to avoid delay out of control.
[0073] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a speech translation tail-end delay optimization method in some embodiments of this application.
[0074] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the target audio synthesis, delivery, and closed-loop update process of a speech translation tail-end delay optimization method in some embodiments of this application.
[0075] Some embodiments of this application also provide a voice translation tail-end delay optimization system, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the method described above and achieves the same beneficial effects.
[0076] Some embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the method described above and achieves the same beneficial effects.
[0077] This invention discloses a method, system, and program product for optimizing end-of-stream latency in speech translation. The method involves preprocessing the audio to be translated, performing streaming recognition, and semantic segmentation to break it down into several clause texts, which are then stored in a buffer. By traversing the buffer texts, the method estimates the end time of the target language audio corresponding to each text to be synthesized. Based on whether the time difference between the end time and the current time meets a preset threshold, it controls whether to continue synthesizing subsequent translated speech. Simultaneously, semantic compression is triggered by combining cumulative duration and text length, and the speech synthesis acceleration rate is dynamically adjusted based on the cumulative latency, thereby optimizing the end-of-stream latency in the streaming translation process. This application can significantly reduce the end-of-stream latency in speech translation, improve the real-time performance and smoothness of translation playback, and is suitable for real-time speech translation scenarios.
[0078] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.
[0079] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.
[0080] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0081] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0082] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.
Claims
1. A method for optimizing tail-end delay in speech translation, characterized in that, include: At least one text to be synthesized is stored in a buffer; wherein each text to be synthesized is a clause split from the audio to be translated; Iterate through each text to be synthesized in the buffer and estimate the end time of playback of the target language audio corresponding to the current text to be synthesized; If the time difference between the playback end time and the current time is less than a preset time difference threshold, the translation speech synthesis will continue for the subsequent text to be synthesized from the current text to be synthesized. If the time difference between the playback end time and the current time is greater than or equal to the preset time difference threshold, the traversal of the buffer ends.
2. The speech translation tail-end delay optimization method according to claim 1, characterized in that, The at least one text to be synthesized is obtained by performing the following operations: The audio to be translated is subjected to speech preprocessing to obtain effective audio; wherein, speech preprocessing includes one or more of speech activity detection, noise reduction and redundancy removal. The effective audio input is processed by the streaming speech recognition engine to obtain streaming text with sentence segmentation markers; Based on the semantic analysis results of the streaming text, the streaming text is split into at least one clause to obtain at least one text to be synthesized.
3. The speech translation tail-end delay optimization method according to claim 1, characterized in that, After storing at least one text to be synthesized in the buffer, the method further includes: If the cumulative original audio duration of at least one text to be synthesized is greater than a first preset duration threshold, semantic compression is performed on each text to be synthesized in the at least one text to be synthesized; wherein, the cumulative original audio duration is determined by combining the start time and end time of the audio segment to be translated corresponding to each text to be synthesized; And / or, If the cumulative text length of at least one text to be synthesized is greater than a preset length threshold, semantic compression is performed on each text to be synthesized.
4. The speech translation tail-end delay optimization method according to claim 3, characterized in that, The semantic compression of each text to be synthesized includes: For each of the at least one texts to be synthesized, text preprocessing is performed on the texts to be synthesized to obtain valid text; wherein, the text preprocessing includes deduplication and / or redundancy removal. A large language model is used to extract semantic information from the effective text based on the constraints input by the user, and to generate a simplified text corresponding to the text to be synthesized. If the semantic similarity between the simplified text and the text to be synthesized is determined to be greater than a preset similarity threshold, the text to be synthesized is updated to the simplified text.
5. The speech translation tail-end delay optimization method according to claim 1, characterized in that, The estimated end time of playback of the target language audio corresponding to the current text to be synthesized includes: The playback end time of the target language audio corresponding to the current text to be synthesized is determined based on the current time, the playback end time of the target language audio corresponding to the previous text to be synthesized, and the playback duration of the target language audio corresponding to the current text to be synthesized.
6. The speech translation tail-end delay optimization method according to claim 1, characterized in that, The process of translating and synthesizing subsequent texts of the current text to be synthesized includes: If the remaining original audio duration of the subsequent text to be synthesized is greater than a second preset duration threshold, the summary information of the subsequent text to be synthesized is used for speech synthesis; wherein, the remaining original audio duration is determined by combining the start and end times of the audio segments to be translated corresponding to each text to be synthesized in the subsequent text to be synthesized; If the remaining original audio duration is less than or equal to the second preset duration threshold, the first text to be synthesized in the subsequent text to be synthesized is translated into speech.
7. The speech translation tail-end delay optimization method according to any one of claims 1 to 6, characterized in that, Before the estimated end time of playback of the target language audio corresponding to the current text to be synthesized, the following is also included: If the current accumulated latency is greater than a preset latency threshold, the acceleration ratio is determined based on the current accumulated latency; wherein, the current accumulated latency is determined based on the current playable duration, the current time, and the start time of the audio segment to be translated corresponding to the first text to be synthesized in the at least one text to be synthesized; The current text to be synthesized is accelerated according to the acceleration ratio to obtain the target language audio corresponding to the current text to be synthesized.
8. The speech translation tail-end delay optimization method according to claim 7, characterized in that, The acceleration factor is: ； Where S is the acceleration factor, L is the current cumulative delay, Z is the preset maximum tolerable delay, and k is the preset acceleration gain coefficient.
9. A voice translation tail-end delay optimization system, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method of any one of claims 1 to 8.
10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.