Voice, picture and subtitle alignment method and system for real-time voice translation synthesis

Abstract

The invention provides a voice, picture and subtitle alignment method and system for real-time voice translation synthesis, and relates to the technical field of voice processing. According to the method, millisecond alignment of translated voice and video is realized through fragment-level parallel processing, compared with a traditional scheme, the delay is greatly reduced, and the problem that translated voice pictures are asynchronous during live broadcast is solved; the method comprises the following steps: automatically generating real-time voice translation according to live broadcast content, and respectively processing to obtain a corresponding srt file and a translated m3u8 fragment; therefore, there is no need to manually translate and generate subtitles in advance, and there is no need to replace the audio of the original video in advance, thereby saving the manpower cost, and providing a more universal live broadcast stream subtitle scheme. Besides, when the languages are switched, the player can automatically select the starting time point, so that the method supports multiple playing modes such as online live broadcast and live broadcast playback.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a method and system for aligning audio, video and subtitles in real-time speech translation synthesis. Background Art

[0002] As the internet and live video streaming technology become increasingly sophisticated, live streaming is being used more and more widely in work and study, especially in the field of education, where online live courses are becoming popular. Users who watch live broadcasts may come from different countries and speak different languages. With the surge in global demand for online courses, multinational users face language barriers, and existing solutions have shortcomings: Problems with missing and delayed voice translation: Current solutions supporting voice translation have significant audio delays, resulting in a serious disconnect between the translated voice and the lecturer's lip movements and courseware switching. When teachers conduct experimental demonstrations in live educational broadcasts, the time difference between the voice explanation and the translated audio will directly lead to misunderstanding among students.

[0003] Weak audio, video, and subtitle synchronization mechanisms: Traditional live broadcast architectures use a single timeline alignment strategy, which cannot adapt to the dynamic latency in multilingual scenarios. When the system performs cascade processing of automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS), the cumulative latency error across each link can reach 3-5 seconds.

[0004] Therefore, the present invention provides a method and system for aligning audio, video and subtitles using real-time speech translation synthesis to solve the technical problems of speech translation omission and delay, as well as weak synchronization mechanism of audio, video and subtitles in the prior art. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a method and system for aligning audio and video subtitles in real-time speech translation synthesis. Through slice-level parallel processing, millisecond-level alignment of translated speech and video is achieved. Dynamic timestamp remapping technology is used to automatically adjust the TTS speaking speed or fill silence, ensuring that each translated audio segment accurately matches the original video duration.

[0006] To achieve the above objectives, the present application proposes a method for aligning audio, video and subtitles synthesized by real-time speech translation, which is used to align subtitles of audio, video and subtitles synthesized by real-time speech translation in a live broadcast scenario, comprising the following steps: Step 1: Record the live content into m3u8 segments and cache the live content; Step 2: Perform speech recognition translation on the live content cache, obtain the translated content and record the timestamp information; Step 3: Save the translated content into srt files with the same name according to the m3u8 segments one by one; Step 4: Perform speech translation synthesis based on the translation content to obtain real-time speech translation; Step 5: According to the timestamp information corresponding to the translation content, adjust the real-time speech translation and the timestamp in the corresponding SRT file to align at the millisecond level; Step 6: Merge the generated real-time speech translation with the video content cache and save it as a translated m3u8 fragment that corresponds one-to-one with the included m3u8 fragment; Step 7: Generate and continuously update by repeating steps 1 to 6 until all m3u8 fragments are converted into corresponding translated m3u8 fragments; Step 8: Get the translation language set by the user, and according to the predetermined switching rules, get the translated m3u8 segment and srt file corresponding to the translation language. The player automatically switches to the translated m3u8 segment for video playback and plays subtitles through the srt file.

[0007] As a further solution, in step 1, perform the following specific steps: Receive live content source streams through a streaming server or FFmpeg; Adaptively adjust the segmentation duration according to the GOP size and forcibly cut the live content source stream at the I-frame boundary to obtain live content fragments; Extract PTS / DTS from the live content source stream as the reference time, and generate the start timestamp and end timestamp for each live content segment; The live content is segmented and cached, and the m3u8 index file is dynamically updated.

[0008] As a further solution, in step 2, perform the following specific steps: Obtain live content cache to obtain live content fragments, perform audio extraction and preprocessing, set the sampling rate within the ASR optimal range, and obtain several segmented audio blocks; Perform streaming speech recognition on the segmented audio blocks, filter out silent segments, suppress ambient noise, and inject domain terms to obtain ASR recognition output results; Select a translation mode based on user settings or the current delay status; translation modes include incremental translation and full sentence translation; Perform machine translation on the ASR recognition output to obtain the translated content; The translated content and the live content segments are marked with corresponding timestamps to obtain timestamp information, including word-level timestamps and sentence-level timestamps.

[0009] As a further solution, in step 3, perform the following specific steps: Convert the translated content into an SRT file through the conversion function, including SRT entry construction and timestamp format conversion; Get the m3u8 segments corresponding to the translated content, and save them as srt files with the same names according to the m3u8 segments.

[0010] As a further solution, in step 4, perform the following specific steps: Get the srt file to be synthesized, parse the data structure, and extract the original text content; Punctuation normalization, number conversion, terminology replacement, and sentence segmentation optimization are performed on the original text content to obtain the text content to be translated; Perform streaming TTS processing on the text content to be translated to obtain the original TTS synthesized speech; The original TTS synthesized speech is rhythmically adjusted, and the emotional features of the original speech are extracted and transferred to the TTS synthesis parameters of the synthesized speech to obtain real-time speech translation.

[0011] As a further solution, lip syncing is fine-tuned for real-time speech translation through the following steps: Obtain the speaker's video frame in the live content segment and extract the speaker's lip contour; Calculate the mouth opening and closing degree through the lip contour, and determine the speaker's pronunciation time through the mouth opening and closing degree; Determine the pronunciation time points of the real-time speech translation to obtain the original translation pronunciation time point set; Obtain the pronunciation time points of each speaker corresponding to the real-time speech translation content to obtain a set of real person pronunciation time points; The original translation pronunciation time point set is aligned one by one with the real person pronunciation time point set based on the principle of nearest matching; if the alignment distance exceeds the fine-tuning range or the nearest pronunciation time point has been matched, the current pronunciation time point is skipped.

[0012] As a further solution, in step 5, adjust the real-time speech translation to millisecond-level alignment with the timestamps in the corresponding srt file by the following steps: Get the start and end timestamps in the srt file; Align the start timestamp of the real-time speech translation with that of the corresponding srt file; Get the tail of the real-time speech translation and compare it with the end timestamp of the corresponding srt file; When the tail of the real-time voice translation exceeds the end timestamp, the playback speed of the real-time voice translation is adjusted to be faster until it aligns with the end timestamp; When there is a gap between the end of the real-time voice translation and the end timestamp, the gap is filled by adding silent content to the real-time voice translation.

[0013] As a further solution, a voice fading effect is added to the end of the real-time voice translation.

[0014] As a further solution, in step 6, perform the following specific steps Obtain live content segments and perform video stream separation to obtain video content segments; Obtain real-time speech translation and perform audio encoding to obtain a translated audio file in the corresponding encoding format; Merge the video content segments and the translated audio files to obtain the corresponding new m3u8 segments; Name the new m3u8 fragment according to the predetermined naming rule to obtain the translated m3u8 fragment; Associate the translated m3u8 fragment with the address corresponding to the m3u8 index file.

[0015] On the other hand, the present invention further provides a system for aligning audio, video and subtitles synthesized by real-time speech translation, which comprises: Live content caching module: collects live content into m3u8 segments and caches the live content at the same time; Speech recognition and translation module: performs speech recognition and translation on the live content cache, obtains the translated content and records the timestamp information; srt file generation module: saves the translated content into srt files with the same name according to the m3u8 segments one by one; Speech translation synthesis module: performs speech translation synthesis based on the translation content to obtain real-time speech translation; Translation time alignment module: This module adjusts the real-time speech translation to align with the timestamp in the corresponding SRT file to the millisecond level according to the timestamp information corresponding to the translation content. Audio and video synchronization merging module: merges the generated real-time speech translation with the video content cache and saves it as translated m3u8 fragments that correspond one-to-one to the included m3u8 fragments; Automatic translation execution module: Generate and continuously update by repeatedly executing steps 1 to 6 until all m3u8 fragments are converted into corresponding translated m3u8 fragments; Playback rule setting module: obtains the translation language set by the user, and obtains the translated m3u8 fragments and srt files corresponding to the translated language according to the predetermined switching rules. The player automatically switches to the translated m3u8 fragments for video playback and plays subtitles through the srt file.

[0016] Compared with related technologies, the method and system for aligning audio, video and subtitles using real-time speech translation synthesis provided by the present invention have the following advantages: The present invention achieves millisecond-level alignment of translated voice and video through slice-level parallel processing, greatly reducing latency compared to traditional solutions and solving the problem of voice and video asynchrony after translation during live broadcast. The present invention automatically generates real-time voice translation based on the live broadcast content, and then processes the corresponding srt files and translated m3u8 slices separately. Therefore, there is no need for manual translation and subtitle generation in advance, nor is there any need to replace the audio of the original video in advance, which saves labor costs and provides a more universal live streaming subtitle solution. In addition, when switching languages, the player can select the starting time point by itself, so that the method supports multiple playback modes such as online live broadcast and live broadcast playback. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 The overall flow chart of the audio-video subtitle alignment method for real-time speech translation synthesis provided by the present invention; Figure 2 This is a structural diagram of an audio-video-caption alignment system for real-time speech translation synthesis provided by the present invention.

[0020] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0022] See also Figure 1 The embodiment of the present application provides a method for aligning audio and video subtitles synthesized by real-time speech translation, which is used to align audio and video subtitles synthesized by real-time speech translation in a live broadcast scenario, including the steps of: Step 1: Record the live content into m3u8 segments and cache the live content; Step 2: Perform speech recognition translation on the live content cache, obtain the translated content and record the timestamp information; Step 3: Save the translated content into srt files with the same name according to the m3u8 segments one by one; Step 4: Perform speech translation synthesis based on the translation content to obtain real-time speech translation; Step 5: According to the timestamp information corresponding to the translation content, adjust the real-time speech translation and the timestamp in the corresponding SRT file to align at the millisecond level; Step 6: Merge the generated real-time speech translation with the video content cache and save it as a translated m3u8 fragment that corresponds one-to-one with the included m3u8 fragment; Step 7: Generate and continuously update by repeating steps 1 to 6 until all m3u8 fragments are converted into corresponding translated m3u8 fragments; Step 8: Get the translation language set by the user, and according to the predetermined switching rules, get the translated m3u8 segment and srt file corresponding to the translation language. The player automatically switches to the translated m3u8 segment for video playback and plays subtitles through the srt file.

[0023] It should be noted that: this embodiment achieves millisecond-level alignment of translated voice and video through slice-level parallel processing, which greatly reduces the delay compared to traditional solutions and solves the problem of asynchronous voice and picture after translation during live broadcast; the present invention automatically generates real-time voice translation according to the live broadcast content, and then processes them separately to obtain the corresponding srt file and translated m3u8 fragments; therefore, there is no need for manual translation and subtitle generation in advance, nor is there any need to replace the audio of the original video in advance, which saves labor costs and provides a more universal live streaming subtitle solution; in addition, when switching languages, the player can select the starting time point by itself, so that the method supports multiple playback modes such as online live broadcast and live playback.

[0024] In step 1, perform the following specific steps: Receive live content source streams through a streaming server or FFmpeg, cut them into HLS standard segments (TS / fMP4), and support protocols: RTMP (mainstream streaming protocol), SRT (anti-network jitter), RTSP (camera live broadcast) and WebRTC (low-latency interactive scenarios), thus adapting to live broadcast scenarios; Furthermore, segment boundary alignment requires forced I-frame splitting, otherwise the player will freeze and display artifacts. The segmentation duration is adaptively adjusted according to the GOP size, and the live content source stream is forced to be split at the I-frame boundary to obtain live content fragments. Extract PTS / DTS from the live content source stream as the reference time, and generate start and end timestamps for each live content segment to ensure the stability and low latency of the live broadcast of the translated content; The live content is segmented and cached, and the m3u8 index file is dynamically updated to adapt to the live content that is constantly generated in the live scene.

[0025] In step 2, perform the following specific steps: Obtain live content cache to obtain live content fragments, perform audio extraction and preprocessing, set the sampling rate within the ASR optimal range (e.g., 16kHz sampling rate, mono), and obtain several segmented audio blocks; Perform streaming speech recognition on the segmented audio blocks, filter out silent segments (use WebRTC VAD to filter silent segments to reduce invalid recognition), suppress ambient noise (such as the RNNoise algorithm to suppress ambient noise in real time), and inject domain terms (such as product names and professional vocabulary) to obtain ASR recognition output results. In addition, we dynamically select the translation mode based on user settings or the current latency status. The translation modes include incremental translation and full-sentence translation. Incremental translation is mainly used in low-latency environments to improve translation feedback efficiency. Full-sentence translation can improve translation quality and is suitable for scenarios with high requirements for live translation accuracy. Perform machine translation on the ASR recognition output to obtain the translated content; The translated content and the live content segments are timestamped to obtain timestamp information, including word-level timestamps and sentence-level timestamps. The corresponding segmentation measurements are shown in Table 1: Table 1 Segmentation strategy table In step 3, perform the following specific steps: Convert the translated content into an SRT file through the conversion function, including SRT entry construction and timestamp format conversion; Get the m3u8 segments corresponding to the translated content, and save them as srt files with the same name according to the m3u8 segments, so as to obtain the srt subtitle content.

[0026] In step 4, to make the translation more consistent with the live audio, perform the following specific steps: Get the srt file to be synthesized, parse the data structure, and extract the original text content; Punctuation normalization, number conversion, terminology replacement, and sentence segmentation optimization are performed on the original text content to obtain the text content to be translated; Perform streaming TTS processing on the text content to be translated to obtain the original TTS synthesized speech; The original TTS synthesized speech is rhythmically adjusted, and the emotional features of the original speech are extracted and transferred to the TTS synthesis parameters of the synthesized speech to obtain real-time speech translation.

[0027] This embodiment uses dynamic timestamp remapping technology to automatically adjust the TTS speech rate or fill silence, ensuring that each translated audio segment accurately matches the original video duration. In addition, this embodiment also introduces lip-driven voice fine-tuning to extract the speaker's mouth shape features and migrate them to the synthesized speech, thereby improving the virtual anchor's lip synchronization. Incremental translation and whole-sentence translation adaptive switching are combined to reduce processing load, bandwidth consumption, and support thousands of concurrent users.

[0028] Specifically, this embodiment further performs lip synchronization fine-tuning on the real-time speech translation through the following steps: Obtain the speaker's video frame in the live content segment and extract the speaker's lip contour; Calculate the mouth opening and closing degree through the lip contour, and determine the speaker's pronunciation time through the mouth opening and closing degree; Determine the pronunciation time points of the real-time speech translation to obtain the original translation pronunciation time point set; Obtain the pronunciation time points of each speaker corresponding to the real-time speech translation content to obtain a set of real person pronunciation time points; The original translation pronunciation time point set is aligned one by one with the real person pronunciation time point set based on the principle of nearest matching; if the alignment distance exceeds the fine-tuning range or the nearest pronunciation time point has been matched, the current pronunciation time point is skipped.

[0029] In addition, in step 5, the real-time speech translation is aligned with the timestamp in the corresponding srt file to the millisecond level through the following steps: Get the start and end timestamps in the srt file; Align the start timestamp of the real-time speech translation with that of the corresponding srt file; Get the tail of the real-time speech translation and compare it with the end timestamp of the corresponding srt file; When the tail of the real-time voice translation exceeds the end timestamp, the playback speed of the real-time voice translation is adjusted to be faster until it aligns with the end timestamp; When there is a gap between the end of the real-time voice translation and the end timestamp, the gap is filled by adding silent content to the real-time voice translation.

[0030] In order to achieve a natural transition between spoken words, we also add a voice fade effect at the end of the real-time voice translation.

[0031] Furthermore, in step 6, the following specific steps are performed: Obtain live content segments and perform video stream separation to obtain video content segments; Obtain real-time speech translation and perform audio encoding to obtain a translated audio file in the corresponding encoding format; Merge the video content segments and the translated audio files to obtain the corresponding new m3u8 segments; Name the new m3u8 fragment according to the predetermined naming rule to obtain the translated m3u8 fragment; Associate the translated m3u8 fragment with the address corresponding to the m3u8 index file.

[0032] Example 2 See also Figure 2 Based on Example 1, the present invention further provides a system for aligning audio, video and subtitles using real-time speech translation synthesis, including: Live content caching module: collects live content into m3u8 segments and caches the live content at the same time; Speech recognition and translation module: performs speech recognition and translation on the live content cache, obtains the translated content and records the timestamp information; srt file generation module: saves the translated content into srt files with the same name according to the m3u8 segments one by one; Speech translation synthesis module: performs speech translation synthesis based on the translation content to obtain real-time speech translation; Translation time alignment module: This module adjusts the real-time speech translation to align with the timestamp in the corresponding SRT file to the millisecond level according to the timestamp information corresponding to the translation content. Audio and video synchronization merging module: merges the generated real-time speech translation with the video content cache and saves it as translated m3u8 fragments that correspond one-to-one to the included m3u8 fragments; Automatic translation execution module: Generate and continuously update by repeatedly executing steps 1 to 6 until all m3u8 fragments are converted into corresponding translated m3u8 fragments; Playback rule setting module: obtains the translation language set by the user, and obtains the translated m3u8 fragments and srt files corresponding to the translated language according to the predetermined switching rules. The player automatically switches to the translated m3u8 fragments for video playback and plays subtitles through the srt file.

[0033] The above are only some embodiments of the present application and are not intended to limit the patent scope of the present application. All equivalent structural transformations made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for aligning audio and video subtitles synthesized by real-time speech translation, which is used to align audio and video subtitles synthesized by real-time speech translation in a live broadcast situation, characterized in that: Including steps: Step 1: Record the live content into m3u8 segments and cache the live content; Step 2: Perform speech recognition translation on the live content cache, obtain the translated content and record the timestamp information; Step 3: Save the translated content into srt files with the same name according to the m3u8 segments one by one; Step 4: Perform speech translation synthesis based on the translation content to obtain real-time speech translation; Step 5: According to the timestamp information corresponding to the translation content, adjust the real-time speech translation and the timestamp in the corresponding SRT file to align at the millisecond level; Step 6: Merge the generated real-time speech translation with the video content cache and save it as a translated m3u8 fragment that corresponds one-to-one with the included m3u8 fragment; Step 7: Generate and continuously update by repeating steps 1 to 6 until all m3u8 fragments are converted into corresponding translated m3u8 fragments; Step 8: Get the translation language set by the user, and according to the predetermined switching rules, get the translated m3u8 segment and srt file corresponding to the translation language. The player automatically switches to the translated m3u8 segment for video playback and plays subtitles through the srt file.

2. The method for aligning audio and video subtitles synthesized by real-time speech translation according to claim 1, characterized in that: In step 1, perform the following specific steps: Receive live content source streams through a streaming server or FFmpeg; Adaptively adjust the segmentation duration according to the GOP size and forcibly cut the live content source stream at the I-frame boundary to obtain live content fragments; Extract PTS / DTS from the live content source stream as the reference time, and generate the start timestamp and end timestamp for each live content segment; The live content is segmented and cached, and the m3u8 index file is dynamically updated.

3. The method for aligning audio and video subtitles synthesized by real-time speech translation according to claim 1, characterized in that: In step 2, perform the following specific steps: Obtain live content cache to obtain live content fragments, perform audio extraction and preprocessing, set the sampling rate within the ASR optimal range, and obtain several segmented audio blocks; Perform streaming speech recognition on the segmented audio blocks, filter out silent segments, suppress ambient noise, and inject domain terms to obtain ASR recognition output results; Select a translation mode based on user settings or the current delay status; translation modes include incremental translation and full sentence translation; Perform machine translation on the ASR recognition output to obtain the translated content; The translated content and the live content segments are marked with corresponding timestamps to obtain timestamp information, including word-level timestamps and sentence-level timestamps.

4. The method for aligning audio and video subtitles synthesized by real-time speech translation according to claim 1, wherein: In step 3, perform the following specific steps: Convert the translated content into an SRT file through the conversion function, including SRT entry construction and timestamp format conversion; Get the m3u8 segments corresponding to the translated content, and save them as srt files with the same names according to the m3u8 segments.

5. The method for aligning audio and video subtitles synthesized by real-time speech translation according to claim 1, characterized in that: In step 4, perform the following specific steps: Get the srt file to be synthesized, parse the data structure, and extract the original text content; Punctuation normalization, number conversion, terminology replacement, and sentence segmentation optimization are performed on the original text content to obtain the text content to be translated; Perform streaming TTS processing on the text content to be translated to obtain the original TTS synthesized speech; The original TTS synthesized speech is rhythmically adjusted, and the emotional features of the original speech are extracted and transferred to the TTS synthesis parameters of the synthesized speech to obtain real-time speech translation.

6. The method for aligning audio, video and subtitles synthesized by real-time speech translation according to claim 5, characterized in that: The lip syncing of the real-time speech translation is also fine-tuned through the following steps: Obtain the speaker's video frame in the live content segment and extract the speaker's lip contour; Calculate the mouth opening and closing degree through the lip contour, and determine the speaker's pronunciation time through the mouth opening and closing degree; Determine the pronunciation time points of the real-time speech translation to obtain the original translation pronunciation time point set; Obtain the pronunciation time points of each speaker corresponding to the real-time speech translation content to obtain a set of real person pronunciation time points; The original translation pronunciation time point set is aligned one by one with the real person pronunciation time point set based on the principle of nearest matching; if the alignment distance exceeds the fine-tuning range or the nearest pronunciation time point has been matched, the current pronunciation time point is skipped.

7. The method for aligning audio, video and subtitles synthesized by real-time speech translation according to claim 3, characterized in that: In step 5, adjust the real-time speech translation to millisecond-level alignment with the timestamp in the corresponding srt file by the following steps: Get the start and end timestamps in the srt file; Align the start timestamp of the real-time speech translation with that of the corresponding srt file; Get the tail of the real-time speech translation and compare it with the end timestamp of the corresponding srt file; When the tail of the real-time voice translation exceeds the end timestamp, the playback speed of the real-time voice translation is adjusted to be faster until it aligns with the end timestamp; When there is a gap between the end of the real-time voice translation and the end timestamp, the gap is filled by adding silent content to the real-time voice translation.

8. The method for aligning audio, video and subtitles synthesized by real-time speech translation according to claim 7, characterized in that: Also adds a voice fade effect at the end of real-time voice translation.

9. The method for aligning audio, video and subtitles synthesized by real-time speech translation according to claim 1, characterized in that: In step 6, perform the following specific steps Obtain live content segments and perform video stream separation to obtain video content segments; Obtain real-time speech translation and perform audio encoding to obtain a translated audio file in the corresponding encoding format; Merge the video content segments and the translated audio files to obtain the corresponding new m3u8 segments; Name the new m3u8 fragment according to the predetermined naming rule to obtain the translated m3u8 fragment; Associate the translated m3u8 fragment with the address corresponding to the m3u8 index file.

10. A system for aligning audio, video and subtitles synthesized by real-time speech translation, comprising: a method for aligning audio, video and subtitles synthesized by real-time speech translation according to any one of claims 1 to 9, characterized in that: include: Live content caching module: collects live content into m3u8 segments and caches the live content at the same time; Speech recognition and translation module: performs speech recognition and translation on the live content cache, obtains the translated content and records the timestamp information; srt file generation module: saves the translated content into srt files with the same name according to the m3u8 segments one by one; Speech translation synthesis module: performs speech translation synthesis based on the translation content to obtain real-time speech translation; Translation time alignment module: This module adjusts the real-time speech translation to align with the timestamp in the corresponding SRT file to the millisecond level according to the timestamp information corresponding to the translation content. Audio and video synchronization merging module: merges the generated real-time speech translation with the video content cache and saves it as translated m3u8 fragments that correspond one-to-one to the included m3u8 fragments; Automatic translation execution module: Generate and continuously update by repeatedly executing steps 1 to 6 until all m3u8 fragments are converted into corresponding translated m3u8 fragments; Playback rule setting module: obtains the translation language set by the user, and obtains the translated m3u8 fragments and srt files corresponding to the translated language according to the predetermined switching rules. The player automatically switches to the translated m3u8 fragments for video playback and plays subtitles through the srt file.

Images