A simultaneous interpretation method and system based on multi-track reservation and double reference synthesis

Abstract

The application provides a simultaneous interpretation method and system based on multi-track reservation and double reference synthesis, which separates original audio from an original audio-video file; divides the original audio into multiple original audio segments; separates a human voice track and a background audio track from each original audio segment; identifies source language text and a timestamp of each speaker from the human voice track; translates the source language text into target language text; extracts emotion slice audio and voiceprint clone audio based on the human voice track; inputs the target language text, the emotion slice audio and the voiceprint clone audio into a general TTS engine to output a target language audio track; and synthesizes the original video picture, each target language audio track and each background audio track to generate a target language audio-video file. The application realizes end-to-end simultaneous interpretation of language independence, identity preservation, emotion restoration and scene integrity through two main pillars of multi-track reservation and double reference synthesis.

Description

Technical Field

[0001] This invention relates to the field of translation technology, and more specifically, to a simultaneous translation method and system based on multi-track preservation and dual-reference synthesis. Background Technology

[0002] With the deepening of globalization, the demand for real-time cross-language communication has surged. Traditional human simultaneous interpretation is costly and has limited coverage.

[0003] Existing technologies such as Google Translator (end-to-end) or Microsoft Translator (cascaded) process all content in a single process, without designing differentiated processing paths for different audio components (human voice / background noise) and speech features (voiceprint / emotion), and are generally limited to specific language pairs (such as English to Chinese), lacking universality.

[0004] In summary, existing simultaneous interpretation methods have the following shortcomings:

[0005] a) Single-track processing paradigm: Treats the input audio as a single data stream, forcing all components (voice / background sound) to follow the same processing path, resulting in the background sound being discarded;

[0006] b) Language binding architecture: Model training and process design are optimized for specific language pairs, making it difficult to extend to new languages;

[0007] c) Rigid processing flow: lacks an adaptive segmentation mechanism for extremely long content, limiting processing capacity. Summary of the Invention

[0008] This invention addresses the technical problems existing in the prior art by providing a simultaneous translation method and system based on multi-track preservation and dual-reference synthesis, thus overcoming the shortcomings of the prior art in simultaneous translation.

[0009] According to a first aspect of the present invention, a simultaneous translation method based on multi-track preservation and dual-reference synthesis is provided, comprising:

[0010] Obtain the original audio and video files, and extract the original audio from the original audio and video files;

[0011] The original audio is divided into multiple original audio segments, and the vocal track and background sound track are separated from each original audio segment;

[0012] The source language text and timestamp of each speaker are identified from the vocal track of each original audio segment, and a structured caption for each speaker is generated. The structured caption includes {[start timestamp-end timestamp, speaker ID, source language text]}.

[0013] Translate the source language text into the target language text;

[0014] Based on the human voice track of each original audio segment, extract emotional slice audio and voiceprint clone audio;

[0015] The target language text, emotional slice audio, and voiceprint clone audio corresponding to each original audio segment are input into a general TTS engine, and the target language audio track is output.

[0016] The original video footage, each target language audio track, and each background sound track are combined to generate a target language audio and video file.

[0017] Based on the above technical solution, the present invention can also be improved as follows.

[0018] Optionally, obtaining the original audio / video file and extracting the original audio from the original audio / video file includes:

[0019] The multimedia decoding engine is invoked to strip the video track from the original audio and video file, extract the original audio stream, and output the standardized original audio waveform file.

[0020] Optionally, the original audio is divided into multiple original audio segments, including:

[0021] Identify all natural pauses in the original audio as cutting points, and cut the original audio into multiple original audio segments based on the cutting points. Each original audio segment is ≤T minutes, where T is a preset duration.

[0022] Optionally, separating the vocal track and background sound track from each of the original audio segments includes:

[0023] For each audio segment, a pre-trained source separation model decouples the original audio segment into a vocal track and a background sound track.

[0024] Optionally, the source language text and timestamp of each speaker are identified from the vocal track of each original audio segment, and structured captions for each speaker are generated, including:

[0025] For each audio segment's voice track, analyze its voiceprint features, segment the audio stream of the voice track according to the speaker's identity, and label the speaker's identity ID for each segment.

[0026] Transcribe the vocal segments of each speaker to generate source language text segments;

[0027] Record the start time tstart and end time tend for each source language text segment;

[0028] Generate a structured subtitle list corresponding to the human voice track for each audio segment. The structured subtitle list is in the form of: {[start timestamp-end timestamp, speaker ID, source language text]}.

[0029] Optionally, translating the source language text into the target language text includes:

[0030] Dynamically load the corresponding machine translation engine based on the target language configured by the user;

[0031] The machine translation engine iterates through the subtitle files in the structured subtitle list and translates the original subtitles into target language text.

[0032] Optionally, generating emotional slice audio and voiceprint clone audio based on the vocal track of each original audio segment includes:

[0033] Based on the start time tstart, end time tend, and speaker identity speaker_id in the structured caption list, original human voice segments that are perfectly aligned with the time of the current sentence to be synthesized are segmented from the human voice track as emotional audio slices;

[0034] For each unique speaker ID (speaker_id), a continuous segment of the speaker's voice with clear sound quality and no background interference of a preset duration is retrieved from the structured subtitles and used as the voiceprint clone audio.

[0035] For each translation task, the corresponding outputs are the translated target language text, the emotional slice audio, and the voiceprint clone audio.

[0036] Optionally, the step of inputting the target language text, emotional slice audio, and voiceprint clone audio corresponding to each original audio segment into a general TTS engine and outputting a target language audio track includes:

[0037] Based on the IndexTTS2 engine, the system generates a speech waveform using the target language text as content by cloning audio from the speaker's voiceprint and emotional slice audio, and outputs the target language audio track.

[0038] According to a second aspect of the present invention, a simultaneous translation system based on multi-track preservation and dual-reference synthesis is provided, comprising:

[0039] The acquisition module is used to acquire the original audio and video files and extract the original audio from the original audio and video files;

[0040] The separation module is used to divide the original audio into multiple original audio segments, and to separate the vocal track and background sound track from each original audio segment;

[0041] The generation module is used to identify the source language text and timestamp of each speaker from the voice track of each original audio segment, and generate structured subtitles for each speaker. The structured subtitles include {[start timestamp-end timestamp, speaker ID, source language text]}. Based on the voice track of each original audio segment, it extracts emotional slice audio and voiceprint clone audio. It is also used to input the target language text, emotional slice audio, and voiceprint clone audio corresponding to each original audio segment into a general TTS engine to generate the target language audio track. Finally, it synthesizes the original video footage, each target language audio track, and each background sound track to generate a target language audio and video file.

[0042] The translation module is used to translate the source language text into the target language text.

[0043] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, the processor being configured to execute a computer program stored in the memory to implement a simultaneous translation method based on multi-track retention and dual-reference synthesis.

[0044] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of a simultaneous translation method based on multi-track preservation and dual-reference synthesis.

[0045] This invention provides a simultaneous translation method and system based on multi-track preservation and dual-reference synthesis. The input audio is separated into a human voice track and a background sound track. The human voice track is processed by translation to generate the target speech, and the background sound track is directly connected to the final synthesis stage. The three tracks (video + background sound + translated audio) are fused and output. This invention achieves end-to-end simultaneous translation that is language-independent, identity-preserving, emotion-restoring, and scene-complete through the two pillars of multi-track preservation and dual-reference synthesis. Attached Figure Description

[0046] Figure 1 A flowchart of a simultaneous translation method based on multi-track preservation and dual-reference synthesis is provided as an embodiment of the present invention;

[0047] Figure 2 The following is an overall flowchart of a simultaneous translation method based on multi-track preservation and dual-reference synthesis, provided as an embodiment of the present invention;

[0048] Figure 3 The diagram below shows a structural block diagram of a simultaneous translation system based on multi-track preservation and dual-reference synthesis, according to an embodiment of the present invention.

[0049] Figure 4 A schematic diagram of a possible hardware structure of an electronic device provided by the present invention;

[0050] Figure 5 This is a schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined with each other to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0052] Figure 1 The following is a flowchart illustrating a simultaneous translation method based on multi-track preservation and dual-reference synthesis according to an embodiment of the present invention. Figure 1 and Figure 2 As shown, the simultaneous translation method proposed in this invention adopts a standardized six-stage pipeline architecture, with the overall process divided into a preprocessing stage, a core processing stage, and a post-processing fusion stage. Each step is linked together through a unified structured metadata chain to ensure the integrity and consistency of information during the flow process.

[0053] See Figure 1 The simultaneous translation method based on multi-track preservation and dual-reference synthesis proposed in this invention includes the following steps:

[0054] Step 1: Obtain the original audio and video files, and extract the original audio from the original audio and video files.

[0055] Understandably, the process involves acquiring the original audio and video files in complex scenarios, separating the original audio and video files, extracting the original audio data from the original audio and video files, and obtaining the original audio and original video footage.

[0056] Specifically, the input is any source audio or video file (MP4, MKV, AVI, MOV, etc.), which is then used by a multimedia decoding engine (such as FFmpeg) to extract the video track and raw audio stream. The audio is then uniformly resampled to a fixed sampling rate (e.g., 16kHz or 48kHz) and in mono / stereo format to eliminate processing biases caused by differences in source file encoding. The final output is a standardized raw audio waveform file (Raw Audio). This step does not involve speech content recognition; it only performs signal-level standardization to ensure language independence in subsequent modules.

[0057] Step 2: Divide the original audio into multiple original audio segments, and separate the vocal track and background sound track from each original audio segment.

[0058] Understandably, the original long audio is divided into segments of ≤T minutes (e.g., 20 minutes) based on silence intervals, and the segmentation logic is independent of language. Specifically, all natural pauses (silence points) in the original audio are identified, and the original long audio is divided into multiple audio data segments using these natural pauses as segmentation points, resulting in multiple original audio segments. The duration of each original audio segment is controlled within a preset duration.

[0059] Specifically, silence detection is performed on the standardized raw audio waveform to identify natural pauses (silent intervals), and a maximum segment duration threshold Tmax (e.g., 20 minutes) is set. Starting from the starting point, when the accumulated duration approaches Tmax, the nearest silent point is searched as the cut-off point; if there is no silence beyond Tmax, a forced cut is performed.

[0060] Overlapping window: To avoid semantic truncation at the cut point, an overlapping area (e.g., 2 seconds) is retained between adjacent segments, and redundancy is removed by timestamp alignment in subsequent processing.

[0061] After intelligent segmentation, a series of audio segments {S1,S2,...,Sn} with global time offset markers are output, along with the corresponding segment metadata.

[0062] The purpose of this step is to enable parallel processing in subsequent steps and improve the system's operating speed.

[0063] Subsequently, each original audio segment is processed in parallel. For each original audio segment, it is separated into a vocal track and a background audio track, resulting in vocal audio and background audio respectively. The vocal audio and background audio are stored independently, and the separation process does not involve speech recognition.

[0064] Specifically, for each original audio segment, a pre-trained audio source separation model (such as a separation network based on U-Net or Transformer architecture) is loaded to decouple the mixed audio into two independent tracks: the VocalTrack, which contains the speech of all speakers, removing background music, environmental noise, and sound effects; and the BackgroundTrack, which contains non-vocal components such as music, applause, and ambient noise. The final output consists of independently stored vocal waveform files and background waveform files.

[0065] Step 3: Identify the source language text and timestamp of each speaker from the vocal track of each original audio segment, and generate structured subtitles for each speaker. The structured subtitles include {[start timestamp-end timestamp, speaker ID, source language text]}.

[0066] The process involves text recognition for each extracted audio segment, identifying the source language text and timestamp for each speaker, and labeling them with a speaker ID. Based on the recognition results, structured subtitles are generated. The structured subtitle format is: [start timestamp - end timestamp], speaker ID, and source language text.

[0067] Specifically, based on each segment of human voice audio, the voices of different speakers are separated. In particular, the voiceprint features in the human voice audio are analyzed, the audio stream is segmented according to the speaker's identity, and the speaker ID of each segment is labeled (such as spk_0, spk_1).

[0068] Then, each speaker's segment is transcribed to generate source language text, precisely recording the start time (tstart) and end time (tend) of each text segment. Finally, a language-independent structured data object is generated, specifically a structured caption list that does not contain any language-specific hard-coding.

[0069] Step 4: Translate the source language text into the target language text.

[0070] Specifically, for each speaker's source language text, a general machine translation engine is invoked to translate the source language text in the structured subtitles into the target language text, while strictly preserving the timestamp and speaker's identity ID.

[0071] Among them, the general machine translation engine does not restrict the types of source and target languages, and can dynamically replace the translation of different language pairs (such as English to Chinese, Japanese to French, German to Spanish), and the translation process of different language pairs remains unchanged.

[0072] Specifically, text translation is performed based on a general machine translation and metadata mapping approach. According to the user-configured target language, the corresponding machine translation engine (API or local model) is dynamically loaded, and the subtitle files in the structured subtitle list are traversed to translate the original subtitles into the target language text. During the translation process, the original start timestamp `tstart`, end timestamp `tend`, and speaker ID `speaker_id` are strictly maintained; only the text content is replaced. Output: a bilingual structured subtitle list.

[0073] Step 5: Extract the emotional reference audio and voiceprint reference audio for each speaker based on each audio segment.

[0074] The process involves generating segmented and cloned audio files for each voice audio segment, serving as the emotion reference audio and voiceprint reference audio, respectively. Specifically, each voice audio segment is segmented according to the timestamp of the structured subtitles, and the segmented voice audio is used as the emotion reference audio. Since each speaker's voice audio is discontinuous, the voice audio of each speaker is identified and extracted, and then a high-quality voice audio segment of fixed duration (e.g., 15 seconds) is spliced ​​together for each speaker as the voiceprint reference audio.

[0075] Specifically, for each speaker, dual-reference audio is dynamically generated. The inputs for this step are the voice track and a list of structured subtitles, which generate emotional slice audio and voiceprint clone audio respectively. The generation of emotional slice audio includes:

[0076] Based on the start time tstart, end time tend, and speaker ID speaker_id in the subtitles, the original human voice segments that are perfectly aligned with the time of the sentence to be synthesized are precisely segmented from the human voice track.

[0077] Function: To extract the speaker's dynamic prosodic features (speech rate, pauses, stress, emotional fluctuations) and solve the problem of "how to say it".

[0078] Generating voiceprint clone audio includes:

[0079] For each unique speaker_id, retrieve the clearest, background-free consecutive segments (fixed duration, such as 15 seconds) of that speaker's voice from the subtitles.

[0080] If a single segment is insufficient, multiple high-quality segments are spliced ​​together.

[0081] Function: Extract the speaker's static voiceprint features (timbre, fundamental frequency range) to solve the "who am I" question.

[0082] The output of this step is: a triplet of (translated text, cloned reference audio, and emotional reference audio) for each translation task.

[0083] Step 6: Input the target language text, emotion reference audio, and voiceprint reference audio of each speaker into the general TTS engine, and output the target language audio.

[0084] The process involves calling a general TTS engine, inputting the target language text, cloned audio (voiceprint reference), and sliced ​​audio (emotion reference) of each speaker in each original audio segment into the general TTS engine, and the general TTS engine outputs the target language audio. The TTS engine can support any target language.

[0085] Specifically, this step implements high-fidelity speech synthesis based on dual references, with the inputs being: translated text (Texttgt), voiceprint reference (Audio Clone), and emotion reference (Audio Slice).

[0086] The processing logic is as follows:

[0087] Cross-language synthesis: The IndexTTS2 engine clones audio and emotional slices based on the speaker's voiceprint and generates speech waveforms with the target language Texttgt as the content.

[0088] Output: Audio track translated into the target language.

[0089] Step 7: Combine the original video footage, each segment of target language audio, and each segment of background audio to generate the target language audio-visual content.

[0090] This step involves fusing multi-track data. Specifically, it synthesizes the original video footage, each segment of target language audio, and each segment of background audio to generate target language audio-visual content. For example, it can translate an English video into a Chinese or other language video.

[0091] Specifically, this step achieves multi-track spatiotemporal alignment and fusion, with the inputs being the original video footage, background audio track, translated audio track, and segmented metadata.

[0092] Processing logic:

[0093] Audio translation synthesis: First, create a full-length blank audio segment. Then, sort all the translated audio segments for insertion. During insertion, the audio is checked from beginning to end. If overlapping audio is found, it is prioritized and moved forward. If an audio segment is too long and overlaps with the next segment, the code first calculates `candidate_start = max_end - duration`. As long as this new starting point does not overlap with the previous segment (>= min_start), the entire segment is moved forward, maintaining the original speed and full length.

[0094] Secondary option: If the audio starts too early and hits the previous sentence, and the forward movement logic is not triggered or applicable, the code will try to set its starting point to min_start (right next to the previous sentence), and move the whole thing to the next sentence as long as it does not hit the next sentence.

[0095] Final compression: Speedup will only be used for variable-speed compression when the original duration of the audio is greater than the available space (duration > available_space).

[0096] Timeline reassembly: Based on the global time offset in the segment metadata, all processed segments {T1...Tn} and {B1...Bn} are stitched onto the global timeline, and the smooth transition (fade-in and fade-out) of the overlapping areas is handled.

[0097] Composite video: The original video footage, background audio track, and translated audio track are combined into the final translated video file.

[0098] Output: The final general-purpose AI simultaneous translation video file.

[0099] See Figure 3 This paper presents a simultaneous translation system based on multi-track preservation and dual-reference synthesis, the system comprising:

[0100] The acquisition module 301 is used to acquire the original audio and video file and extract the original audio from the original audio and video file;

[0101] The separation module 302 is used to divide the original audio into multiple original audio segments, and separate the vocal track and background sound track from each original audio segment;

[0102] The generation module 303 is used to identify the source language text and timestamp of each speaker from the voice track of each original audio segment, and generate structured subtitles for each speaker, wherein the structured subtitles include {[start timestamp-end timestamp, speaker ID, source language text]}; extract emotional slice audio and voiceprint clone audio based on the voice track of each original audio segment; input the target language text, emotional slice audio and voiceprint clone audio corresponding to each original audio segment into a general TTS engine to generate a target language audio track; and synthesize the original video frame, each target language audio track and each background sound track to generate a target language audio and video file;

[0103] Translation module 304 is used to translate the source language text into the target language text.

[0104] It is understood that the simultaneous translation system based on multi-track preservation and dual-reference synthesis provided by the present invention corresponds to the simultaneous translation method based on multi-track preservation and dual-reference synthesis provided in the foregoing embodiments. The relevant technical features of the simultaneous translation system based on multi-track preservation and dual-reference synthesis can be referred to the relevant technical features of the simultaneous translation method based on multi-track preservation and dual-reference synthesis, and will not be repeated here.

[0105] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 4 As shown, an embodiment of the present invention provides an electronic device 400, including a memory 410, a processor 420, and a computer program 411 stored in the memory 410 and executable on the processor 420. When the processor 420 executes the computer program 411, it implements the steps of a simultaneous translation method based on multi-track preservation and dual-reference synthesis.

[0106] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. (See diagram below.) Figure 5 As shown, this embodiment provides a computer-readable storage medium 500 on which a computer program 511 is stored. When the computer program 511 is executed by a processor, it implements the steps of a simultaneous translation method based on multi-track preservation and dual-reference synthesis.

[0107] The simultaneous translation method and system based on multi-track preservation and dual-reference synthesis provided in this invention have the following advantages compared with the prior art:

[0108] (1) By decoupling language-related modules (translation / TTS) from language-independent processes, new languages ​​can be supported simply by changing the engine without refactoring the system. Actual tests show that it supports multiple languages ​​such as English-Chinese, Japanese-French, German-Spanish, etc., with a 100% reuse rate of process code.

[0109] (2) The multi-track architecture achieves intelligent preservation of background sound for the first time in AI simultaneous translation. User surveys show that the immersive experience of the scene is improved by 60%, which is especially suitable for scenarios that are sensitive to ambient sound, such as meetings, interviews, and movies.

[0110] (3) The dual reference mechanism enables the system to achieve a high accuracy in maintaining identity in a 4-person dialogue scenario, while the MOS sentiment score reaches 4.1 (2.8 in the traditional scheme), taking into account both identity consistency and naturalness of expression.

[0111] (4) The standardized segmented-parallel architecture enables the system to process content of arbitrary duration, breaking through the duration limit of existing systems.

[0112] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0113] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0117] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0118] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A simultaneous translation method based on multi-track preservation and dual-reference synthesis, characterized in that, include: Obtain the original audio and video files, and extract the original audio from the original audio and video files; The original audio is divided into multiple original audio segments, and the vocal track and background sound track are separated from each original audio segment; The source language text and timestamp of each speaker are identified from the vocal track of each original audio segment, and a structured caption for each speaker is generated. The structured caption includes {[start timestamp-end timestamp, speaker ID, source language text]}. Translate the source language text into the target language text; Based on the human voice track of each original audio segment, extract emotional slice audio and voiceprint clone audio; The target language text, emotional slice audio, and voiceprint clone audio corresponding to each original audio segment are input into a general TTS engine, and the target language audio track is output. The original video footage, each target language audio track, and each background sound track are combined to generate a target language audio and video file.

2. The simultaneous interpretation method according to claim 1, characterized in that, The step of obtaining the original audio and video file and extracting the original audio from the original audio and video file includes: The multimedia decoding engine is invoked to strip the video track from the original audio and video file, extract the original audio stream, and output the standardized original audio waveform file.

3. The simultaneous interpretation method according to claim 1, characterized in that, The original audio is divided into multiple original audio segments, including: Identify all natural pauses in the original audio as cutting points, and cut the original audio into multiple original audio segments based on the cutting points. Each original audio segment is ≤T minutes, where T is a preset duration.

4. The simultaneous interpretation method according to claim 1, characterized in that, The process of separating the vocal track and background audio track from each of the original audio segments includes: For each audio segment, a pre-trained source separation model decouples the original audio segment into a vocal track and a background sound track.

5. The simultaneous interpretation method according to claim 1, characterized in that, The source language text and timestamp of each speaker are identified from the vocal track of each original audio segment, and structured captions for each speaker are generated, including: For each audio segment's voice track, analyze its voiceprint features, segment the audio stream of the voice track according to the speaker's identity, and label the speaker's identity ID for each segment. Transcribe the vocal segments of each speaker to generate source language text segments; Record the start time tstart and end time tend for each source language text segment; Generate a structured subtitle list corresponding to the human voice track for each audio segment. The structured subtitle list is in the form of: {[start timestamp-end timestamp, speaker ID, source language text]}.

6. The simultaneous interpretation method according to claim 1, characterized in that, The process of translating the source language text into the target language text includes: Dynamically load the corresponding machine translation engine based on the target language configured by the user; The machine translation engine iterates through the subtitle files in the structured subtitle list and translates the original subtitles into target language text.

7. The simultaneous interpretation method according to claim 5, characterized in that, The extraction of emotional slice audio and voiceprint clone audio based on the human voice track of each original audio segment includes: Based on the start time tstart, end time tend, and speaker identity speaker_id in the structured caption list, original human voice segments that are perfectly aligned with the time of the current sentence to be synthesized are segmented from the human voice track as emotional audio slices; For each unique speaker ID (speaker_id), a continuous segment of the speaker's voice with clear sound quality and no background interference of a preset duration is retrieved from the structured subtitles and used as the voiceprint clone audio. For each translation task, the corresponding outputs are the translated target language text, the emotional slice audio, and the voiceprint clone audio.

8. The simultaneous interpretation method according to claim 1, characterized in that, The process involves inputting the target language text, emotional slice audio, and voiceprint clone audio corresponding to each original audio segment into a general TTS engine, and outputting a target language audio track, including: Based on the IndexTTS2 engine, the system generates a speech waveform using the target language text as content by cloning audio from the speaker's voiceprint and emotional slice audio, and outputs the target language audio track.

9. A simultaneous translation system based on multi-track preservation and dual-reference synthesis, characterized in that, include: The acquisition module is used to acquire the original audio and video files and extract the original audio from the original audio and video files; The separation module is used to divide the original audio into multiple original audio segments, and to separate the vocal track and background sound track from each original audio segment; The generation module is used to identify the source language text and timestamp of each speaker from the voice track of each original audio segment, and generate structured subtitles for each speaker. The structured subtitles include {[start timestamp-end timestamp, speaker ID, source language text]}. Based on the voice track of each original audio segment, it extracts emotional slice audio and voiceprint clone audio. It is also used to input the target language text, emotional slice audio, and voiceprint clone audio corresponding to each original audio segment into a general TTS engine to generate the target language audio track. Finally, it synthesizes the original video footage, each target language audio track, and each background sound track to generate a target language audio and video file. The translation module is used to translate the source language text into the target language text.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the simultaneous translation method based on multi-track preservation and dual-reference synthesis as described in any one of claims 1-8.

Images