Multi-language off-line simultaneous interpretation method and device based on small bidirectional translation model

Abstract

The invention discloses a multilingual off-line simultaneous interpretation method based on a small bidirectional translation model. According to the method, voice front-end enhancement, automatic voice recognition, neural machine translation and text-to-voice conversion are sequentially executed locally on a terminal, and a unified sub-word list and a shared model are adopted to automatically judge translation between at least two languages and generate a target text / voice in a stream mode. In order to adapt to the mobile terminal, the translation model is quantized by an integer and deployed in a lightweight format, and the time delay is reduced in combination with beam width search, early stop and cache mechanisms. The voice front end adopts a causal U-shaped network to jointly suppress reverberation and noise, room impulse response and multi-noise are introduced in the training stage, a signal-to-noise ratio strategy is configured, and time sequence weighted filtering and mean square error gain are combined in the reasoning stage to improve intelligibility. The device is composed of a mobile terminal and a Bluetooth earphone, supports a collaborative chain of earphone side front enhancement and mobile phone side decoding, and realizes stable simultaneous transmission output in a weak network / no network scene.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing technology, and to the integrated processing of terminal-side speech denoising, automatic speech recognition, neural machine translation and text-to-speech, specifically to a method and apparatus for performing multilingual offline simultaneous interpretation locally on a mobile terminal. Background Technology

[0002] With globalization, the demand for cross-language real-time communication is growing rapidly, and the application of simultaneous interpretation has gradually expanded from professional conferences to a wider range of scenarios such as business travel, learning, and daily social interactions. To meet the requirements of low latency, high intelligibility, and ease of use, mobile simultaneous interpretation needs to simultaneously address multiple technical challenges, including offline capabilities, limited device resources, and ensuring voice quality in complex acoustic environments.

[0003] Existing mobile translation technologies face several major bottlenecks: First, high-quality translation services heavily rely on large cloud-based models, which cannot provide reliable services in unstable or offline environments, limiting their availability in offline scenarios. Second, common offline implementations typically link automatic speech recognition, neural machine translation, and speech synthesis, deploying independent models (direction-specific models) for each translation direction. For example, in Chinese-English translation, each of the two unidirectional models is approximately 312MB, resulting in a large total size. Combined with offline automatic speech recognition and speech synthesis modules, this leads to excessive storage and RAM consumption on mobile devices, making it difficult to install or run smoothly on low-end devices. Third, although large models supporting multilingual translation exist, their large number of parameters and high computational latency prevent local deployment on resource-constrained mobile devices. Fourth, bidirectional translation solutions often rely on external language recognition or manual switching of translation directions, making the process cumbersome and unsuitable for rapid and continuous simultaneous interpretation scenarios.

[0004] Meanwhile, speech noise reduction and reverberation cancellation are crucial for ensuring the quality of simultaneous interpretation. In large conference rooms and venues equipped with sound reinforcement systems, reverberation and reflections have a particularly significant impact on speech intelligibility: large room volumes, strong wall and ceiling reflections, playback from sound reinforcement systems, and concurrent speaking by multiple speakers all lead to long-term reverberation and complex reverse sound fields, which in turn cause speech blurring, interference between the original and translated speech, and howling or feedback between the microphone and speaker. Traditional speech noise reduction methods are effective in steady-state noise scenarios, but when dealing with non-steady-state noise (such as transient noises like clapping, keyboard clicks, and footsteps) or noise overlapping with speech frequencies (such as wind noise), it is often difficult to achieve a good balance between noise reduction effectiveness and speech naturalness and intelligibility. Deep learning-based methods, through the powerful fitting ability of neural networks, can usually achieve better noise reduction results than traditional algorithms in complex noisy environments, and can better preserve speech details and intelligibility. However, these models generally suffer from high computational complexity, large number of parameters, high memory consumption, and difficulty in meeting real-time processing requirements on resource-constrained devices, thus limiting their widespread deployment in scenarios such as Bluetooth headsets, mobile devices, and embedded conferencing equipment.

[0005] Some existing solutions have attempted to implement offline voice translation on terminal devices to improve the portability and real-time performance of simultaneous interpretation. For example, Chinese patent application CN109614628A discloses a translation method and system based on smart hardware, which integrates offline and online translation modules by combining a wireless earphone storage box with a translator, allowing users to complete voice translation without connecting to a mobile app or even without a network connection. This system utilizes the microphone / speaker on the earphone and the integrated terminal to achieve bidirectional voice translation of preset language pairs: the terminal's offline translation module recognizes and translates the speech of one language into another according to a pre-set translation mode, and then broadcasts the translation through the speaker of the terminal or earphone. This design reduces the awkwardness of device transmission during communication to some extent and improves ease of use. However, this solution is not specifically optimized for the model size and computing power limitations of terminal devices, making it difficult to balance memory usage and speed requirements for real-time translation. Especially in complex environments, this solution has limited support for speech noise reduction and recognition stability.

[0006] In addition, another Chinese patent, CN105210056A, proposes a hybrid offline / online speech translation system. This system allows client computing devices to translate speech from one language to another using a locally deployed translation model in offline mode, and to send the speech to a remote server for translation and return the result in online mode with a network connection. To balance offline translation performance and accuracy, the backend server can continuously train and optimize the large-scale translation model in the cloud based on user data, and distribute updated, streamlined versions of the model to the client, making the local model smaller, more efficient, and more relevant to the user's domain, thus improving offline translation quality. This solution addresses the need for providing machine translation services without a network connection, enabling offline translation to be performed to some extent using a streamlined model. However, its limitations lie in the fact that the offline model is still derived from a large cloud model, often requiring pre-built models or data for different language pairs for multilingual translation scenarios. It also struggles to support high-precision multilingual translation simultaneously on resource-constrained mobile devices. Furthermore, this solution primarily focuses on online / offline mode switching and model updates, without addressing issues such as simultaneous interpretation latency, incremental output stability, and recognition in noisy environments. Therefore, existing technologies still struggle to achieve multilingual offline simultaneous interpretation supported by small models on mobile devices, and there is room for improvement in terms of translation latency, accuracy, and adaptability to complex environments.

[0007] Existing technologies for offline simultaneous interpretation still suffer from common shortcomings: First, the systematic constraints on lightweight models at the receiving end are insufficient. Multilingual bidirectional translation often relies on the coexistence of multiple models specific to a particular direction or cloud rollback, making it difficult to implement in low- to mid-range devices due to the large size and memory overhead. Second, the actual acoustic conditions of "coexistence of reverberation and noise" are often handled in a fragmented manner, lacking joint modeling within a single causal network and an engineering path aligned with the timing of streaming decoding, indirectly weakening recognition and translation accuracy and increasing end-to-end latency. Third, in the implementation scenario of Bluetooth headsets and mobile phones working together, there is a lack of link design and resource scheduling specifications for headset-side preprocessing and mobile phone-side decoding, and thread affinity, weighted segmented loading, cache reuse, and power consumption feedback are not incorporated into the same optimization framework. Fourth, for high-reverberation scenarios such as conference rooms / reverse sound fields, a database-based adaptation and parameter selection process indexed by room impulse response has not yet been formed, making it difficult to stably guarantee pre-translation speech quality and continuous output. Based on this, the edge-side multilingual offline simultaneous interpretation solution proposed in this application aims to unify the causal front-end of "reverberation-noise joint modeling", the quantitative inference of the small bidirectional translation model with shared sub-words, and the link / thread / power consumption coordination of the headset-mobile phone into the same engineering baseline within the same device system. It takes into account size, computing power, latency and stability under weak network or offline conditions, and provides deployable and scalable basic capabilities for wide-area mobile scenarios and multi-participant conferences. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides a multilingual offline simultaneous interpretation method based on a small bidirectional translation model. This method solves the problems of coordinated de-reverberation and noise reduction, automatic translation direction determination, efficient local deployment of the miniaturized model, multilingual distribution, and time-delay energy consumption management in edge-side offline bidirectional simultaneous interpretation.

[0009] A multilingual offline simultaneous interpretation method based on a small bidirectional translation model, running locally on a terminal device, includes: receiving source language speech; inputting the source language speech into a speech denoising front-end optimized by inference to obtain denoised speech; obtaining source language text through local automatic speech recognition or receiving source language text input by the user; automatically determining the translation direction and generating target language text in real time by a neural machine translation model without the need for external switches; displaying the target language text on the terminal device and / or broadcasting the target language text through a local text-to-speech module; wherein the translation model and the word segmenter are shared between at least two languages, and are deployed in a lightweight inference format after integer quantization to meet the offline real-time inference needs of mobile devices.

[0010] Furthermore, this includes: performing low-bit integer quantization on the trained bidirectional translation model and using symmetric scaling factors, followed by linear dequantization during the inference phase; converting the quantized model into a terminal-specific inference format, which includes binary weights, a unified sub-vocabulary, and model configuration; running the model on a general-purpose processor on a local inference engine, employing a small beamwidth search combined with a repetition suppression strategy, and allocating threads according to a preset ratio of available processing resources; configuring an early stopping criterion and setting a single-step generation waiting latency threshold during the decoding phase; using streaming coding and attention caching mechanisms to incrementally decode the source language text from automatic speech recognition during the neural machine translation phase, and configuring lookahead windows and stabilization windows to reduce short-term write-back and output flicker; setting memory and power management strategies, reducing peak resource consumption through weight segmentation loading, cache reuse, and low-power scheduling, and handling off-vocabulary symbols with unknown sub-vocabulary backoff rules to maintain decoding continuity.

[0011] Furthermore, it employs thread affinity binding and dynamic thread allocation strategies to adapt to the terminal's thermal and power consumption status, including migrating inference threads between cores based on power consumption feedback, and adjusting the thread ratio of the decoding and noise reduction front-end according to whether the voice input is continuous and whether text-to-speech playback is enabled.

[0012] Furthermore, it also includes a link identification and distribution mechanism: before the session begins, the corresponding target language link is pre-identified for each participant's receiving device. When a participant speaks, their source language speech content is translated into at least one target language, and the corresponding translation result is distributed to the receiving device of the corresponding target language according to the link identification.

[0013] Furthermore, the translation model employs a shared sub-word segmenter jointly trained on at least two languages ​​to construct a unified vocabulary, and uses the unified vocabulary to uniformly encode and decode the source and target languages; wherein: the shared sub-word segmenter learns the set of sub-word units and mapping relationships from mixed corpora containing multilingual text based on a statistical sub-word segmentation algorithm; the unified vocabulary sets vocabulary size and character coverage parameters and includes functional markers for sequence start and end, unknown symbols, and padding; during the inference phase, out-of-vocabulary symbols are processed according to a backoff rule at a finer granularity to ensure the continuity and stability of cross-language decoding.

[0014] Furthermore, the translation direction determination mechanism is implemented in the following way: hybrid training with bidirectional parallel corpora is adopted, source and target pairs of different languages ​​are randomly mixed and balanced in proportion, and end-to-end training is carried out under a unified sub-word list, so that the model learns the input language features and corresponding translation direction mapping from the input representation; the global semantic representation and attention context summary at the encoding end constitute the language indication representation, and the decoding end uses the representation to drive the target space selection of the initial step to trigger the generation of the corresponding translation direction.

[0015] Furthermore, during end-to-end training, a unified word segmenter is used to encode the source and target language texts, and the translation model is jointly trained under a single shared vocabulary. This allows the model to simultaneously learn the semantic representation of the source sentence and the generation of the target sentence within the same representation space. The unified word segmenter learns the set of word units and their mapping relationships based on a statistical word segmentation algorithm. It sets the vocabulary size and character coverage parameters and introduces functional markers to handle sequence start and end, unknown symbols, and padding. During the inference phase, a rule of backtracking to finer-grained units is used for out-of-vocabulary symbols to ensure the continuity of encoding and decoding. The vocabulary is expanded through incremental retraining, and the translation model is jointly or sequentially retrained to adapt to the expanded word segmenter, thereby ensuring compatibility with newly added languages.

[0016] Furthermore, the speech denoising front-end operates according to the following process during training and inference: Step S1, acquire room impulse response, clean speech, and noise data; convolve the room impulse response adjusted for reverberation duration with the clean speech to obtain reverberated speech; and then perform enhancement and amplitude consistency normalization on the reverberated speech and noise in the spectral domain, followed by mixing according to a preset signal-to-noise ratio strategy to obtain training input, with the clean speech serving as the training target; Step S2, construct a U-shaped neural network comprising an encoding part, an enhancement part, and a decoding part; the encoding part downsamples the input spectrum to extract features; the enhancement part uses a channel-grouped recurrent neural network for causal modeling and normalization fusion; the decoding part upsamples and reconstructs the enhanced features; the encoding part and the decoding part... Step S3: Skip connections are set between components and the features are passed by feature addition; Step S4: Short-time Fourier transform is performed on the training input to obtain spectral features and input into the network to obtain a mask. A joint loss function consisting of a time-domain signal-to-noise ratio term, a speech component term, and a noise component term is adopted, and weighting coefficients are set for different frequency sub-bands. The network parameters are updated based on the joint loss; Step S5: In the inference stage, short-time Fourier transform is performed on the noisy speech and input into the network to obtain an initial mask. An optimized mask is generated based on time-weighted depth filtering that only uses the time-frequency information of the current frame and its preceding frames. The gain is calculated based on the minimum mean square error post-processing to correct the mask. The corrected mask is multiplied by the noisy spectrum and output as denoised speech by inverse short-time Fourier transform.

[0017] Furthermore, step S1 includes: acquiring room impulse response, clean speech and noise data; adjusting the room impulse response to a preset reverberation time range to maintain its attenuation characteristics; convolving it with clean speech to obtain reverberant speech; selecting one or more types of noise according to a preset probability; performing enhancement and amplitude consistency normalization on the reverberant speech and / or the noise in the spectral domain and normalizing at least one signal level to a preset range; mixing them according to a preset signal-to-noise ratio strategy to form training input and using clean speech as the training target.

[0018] Furthermore, step S2 includes: constructing a causal U-shaped neural network consisting of encoding, enhancement, and decoding parts; setting skip connections between encoding and decoding and performing channel matching before fusion; the encoding part performing downsampling feature extraction on the input spectrum; the enhancement part using time-series modeling units grouped by channel to model causally based on the current frame and previous frames and fusing them after normalization; the decoding part upsampling and reconstructing the enhancement features to restore frequency resolution; and the network output being a complex-valued mask that weights the real and imaginary parts of the input complex spectrum and limits the amplitude to 0 to 1.

[0019] Furthermore, step S3 includes: performing a short-time Fourier transform on the training input and inputting it into the network to obtain a mask; employing a joint loss including a time-domain signal-to-noise ratio term, a speech component term, and a noise component term and training under subband weighting, wherein the subband weights are preset or learnable and can be adaptively adjusted based on the training process, sample signal-to-noise ratio, or noise steady state; and performing backpropagation accordingly to update the network parameters.

[0020] Furthermore, step S4 includes: performing a short-time Fourier transform on the noisy speech during the inference stage to obtain an initial mask; using causal temporal weighted deep filtering to generate an optimized mask based solely on the time-frequency information of the current frame and its preceding frames at the current and neighboring frequencies, and using non-negative and normalized weighting coefficients; combining post-processing based on minimum mean square error and using prior signal-to-noise ratio estimation to correct the mask; multiplying the corrected mask with the noisy spectrum and outputting the temporal-domain denoised speech through an inverse short-time Fourier transform.

[0021] An offline simultaneous interpretation device, employing any of the methods described above, comprises a mobile terminal and a Bluetooth headset: the Bluetooth headset includes a microphone for acquiring source language speech and a speaker for playing target language speech; the mobile terminal includes a display screen and / or a speaker and a processing unit, the processing unit being configured to locally execute a speech noise reduction front-end, automatic speech recognition, neural machine translation, and text-to-speech processing flow sequentially, automatically determining the translation direction and generating a target language translation and its speech; the source language speech can be acquired by the microphone of the Bluetooth headset or the microphone built into the mobile terminal, and the source language speech and / or the target language speech are transmitted between the Bluetooth headset and the mobile terminal via a Bluetooth link; the translation can be displayed on the mobile terminal's display screen and / or played by the mobile terminal's speaker or the Bluetooth headset's speaker; the speech noise reduction front-end operates causally and is located between the audio input and automatic speech recognition, and the device achieves real-time simultaneous interpretation from speech input to target language speech output in an offline state.

[0022] The multilingual offline simultaneous interpretation method based on a small bidirectional translation model provided by this invention has the following advantages:

[0023] This invention provides a multilingual offline simultaneous interpretation method based on a small bidirectional translation model. Its advantages include: completing the entire local processing chain of "speech front-end enhancement—automatic speech recognition—neural machine translation—text-to-speech" on the terminal side; employing a miniaturized, integer-quantized bidirectional translation model and a shared sub-word segmenter; requiring only one model to cover bidirectional translation of at least two languages, avoiding the increased size and memory consumption associated with traditional "dual-model, bidirectional" parallel processing; deploying the model in a terminal-specific lightweight inference format; combining small-beamwidth search, repetition suppression, early stopping, and single-step waiting threshold decoding controls; and using streaming encoding and attention caching to achieve incremental output, reducing end-to-end latency and its fluctuations; introducing weighted segmented loading, cache reuse, and low-power scheduling at the resource layer; and using thread affinity and dynamic thread allocation to migrate inference threads between multiple cores based on power consumption and load, balancing performance and battery life; and automatically determining the translation direction by the model at the interaction layer without external switching, eliminating the need for manual switching and enabling "seamless translation" in continuous dialogue scenarios. The synergy between the aforementioned offline capabilities and small-scale model deployment enables the system to continue providing stable services in weak or no network environments, and maintains its installable, runnable, and scalable engineering characteristics on low-to-medium computing power terminals.

[0024] This invention unifies the modeling of room reverberation and environmental noise through a single causal network. During the training phase, room impulse response adjusted for reverberation duration and multi-source noise are introduced. Spectral domain enhancement and amplitude uniformity are employed, combined with a signal-to-noise ratio (SNR) strategy to generate training samples. The network is implemented with a causal structure consisting of encoding, enhancement, and decoding parts, incorporating skip connections and channel matching. The enhancement part performs temporal causal modeling based on the current frame and historical frames, outputting a mask weighted by a complex spectrum. The loss function consists of a temporal SNR term, a speech component term, and a noise component term, with sub-band weighting. During the inference phase, the mask is further corrected using temporally weighted depth filtering and gain post-processing based on the minimum mean square error criterion. Compared with the cascaded schemes of "separate noise reduction" and "separate dreverberation", this joint path avoids repeated feature extraction and dual-model superposition inference, significantly reducing the front-end computational burden and latency accumulation. More importantly, the front-end output is strictly aligned with the incremental decoding in terms of timing. After the clarity and stability of the speech recognition input are improved, the translation cascade error and short-term write-back phenomenon are reduced accordingly, forming a continuous gain from the acoustic side to the semantic side.

[0025] The two types of capabilities mentioned above form a mutually reinforcing closed-loop effect in the Bluetooth headset and mobile phone collaborative system. On the one hand, the offline miniaturized bidirectional model and lightweight inference format ensure that the mobile phone can still complete the local voice front-end enhancement, automatic speech recognition, neural machine translation, and text-to-speech processing even under conditions of limited computing power and battery. On the other hand, the joint front-end enhancement can be moved to the Bluetooth headset side to operate in a causal manner. After suppressing reverberation and noise during the acquisition phase, audio or incremental text is sent up via the Bluetooth link, reducing air interface occupation and link round-trip latency from the source, and maintaining stable operation of the headset with low power consumption. For scenarios that rely more on the mobile phone's computing power, the system can also process the recording directly on the mobile phone and send the translated speech back to the headset for playback. The two paths can be flexibly switched according to device capabilities and power consumption strategies. With thread affinity and dynamic thread allocation, weighted segmented loading and cache reuse, as well as thread migration based on power consumption feedback, the end-to-end latency and its fluctuations can be kept controllable under different models and thermal-power states, taking into account both real-time performance and battery life. Furthermore, through pre-meeting link calibration and target language distribution mechanisms, each receiver in a multi-person, multilingual conversation can stably obtain the corresponding language translation. An optional room impulse response database, characterized by reverberation time parameters, early reflection energy ratios, and room type estimation, can be established and indexed during training synthesis and inference selection. This allows the headphone-side front-end and mobile phone-side decoding to quickly adapt to high-reverberation environments such as conference rooms and reverse sound fields. Thus, the offline small model and the joint front-end support each other in the Bluetooth system: the front-end improves the quality of the recognized input to indirectly improve translation accuracy, while offline bidirectional decoding ensures continuous output under network-free conditions. The combination of these two features achieves a simultaneous interpretation experience with controllable latency, energy efficiency, and multilingual scalability. Attached Figure Description

[0026] Appendix Figure 1 : Flowchart of the simultaneous interpretation method of the present invention.

[0027] Appendix Figure 2 : A structural diagram of the bidirectional translation model of this invention.

[0028] Appendix Figure 3 : A diagram of the front-end network structure for speech noise reduction in this invention.

[0029] Appendix Figure 4 : Detailed structural diagram of the U-Net voice noise reduction front-end of the present invention.

[0030] Appendix Figure 5 : Flowchart of the speech noise reduction front-end training process of this invention.

[0031] Appendix Figure 6 : Flowchart of the front-end inference process for speech noise reduction in this invention. Detailed Implementation

[0032] The technical solution of the present invention will now be clearly and completely described in conjunction with the accompanying drawings. In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. The terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0033] This invention provides a speech denoising method based on inference optimization, applicable to speech front-end processing in offline simultaneous interpretation scenarios. This method achieves efficient noise suppression and reverberation removal through a deep learning model. Addressing the shortcomings of existing technologies, such as the challenges of handling non-steady-state noise, removing low-frequency noise, resource constraints, and real-time requirements, a comprehensive optimization scheme is proposed. The specific embodiments of this invention are described in detail below with reference to the accompanying drawings.

[0034] like Figure 1 As shown, the system of this invention includes a voice acquisition module, a voice noise reduction front-end module, an automatic speech recognition module, a bidirectional neural machine translation module, and a text-to-speech output module. These modules are connected sequentially according to the signal flow direction, forming a complete closed loop from source language speech input to target language speech output. Throughout the processing, all models and algorithms run locally on the mobile terminal, and inference calculations are performed by the terminal's internal processing unit. The system can achieve offline simultaneous interpretation of speech to speech even without a network connection.

[0035] During system operation, users can collect voice signals via Bluetooth headsets or the microphones built into mobile devices. The collected raw voice is first input to the voice noise reduction front-end module. This module uses a causal neural network model to denoise and dederease the input voice, effectively eliminating environmental noise, wind noise, and spatial reflection effects, thereby improving the signal-to-noise ratio and speech clarity. The processed high-quality voice signal is then transmitted in real time to the automatic speech recognition module.

[0036] The automatic speech recognition module employs a lightweight streaming engine to extract acoustic features and decode language from the input speech, outputting the corresponding source language text. When the user directly inputs text, this module can automatically skip the speech recognition step and pass the input text to subsequent modules. The recognized source language text is automatically sent by the system to the bidirectional neural machine translation module.

[0037] The neural machine translation module is the core of this system. Based on a lightweight bidirectional Transformer model structure, this module incorporates a unified multilingual shared vocabulary and a SentencePiece word segmenter, automatically determining the language category and translation direction of the input text to achieve bidirectional translation within the same model. The model employs a streaming incremental inference approach, progressively generating the corresponding target language output while receiving the input text, thus achieving real-time translation. Once the target language text is generated, the system synchronously outputs the result to the display and invokes the text-to-speech module.

[0038] The text-to-speech module is responsible for converting the translation results into natural and fluent speech output. Based on the acoustic features and pronunciation rules of the target language, this module uses a lightweight speech synthesis model to generate the target language speech signal. The output can be played through the mobile terminal's speaker or transmitted via Bluetooth to headphones for playback, enabling an instant voice translation experience.

[0039] To ensure low latency and high stability in the processing, the system's internal modules operate collaboratively through a unified memory cache and data channel. The output of the speech denoising module directly serves as the input to the speech recognition module. The incremental results from the recognition module are fed into the translation module via a streaming interface, and the translation results are transmitted to the speech synthesis module in real time, forming a continuous speech-to-speech data stream. The system employs a multi-threaded parallel architecture, effectively controlling end-to-end latency while maintaining smooth translation output, thus ensuring the real-time performance of simultaneous interpretation.

[0040] Regarding resource consumption, the overall system parameters of this invention have undergone quantization compression and computation graph optimization, resulting in a bidirectional translation model size of only about 78MB. The overall memory consumption is less than 1 / 5 of traditional solutions, enabling real-time inference on ordinary mobile terminal chips. The system design supports multi-language expansion; when adding new language pairs, only incremental updates to the vocabulary and fine-tuning of the model are required, thus providing excellent scalability and deployment flexibility.

[0041] Through the above structural design, this invention realizes multilingual offline simultaneous interpretation functionality on mobile terminals. The entire system can complete the entire processing chain, including voice acquisition, voice enhancement, voice recognition, bidirectional translation, and voice synthesis, even in a network-free environment. It can automatically recognize the input language, automatically determine the translation direction, and output the corresponding voice result, achieving a natural, fluent, and low-latency cross-language communication experience.

[0042] like Figure 2As shown in the figure, the present invention uses a bidirectional neural machine translation model. This model adopts a lightweight Transformer structure, combined with a unified SentencePiece tokenization mechanism and quantization inference optimization, to achieve the offline real-time processing ability of Chinese-English bidirectional translation. The model structure mainly includes an embedding layer, an encoder, a decoder, and an attention interaction module. Each part works together in the system to complete the automatic conversion from the source language text to the target language text.

[0043] The embedding layer is responsible for converting the input source language text into a sub-word level embedding representation. The model uses a unified shared vocabulary, and both Chinese and English use the same sub-word segmentation logic and encoding rules, thus eliminating the need for switching between different languages at the model level. When the user inputs Chinese, such as "How much is this?", the system decomposes the sentence into several sub-word units and converts them into a corresponding digital ID sequence and inputs it into the model; when the user inputs English, such as "What time is it now", the system also encodes it into an ID sequence with the same tokenizer to achieve cross-language unified processing.

[0044] The encoder performs feature extraction and context modeling on the input ID sequence. This part consists of multiple layers of self-attention networks and feed-forward layers, capturing the word dependencies through the global attention mechanism and generating semantic representations. The decoder generates the sub-word output of the target language step by step according to the semantic representation output by the encoder, combined with the attention weight distribution. Since the model has been trained with bidirectional mixed corpus during the training stage, it can automatically determine the translation direction according to the input language. For example, when the input is Chinese, the model automatically activates the English output space; when the input is English, it activates the Chinese output space, thus achieving automatic determination of bidirectional translation during inference.

[0045] When the user starts the offline simultaneous interpretation application of the present invention on a mobile terminal, the system first performs an initialization process. The application loads a 78MB quantized bidirectional translation model, a unified SentencePiece tokenizer, an automatic speech recognition (ASR) module, and a text-to-speech (TTS) module into the local memory at startup. Each module is deployed in a lightweight format, and both the model size and the running resources are much lower than the traditional multi-model solution. After initialization, the system enters the listening and waiting-for-translation state, ready to receive the user's voice input at any time.

[0046] Taking the actual communication scenario of Chinese-English dual users as an example, assume that a Chinese user says to an English user through a mobile terminal installed with this system in a network-free environment: "How much is this?" After the voice signal is collected by the microphone, it first enters the voice denoising front-end module, which performs denoising and reverberation removal on the voice and outputs a clear voice waveform. The optimized voice signal is sent to the ASR module, and the ASR engine transcribes it into Chinese text "How much is this?" in real time. Subsequently, the text is passed to the bidirectional word segmentation module, and the unified word segmenter decomposes the text into sub-word units and converts it into a corresponding ID sequence. After receiving this sequence, the core translation engine automatically recognizes the input language as Chinese and determines the translation direction as "Chinese → English". The model performs forward calculation and generates an ID sequence of the English result. The word segmentation module decodes this output sequence and restores it to the target language text "How much is this?". Finally, the output module displays the translated content on the terminal display screen, and at the same time calls the TTS engine to synthesize the translated text into English speech, which is broadcast through the speaker or earphone, realizing instant translation from Chinese speech to English speech.

[0047] After listening to the translation, the English user replies: "Twenty dollars." The system also collects its voice signal through the microphone, and after being processed by the voice denoising module, it is sent to the ASR module and transcribed into English text "Twenty dollars." The text is segmented and encoded by the bidirectional word segmenter and then input to the translation engine. The model automatically recognizes its language as English according to the input features and switches the translation direction to "English → Chinese". The translation engine performs forward inference and generates an ID sequence of the Chinese result, and the word segmentation module decodes it into the text "Twenty dollars." The output module synchronously displays the translation, and the TTS engine broadcasts it in Chinese speech. The whole process is completed within a millisecond-level response time, without manual intervention or manual switching of the language direction.

[0048] In the above bidirectional interaction process, the user does not need to perform any language selection or direction setting operations. The system can automatically determine the language according to the text features at the input stage through the unified word segmenter and the bidirectional model structure, and automatically determine the output direction at the translation stage. All inferences and calculations are completed locally on the terminal, without relying on network connection, ensuring high real-time performance and high stability in an offline environment. The system adopts a streaming incremental translation mechanism, which can start translation while the voice recognition outputs partial results, thus realizing an instant response mode of recognizing and translating simultaneously. Through multi-threaded task scheduling and asynchronous buffer design, the translation delay is controlled within 100 milliseconds, and the translated speech can be output almost synchronously with the original speech, realizing true offline simultaneous interpretation.

[0049] Throughout the dialogue, this system achieved automatic bidirectional translation between Chinese and English with extremely low translation latency and natural, fluent speech output. Compared to traditional solutions that require deploying two separate unidirectional models, this invention, through unified word segmentation and a shared vocabulary structure, only requires loading one bidirectional model to complete the translation task, significantly reducing model size and memory consumption. Simultaneously, quantization compression and inference optimization further reduce computational load, enabling the system to run stably on ordinary mobile terminals and supporting 24 / 7 offline communication scenarios. Functionally, this solution achieves fully automated language identification and translation direction determination; in terms of user experience, it eliminates the manual language switching required in traditional applications, greatly improving the fluency and naturalness of communication, and providing an efficient and lightweight technical solution for cross-language, offline, and real-time communication.

[0050] like Figure 3 and Figure 4 As shown, the speech denoising front-end module of this invention is implemented using a causal U-shaped neural network structure (U-Net), mainly used to jointly suppress reverberation and noise signals during the speech input stage, thereby improving the input quality of the speech recognition and translation module. This module includes three main functional units: an encoding part, an enhancement part, and a decoding part. These three parts achieve the fusion of features at different levels through skip connections, thus maintaining real-time performance while ensuring speech clarity and detail fidelity. This invention aims to address the shortcomings of existing speech denoising technologies in handling non-steady-state noise (such as clapping, keyboard sounds, and footsteps) and noise with frequencies close to speech (such as wind noise), including the challenges of non-steady-state noise processing, low-frequency noise removal, resource limitations of deep learning models, and real-time requirements.

[0051] When a user speaks, the speech signal captured by the microphone typically contains various environmental interferences, such as street noise, conference room reverberation, wind noise, and keyboard typing. To ensure the accuracy of recognition and translation, the input signal is first fed into the speech denoising front-end module. This module operates in real time in a causal manner, with its input being a time-domain speech signal and its output being the denoised speech waveform. The input speech is converted into a complex spectrum through a Short-Time Fourier Transform (STFT), where the real and imaginary parts represent the amplitude and phase information of the speech, respectively. The encoding part downsamples and extracts multi-scale features from this spectrum, extracting key speech information and compressing the time-frequency dimension through convolution and normalization layers. The enhancement part, as the core module, uses a grouped recurrent neural network (such as a grouped LSTM or GRU) to perform sequence modeling of features in the time dimension, distinguishing noise from speech by capturing the speech context relationship between consecutive frames. Because it adopts a causal modeling approach, the enhancement part only relies on the current frame and its historical frame information, without using future frame data, thus ensuring the real-time operation characteristics of the system.

[0052] The decoding section reconstructs the enhanced features layer by layer through upsampling and incorporates a skip connection mechanism to recover speech details. Each decoding layer receives high-resolution features from the corresponding coding layer and fuses them with the current layer's features to prevent high-frequency information loss. The final output is a complex spectral mask, where the real and imaginary parts correspond to the weighting coefficients of the real and imaginary parts of the input spectrum, respectively. The mask values ​​are constrained between 0 and 1 to ensure that noise is suppressed without amplifying the speech signal. After inverse short-time Fourier transform (ISTFT) reconstruction, the output is the time-domain denoised speech.

[0053] To ensure the robustness of the speech denoising front-end in complex environments, this invention designs a systematic training process, including three stages: data preparation, network construction, and model optimization. In the data preparation stage, the system first constructs a multi-source hybrid dataset. The dataset contains three basic types of samples: clean speech samples, noise samples, and room impulse response (RIR) samples. The clean speech samples are derived from a bilingual Chinese-English corpus, covering different speakers, speech rates, and intonations; the noise samples include various types such as street noise, wind noise, keyboard noise, and traffic noise; the RIR samples are used to simulate the reverberation characteristics of different rooms.

[0054] In the data mixing stage, the system uses the Target Reverberation Time Shortening (RTS) method to process RIR samples, limiting their reverberation time T60 to 0.15 seconds while maintaining exponential decay characteristics. The RTS-processed RIR samples are convolved with clean speech to generate reverberant speech, which is then mixed with noise samples within a set signal-to-noise ratio (SNR) range of [-5, 15] dB to form noisy speech. To enhance the model's generalization ability, the system randomly selects noise types and performs spectral augmentation and same scaling, ultimately generating training sample pairs covering multiple noise, reverberation, and language conditions. The generation of training data is fundamental to model performance; this invention designs a processing pipeline for mixed reverberation and noise data. The specific steps include: (1) preparing three types of basic data, including room impulse response (RIR), clean speech (Speech), and noise (Noise) files; (2) using RTS target processing to generate RTS RIR (target reverberation time T60=0.15 seconds, maintaining exponential decay characteristics); (3) convolving the speech signal with RTS RIR to generate reverberant speech, selecting noise with a probability p<0.5, setting the SNR range to [-5,15] dB; adjusting the full-range dBfs level of stationary noise to [-55,-65]; performing spectral enhancement and the same scaling on the speech and noise respectively, and uniformly adjusting the dBfs level to [-20,-35]; (4) mixing the reverberant speech with the processed noise to obtain the training input, and using the corresponding clean speech (speech processed by RTS) as the training target.

[0055] After the network is built, the system trains the model using a joint loss function. The loss function consists of three parts: temporal signal-to-noise ratio (SNR) loss, speech component preservation loss, and noise component suppression loss. The temporal SNR loss constrains the temporal difference between the output speech and the target clean speech, thus maintaining the phase consistency of the speech waveform; the speech component preservation loss constrains the model to retain the structural features and clarity of the speech as much as possible; and the noise component suppression loss enhances the model's noise removal capability under high-noise conditions. To further balance the noise reduction intensity and speech naturalness across different frequency ranges, this invention introduces a frequency-band weighting mechanism into the joint loss function. Specifically, the spectrum is divided into multiple sub-bands, and different weight coefficients are assigned to each band, allowing the model to focus more on noise suppression in the low-frequency range and preserve speech details in the mid-to-high-frequency range. This strategy significantly improves the model's suppression effect on low-frequency noise (such as wind noise) and avoids high-frequency distortion. During training, mini-batch samples are input into the network, and parameters are updated using the backpropagation algorithm. The model is trained alternately on samples containing various noises and reverberations, giving it strong robustness under multilingual and complex acoustic conditions. After training converges, the model parameters are quantized and compressed into a terminal-loadable format, which includes two parts: a binary weight file and a structure configuration file.

[0056] This invention's deep learning-based speech denoising (including reverberation reduction) process achieves noise suppression and speech preservation through the synergistic effect of Short-Time Fourier Transform (STFT), Neural Network (NN), Inverse Short-Time Fourier Transform (ISTFT), and Loss Function (LOSS). The input noisy speech signal is first converted into spectral features (containing real and imaginary parts) by the STFT module; these spectral features are then processed by the feature extraction module and input into the NN model; the predicted spectrum output by the NN model and the target clean spectrum are compared using the LOSS module to calculate the loss, which is then used for model training; after training, the NN model denoises the input spectral features, and the output clean spectrum is converted into a time-domain signal by the ISTFT module to obtain the final denoised speech.

[0057] During the inference phase, the noise reduction model can be independently deployed in a Bluetooth headset or mobile terminal. For headset-side deployment, the model runs on the local processing chip in integer quantization form, enabling real-time noise reduction at the acquisition end. For terminal-side deployment, the speech signal is transmitted to the terminal via Bluetooth, where this module performs noise reduction and enhancement processing before being sent to the speech recognition module. The two deployment methods can be automatically switched according to the device's computing power, power consumption, and real-time requirements. The system adopts a streaming inference mechanism, processing each frame of input independently and outputting it in real time, ensuring the temporal alignment and synchronization of the continuous speech stream. Through the above structure and training scheme, the speech noise reduction front-end of this invention can achieve efficient and low-latency real-time noise reduction under the resource constraints of mobile terminals. This module can not only effectively eliminate non-steady-state noise (such as keyboard sounds and footsteps), but also maintain the naturalness and intelligibility of speech under low-frequency overlapping noise conditions. Its output speech signal is significantly better than traditional methods in terms of signal-to-noise ratio and subjective clarity, providing a high-quality input guarantee for automatic speech recognition and neural machine translation, thus enabling the entire offline simultaneous interpretation system to operate stably and smoothly in complex environments.

[0058] This invention employs an Encoder-enhance-Decoder structure (primarily a U-Net model) and reduces computational cost through various improvements. The Encoder part (EncBlock) consists of multiple convolutional layers that downsample the input spectral features to extract multi-scale features. The enhancement part, located between the Encoder and Decoder, uses a recurrent neural network (such as LSTM / GRU) to process the high-dimensional features output by the Encoder, enhancing contextual information modeling. The Decoder part (DecBlock) consists of multiple deconvolutional layers that upsample the enhanced features, gradually restoring the output features that match the input spectral size. Each EncBlock and its corresponding DecBlock transfer features through skip connections, achieving high- and low-dimensional feature fusion.

[0059] The model improvement scheme includes: (a) Convolutional kernel size optimization: the kernel size of the convolutional layers in EncBlock and the deconvolutional layers in DecBlock is adjusted from 3×2 to 3×1; only the input layer and the last deconvolutional layer retain 5×2 convolutional kernels to utilize historical frame information. This optimization reduces the computational cost of convolution / deconvolution, reduces the memory access overhead of buffered data during real-time inference, and does not lose the denoising effect. (b) Deconvolutional layer improvement: the ordinary deconvolutional layer is replaced by a sub-pixel convolutional layer to implement the original deconvolutional layer function, which is more convenient for hardware and inference engine implementation. This modification reduces spectrogram artifacts and improves the denoising effect. (c) Grouped recurrent neural network design: grouped LSTM is used instead of ordinary LSTM. The input features and LSTM units are divided into two groups, and each group is calculated independently and then normalized and fused through an InstantNorm layer. This design reduces the computational cost of LSTM by 50% by adding only one InstantNorm layer, with an acceptable decrease in denoising performance. It can be further expanded to grouped GRU or 4-group partitioning to further reduce computational cost. (d) Improved skip connections: Before the improvement, the features of EncBlock and DecBlock were passed through channel concatenation, resulting in high computational cost. After the improvement, a 1×1 conv2d layer is added to the skip connections to compress the output features of EncBlock to the same number of channels as the input features of DecBlock, and the features are passed through feature addition (⊕). Additionally, gated convolution (with sigmoid activation controlling feature weights) is attempted. This improvement only increases the computational cost of the 1×1 convolution by a small amount, significantly reducing the computational cost of the deconvolution part, with only a slight decrease in performance and a reduction in SNR of approximately 0.1dB after denoising.

[0060] The loss function of this invention consists of multiple parts, which balance the noise reduction and speech distortion through frequency band weighting. Specifically, it includes: (a) a time-domain negative signal-to-noise ratio loss function: L_time = -SNR(s, s_hat), where s is the clean speech time-domain signal, and s_hat is the model prediction time-domain signal, used to implicitly constrain phase information. (b) speech component loss: L_speech =|S - S_hat|, where S is the spectral feature of the clean speech, and S_hat is the predicted feature. (c) noise component loss: L_noise = |N - N_hat|, where N is the spectral feature of the noise, and N_hat is the noise prediction feature. The total loss function is L = α * L_speech + β * L_noise + γ * L_time, where α, β, and γ are weighting coefficients, with values ​​ranging from [0,1]. Different α and β values ​​are set for different frequency bands, mainly to enhance low-frequency noise suppression: 0-500Hz (α=0.3, β=0.7), 500-2000Hz (α=0.35, β=0.65), 2000-4000Hz (α=0.4, β=0.6), 4000-8000Hz (α=0.5, β=0.5).

[0061] The post-processing module is used to optimize the mask output by the model and suppress residual noise, using the deepmmse method. The specific process is as follows: (1) Input the real and imaginary parts of the spectrum of the noisy speech, and the real and imaginary parts of the mask predicted by the model. (2) Estimate the clean speech spectrum: Calculate the estimated real and imaginary parts of the clean speech spectrum using a complex-valued mask: S_real_hat = mask_real * Y_real - mask_imag * Y_imag, S_imag_hat = mask_real * Y_imag + mask_imag * Y_real. (3) Calculate the power spectrum and signal-to-noise ratio: power spectrum of noisy speech P_y = |Y|^2, estimate power spectrum of clean speech P_s = |S_hat|^2, estimate power spectrum of noise P_n = P_y - P_s, prior signal-to-noise ratio ξ = P_s / P_n, posterior signal-to-noise ratio γ = P_y / P_n. (4) Calculate the gain and optimize the mask: gain G = sqrt(ξ / (ξ+ 1)) * exp(0.5 * integral), optimized mask: mask_opt = mask * G.

[0062] This invention employs a complex masking strategy combined with deep filtering. It addresses the issue of latency introduced by using future information in deep filtering by eliminating the use of future information and processing only the current frame and previous frames, thus improving background smoothness after noise reduction. Traditional masking schemes directly perform a one-to-one mapping of individual frequency points; complete deep filtering schemes fuse the frequency points of the current frame plus the preceding and following frames; the improved deep filtering scheme only covers the current frame plus the preceding frame, performing weighted fusion to avoid introducing latency from subsequent frames, making it suitable for low-latency scenarios.

[0063] like Figure 5 and Figure 6 As shown, the speech denoising front-end of this invention adopts a causal structure to run in real time during the inference stage, so as to realize a low-latency processing mode of simultaneous input, enhancement, and output of continuous speech stream. The entire real-time inference process includes five main steps: short-time Fourier transform, complex mask prediction, temporal weighted deep filtering, post-processing correction, and speech reconstruction. Each step is executed collaboratively in a pipeline manner in the multi-threaded environment of the mobile terminal, ensuring that the system has high stability and real-time performance during simultaneous interpretation.

[0064] When a user begins speaking, the input module acquires the speech signal and buffers it in the audio input buffer. The speech data is then framed and input into the Short-Time Fourier Transform (STFT) module, converting it into a complex spectral representation. This spectrum contains a mixture of speech signal and ambient noise; its real and imaginary parts correspond to the amplitude and phase information of the signal, respectively, providing time-frequency domain features for subsequent neural network processing. The STFT-processed spectrum is input to the speech denoising network, where it is processed sequentially by the trained encoding, enhancement, and decoding parts. The model uses a causal loop structure to extract features and enhance semantics from the input frames, relying only on the current and previous frames for prediction, without using future frames, thus eliminating the latency accumulation caused by contextual dependencies. The network output is the initial complex mask for the current frame, where its real and imaginary parts represent the weighting ratios of the real and imaginary parts of the input spectrum, respectively, used to distinguish the energy distribution of speech and noise. This mask initially achieves speech enhancement, but in high-noise or reverberant environments, local oversuppression or residual noise may still exist.

[0065] To further improve the smoothness and consistency of the output speech, this invention introduces a temporal weighted deep filtering mechanism during the inference stage. This mechanism performs weighted smoothing on the mask prediction results of several consecutive frames without increasing network parameters. The system assigns adaptive weight coefficients to each frame, with weights satisfying non-negative normalization constraints to ensure overall energy conservation. This filtering operation can be viewed as a time-recursive smoothing process used to suppress random noise fluctuations and transient interference. For example, when short-term non-steady-state noise (such as sudden keyboard clicks or wind noise) appears in the speech, the mask prediction results of adjacent frames can be used to correct the weights of the abnormal frames during the filtering process, thereby restoring speech continuity and preventing acoustic artifacts. This temporal filtering mechanism is seamlessly integrated with the network output stage, operating with constant time complexity in real-time streaming inference, thus improving speech stability while maintaining the system's low latency characteristics.

[0066] After mask smoothing, the system further performs a post-processing correction step. The post-processing module calculates the gain factor for each frequency point based on the minimum mean square error (MMSE) criterion to balance speech fidelity and noise suppression strength. Specifically, the system first estimates the prior and posterior signal-to-noise ratios (SNR) based on the power spectra of the noisy and enhanced speech, and then calculates the gain function based on the estimates. This gain function is multiplied by the corrected mask to obtain the final complex mask. When the prior SNR is high, the gain approaches 1 to preserve speech details; when the SNR is low, the gain automatically decreases to enhance noise reduction capabilities. Through this gain correction process, the model further reduces residual noise while maintaining speech clarity, effectively reducing the "muffled" or "metallic" sound of the speech and improving the naturalness and subjective listening experience of the output speech.

[0067] The optimized mask is multiplied point-by-point with the input spectrum and then fed into the Inverse Short Time Fourier Transform (ISTFT) module for time-domain reconstruction to obtain the real-time enhanced speech signal. The system employs a frame-level pipeline mechanism to process speech data, with input, inference, and output of each frame occurring simultaneously to ensure seamless continuity of the speech stream. To maintain the continuity of simultaneous interpretation, the system uses a frame buffer synchronization mechanism in the output stage; that is, while the speech recognition module receives a new frame, the noise reduction module outputs the result of the previous frame, thus achieving strict alignment in the time domain. The end-to-end latency of the entire noise reduction and recognition path is controlled within 20 milliseconds, meeting the requirements of real-time interaction.

[0068] The noise reduction inference module of this invention can be flexibly deployed on either a headset or a mobile terminal. Specifically: When deployed on a headset, the model is stored in integer quantization format on the Bluetooth headset chip. During runtime, spectrum analysis and mask inference are performed locally. The enhanced audio stream is transmitted to the mobile terminal via the Bluetooth link, reducing bandwidth usage and transmission latency at the source. When deployed on a terminal, the model runs in parallel on the mobile terminal's multi-core CPU. The timing filtering and post-processing modules are allocated independent threads and work in conjunction with the speech recognition module to perform pipelined processing, further improving real-time performance and resource utilization. Under different deployment conditions, the system can dynamically adjust the inference thread priority based on device load and power consumption: when intermittent speech input is detected, the system enters a low-power mode; when continuous speech is detected, the system automatically increases the computational resource allocation for the noise reduction and translation modules, achieving a balance between performance and energy consumption.

[0069] Through the aforementioned inference optimization mechanism, this invention achieves efficient non-steady-state noise suppression and low-frequency noise removal in the real-time noise reduction stage. Experimental results show that under wind noise and keyboard noise interference conditions, the speech signal-to-noise ratio (SNR) is improved by approximately 6 dB compared to the traditional spectral subtraction method; in complex reverberation environments, the speech intelligibility index (STOI) is improved by approximately 9%, and the speech intelligibility retention rate exceeds 95%. Compared to the traditional deep filtering method based on future frames, the causal inference scheme of this invention reduces system latency from 50 ms to less than 20 ms while maintaining a comparable noise reduction effect, enabling stable operation in scenarios such as real-time calls, simultaneous interpretation in conferences, and mobile interaction.

[0070] In summary, the speech denoising inference and optimization mechanism of this invention achieves efficient real-time suppression of non-steady-state noise, low-frequency noise, and reverberation signals on resource-constrained terminal devices. Its structural design balances computational efficiency with speech naturalness, providing a reliable speech front-end input guarantee for offline simultaneous interpretation systems, enabling the system to maintain accurate, fluent, and natural speech translation performance even in complex acoustic environments.

[0071] This invention also provides an offline simultaneous interpretation device for implementing the aforementioned multilingual offline simultaneous interpretation method based on a small bidirectional translation model. The device consists of a mobile terminal and a Bluetooth headset, which are interconnected via a Bluetooth link to collaboratively complete the entire local processing chain from voice acquisition to translation playback. The mobile terminal is the core processing unit, and the Bluetooth headset serves as the voice input and output terminal.

[0072] The Bluetooth headset includes two main components: a microphone and a speaker. The microphone is used to collect the source language speech signal of the user and output it to the Bluetooth module in the form of a continuous audio stream. The speaker is used to play the target language speech result generated by translation. The output end of the microphone is electrically connected to the input end of the Bluetooth module, and is used to digitize the collected speech signal and send it to the mobile terminal; the input end of the speaker is connected to the output end of the Bluetooth module, and is used to receive the target language audio signal sent back by the mobile terminal and convert it into audible speech for output. The Bluetooth chip integrated inside the headset supports the low-latency audio transmission protocol and can complete the upload and backhaul of speech data within milliseconds.

[0073] The mobile terminal includes a display screen, a speaker, and a processing unit. The display screen is used to display the generated target language translation on the visual interface, and the speaker can be used as an optional output end to directly play the translation result. The processing unit is the core control module of the device, and multiple functional modules are integrated inside it, corresponding to the voice noise reduction front end, the automatic speech recognition module, the neural machine translation module, and the text-to-speech module respectively. Each module is connected in sequence according to the signal flow direction to form a complete voice-to-voice offline translation processing link.

[0074] When the device is working, the user inputs the source language speech signal through the microphone of the Bluetooth headset, such as the Chinese sentence "What time is it now". After the speech signal is transmitted to the mobile terminal through the Bluetooth link, it first enters the voice noise reduction front end module. The voice noise reduction front end performs short-time Fourier transform, feature extraction, mask prediction, and inverse transform on the input signal through a causal U-shaped neural network, and outputs the noise-reduced speech. The noise-reduced speech enters the automatic speech recognition module and is transcribed into the source language text. The recognition result is then passed to the bidirectional neural machine translation module. Without any external language switching instructions, this module automatically determines the translation direction according to the language characteristics of the input text and generates the corresponding target language text. For example, when it detects that the input is Chinese, the translation module automatically outputs the English translation "What time is it now"; if the input is the English "What time is it now", it generates the Chinese translation "现在几点". The generated target language text is sent to the text-to-speech module for synthesis, and the corresponding speech signal is output.

[0075] The target language speech can be output through two paths. One is to be directly played by the speaker of the mobile terminal for the user to listen to locally; the other is to be backhauled to the speaker end of the Bluetooth headset through the Bluetooth link for playback to achieve a wearable simultaneous interpretation experience. The two paths can be automatically switched according to user settings or the system environment. When the user wears the headset to communicate in a meeting scenario, the translated speech is backhauled to the headset for playback; when the device is used for single-device learning or travel scenarios, the translated speech can be directly output by the speaker of the mobile terminal.

[0076] To ensure a balance between real-time performance and energy efficiency on mobile devices, the processing unit is equipped with thread affinity binding and dynamic thread allocation mechanisms. The system rationally allocates threads across the multi-core processor based on the task characteristics and power consumption requirements of different modules. The speech noise reduction front-end and automatic speech recognition module are bound to cores with higher clock speeds to ensure real-time response; the translation and text-to-speech modules automatically migrate to low-power cores when system temperature rises or battery power decreases. The system dynamically adjusts thread allocation based on the continuity of voice input. When continuous speech is detected, the resource allocation ratio for the translation module is increased; when pauses or silence are detected, the inference thread load is reduced and a low-power mode is entered, thereby extending device battery life.

[0077] The device also supports a weight segmentation loading and cache reuse mechanism. Model weight files are stored in segments according to functional modules. During execution, the system loads the appropriate parts as needed and releases the cache after inference to avoid peak memory usage. The noise reduction and recognition modules share an audio cache, while the translation and synthesis modules share a text cache, achieving seamless integration between modules. In this way, the system effectively reduces memory read / write operations while maintaining real-time performance, improving overall processing efficiency.

[0078] In practical applications of Chinese-English translation, the device can automatically determine the input language and switch the translation accordingly. For example, when a Chinese user says "What time is it now?", the earphones capture the speech, and the mobile terminal performs noise reduction, recognition, and translation, ultimately playing the English translation "What time is it now?" through the earphone speakers. When a foreign user says "What time is it now?", the system reverses the process, generating a Chinese translation and playing "What time is it now?". The entire process requires no network connection or manual intervention; the voice input and translation output are almost synchronous, achieving a natural and fluent offline simultaneous interpretation experience.

[0079] Through the above structural design, the offline simultaneous interpretation device of this invention achieves local closed-loop processing of voice acquisition, signal enhancement, speech recognition, bidirectional translation, and speech synthesis within a single system. The collaborative architecture between the Bluetooth headset and the mobile terminal enables the system to operate independently in weak network or even no network environments, meeting the real-time communication needs in multilingual scenarios. Thread affinity binding and power scheduling mechanisms ensure stable operation even on low-end and mid-range devices, while segmented loading and cache reuse strategies reduce peak system resource consumption, and Bluetooth link optimization reduces end-to-end transmission latency. This device features a simple structure, flexible deployment, and rapid response, and can be widely applied in scenarios such as international conferences, cross-border business, tourism learning, and multilingual teaching, providing a stable, efficient, and scalable technical implementation path for mobile simultaneous interpretation.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multilingual offline simultaneous interpretation method based on a small-scale bidirectional translation model, running locally on a terminal device, characterized in that, include: The system receives source language speech, inputs the source language speech into a speech denoising front-end optimized by inference to obtain denoised speech, obtains source language text through local automatic speech recognition or receives source language text input by the user, and automatically determines the translation direction and generates target language text in real time by a neural machine translation model without the need for external switches. The target language text is displayed on the terminal device and / or the target language text is broadcast through a local text-to-speech module. The translation model and word segmenter are shared between at least two languages ​​and are deployed in a lightweight inference format after integer quantization to meet the offline real-time inference needs of mobile devices.

2. The method according to claim 1, characterized in that, These include: performing low-bit integer quantization on the trained bidirectional translation model and using symmetric scaling factors, followed by linear dequantization during the inference phase; converting the quantized model into a terminal-specific inference format, which includes binary weights, a unified sub-vocabulary, and model configuration; running the model on a general-purpose processor on a local inference engine, employing a small beamwidth search combined with a repetition suppression strategy, and allocating threads according to a preset ratio of available processing resources; configuring an early stopping criterion and setting a single-step generation waiting latency threshold during the decoding phase; using streaming coding and attention caching mechanisms to incrementally decode the source language text from automatic speech recognition during the neural machine translation phase, and configuring lookahead windows and stabilization windows to reduce short-term write-back and output flicker; setting memory and power management strategies, reducing peak resource consumption through segmented weight loading, cache reuse, and low-power scheduling, and handling off-vocabulary symbols with unknown sub-vocabulary backoff rules to maintain decoding continuity.

3. The method according to claim 2, characterized in that, The strategy employs thread affinity binding and dynamic thread allocation to adapt to the terminal's thermal and power consumption status. This includes migrating inference threads between cores based on power consumption feedback, and adjusting the thread ratio of the decoding and noise reduction front-end according to whether the voice input is continuous and whether text-to-speech playback is enabled.

4. The method according to claim 1, characterized in that, It also includes a link identification and distribution mechanism: before the session starts, the corresponding target language link is pre-identified for each participant's receiving device. When a participant speaks, the source language speech content is translated into at least one target language, and the corresponding translation result is distributed to the receiving device of the corresponding target language according to the link identification.

5. The method according to claim 1, characterized in that, The translation model employs a shared sub-word segmenter jointly trained on at least two languages ​​to construct a unified vocabulary, and uses the unified vocabulary to uniformly encode and decode the source and target languages. Specifically, the shared sub-word segmenter learns the set of sub-word units and mapping relationships from mixed corpora containing multilingual text based on a statistical sub-word segmentation algorithm. The unified vocabulary sets vocabulary size and character coverage parameters and includes functional markers for sequence start and end, unknown symbols, and padding. During the inference phase, out-of-vocabulary symbols are processed according to a more granular backoff rule to ensure the continuity and stability of cross-language decoding.

6. The method according to claim 1, characterized in that, The translation direction determination mechanism is implemented in the following way: it adopts hybrid training with bidirectional parallel corpora, randomly mixes and balances the source and target pairs of different languages, and conducts end-to-end training under a unified sub-word list, so that the model learns the input language features and corresponding translation direction mapping from the input representation; the global semantic representation and attention context summary at the encoding end constitute the language indication representation, and the decoding end uses this representation to drive the target space selection of the initial step to trigger the generation of the corresponding translation direction.

7. The method according to claim 5, characterized in that, In the end-to-end training process, a unified word segmenter is used to encode the source and target language texts, and the translation model is jointly trained under a single shared vocabulary. This allows the model to simultaneously learn the semantic representation of the source sentence and the generation of the target sentence within the same representation space. The unified word segmenter learns the set of word units and their mapping relationships based on a statistical word segmentation algorithm. It sets the vocabulary size and character coverage parameters and introduces functional markers to handle sequence start and end, unknown symbols, and padding. During the inference phase, a rule of backtracking to finer-grained units is used for out-of-vocabulary symbols to ensure the continuity of encoding and decoding. The vocabulary is expanded through incremental retraining, and the translation model is jointly or sequentially retrained to adapt to the expanded word segmenter, thereby ensuring compatibility with newly added languages.

8. The method according to claim 1, characterized in that, The speech noise reduction front-end operates according to the following process during training and inference: Step S1: Acquire room impulse response, clean speech, and noise data. Convolve the room impulse response (adjusted for reverberation duration) with the clean speech to obtain reverberated speech. Then, enhance and normalize the reverberated speech and noise in the spectral domain, and mix them according to a preset signal-to-noise ratio strategy to obtain the training input. The clean speech serves as the training target. Step S2: Construct a U-shaped neural network containing an encoding part, an enhancement part, and a decoding part. The encoding part downsamples the input spectrum to extract features. The enhancement part uses a channel-grouped recurrent neural network for causal modeling and normalization fusion. The decoding part upsamples and reconstructs the enhanced features. The encoding part and the decoding part... Step S3: Set up skip connections and pass them by feature addition; Step S4: Perform short-time Fourier transform on the training input to obtain spectral features and input them into the network to obtain a mask. Use a joint loss function consisting of a time-domain signal-to-noise ratio term, a speech component term, and a noise component term, and set weighting coefficients for different frequency sub-bands. Update the network parameters based on the joint loss; Step S5: In the inference phase, perform short-time Fourier transform on the noisy speech and input it into the network to obtain an initial mask. Generate an optimized mask based on time-weighted deep filtering that only uses the time-frequency information of the current frame and its preceding frames. Calculate the gain based on minimum mean square error post-processing to correct the mask. Multiply the corrected mask by the noisy spectrum and output the denoised speech through inverse short-time Fourier transform.

9. The method according to claim 8, characterized in that, Step S1 includes: acquiring room impulse response, clean speech and noise data; adjusting the room impulse response to a preset reverberation time range to maintain its attenuation characteristics; convolving it with clean speech to obtain reverberant speech; selecting one or more types of noise according to a preset probability; performing enhancement and amplitude consistency normalization on the reverberant speech and / or the noise in the spectral domain and normalizing at least one signal level to a preset range; mixing them according to a preset signal-to-noise ratio strategy to form training input and using clean speech as the training target.

10. The method according to claim 8, characterized in that, Step S2 includes: constructing a causal U-shaped neural network consisting of encoding, enhancement, and decoding parts; setting skip connections between encoding and decoding and performing channel matching before fusion; the encoding part extracting downsampled features from the input spectrum; the enhancement part using time-series modeling units grouped by channel to model causally based on the current frame and previous frames and fusing them after normalization; the decoding part upsampling and reconstructing the enhanced features to restore frequency resolution; and the network outputting a complex-valued mask that weights the real and imaginary parts of the input complex spectrum and limits the amplitude to 0 to 1.

11. The method according to claim 8, characterized in that, Step S3 includes: performing a short-time Fourier transform on the training input and inputting it into the network to obtain a mask; using a joint loss including a time-domain signal-to-noise ratio term, a speech component term, and a noise component term and training under subband weighting, wherein the subband weights are preset or learnable and can be adaptively adjusted based on the training process, sample signal-to-noise ratio, or noise steady state; and performing backpropagation accordingly to update the network parameters.

12. The method according to claim 8, characterized in that, Step S4 includes: performing a short-time Fourier transform on the noisy speech during the inference stage to obtain an initial mask; using causal temporal weighted deep filtering to generate an optimized mask based solely on the time-frequency information of the current frame and its preceding frames at the current and neighboring frequencies, and using non-negative and normalized weighting coefficients; combining post-processing based on minimum mean square error and using prior signal-to-noise ratio estimation to correct the mask; multiplying the corrected mask by the noisy spectrum and outputting the temporal denoised speech through an inverse short-time Fourier transform.

13. An offline simultaneous interpretation device, employing the method described in any one of claims 1 to 12, characterized in that, The device includes a mobile terminal and a Bluetooth headset: the Bluetooth headset includes a microphone for capturing source language speech and a speaker for playing target language speech; the mobile terminal includes a display screen and / or a speaker and a processing unit, the processing unit being configured to locally execute a speech noise reduction front-end, automatic speech recognition, neural machine translation, and text-to-speech processing flow sequentially, automatically determining the translation direction and generating the target language translation and its speech; the source language speech can be captured by the microphone of the Bluetooth headset or the microphone built into the mobile terminal, and the source language speech and / or the target language speech are transmitted between the Bluetooth headset and the mobile terminal via a Bluetooth link; the translation can be displayed on the display screen of the mobile terminal and / or played by the speaker of the mobile terminal or the speaker of the Bluetooth headset; the speech noise reduction front-end operates in a causal manner and is located between the audio input and the automatic speech recognition, and the device achieves real-time simultaneous interpretation from speech input to target language speech output in offline mode.

Images