Multi-language voice real-time translation method and system based on large model
By introducing real-time timbre transfer technology and optimizing data transmission schemes, the problems of lack of timbre and high latency in traditional cross-language speech translation systems have been solved, enabling personalized speech synthesis and low-latency translation, thus improving user experience and system performance.
Patent Information
- Application Number
- CN202511740433.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-25
- Publication Date
- 2026-03-17
AI Technical Summary
Traditional cross-language real-time speech translation systems lack modules for learning and converting speaker timbre, have poor inter-module coordination, rigid data interaction methods, and high network transmission latency, resulting in output results that lack personalization and naturalness, and are inefficient and inaccurate.
Real-time timbre transfer technology is introduced, deep learning algorithms are used to extract timbre features and apply them in the speech synthesis stage, and data transmission is optimized by combining HTTP/2 protocol and local caching. Speech recognition and synthesis are performed through a large model translation module.
It achieves timbre preservation in speech translation, improves user experience and system performance, reduces end-to-end translation latency, and meets the needs of real-time dialogue and simultaneous interpretation in conferences.
Smart Images
Figure CN121687009A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of real-time speech translation technology, specifically relating to a method and system for real-time multilingual speech translation based on a large model. Background Technology
[0002] In existing cross-language real-time speech translation systems, such as Figure 1 As shown, traditional solutions generally employ a cascaded architecture of speech recognition, machine translation, and speech synthesis. While this approach covers the basic process from source language speech to target language speech, it suffers from the following drawbacks in practical applications: (1) Traditional solutions only include three core modules and lack a module for learning and converting the speaker’s timbre. This results in the final synthesized speech only using general, preset acoustic features, which cannot preserve and restore the original speaker’s unique timbre and rhythm, making the output results lack personalization and naturalness, and providing a poor experience in application scenarios that require identity recognition or emotional transmission.
[0003] (2) The data interaction between modules is too simple and rigid, mainly manifested as one-way, sequential text transmission. This highly independent black-box design leads to poor inter-module collaboration and makes cross-module joint optimization impossible. For example, pronunciation ambiguity or sentence segmentation problems in the speech recognition stage will be directly transmitted and amplified due to the lack of semantic feedback from the translation module, forming an error accumulation effect and ultimately affecting the overall output quality.
[0004] (3) The speech recognition model is not efficient and accurate enough in noisy environments or dialect scenarios; the machine translation module usually relies on remote servers, which cause high latency and slow response due to network transmission.
[0005] In summary, traditional solutions have inherent defects in terms of functional completeness, system coordination, and depth of technical optimization, which seriously restrict the performance, efficiency, and user experience of cross-language speech conversion systems. Summary of the Invention
[0006] This invention addresses the shortcomings of existing technologies by providing a real-time multilingual speech translation method and system based on a large model. By introducing real-time timbre transfer technology, it enhances the user experience. Furthermore, by transmitting data to the large model translation module and via HTTP / 2 protocol through an optimized internal interface, it reduces end-to-end translation latency, thus meeting the requirements of application scenarios with extremely high latency requirements, such as real-time dialogue and simultaneous interpretation in conferences.
[0007] This invention provides the following technical solution: Firstly, a real-time multilingual speech translation system based on a large model is provided, including: The voice acquisition module is used to pre-acquire the speaker's voice sample and acquire the voice to be translated in real time; The speech recognition module is used to convert the speech to be translated into source language text using a speech recognition algorithm; The large model translation module is used to send source language text to the selected large model API via network request, and receive the target language text translated into real time by the selected large model. The target language parameter is specified when sending the translation request. The timbre feature extraction module is used to extract timbre features from pre-collected speech samples; The speech synthesis and output module uses a speech synthesis model to generate target language speech about the speaker based on the extracted timbre features and the target speech text.
[0008] Optionally, when the speech acquisition module acquires speech to be translated in real time, it controls the intensity of the acquired speech signal through adaptive gain control. Specifically, it initially amplifies the acquired speech and monitors the intensity of the amplified speech signal in real time. If the signal is within a set threshold range, it maintains the current gain. If the signal exceeds the set threshold range, it adjusts the current gain. Specifically, if the signal exceeds a set high value, it quickly reduces the gain. If the signal is below a set low value, it slowly increases the gain.
[0009] Optionally, the voice acquisition module adjusts the current gain when the signal amplitude exceeds a set threshold range, specifically by obtaining the target gain value based on the ratio of the current signal amplitude to the target signal amplitude. ;according to Obtain the adjusted gain ,in, For the current gain, To adjust the coefficient, select the set value when rapidly reducing the gain. Select the set value while slowly increasing the gain. , .
[0010] Optionally, the speech recognition module converts the speech to be translated into source language text using a speech recognition algorithm, specifically by using an end-to-end deep learning model encapsulated in the SDK to decode the speech signal into a text sequence.
[0011] Optionally, the large model translation module uses the HTTP / 2 protocol for data transmission when sending translation requests, and performs Gzip compression on the request data; The large model selected by the large model translation module adopts the Transformer architecture and captures long-distance dependencies in the source language text through a multi-head attention mechanism; The large model translation module adopts a local caching mechanism. Before performing translation, it monitors whether there are cached results for the source language text. If there are, the cached results are used directly. If not, the selected large model is used to translate the language text.
[0012] Optionally, the timbre feature extraction module extracts timbre features from the pre-collected speech samples by using a deep learning-based feature extraction algorithm to extract the timbre features of the pre-collected speech samples, including fundamental frequency, formants, and spectral envelope. The deep learning-based feature extraction algorithm is a variational autoencoder or a generative adversarial network.
[0013] Secondly, a real-time multilingual speech translation method based on a large model is provided, including: Pre-collect speaker's voice samples and collect the voice to be translated in real time; The speech to be translated is converted into source language text using a speech recognition algorithm; The source language text is sent to the selected large model API via a network request, and the target language text is received after real-time translation of the source language text by the selected large model. The target language parameter is specified when sending the translation request. Phonic features are extracted from the pre-collected speech samples; The extracted timbre features and target speech text are used to generate the target language speech about the speaker through a speech synthesis model.
[0014] Optionally, when sending a translation request, the HTTP / 2 protocol is used for data transmission, and the request data is compressed using Gzip. Before performing the translation, the system checks whether there are cached results for the source language text. If they exist, the cached results are used directly; otherwise, the selected large model is used to translate the language text.
[0015] Thirdly, a computer device is provided, including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the steps of the real-time multilingual speech translation method based on a large model as described in any of the second aspects.
[0016] Fourthly, a computer-readable storage medium is provided for storing a computer program; when the computer program is executed by a processor, it implements the steps of the real-time multilingual speech translation method based on a large model as described in any of the second aspects.
[0017] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention introduces real-time timbre transfer technology, which extracts the speaker's timbre features during the speech acquisition stage and applies these features to the generation of target language audio during the speech synthesis stage. Deep learning algorithms are used to model and transfer the timbre features, ensuring that the translated speech retains the speaker's unique timbre, such as pitch, tone, and intonation. This personalized speech synthesis method greatly enhances the user experience. Furthermore, after recognizing the text, this invention directly transmits the data to the large model translation module through an optimized internal interface, avoiding the data loss and delay during conversion between different interfaces in traditional solutions. This achieves seamless data flow and efficient processing between modules, significantly improving the system's performance and stability.
[0018] (2) The HTTP / 2 protocol is used to improve data transmission efficiency and the data is compressed. In the large model translation process, local caching and optimized request strategies are used to reduce API call time and significantly reduce end-to-end translation latency, which can meet the application scenarios with extremely high latency requirements such as real-time dialogue and conference simultaneous interpretation. Attached Figure Description
[0019] Figure 1 This is a block diagram of the existing cross-language real-time speech translation system; Figure 2 This is a block diagram of the multilingual speech real-time translation system based on a large model according to the present invention; Figure 3 This is a flowchart of the multilingual speech real-time translation method based on a large model according to the present invention. Detailed Implementation
[0020] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be used to limit the scope of protection of the present invention. It should be noted that the term "comprising" and any variations thereof in the specification, claims and the above-mentioned drawings of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or devices.
[0021] Example 1 like Figure 2 As shown, a multilingual speech real-time translation system based on a large model is provided, including: I. Voice Acquisition Module.
[0022] Used to pre-collect speech samples of speakers and to collect speech to be translated in real time.
[0023] On Android devices, the built-in microphone driver is used to acquire voice signals. To improve voice acquisition quality, adaptive gain control technology is employed to automatically adjust the microphone's input gain, ensuring clear and stable voice signals are obtained under varying ambient volume conditions.
[0024] Specifically, the system performs initial amplification on the real-time acquired speech and monitors the strength of the amplified speech signal in real time. If the signal is within a set threshold range, the current gain is maintained. If the signal exceeds the set threshold range, the current gain is adjusted. Specifically, if the signal exceeds the set high value, the gain is rapidly reduced. If the signal is below the set low value, the gain is slowly increased.
[0025] In this embodiment, the target gain value is obtained based on the ratio of the current signal amplitude to the target signal amplitude. ;according to Obtain the adjusted gain ,in, For the current gain, To adjust the coefficient, select the set value when rapidly reducing the gain. Select the set value while slowly increasing the gain. , .
[0026] The pre-collected speech text, typically 30 seconds to 1 minute long, is used to extract the timbre.
[0027] II. Speech Recognition Module.
[0028] Used to convert speech to be translated into source language text using speech recognition algorithms.
[0029] Specifically, the process involves using an end-to-end deep learning model encapsulated in the SDK to decode speech signals into text sequences. This end-to-end deep learning model can optionally be a LAS (Local Alignment) model, but acoustic and language models can also be used. The acoustic model accurately identifies phoneme features in the speech signal, while the language model optimizes phoneme combinations based on grammatical rules and semantic information to improve recognition accuracy. Furthermore, during LAS model training, the inherent alignment relationship between speech and text is automatically learned, and during decoding, algorithms such as beam search are used to find the most probable text sequence based on this learned relationship.
[0030] Of course, in some other embodiments, the Dynamic Time Warping (DTW) algorithm can also be used during the training process to solve the time alignment problem of speech signals at different speech rates.
[0031] III. Large Model Translation Module.
[0032] This tool is used to send source language text to a selected large model API via a network request and to receive the target language text translated in real time by the selected large model. The target language parameter is specified when sending the translation request.
[0033] API (Application Programming Interface): An application programming interface is a bridge for communication between different software or hardware, consisting of a series of predefined functions. With APIs, developers can access specific functional routines without needing to understand the complex source code and working mechanisms of the software or hardware. In this solution, Alibaba Cloud's Tongyi Qianwen, Baidu's Wenxin Yiyan, and iFlytek's Xinghuo Big Model, among others, provide translation functionality to Android terminals through APIs. Developers only need to pass in parameters such as the text to be translated and the target language according to the API rules to obtain the translation results from the big model, easily achieving the integration of powerful translation services and endowing applications with efficient translation capabilities.
[0034] The large-scale model translation module utilizes the Transformer architecture, employing a multi-head attention mechanism to capture long-distance dependencies in the source language text. Transformer is an innovative deep learning architecture based on attention mechanisms, initially emerging in the field of natural language processing. It breaks away from the traditional structural patterns of recurrent neural networks (RNNs) and convolutional neural networks (CNNs), directly capturing dependencies between different positions in the input sequence through multi-head attention and supporting parallel data processing. This makes Transformer more efficient and perform better than traditional architectures when processing long sequences of data. In the large-scale model translation module of this solution, the Transformer architecture assists the large model in performing in-depth semantic understanding and efficient encoding and decoding operations on the source language text, thereby achieving accurate translation results.
[0035] To further reduce network transmission latency, the HTTP / 2 protocol is used for data transmission, and the requested data is compressed using Gzip.
[0036] HTTP / 2, the second major version of the Hypertext Transfer Protocol, represents a significant performance breakthrough compared to HTTP / 1.1. HTTP / 2 employs a binary framing layer and supports multiplexing, allowing multiple requests and responses to be sent and received simultaneously over a single connection, significantly improving data transmission efficiency. Furthermore, it features header compression and server push functionality, effectively reducing network latency and bandwidth consumption. In the large-scale model translation module of this solution, the use of the HTTP / 2 protocol optimizes network transmission performance, reduces data transmission latency, and ensures the real-time performance and smoothness of the translation process.
[0037] Gzip is both a data compression algorithm and a file format. It processes data using lossless compression, significantly reducing data size without losing information. In network transmission scenarios, Gzip-compressed data reduces bandwidth usage and shortens transmission time. In this solution, data sent to the large model API is Gzip-compressed to ensure data integrity while accelerating data transmission, further reducing overall translation latency and improving system response efficiency.
[0038] In some other embodiments, the large model translation module also employs a local caching mechanism. Before performing translation, it checks whether cached results exist for the source language text. If they do, the cached results are used directly; otherwise, the selected large model is used for translation. Prioritizing retrieval from the cache reduces repeated calls to the large model API, thereby lowering overall translation latency.
[0039] Specifically, on the Android client, the Room library is primarily used to create a local SQLite database, storing the source text, target language, translation results, and timestamps. Before translation, the database is queried using the MD5 hashes of the source text and target language as unique keys. If a match is found, the result is returned directly; otherwise, after a successful translation of the selected large model, the new result is asynchronously stored in the database. Simultaneously, the cache lifecycle is managed through periodic cleanup or an LRU (Least Recently Used) strategy to prevent storage space from being occupied indefinitely.
[0040] IV. Timbre Feature Extraction Module.
[0041] Used to extract timbre features from pre-collected speech samples.
[0042] The timbre features of the pre-collected speech samples are extracted using a deep learning-based feature extraction algorithm, including fundamental frequency, formants, and spectral envelope. The deep learning-based feature extraction algorithm is either a variational autoencoder or a generative adversarial network.
[0043] Variational autoencoders (VAEs) map speech signals to a low-dimensional feature space by constructing encoder and decoder structures, extracting parameters containing the speaker's timbre characteristics, such as fundamental frequency, formants, and spectral envelope. The encoder is responsible for mapping the input data to a low-dimensional latent space and extracting the feature representation of the data (i.e., feature parameters); the decoder then attempts to reconstruct the original data based on the feature parameters of the latent space.
[0044] Generative Adversarial Networks (GANs) consist of two adversarial neural networks: a generator and a discriminator. The generator's task is to generate samples similar to real data, while the discriminator is responsible for distinguishing between the generated samples and real samples. During adversarial training, both continuously optimize their performance, with the generator generating increasingly realistic data and the discriminator's ability to distinguish between real and generated samples becoming increasingly powerful.
[0045] V. Speech Synthesis and Output Module.
[0046] The extracted timbre features and target speech text are used to generate the target language speech about the speaker through a speech synthesis model.
[0047] Speech synthesis models are typically VITS (Virtual Interpretive Synthesis) or Conditional Generative Adversarial Networks (cGANs). cGANs introduce additional conditional information (such as the translated text in this scheme) into the inputs of the generator and discriminator, enabling the generator to produce corresponding samples according to specific conditions. During the speech synthesis stage, cGAN uses the translated text as a condition, fusing timbre feature parameters extracted from the speaker's speech to generate target language audio that meets the requirements, achieving a highly personalized speech synthesis effect and making the translated speech more realistic and unique.
[0048] It's worth noting that during the synthesis process, parameters such as speech rate, tone, and volume can be flexibly controlled. For example, in business scenarios, a more composed voice with a moderate pace can be set, and the target language can be selected; for entertainment scenarios, a more lively voice with rich intonation can be used. Finally, the synthesized audio is output from the speaker through the Android terminal's audio playback driver, completing the entire real-time speech translation process.
[0049] By modeling and transferring timbre features using deep learning algorithms, the translated speech retains the speaker's unique timbre, such as pitch, tone, and intonation. This personalized speech synthesis method greatly enhances the user experience, making users feel as if they are speaking the target language themselves when they hear the translated speech, effectively reducing the sense of unfamiliarity and barriers in cross-language communication. Speech recognition and timbre feature extraction are performed in parallel during speech synthesis, thereby accelerating audio generation.
[0050] Example 2 like Figure 3 As shown, a real-time multilingual speech translation method based on a large model is provided, including: Pre-collect speaker's voice samples and collect the voice to be translated in real time; The speech to be translated is converted into source language text using a speech recognition algorithm; The source language text is sent to the selected large model API via a network request, and the target language text is received after real-time translation of the source language text by the selected large model. The target language parameter is specified when sending the translation request. Phonic features are extracted from the pre-collected speech samples; The extracted timbre features and the target speech text are used to generate the target language speech about the speaker through a speech synthesis model.
[0051] When sending a translation request, the HTTP / 2 protocol is used for data transmission, and the request data is compressed using Gzip. Before performing the translation, the system checks whether there are cached results for the source language text. If they exist, the cached results are used directly; otherwise, the selected large model is used to translate the language text.
[0052] For more detailed information on the above methods, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.
[0053] Example 3 The present invention provides a computer device, including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the steps of the above-described multilingual speech real-time translation method based on a large model.
[0054] For more detailed information on the above methods, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.
[0055] Example 4 This invention provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, it implements the steps of the above-described multilingual speech real-time translation method based on a large model.
[0056] For more detailed information on the above methods, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.
[0057] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The devices and storage media disclosed in the embodiments are described simply because they correspond to the methods disclosed in the embodiments; relevant parts can be referred to the method section.
[0058] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.
[0059] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.
Claims
1. A large model-based multilingual speech real-time translation system, characterized by, Comprise: A voice collection module for pre-collecting a voice sample of a speaker and collecting a voice to be translated in real time; A voice recognition module for converting the voice to be translated into a source language text through a voice recognition algorithm; A large model translation module for sending the source language text to a selected large model API through a network request and receiving a target language text translated by the selected large model in real time, wherein a target language parameter is specified when sending the translation request; A timbre feature extraction module for extracting timbre features from the pre-collected voice sample; A voice synthesis and output module for generating a target language voice about the speaker through a voice synthesis model based on the extracted timbre features and the target voice text. 2.The large model-based multilingual speech real-time translation system according to claim 1, wherein, When the voice collection module collects the voice to be translated in real time, the voice signal intensity collected through adaptive gain control, specifically: the initially amplified real-time collected voice is monitored in real time, if it is within the set threshold interval, the current gain is maintained, if it is beyond the set threshold interval, the current gain is adjusted, specifically, if it exceeds the set high value, the gain is quickly reduced, if it is below the set low value, the gain is slowly increased. 3.The large model based multilingual speech real-time translation system according to claim 2, wherein, The voice acquisition module adjusts the current gain beyond a set threshold interval, specifically: according to the ratio of the current signal amplitude and the target signal amplitude, a target gain value is obtained ; according to , an adjusted gain is obtained, wherein is the current gain, is an adjustment coefficient, a set value is selected when the gain is rapidly reduced , and a set value is selected when the gain is slowly increased , . 4.The large model-based multilingual speech real-time translation system according to claim 1, wherein, The voice recognition module converts the voice to be translated into a source language text through a voice recognition algorithm, specifically: an end-to-end deep learning model packaged in an SDK is used to decode the voice signal into a text sequence. 5.The large model based multilingual speech real-time translation system according to claim 1, wherein, The large model translation module uses HTTP / 2 protocol for data transmission when sending the translation request, and performs Gzip compression processing on the request data; The large model translation module uses a Transformer architecture, which captures long-distance dependency relationships in the source language text through a multi-head attention mechanism; The large model translation module uses a local cache mechanism to monitor whether there is a cache result for the source language text before performing translation, if there is, the cache result is used directly, if not, the selected large model is used to translate the language text. 6.The large model based multilingual speech real-time translation system according to claim 1, wherein, The timbre feature extraction module extracts timbre features from the pre-collected voice sample, specifically: a deep learning-based feature extraction algorithm is used to extract the timbre features of the pre-collected voice sample, including fundamental frequency, formant and spectral envelope; The deep learning-based feature extraction algorithm is a variational autoencoder or a generative adversarial network.
7. A large model-based multilingual speech real-time translation method, characterized in that, Comprise: Pre-collecting a voice sample of a speaker and collecting a voice to be translated in real time; Converting the voice to be translated into a source language text through a voice recognition algorithm; Sending the source language text to a selected large model API through a network request and receiving a target language text translated by the selected large model in real time, wherein a target language parameter is specified when sending the translation request; Extracting timbre features from the pre-collected voice sample; Generating a target language voice about the speaker through a voice synthesis model based on the extracted timbre features and the target voice text. 8.The large model-based multilingual speech real-time translation method according to claim 7, characterized in that, When sending the translation request, HTTP / 2 protocol is used for data transmission, and Gzip compression processing is performed on the request data; Before performing the translation, it is monitored whether there is a cache result for the source language text, if there is, the cache result is directly used, if not, a selected large model is used to translate the language text.
9. A computer device, comprising: A computer program product comprising a computer readable medium storing a computer program, wherein the computer program is executable by a processor to implement the steps of the method of real-time translation of multilingual speech based on a large model according to claim 7 or claim 8.
10. A computer-readable storage medium, characterized in that, A computer program product comprising a computer readable medium storing a computer program, wherein the computer program is executable by a processor to implement the steps of the method of real-time translation of multilingual speech based on a large model according to claim 7 or claim 8.