Simultaneous interpretation method, device and system

Abstract

The invention relates to the technical field of communication, and provides a simultaneous interpretation method, device and system, and the method comprises the steps: building an audio transmission channel with a user terminal; receiving an original audio output by the user terminal through the audio transmission channel; in the process of playing the original audio, synchronously translating the original audio to generate a translated audio of the original audio; and superposing and playing the translated audio and the original audio. According to the invention, the audio transmission channel with the user terminal is established and the audio output is taken over, so that the versatility compatibility of call application and equipment is not limited; translation processing is synchronously performed in parallel in the process of playing the original sound audio, and the generated translated text audio and the original sound audio are played in a superposed manner, so that double-flow parallel output of direct transmission of the original sound and simultaneous transmission of the translated text is realized.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a simultaneous interpretation method, device and system. Background Technology

[0002] With increasingly frequent cross-language exchanges globally, voice calls have become one of the most efficient ways to communicate remotely. However, when the two parties in a call do not speak the same language, communication becomes a significant barrier, making the need for real-time call translation technology increasingly urgent.

[0003] Existing call translation methods typically employ a translation headset in conjunction with a mobile application. This method connects to the phone via Bluetooth, transmitting audio to the application for speech recognition, machine translation, and speech synthesis, before sending the translated audio back to the headset for playback. However, this method is highly dependent on specific applications, limiting its applicability; furthermore, the back-and-forth audio transmission between the headset and the application results in significant communication latency, making it difficult to meet the smoothness requirements of full-duplex real-time communication. Summary of the Invention

[0004] This invention provides a simultaneous interpretation method, device, and system to address the deficiencies in the existing technology.

[0005] This invention provides a simultaneous interpreting method, comprising the following steps: Establish an audio transmission channel with the user terminal; The original audio output by the user terminal is received through the audio transmission channel; During the playback of the original audio, the original audio is simultaneously translated to generate a translated audio. The translated audio is played overlaid with the original audio.

[0006] According to a simultaneous interpretation method provided by the present invention, establishing an audio transmission channel with a user terminal includes: A wireless communication link is established with the user terminal based on a wireless transmission protocol. The device configuration signaling is sent to the user terminal through the wireless communication link to trigger the user terminal to switch the system-level audio input and output routing to the wireless communication link, thereby forming the audio transmission channel.

[0007] According to a simultaneous interpretation method provided by the present invention, the wireless transmission protocol includes a hands-free configuration protocol and an advanced audio distribution configuration protocol; The establishment of a wireless communication link with the user terminal based on a wireless transmission protocol includes: Based on the hands-free configuration protocol, a synchronous connection link for transmitting two-way voice data is established as the wireless communication link; or, Based on the aforementioned advanced audio distribution configuration protocol, an asynchronous connection link for transmitting unidirectional media data is established as the wireless communication link.

[0008] According to a simultaneous interpretation method provided by the present invention, the step of translating the original audio to generate a translated audio is as follows: The original audio is subjected to speech recognition to obtain the source language text; The source language text is machine translated to obtain the target language text; The target language text is processed by speech synthesis to obtain the translated audio.

[0009] According to a simultaneous interpretation method provided by the present invention, the step of performing machine translation on the source language text to obtain the target language text includes: Based on the speech energy characteristics and pause characteristics of the original audio, the source language text is segmented into sentences to obtain segmented text; The segmented text is machine translated to obtain the target language text.

[0010] According to a simultaneous interpretation method provided by the present invention, the step of superimposing the translated audio with the original audio includes: Real-time detection of the playback status of the original audio and the generation time of the translated audio; If the translated audio is about to be played but the original audio has not yet finished playing, reduce the playback volume of the original audio. The original audio, with its volume reduced, is mixed with the translated audio and then played back.

[0011] According to a simultaneous interpretation method provided by the present invention, after superimposing the translated audio with the original audio, the method further includes: Collect response audio in response to the original audio; The reply audio is sent to the user terminal through the audio transmission channel, so that the user terminal can forward the reply audio to the other end of the call; During the process of sending the response audio to the user terminal, the response audio is simultaneously translated to generate a translated response audio. The translated audio response is sent to the user terminal via the audio transmission channel, so that the user terminal can forward the translated audio response to the other end of the call.

[0012] The present invention also provides a simultaneous interpretation device, comprising the following modules: A wireless communication module is used to establish an audio transmission channel with a user terminal and to receive the original audio output by the user terminal through the audio transmission channel. An audio playback module is used to play audio signals. The processor, connected to the wireless communication module and the audio playback module, is used to perform the following operations: Control the audio playback module to play the original audio; During the playback of the original audio, the original audio is simultaneously translated to generate a translated audio. The translated audio is superimposed on the original audio, and the audio playback module is controlled to play the superimposed mixed audio.

[0013] The simultaneous interpretation device provided by the present invention further includes: An audio acquisition module is used to acquire response audio in response to the original audio. The processor is also used to perform the following operations: The wireless communication module is controlled to send the reply audio to the user terminal through the audio transmission channel, so that the user terminal can forward the reply audio to the other end of the call; During the process of sending the response audio to the user terminal, the response audio is simultaneously translated to generate a translated response audio. The wireless communication module is controlled to send the translated audio response to the user terminal through the audio transmission channel, so that the user terminal can forward the translated audio response to the other end of the call.

[0014] The present invention also provides a simultaneous interpretation system, comprising: the simultaneous interpretation device as described above, a user terminal, and a call user terminal, wherein the simultaneous interpretation device and the user terminal are wirelessly connected.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the simultaneous interpretation method as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the simultaneous interpretation method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the simultaneous interpretation method as described above.

[0018] The simultaneous interpretation method, device, and system provided by this invention achieve universal compatibility across various call applications and devices by establishing an audio transmission channel with the user terminal and taking over the audio output. By performing translation processing synchronously and in parallel while the original audio is playing, and then superimposing the generated translated audio with the original audio, dual-stream parallel output of direct original audio transmission and simultaneous translation is achieved. Because the original audio is played immediately and the translated audio is generated synchronously in the background and then superimposed, the waiting delay during the call is greatly reduced, avoiding the communication interruptions and fragmentation caused by waiting for translation completion in traditional translation methods. This provides users with a smooth, natural, and full-duplex real-time cross-language communication solution. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the simultaneous interpretation method provided by the present invention.

[0021] Figure 2 This is a timing interaction diagram of the downlink audio processing flow provided by the present invention.

[0022] Figure 3 This is a timing interaction diagram of the uplink audio processing flow provided by the present invention.

[0023] Figure 4 This is a schematic diagram of the simultaneous interpretation system provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] With the increasing demand for cross-language communication, voice calls, as one of the most direct and efficient methods of remote communication, play an important role in daily life and work. However, when the two parties in a call do not speak the same language, communication becomes a significant obstacle. Currently, although many translation software and devices exist on the market, their application in the specific scenario of "real-time call translation" still faces many inconveniences.

[0026] Current call translation technologies mainly fall into two categories: one is the call translation function built into specific brand mobile phones, which works by acquiring the uplink and downlink audio during a phone call and then performing speech recognition, machine translation, and speech synthesis to achieve call translation; the other is a translation headset, which works by connecting the headset to the phone, taking over the audio from the phone, and then transmitting the audio to an application on the phone for the aforementioned speech recognition, translation, and speech synthesis processing.

[0027] However, the above solutions have significant drawbacks in practical applications. For built-in mobile phone translation solutions, their functionality is typically highly tied to specific brand hardware devices, resulting in limited coverage and very limited support for third-party calling software, often requiring third-party software to actively adapt before use. For translation headset solutions, they heavily rely on mobile applications, leading to limited platform support, such as typically not supporting personal computers. More importantly, since the headset itself lacks the ability to recognize, translate, and synthesize audio, audio needs to be transmitted back and forth between the headset and the mobile application. This long transmission link leads to high overall translation latency, making it difficult to meet the requirements for smooth communication. Therefore, there is an urgent need for a universal solution that is not limited by calling applications and devices, has low latency, and enables full-duplex communication.

[0028] To address this, the present invention provides a simultaneous interpretation method, which aims to establish an audio transmission channel between the present invention and a user terminal. The present invention receives the original audio output by the user terminal through the audio transmission channel, and simultaneously translates the original audio during playback to generate a translated audio. The translated audio is then superimposed on the original audio for playback, thereby achieving a universal, unrestricted, and unrestricted real-time call translation with clear sound quality, low latency, and full-duplex capability.

[0029] It should be noted that all actions involving the acquisition of signal information or data in this invention are carried out in compliance with the relevant data protection laws and regulations of the country where the invention is located, and with the authorization granted by the owner of the relevant device.

[0030] in, Figure 1 This is a flowchart illustrating the simultaneous interpretation method provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 110, 120, 130 and 140.

[0031] Step 110: Establish an audio transmission channel between the user terminal and the user terminal.

[0032] Here, the user terminal can be understood as the device served by the simultaneous interpretation method provided in this embodiment. It can be any electronic device that supports audio output and wireless communication functions, including but not limited to smartphones, tablets, personal computers, smartwatches, and other wearable devices. The user terminal typically runs applications that support voice calls, such as the system's built-in phone application or third-party instant messaging software.

[0033] An audio transmission channel refers to a logical or physical link used for bidirectional or unidirectional transmission of audio data between the device executing this method (hereinafter referred to as "this device") and the user terminal. After the channel is established, this device can act as an external audio device for the user terminal, taking over the audio input and output of the user terminal.

[0034] There are various ways to establish an audio transmission channel between the user terminal and the device. As an alternative embodiment, a wired connection can be established, such as via a Type-C interface or a 3.5mm audio jack. As a more preferred embodiment, to improve portability and versatility, a wireless communication technology can be used to establish the connection. For example, short-range wireless communication technologies such as Bluetooth or Wi-Fi Direct can be used to pair with and establish a connection with the user terminal.

[0035] It should be noted that the establishment of the channel in this embodiment is not only a physical connection, but also includes a handshake and configuration at the protocol level, so that the user terminal recognizes that the device has audio processing capabilities and is willing to direct the call audio stream to the device.

[0036] Step 120: Receive the original audio output from the user terminal through the audio transmission channel.

[0037] Specifically, the original audio can be understood as the downlink audio stream received by the user terminal during a call, that is, the voice signal emitted by the other end of the call (i.e., the other party). For example, when a user answers a call on a mobile phone, the voice data of the other party speaking is the original audio. Before the audio transmission channel in this embodiment is established, the original audio is usually played by the user terminal's own earpiece or speaker; after the audio transmission channel is established, the original audio is transmitted to this device through the audio transmission channel.

[0038] Receiving the original audio through the audio transmission channel means that this device actually takes over the audio output function of the user terminal. This takeover mechanism makes the method of this embodiment highly versatile, not restricting the specific type of the user terminal nor the specific call application. As long as the audio output routing of the user terminal points to this device, this device can obtain the original audio.

[0039] Step 130: During the process of playing the original audio, synchronously perform translation processing on the original audio to generate a translated audio of the original audio.

[0040] Specifically, playing the original audio means that after this device receives the original audio, it uses its own playback component to restore the sound, such as playing through its own speaker or earphone driver unit for the user to listen. To ensure the real-time nature of the call, the received original audio can be played immediately to preserve the real-time voice environment of the call, enabling the user to perceive the tone, emotion, and intonation of the other party in a timely manner.

[0041] Synchronously performing translation processing on the original audio means that while playing the original audio, the same original audio is computationally processed in parallel in the background. This is a dual-channel parallel processing architecture, that is, one channel is the live broadcast of the original sound, and the other channel is the translation calculation. The specific process of the translation processing usually includes converting the speech signal in the original audio into source language text, translating the source language text into target language text, and synthesizing the target language text into target language speech, so as to finally generate the translated audio.

[0042] Generating the translated audio of the original audio means obtaining the sound form of the translation result corresponding to the other party's words. For example, if the original audio is the English "Hello, how are you today?", the generated translated audio may be the Chinese "你好,今天怎么样?".

[0043] By adopting this dual-channel parallel processing mechanism in this embodiment, the serial logic of traditional translation devices that listen first and then translate or translate first and then play is broken, greatly shortening the user's sense of waiting time.

[0044] For example, at T=0ms, after receiving the original audio "Hello, how are you today?", the device immediately begins playing the original audio. Between T=0 and T=1000ms, while the user hears the original audio, the device simultaneously performs recognition, translation, and synthesis processing on the original audio in the background. At T=1000ms, the translated audio is generated and ready for playback. Thus, it can be seen that the playback of the original audio begins immediately at T=0ms, and the translated audio is ready at T=1000ms. The user only waits approximately 1000ms to hear the translated audio. Because the user is continuously listening to the original audio during this time, their attention is effectively filled, and the user hardly perceives any tedious silent waiting, achieving a near real-time simultaneous interpretation experience.

[0045] Step 140: Overlay the translated audio with the original audio.

[0046] Specifically, overlay playback refers to overlapping or mixing the generated translated audio with the currently playing original audio on the timeline. This does not mean that the two must start or end completely simultaneously, but rather that the original audio and the translated audio are intertwined and coexist in the user's auditory perception, creating an effect similar to human simultaneous interpretation.

[0047] Typically, because translation processing itself requires a certain amount of computation time, the playback output of the translated audio will naturally lag behind the playback output of the corresponding original audio. Therefore, overlay playback often manifests as the user first hearing the beginning of a sentence of the original audio, and then the translated audio overlaying in while the original audio is still playing or has just finished playing.

[0048] As an optional implementation, to ensure the translated audio is clearly distinguishable, the audio can be mixed during overlay playback. For example, when the translated audio is detected as ready to play, if the original audio has not yet finished playing, the volume of the original audio can be appropriately reduced, such as to 20%-40%, using the original audio as background noise while the translated audio is played at a normal or higher volume. This allows the user to hear the translated content clearly while preserving the background atmosphere of the original audio.

[0049] Through the steps described above, this embodiment achieves a full-duplex simultaneous interpretation experience. Users no longer need to wait for the other party to finish speaking and for the machine to finish translating before they can hear the content, as with walkie-talkies. Instead, they can listen to the original audio and the translated text simultaneously, making the communication process smoother and more natural.

[0050] The simultaneous interpretation method provided in this embodiment achieves universal compatibility across various call applications and devices by establishing an audio transmission channel with the user terminal and taking over the audio output. By performing translation processing synchronously and in parallel while the original audio is played, and then superimposing the generated translated audio with the original audio, it achieves dual-stream parallel output of direct original audio transmission and simultaneous translation. Because the original audio is played immediately and the translated audio is generated synchronously in the background and then superimposed, the waiting delay during the call is greatly reduced, avoiding the communication interruptions and fragmentation caused by waiting for translation completion in traditional translation methods. This provides users with a smooth, natural, and full-duplex real-time cross-language communication solution.

[0051] in, Figure 2 This is a timing interaction diagram of the downlink audio processing flow provided by the present invention, as shown below. Figure 2 As shown, the other end of the call sends the original audio, which reaches the user terminal via the communication network. The user terminal then transmits the received original audio to this device through the established audio transmission channel. Upon receiving the original audio, this device immediately buffers the audio. Then, this device executes the following parallel processing logic: On one hand, it immediately plays the original audio through the speaker or connected headphones, ensuring the user can hear the other party's original voice instantly without significant delay; on the other hand, it simultaneously performs parallel translation processing in the background, specifically including speech recognition (converting English speech to English text), text translation (translating English text to Chinese text), and speech synthesis (converting Chinese text to Chinese audio), generating and playing the translated audio. Next, this device overlays the generated translated audio with the original audio, allowing the user to hear the original audio first, followed by the translated audio, thus achieving a full-duplex simultaneous interpretation experience.

[0052] In order to achieve a convenient and universal connection experience, avoid cumbersome cable connection operations for users, and ensure that the device can be recognized as a legitimate audio peripheral by various user terminals, this embodiment preferably adopts a channel construction method based on standard wireless protocols in step 110.

[0053] Specifically, step 110, establishing an audio transmission channel with the user terminal, includes: Establish a wireless communication link with the user terminal based on the wireless transmission protocol; The device configuration signaling is sent to the user terminal through the wireless communication link to trigger the user terminal to switch the system-level audio input and output routing to the wireless communication link, thus forming an audio transmission channel.

[0054] Specifically, wireless transmission protocols typically refer to general, standardized short-range wireless communication standards, such as the Bluetooth protocol. Establishing a wireless communication link refers to the process by which the device and the user terminal complete the physical layer radio frequency handshake, the link layer connection establishment, and the security layer pairing and authentication. For example, when a user clicks to connect to this device in the Bluetooth settings interface of their mobile phone, the two parties complete the underlying wireless link establishment based on the Bluetooth protocol.

[0055] Establishing the underlying wireless communication link only signifies a communication connection between the two devices; it does not mean that audio will automatically flow to this device. To achieve audio takeover, this device needs to actively declare its identity and capabilities to the user terminal via the wireless communication link.

[0056] Device configuration signaling is used to carry this identity and capability description information. Specifically, this device can send configuration signaling containing a specific service class and configuration description to the user terminal through standard device discovery and service record mechanisms. This configuration signaling identifies this device as a standard audio peripheral, such as a Bluetooth headset or hands-free device, rather than an ordinary data transmission device.

[0057] Once the user terminal receives the configuration signaling and confirms the audio peripheral's identity, the user terminal's operating system's audio management subsystem is automatically triggered. This triggers an audio routing switch, forcibly switching all system-level audio output routes—originally pointing to the user terminal's earpiece or speaker, and system-level audio input routes originally pointing to the user terminal's microphone—to the current wireless communication link. At this point, a complete audio transmission channel taking over the user terminal's system-level input and output is formally established.

[0058] This embodiment, through its standard protocol stack-based link establishment and device role configuration strategy, eliminates the need to install any specific applications or drivers on the user terminal. It leverages the system's own mechanisms to seamlessly manage all call audio (such as telephone calls, WeChat, and conferencing software), greatly enhancing compatibility and user experience.

[0059] Considering that different audio application scenarios have different requirements for transmission bandwidth, latency, and two-way interactivity—for example, calls require low-latency two-way transmission, while listening to music requires high-quality one-way transmission—this embodiment supports multiple specific protocol configurations when establishing a wireless communication link to adapt to different service needs. Specifically, the wireless transmission protocols include the Hands-Free Profile (HFP) protocol and the Advanced Audio Distribution Profile (A2DP) protocol.

[0060] Based on this, a wireless communication link is established between the user terminal and the wireless transmission protocol, including: Based on the hands-free configuration protocol, a synchronous connection link for transmitting two-way voice data is established as a wireless communication link. or, An asynchronous connection link for transmitting unidirectional media data is established based on an advanced audio distribution configuration protocol as a wireless communication link.

[0061] Specifically, when a user terminal is in a call, such as a telephone call, WeChat voice call, or conference call, it needs to simultaneously transmit the reply audio recorded by the user terminal's microphone and the original audio sent by the other end of the call, with extremely high real-time requirements. In this case, this device connects to the user terminal via HFP. The HFP protocol establishes a synchronous connection link based on a synchronous directional connection or extended synchronous directional connection channel. This link is specifically designed for voice, ensuring low-latency transmission of bidirectional voice data and ensuring that both parties in the call are listening and speaking simultaneously.

[0062] When the user terminal is playing media, such as watching foreign language videos, listening to foreign language podcasts, or listening to music, the primary requirement is to receive high-quality original audio from the other end of the call; response audio is not required. In this case, this device connects to the user terminal via A2DP. A2DP establishes an asynchronous connection link based on an asynchronous connectionless channel. This link supports high-bitrate audio codec formats such as SBC and AAC, enabling the transmission of high-fidelity one-way media data, making it suitable for one-way real-time translation scenarios for streaming media content.

[0063] By supporting the two protocols mentioned above, this embodiment can flexibly switch between call translation mode and media translation mode, covering various scenarios of users' cross-language communication and content consumption.

[0064] In order to achieve efficient and accurate conversion from original audio to translated audio, and to ensure the modularity and scalability of the entire processing, this embodiment adopts a phased pipeline processing method to generate translated audio.

[0065] Specifically, step 130 involves translating the original audio to generate a translated audio, including: Speech recognition is performed on the original audio to obtain the source language text; Machine translation is performed on the source language text to obtain the target language text; The target language text is processed into speech to obtain the translated audio.

[0066] Specifically, speech recognition is the entry point of the entire translation pipeline. This device uses an internally integrated speech recognition engine to perform real-time analysis on the received original audio. The speech recognition engine extracts acoustic features from the original audio, such as Mel Frequency Cepstral Coefficients (MFCC), and combines the acoustic model and language model to convert the continuous speech signal into the corresponding text sequence, i.e., the source language text. To adapt to the simultaneous interpretation scenario, speech recognition here usually adopts streaming recognition technology, that is, outputting text while receiving audio, rather than waiting for the entire sentence of audio to end before starting recognition, thereby minimizing the recognition latency of the first word.

[0067] Next, this device inputs the recognized source language text into the built-in neural network machine translation engine. The machine translation engine is based on deep learning models, such as the Transformer architecture, and accurately translates it into the set target language text according to the context semantics of the source language text. For example, translating the source language text "Hello" into the target language text "你好". This process also supports streaming translation, and as the source language text is continuously generated, the target language text will be updated and output in real time.

[0068] Finally, this device uses the speech synthesis engine to convert the target language text generated by translation into natural and fluent speech waveform data, i.e., the translated audio. To enhance the user's listening experience, the speech synthesis engine usually adopts high-quality synthesis technology based on neural networks, which can generate speech with anthropomorphic intonation, emotion, and natural sentence breaks, making the translated audio sound more like a real person speaking rather than a mechanical electronic voice.

[0069] Through the close connection and parallel pipeline operation of the above three steps, this embodiment can complete the entire process from listening to translation and then to speaking in a very short time, providing timely translated audio data for overlay playback.

[0070] Considering that in continuous natural conversations, the speech stream is often continuous. If the continuous source language text is directly input into the translation engine, it may lead to an overly long translation context, semantic confusion, or even memory overflow, affecting the accuracy and real-time performance of translation. Therefore, a mechanism is needed to split the continuous text stream into short sentences or segments with relatively complete semantics.

[0071] Based on this, performing machine translation on the source language text to obtain the target language text includes: Segmenting the source language text according to the speech energy characteristics and pause characteristics of the original audio to obtain segmented text; Performing machine translation on the segmented text to obtain the target language text.

[0072] Specifically, speech energy features refer to the characteristics of the amplitude or power of an audio signal changing over time. These features can be extracted from the original audio by calculating the short-time energy or root-mean-square amplitude of the audio frame. Pause features refer to the duration of silent or low-energy segments in a speech signal. These pause features can be extracted from the original audio by monitoring the energy state of consecutive audio frames using speech activity detection algorithms.

[0073] Since speech energy features characterize the presence of valid human voice input at a given moment from an acoustic intensity perspective, they can accurately distinguish speech segments from background noise, avoiding segmentation errors caused by misjudging environmental noise as valid speech. Pause features characterize the speaker's phrasing habits from a temporal rhythm perspective, accurately capturing the natural intervals between sentences and avoiding forced segmentation that leads to semantic fragmentation. Therefore, this embodiment combines speech energy features and pause features to segment the source language text, resulting in semantically complete and rhythmically natural segmented text.

[0074] As an optional implementation, when the detected speech energy is below a preset threshold and the duration exceeds a set pause threshold, the current sentence is determined to have ended or a natural pause has occurred. Based on this physical feature breakpoint, the source language text sequence identified up to the current moment is truncated to form an independent segmented text. This segmentation method can match the speaker's natural speaking speed and breathing rhythm, ensuring that the segmented text is usually a relatively complete sentence or phrase semantically.

[0075] After obtaining the individual text segments, they can be input into the machine translation engine as a complete translation unit. The machine translation engine then performs contextual understanding and transformation on these segments to generate the corresponding target language text.

[0076] By employing this dynamic segmentation strategy based on speech features, this embodiment can break down long speech streams into smaller segments, ensuring that the translation engine processes text of appropriate length each time, thus improving translation accuracy, and enabling the translation to be generated sentence by sentence in accordance with the speaker's pauses and rhythms.

[0077] Considering that in scenarios where the original audio and the translated audio are played simultaneously at maximum volume, the sound may become mixed and mask each other, making it difficult for users to hear the translated content clearly, and also negating the role of the original audio as background emotional support. To resolve this auditory conflict and achieve the best mixing effect where the translated audio is clear and the original audio is audible, this embodiment adopts a dynamic mixing strategy.

[0078] Specifically, step 140 involves overlaying the translated audio with the original audio, including: Real-time detection of the playback status of the original audio and the generation time of the translated audio; When the translated audio is about to be played but the original audio has not yet finished playing, lower the playback volume of the original audio. The original audio, with its volume reduced, is mixed with the translated audio before being played.

[0079] Specifically, this device continuously monitors the status of both the original audio and the translated audio. For the original audio, it tracks its current playback status in real time to determine whether the current sentence has finished playing or how much time remains. Simultaneously, it monitors the output queue of the translation engine. Once a new translated audio segment is synthesized and pushed into the playback buffer, that point in time is marked as the generation time of the translated audio, meaning the translation is ready for playback.

[0080] When the translated audio is detected as generated and ready to be played, but the corresponding original audio has not yet finished playing, in order to avoid acoustic masking effects between the two high-volume audio streams, which would reduce information recognizability, this embodiment can lower the playback volume of the original audio to a preset background volume level. For example, attenuating the volume of the original audio to 20%-40% of its original volume ensures that the translation is not masked while allowing the user to faintly hear the tone and emotion of the original audio.

[0081] After adjusting the volume, digital signal processing technology can be used to digitally mix the attenuated original audio with the translated audio at a normal volume. The mixed data is then sent to a digital-to-analog converter and used to drive the speakers. In this way, to the user's auditory perception, the translated audio appears as a clear foreground sound, while the original audio smoothly recedes into the background sound. The two create a clear hierarchy in both the frequency and time domains, greatly improving the listening comfort of simultaneous interpretation.

[0082] Considering that true simultaneous interpretation should be a full-duplex two-way communication, meaning that not only should the user understand what the other party is saying, but the other party should also understand what the user is saying, this embodiment, in addition to implementing downlink translation, further provides a parallel translation processing mechanism for the user's uplink speech, thereby closing the entire cross-language communication link.

[0083] Specifically, after overlaying the translated audio with the original audio, it also includes: Collect response audio based on the original audio; The reply audio is sent to the user terminal through the audio transmission channel, so that the user terminal can forward the reply audio to the other end of the call; During the process of sending the audio response to the user's terminal, the audio response is translated simultaneously to generate a translated audio response. The translated audio response is sent to the user terminal via an audio transmission channel, so that the user terminal can forward the translated audio response to the other end of the call.

[0084] Specifically, when a user listens to both the original audio and the translated audio from the other end of the call and wishes to respond, this device can use its built-in microphone array to capture the user's response audio in real time. To ensure capture quality, the captured raw response audio may undergo preprocessing such as noise reduction and echo cancellation to extract clear human voice and obtain the response audio. Here, the other end of the call refers to the remote communication device that has established a call connection with the user's terminal, and the response audio refers to the voice signal emitted by the user in response to the original audio.

[0085] To ensure the other end of the call can perceive the user's genuine tone and emotions, this device, upon acquiring the response audio, can immediately transmit it to the user's terminal via an established audio transmission channel. Upon receiving the response audio, the user terminal treats it as a regular microphone input and forwards it to the other end of the call via cellular network or the internet.

[0086] While sending the reply audio, the device simultaneously performs speech recognition, machine translation, and speech synthesis on the same reply audio in the background. This process converts the user's reply audio into a translated reply audio, which can be understood as speech data that is semantically consistent with the reply audio but in a different language.

[0087] Once the translated audio response is generated, the device will send it to the user terminal again via the audio transmission channel, and the user terminal will also forward this audio to the other end of the call.

[0088] Using the above method, the other end of the call will first hear the user's original voice response, such as "Hello," followed by (approximately 1-2 seconds later) the translated voice, such as "Hello." This bidirectional, dual-stream, and parallel processing mechanism allows both parties to understand each other's semantics without barriers while preserving genuine emotional exchange, achieving true full-duplex real-time cross-language communication.

[0089] in, Figure 3 This is a timing interaction diagram of the uplink audio processing flow provided by the present invention, as shown below. Figure 3As shown, when a user sends a response audio to the original audio, this device captures the response audio using its built-in microphone. Then, the device initiates a dual-channel independent audio processing flow: First, it immediately transmits the original audio, sending the real-time captured response audio to the user terminal via the established audio transmission channel. Upon receiving the audio, the user terminal immediately forwards it to the other end of the call via the communication network, ensuring the other party hears the response audio immediately, preserving the user's authentic tone. Next, while sending the response audio, the device simultaneously performs independent translation processing, concurrently performing speech recognition (Chinese audio to Chinese text), text translation (Chinese text to English text), and speech synthesis (English text to Chinese audio) to generate a translated response audio. Once the synthesis is complete, the device immediately sends the generated translated response audio to the user terminal via the audio transmission channel. The user terminal then forwards the translated response audio to the other end of the call, ensuring the other party hears the translated response audio after hearing the original audio, thus achieving full-duplex translation on the uplink.

[0090] The simultaneous interpretation device provided by the present invention is described below. The simultaneous interpretation device described below and the simultaneous interpretation method described above can be referred to in correspondence.

[0091] Considering that current translation headsets rely on mobile apps for calculations, resulting in high latency and poor compatibility, and that built-in translation functions on mobile phones are limited to specific brands, this embodiment provides a hardware solution that can independently perform edge computing and directly take over the audio stream by integrating a high-performance computing unit locally on the device.

[0092] Specifically, this embodiment provides a simultaneous interpretation device, including: The wireless communication module is used to establish an audio transmission channel with the user terminal and to receive the original audio output by the user terminal through the audio transmission channel. An audio playback module is used to play audio signals. The processor, connected to the wireless communication module and the audio playback module, performs the following operations: Control the audio playback module to play the original audio; During the playback of the original audio, the original audio is translated simultaneously to generate a translated audio version of the original audio. The translated audio is superimposed on the original audio, and the audio playback module is controlled to play the superimposed mixed audio.

[0093] Specifically, the wireless communication module integrates an RF chip and baseband circuitry that support standard wireless transmission protocols (such as Bluetooth 5.0 and above). This module is configured to support hands-free configuration protocols and advanced audio distribution configuration protocols. During operation, the wireless communication module broadcasts device configuration signaling, identifying the simultaneous interpretation device as a standard audio peripheral. Once a connection is established with the user terminal, the module takes over the user terminal's system-level audio output routing, thus enabling real-time reception of the original audio output from the user terminal.

[0094] The audio playback module includes a digital-to-analog converter, an audio amplifier, and a sound-emitting unit. Its function is to convert the digital audio signal output by the processor into an analog signal and drive the sound-emitting unit to emit sound for the user to listen to.

[0095] The processor, connected to the wireless communication module and audio playback module, is the core of the entire device's control and computing. The processor can be a high-performance system-on-a-chip (SoC) integrating digital signal processing and neural network processing units.

[0096] The processor is configured to perform the following operations to enable full-duplex simultaneous interpretation: When the wireless communication module receives the original audio, the processor immediately places it into the playback buffer and controls the audio playback module to decode and play it. This is a direct audio path with extremely low processing latency, typically in the millisecond range, ensuring that users can hear the original audio in real time.

[0097] While sending the original audio to the playback module, the processor utilizes its integrated speech recognition engine, machine translation engine, and speech synthesis engine to perform parallel computations on the same original audio. Specifically, the processor first identifies the original audio as source language text, then translates it into target language text, and finally synthesizes the translated audio. Because these computations are performed entirely on the local processor, without uploading to the cloud or relying on a mobile app, it boasts extremely high response speed and privacy security.

[0098] Once the translated audio is generated, the processor's internal mixing algorithm module digitally superimposes the current translated audio data stream with the original audio that is currently playing or waiting to be played in the buffer. During this process, the processor can also dynamically adjust the gain of the two audio streams, such as reducing the gain of the original audio, to generate a well-defined mixed audio. Finally, the processor controls the audio playback module to output the mixed audio.

[0099] The simultaneous interpretation device provided in this embodiment integrates audio transmission, playback, and core translation calculations through a modular design at the hardware level, achieving independent simultaneous interpretation functionality without relying on external computing power.

[0100] To meet the two-way communication needs of users in cross-language communication—namely, to understand the other party and to make the other party understand them—this embodiment adds an audio acquisition module and expands the processor's functionality on the basis of the aforementioned simultaneous interpretation device to support uplink simultaneous interpretation.

[0101] Specifically, the simultaneous interpretation equipment provided in this embodiment also includes: The audio acquisition module is used to acquire response audio for the original audio. The processor is also used to perform the following operations: The wireless communication module controls the transmission of reply audio to the user terminal via the audio transmission channel, so that the user terminal can forward the reply audio to the other end of the call; During the process of sending the audio response to the user's terminal, the audio response is translated simultaneously to generate a translated audio response. The control wireless communication module sends the translated audio of the reply to the user terminal through the audio transmission channel, so that the user terminal can forward the translated audio of the reply to the other end of the call.

[0102] The audio acquisition module typically includes a single microphone or a microphone array consisting of multiple microphones, along with a corresponding analog-to-digital converter. When a user responds to the original audio and translated audio, the audio acquisition module picks up the user's voice signal and converts it into digital response audio. To improve sound pickup quality, this module can also work with a processor to perform preprocessing algorithms such as noise reduction, beamforming, and echo cancellation to ensure that the acquired response audio is clear and pure.

[0103] Based on the newly added audio acquisition module, the processor is also configured to perform the following uplink processing operations: Once the audio acquisition module captures the response audio, the processor immediately transmits it to the user terminal via the wireless communication module. Since the device is disguised as a standard Bluetooth headset, the user terminal treats the received response audio as a microphone input signal and forwards it in real-time to the other end of the call, ensuring that the other party can hear the user's original voice immediately and perceive the user's tone and emotions.

[0104] During the process of sending the response audio to the user terminal, the processor simultaneously initiates an uplink translation process locally. The processor uses a speech recognition engine to convert the response audio into source language text, uses a machine translation engine to translate it into target language text, and finally uses a speech synthesis engine to generate the translated response audio.

[0105] Once the translated audio response is generated, the processor will call the wireless communication module again to send the translated audio response to the user terminal, which will then forward it as a microphone input signal to the other end of the call.

[0106] Through the above mechanism, the simultaneous interpretation device provided in this embodiment realizes complete two-way full-duplex simultaneous interpretation, that is, in the downlink direction, the user hears the original sound + translation, and in the uplink direction, the other party hears the original sound + translation.

[0107] Based on any of the above embodiments Figure 4 This is a schematic diagram of the simultaneous interpretation system provided by the present invention, as shown below. Figure 4 As shown, the system includes: simultaneous interpretation equipment, a user terminal, and a calling user terminal. The simultaneous interpretation equipment establishes a wireless communication connection with the user terminal via a Bluetooth module, simulating a Bluetooth headset device to take over the audio input / output channels for the user terminal's calls. The user terminal runs calling apps such as phone and WeChat, and communicates remotely with the other end of the call via cellular or Wi-Fi networks.

[0108] Then, the processing flow for downlink audio is as follows: the original audio sent by the other end of the call is transmitted to the user terminal's call APP via cellular or WiFi network; the user terminal transmits the original audio to the simultaneous interpretation device via Bluetooth; after receiving the original audio, the simultaneous interpretation device transmits it directly to the speaker / headphones for playback, and at the same time transmits it in parallel to the internal translation service module (including speech recognition engine, machine translation engine, and speech synthesis engine) for processing. The generated translated audio is then also transmitted to the speaker / headphones and played overlaid with the original audio.

[0109] Next, the uplink audio processing flow is as follows: the user records a response audio through the microphone; this response audio is sent to the Bluetooth module and directly transmitted to the user terminal as uplink audio; simultaneously, the response audio is also sent to the translation service module for processing, and the generated translated response audio is then sent to the user terminal via the Bluetooth module. The user terminal forwards the received response audio and translated response audio to the other end of the call via the network, thus completing the full-duplex call translation process.

[0110] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to perform the simultaneous interpretation methods provided by the above methods.

[0111] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the simultaneous interpretation methods provided by the methods described above.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A simultaneous interpreting method, characterized in that, include: Establish an audio transmission channel with the user terminal; The original audio output by the user terminal is received through the audio transmission channel; During the playback of the original audio, the original audio is simultaneously translated to generate a translated audio. The translated audio is played overlaid with the original audio.

2. The simultaneous interpretation method according to claim 1, characterized in that, The establishment of the audio transmission channel with the user terminal includes: A wireless communication link is established with the user terminal based on a wireless transmission protocol. The device configuration signaling is sent to the user terminal through the wireless communication link to trigger the user terminal to switch the system-level audio input and output routing to the wireless communication link, thereby forming the audio transmission channel.

3. The simultaneous interpretation method according to claim 2, characterized in that, The wireless transmission protocol includes a hands-free configuration protocol and an advanced audio distribution configuration protocol; The establishment of a wireless communication link with the user terminal based on a wireless transmission protocol includes: Based on the hands-free configuration protocol, a synchronous connection link for transmitting two-way voice data is established as the wireless communication link; or, Based on the aforementioned advanced audio distribution configuration protocol, an asynchronous connection link for transmitting unidirectional media data is established as the wireless communication link.

4. The simultaneous interpretation method according to any one of claims 1 to 3, characterized in that, The process of translating the original audio to generate a translated audio file includes: The original audio is subjected to speech recognition to obtain the source language text; The source language text is machine translated to obtain the target language text; The target language text is processed by speech synthesis to obtain the translated audio.

5. The simultaneous interpretation method according to claim 4, characterized in that, The process of machine translating the source language text to obtain the target language text includes: Based on the speech energy characteristics and pause characteristics of the original audio, the source language text is segmented into sentences to obtain segmented text; The segmented text is machine translated to obtain the target language text.

6. The simultaneous interpretation method according to any one of claims 1 to 3, characterized in that, The step of superimposing the translated audio with the original audio includes: Real-time detection of the playback status of the original audio and the generation time of the translated audio; If the translated audio is about to be played but the original audio has not yet finished playing, reduce the playback volume of the original audio. The original audio, with its volume reduced, is mixed with the translated audio and then played back.

7. The simultaneous interpretation method according to any one of claims 1 to 3, characterized in that, After superimposing the translated audio with the original audio, the method further includes: Collect response audio in response to the original audio; The reply audio is sent to the user terminal through the audio transmission channel, so that the user terminal can forward the reply audio to the other end of the call; During the process of sending the response audio to the user terminal, the response audio is simultaneously translated to generate a translated response audio. The translated audio response is sent to the user terminal via the audio transmission channel, so that the user terminal can forward the translated audio response to the other end of the call.

8. A simultaneous interpretation device, characterized in that, include: A wireless communication module is used to establish an audio transmission channel with a user terminal and to receive the original audio output by the user terminal through the audio transmission channel. An audio playback module is used to play audio signals. The processor, connected to the wireless communication module and the audio playback module, is used to perform the following operations: Control the audio playback module to play the original audio; During the playback of the original audio, the original audio is simultaneously translated to generate a translated audio. The translated audio is superimposed on the original audio, and the audio playback module is controlled to play the superimposed mixed audio.

9. The simultaneous interpretation equipment according to claim 8, characterized in that, Also includes: An audio acquisition module is used to acquire response audio in response to the original audio. The processor is also used to perform the following operations: The wireless communication module is controlled to send the reply audio to the user terminal through the audio transmission channel, so that the user terminal can forward the reply audio to the other end of the call; During the process of sending the response audio to the user terminal, the response audio is simultaneously translated to generate a translated response audio. The wireless communication module is controlled to send the translated audio response to the user terminal through the audio transmission channel, so that the user terminal can forward the translated audio response to the other end of the call.

10. A simultaneous interpretation system, characterized in that, include: The simultaneous interpretation device, user terminal, and call user terminal as described in claim 8, wherein the simultaneous interpretation device and the user terminal are wirelessly connected.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the simultaneous interpretation method as described in any one of claims 1 to 7.

Images