A method and device for translating voice messages.

Abstract

This application relates to the field of two-way voice translation technology, providing a method and device for translating voice during calls, enabling real-time translation of two-way voice conversations. When a user makes a call using a wearable device, since the first mobile terminal lacks permission to input data during the call, the wearable device acts as a relay. Utilizing the existing Bluetooth connection with the first mobile terminal, it sends the translated version of the user's voice and the other party's voice to the first mobile terminal via Bluetooth. The first mobile terminal then forwards this to a second mobile terminal, thus achieving real-time translation of the two-way voice conversation. Furthermore, voice compression is performed during data transmission between the wearable device and the first mobile terminal. Despite the limited bandwidth and potential instability of Bluetooth transmission, high-quality, low-latency, and highly reliable two-way real-time voice communication and translation are achieved, effectively breaking down language barriers.

Description

Technical Field

[0001] This application relates to the field of two-way voice translation technology, and provides a method and device for translating voice during a conversation. Background Technology

[0002] With the continuous development of real-time speech translation technology, translation systems that work in collaboration with portable wearable devices (such as smart glasses and Bluetooth headsets) and mobile terminals such as smartphones have gradually matured. Currently, common real-time translation systems typically adopt an architecture where the smartphone serves as the core processing unit and the wearable device acts as the audio input / output unit. Specifically, the mobile terminal has a translation application (App) installed, responsible for performing computationally intensive tasks such as speech recognition, machine translation, and speech synthesis, while the wearable device is mainly responsible for audio acquisition and playback.

[0003] Data streams in existing real-time translation systems are typically transmitted in a one-way mode, such as... Figure 1 As shown, after the microphone of the wearable device receives the speech in the first language, it transmits it to the mobile terminal via Bluetooth. The translation app on the mobile terminal converts the speech into the second language and returns it to the wearable device, which then plays it to the user through the speaker or bone conduction device.

[0004] However, this one-way translation transmission mode only considers the needs of "users listening to the translation content themselves" and cannot be applied to translation scenarios where multiple users conduct two-way real-time communication. Summary of the Invention

[0005] This application provides a method and device for translating voice during a phone call, used to achieve voice translation during two-way communication.

[0006] In a first aspect, embodiments of this application provide a method for translating voice call audio, applied to a wearable device, wherein the wearable device is connected to a first mobile terminal via Bluetooth, and the method includes: The system collects a first voice stream from a first user speaking in a first language, compresses and encodes the first voice stream, and sends it to the first mobile terminal so that the first mobile terminal can translate the first voice stream into a first target audio stream in a second language. The system receives the compressed and encoded first target audio stream returned by the first mobile terminal, decodes the first target audio stream, and sends it back to the first mobile terminal, so that the first mobile terminal sends the decoded first target audio stream to the second user's second mobile terminal.

[0007] In the above solution: Addressing the need for real-time two-way translation during voice calls, when the first user uses a wearable device for a call, the first mobile terminal lacks the authority to directly transmit data during the call. Therefore, in the uplink communication link, after receiving the first target audio stream translated from the first voice stream sent by the first mobile terminal, the wearable device compresses and encodes the first target audio stream, and sends the compressed and encoded first target audio stream back to the first mobile terminal, which then forwards it to the other party's second mobile terminal. By adding the process of the wearable device sending the received translation back to the first mobile terminal, and then the first mobile terminal forwarding it to the other party's second mobile terminal, the transmission of translation between devices is realized, thus supporting real-time two-way voice translation. Simultaneously, by reusing the existing Bluetooth channel between the wearable device and the first terminal device, and instead using the glasses as a relay to transmit the translation, there is no need to develop an additional network interface for the first terminal device to send data, simplifying system implementation and improving the compatibility and stability of data transmission.

[0008] Optionally, the step of compressing and encoding the first speech stream and sending it to the first mobile terminal so that the first mobile terminal can translate the first speech stream into a first target audio stream in a second language includes: The first speech stream is PCM encoded to obtain a first digital audio stream, and the first digital audio stream is Opus compressed to obtain a first compressed audio stream. The first compressed audio stream is sent to the first mobile terminal so that the first mobile terminal can translate the first speech stream into a first target audio stream in a second language. The step of receiving the compressed and encoded first target audio stream returned by the first mobile terminal, and decoding the first target audio stream and sending it to the first mobile terminal includes: The system receives the first target audio stream after Opus compression encoding returned by the first mobile terminal, performs Opus decompression and decoding on the first target audio stream to obtain the first target audio stream in PCM format, and sends the first target audio stream in PCM format to the first mobile terminal.

[0009] In the above solution: the wearable device compresses the PCM format speech into Opus format before sending it to the first mobile terminal, and the first mobile terminal translates the speech and then re-compresses the translation into Opus format before returning it to the wearable device. This can significantly reduce the amount of data transmission on the Bluetooth link, save Bluetooth bandwidth and reduce power consumption. At the same time, Opus encoding has low latency, high compression rate and good anti-packet loss performance, which can effectively ensure the fluency and speech clarity of real-time translation.

[0010] Optionally, the method further includes: The system receives a second voice stream from the first mobile terminal when the second user is making a call in the second language. The second voice stream is collected by the second mobile terminal and sent to the first mobile terminal. The second speech stream is compressed and encoded and then sent to the first mobile terminal so that the first mobile terminal can translate the second speech stream into a second target audio stream in the first language. The system receives the compressed and encoded second target audio stream returned by the first mobile terminal and decodes and plays it.

[0011] In the above solution: Addressing the need for real-time two-way translation during voice calls, when the first user uses a wearable device for a call, the first mobile terminal lacks the permission to directly transmit data into the call. Therefore, in the downlink communication link, after the first mobile terminal receives the second voice stream sent by the other device, it transmits it to the wearable device via the existing Bluetooth connection. The wearable device acts as a relay, bringing the second voice stream into the call. This eliminates the need for an additional data transmission network interface, simplifying system implementation and improving data transmission compatibility and stability. Simultaneously, the wearable device compresses the second voice stream before sending it to the first mobile terminal for translation, shortening the overall latency from receiving the translation to playback and improving the real-time conversation experience.

[0012] Optionally, the step of compressing and encoding the second speech stream and sending it to the first mobile terminal so that the first mobile terminal translates the second speech stream into a second target audio stream in the first language includes: The second speech stream is PCM encoded to obtain a second digital audio stream, and the second digital audio stream is Opus compressed to obtain a second compressed audio stream. The second compressed audio stream is sent to the first mobile terminal so that the first mobile terminal can translate the second speech stream into a second target audio stream in the first language. The step of receiving the compressed and encoded second target audio stream returned by the first mobile terminal and decoding and playing it includes: The system receives the second target audio stream after Opus compression encoding returned by the first mobile terminal, and performs Opus decompression and decoding on the second target audio stream to obtain a second target audio stream in PCM format for playback.

[0013] In the above solution: the AR glasses compress the PCM format speech into Opus format before sending it to the first mobile terminal, and the first mobile terminal translates the speech and then compresses the translation back into Opus format before returning it to the AR glasses. This can significantly reduce the amount of data transmitted over the wireless link, save Bluetooth bandwidth and reduce power consumption. At the same time, Opus encoding has low latency, high compression rate and good anti-packet loss performance, which can effectively ensure the fluency and speech clarity of real-time translation.

[0014] Secondly, embodiments of this application provide a method for translating voice call audio, applied to a first mobile terminal, the first mobile terminal being connected to a wearable device via Bluetooth, the method comprising: The wearable device receives a compressed and encoded first voice stream, which is collected by the wearable device when the first user is making a call in a first language. The first speech stream is translated into a first target audio stream in a second language, and the first target audio stream is compressed and encoded before being sent to the wearable device for decoding. The device receives the decoded first target audio stream sent by the wearable device and sends the decoded first audio stream to the second user's second mobile terminal.

[0015] In the above solution: Addressing the need for real-time two-way translation during voice calls, when the first user uses a wearable device for a call, the first mobile terminal lacks the authority to directly transmit data during the call. Therefore, in the uplink communication link, after receiving the translated first target audio stream from the first mobile terminal, the wearable device compresses and encodes the first target audio stream, then sends the compressed and encoded first target audio stream back to the first mobile terminal, which then forwards it to the other party's second mobile terminal. By adding the process of the wearable device sending the received translation back to the first mobile terminal, and then the first mobile terminal forwarding it to the other party's second mobile terminal, the transmission of translation across devices is achieved, thus supporting real-time two-way voice translation. Simultaneously, by reusing the existing Bluetooth channel between the wearable device and the first terminal device, and instead using the glasses as a relay to transmit the translation, there is no need to develop an additional network interface for the first terminal device to send data, simplifying system implementation and improving data transmission compatibility and stability.

[0016] Optionally, when the first speech stream is a first compressed audio stream encoded with Opus, the step of translating the first speech stream into a first target audio stream in a second language, and compressing and encoding the first target audio stream before sending it to the wearable device for decoding includes: The first compressed audio stream is decompressed and decoded using Opus to obtain a first digital audio stream; the first digital audio stream is obtained by the wearable device through PCM encoding of the first speech stream. The first digital audio stream encoded by PCM is translated into a first target audio stream in a second language, and the first target audio stream is compressed and encoded by Opus to obtain the first target audio stream. The first target audio stream, after Opus compression encoding, is sent to the wearable device so that the wearable device converts the first target audio stream into PCM format.

[0017] In the above solution: the wearable device compresses the PCM format speech into Opus format before sending it to the first mobile terminal, and the first mobile terminal translates the speech and then re-compresses the translation into Opus format before returning it to the wearable device. This can significantly reduce the amount of data transmission on the Bluetooth link, save Bluetooth bandwidth and reduce power consumption. At the same time, Opus encoding has low latency, high compression rate and good anti-packet loss performance, which can effectively ensure the fluency and speech clarity of real-time translation.

[0018] Optionally, the method further includes: Receive a second voice stream from a second mobile terminal when a second user is making a call in a second language, and send the second voice stream to the wearable device for compression encoding; Receive the compressed and encoded second voice stream sent by the wearable device; The second speech stream is translated into a second target audio stream in the first language, and the second target audio stream is compressed and encoded before being sent to the wearable device for decoding and playback.

[0019] In the above solution: Addressing the need for real-time two-way translation during voice calls, when the first user uses a wearable device for a call, the first mobile terminal lacks the permission to directly transmit data into the call. Therefore, in the downlink communication link, after the first mobile terminal receives the second voice stream sent by the other device, it transmits it to the wearable device via the existing Bluetooth connection. The wearable device acts as a relay, bringing the second voice stream into the call. This eliminates the need for an additional data transmission network interface, simplifying system implementation and improving data transmission compatibility and stability. Simultaneously, the wearable device compresses the second voice stream before sending it to the first mobile terminal for translation, shortening the overall latency from receiving the translation to playback and improving the real-time conversation experience.

[0020] Optionally, when the second speech stream is Opus compressed and encoded into a second compressed audio stream, the step of translating the second speech stream into a second target audio stream in the first language, and compressing and encoding the second target audio stream before sending it to the wearable device for decoding and playback includes: The second compressed audio stream is decompressed and decoded using Opus to obtain a second digital audio stream; the second digital audio stream is obtained by PCM encoding the second voice stream of the second user when making a call in the second language by the wearable device. The second digital audio stream is translated into a second target audio stream in the first language, and the second target audio stream is compressed and encoded using Opus to obtain the second target audio stream. The second target audio stream, after Opus compression encoding, is sent to the wearable device so that the wearable device can decompress and decode the second target audio using Opus and play it in PCM format.

[0021] In the above solution: the wearable device compresses the PCM format speech into Opus format before sending it to the first mobile terminal, and the first mobile terminal translates the speech and then re-compresses the translation into Opus format before returning it to the wearable device. This can significantly reduce the amount of data transmission on the Bluetooth link, save Bluetooth bandwidth and reduce power consumption. At the same time, Opus encoding has low latency, high compression rate and good anti-packet loss performance, which can effectively ensure the fluency and speech clarity of real-time translation.

[0022] Thirdly, embodiments of this application provide a wearable device, including a processor, a memory, a Bluetooth module, a microphone, and a playback device, wherein the playback device, the microphone, the Bluetooth module, the memory, and the processor are connected via a bus; The playback device is used to play call audio; The microphone is used to capture call audio; The Bluetooth module is used to communicate with the mobile terminal to send and receive audio data. The memory stores a computer program, and the processor executes the steps of the method described in any one of the first aspects according to the computer program.

[0023] Fourthly, embodiments of this application provide a mobile terminal, including a processor, a memory, and a Bluetooth module, wherein the Bluetooth module, the memory, and the processor are connected via a bus; The Bluetooth module is used to communicate with wearable devices for audio data transmission and reception; The memory stores a computer program, and the processor executes the steps of the method described in any one of the second aspects according to the computer program.

[0024] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions that, when executed, can implement the steps of any of the above-described methods for translating voice calls.

[0025] The technical effects of any of the implementation methods in the third to fifth aspects can be found in the technical effects of the corresponding implementation methods in the first and second aspects, and will not be repeated here. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A schematic diagram of a one-way translation transmission mode provided in an embodiment of this application; Figure 2 This is a schematic diagram of a two-way voice call real-time translation mode provided in an embodiment of this application; Figure 3 A timing interaction diagram for uplink voice translation provided in an embodiment of this application; Figure 4 A timing interaction diagram for downlink voice translation provided in an embodiment of this application; Figure 5 This is a diagram of a complete uplink and downlink two-way voice call translation architecture; Figure 6 A flowchart for converting PCM format to Opus format provided in this application embodiment; Figure 7 A flowchart for converting Opus format to PCM format provided in this application embodiment; Figure 8 A functional structure diagram of a wearable device provided in an embodiment of this application; Figure 9 This application provides a functional structure diagram of a mobile terminal according to an embodiment of the present application. Figure 10 A hardware structure diagram of a wearable device provided in an embodiment of this application; Figure 11 This is a hardware structure diagram of a mobile terminal provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0029] Based on the exemplary embodiments shown in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete technical solution on its own.

[0030] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0031] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related products or entities and do not necessarily imply a specific order or sequence, unless otherwise indicated. It should be understood that such terms can be used interchangeably where appropriate, for example, in situations where implementation can proceed in a sequence other than those given in the embodiments illustrated or described in this application.

[0032] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0033] Two-way real-time voice calls and translation break down language barriers, allowing users with different native languages ​​to communicate instantly and without obstacles, just like using the same language. However, the one-way transmission mode of the translation in existing real-time translation systems cannot meet the needs of two-way real-time voice translation.

[0034] In view of this, this application provides a method for translating voice calls. When making a voice call using a wearable device, only the wearable device has the authority to transmit data during the call. Therefore, after receiving the translation of the first voice stream returned by the first mobile terminal, the wearable device decodes the translation and sends the decoded translation to the first mobile terminal via an existing Bluetooth connection. The first mobile terminal then forwards the translation to the second mobile terminal, enabling the other party to listen to the voice. Additionally, after receiving the second voice stream from the second mobile terminal, the first mobile terminal sends the second voice stream to the wearable device via Bluetooth. The wearable device compresses and encodes the second voice stream before sending it to the first mobile terminal for translation. It then decodes and plays the translation returned by the first mobile terminal, enabling the other party to listen to the voice. Thus, real-time translation of two-way voice calls is achieved without developing additional network interfaces.

[0035] like Figure 2 The diagram shown is an architecture diagram of the real-time translation system for two-way voice communication provided in this application. In this system, the solid data stream represents the uplink that sends its own voice, and the dashed data stream represents the downlink that receives the other party's voice.

[0036] In the uplink, the wearable device collects the first voice stream when the first user is talking in the first language, and sends the first voice stream to the first mobile terminal via Bluetooth. The first mobile terminal translates the first voice stream into the first target audio stream in the second language used by the second user and returns it to the wearable device. The wearable device sends the first target audio stream to the first mobile terminal, which then sends it to the second user on the second mobile terminal side for listening.

[0037] In the uplink, to address the need for bidirectional real-time translation during voice calls, when the first user uses a wearable device for a call, the first mobile terminal does not have permission to directly transmit data during the call. Therefore, in the uplink communication link, after the wearable device receives the first target audio stream translated from the first voice stream sent by the first mobile terminal, it compresses and encodes the first target audio stream and sends the compressed and encoded first target audio stream back to the first mobile terminal, which then forwards it to the other party's second mobile terminal. By adding the process of the wearable device sending the received translation back to the first mobile terminal, and then the first mobile terminal forwarding it to the other party's second mobile terminal, the transmission of translation between devices is realized, thus supporting bidirectional real-time voice translation. At the same time, by reusing the existing Bluetooth channel between the wearable device and the first terminal device, and instead using the glasses as a relay to transmit the translation, there is no need to develop an additional network interface for the first terminal device to send data, simplifying the system implementation and improving the compatibility and stability of data transmission.

[0038] In the downlink, the first mobile terminal receives the second voice stream when the second user makes a call in the second language using the second mobile terminal, and sends the second voice stream to the wearable device via Bluetooth connection. The wearable device sends the second voice stream to the first mobile terminal. The first mobile terminal translates the second voice stream into a second target audio stream in the first language used by the first user and returns it to the wearable device. The wearable device plays the second target audio stream to the first user for listening.

[0039] In the downlink, to address the need for two-way real-time translation during voice calls, when the first user uses a wearable device for a call, the first mobile terminal does not have permission to directly transmit data into the call. Therefore, in the downlink communication link, after the first mobile terminal receives the second voice stream sent by the other device, it sends it to the wearable device via the existing Bluetooth connection. The wearable device acts as a relay, bringing the second voice stream into the call. This eliminates the need for an additional data transmission network interface, simplifying system implementation and improving data transmission compatibility and stability. Simultaneously, the wearable device compresses the second voice stream before sending it to the first mobile terminal for translation, shortening the overall latency from receiving the translation to playback and improving the real-time conversation experience.

[0040] See Figure 3 The timing interaction diagram for uplink voice translation provided in this application embodiment mainly includes the following steps: S301: The wearable device collects the first voice stream when the first user is speaking in the first language.

[0041] The wearable device is equipped with a microphone that has noise reduction and echo cancellation functions, and can process the first voice signal collected when the first user is talking in the first language into a clean first voice stream.

[0042] S302: After compressing and encoding the first voice stream, the wearable device sends it to the first mobile terminal via Bluetooth.

[0043] Typically, the first audio stream captured by the microphone is an analog signal, which is then modulated by Pulse Code Modulation (PCM) to obtain a digitized first digital audio stream.

[0044] To reduce Bluetooth transmission bandwidth usage and latency while ensuring voice quality, the first voice stream in PCM format can be efficiently compressed.

[0045] Opus coding, with its extremely low algorithm latency, excellent compression ratio, and adaptability to network jitter, has become an ideal choice for speech compression. Therefore, after obtaining the first digitized digital audio stream, the first digital audio stream can be first compressed and encoded using Opus to obtain the first compressed audio stream, and then the first compressed audio stream can be sent to the first mobile terminal via an SPP / BLE link.

[0046] By converting the first audio stream from PCM format to Opus format, transmission efficiency and system stability can be significantly improved in real-time translation scenarios.

[0047] S303: The first mobile terminal translates the first voice stream into a first target audio stream in a second language, compresses and encodes the first target audio stream, and then sends it to the wearable device via Bluetooth connection.

[0048] The first voice stream received by the first mobile terminal is a compressed audio stream encoded with Opus. Therefore, before translation, the first compressed audio stream can be decompressed and decoded using Opus to obtain a first digital audio stream encoded with PCM. Then, the language translation model is called to translate the first digital audio stream into a first target audio stream in the second language. At this time, the first target audio stream is a digital audio signal in PCM format. After Opus compression encoding, the first target audio stream in Opus format is obtained.

[0049] S304: Sends the compressed first target audio stream to the wearable device via Bluetooth connection.

[0050] Since the first mobile terminal does not have permission to input data into the call, it needs to send the Opus-compressed first target audio stream to the wearable device again via the SPP / BLE link, and the wearable device will then send the translated first target audio stream into the call.

[0051] S305: The wearable device decompresses and decodes the first target audio stream sent by the first mobile terminal, and sends the decoded first target audio stream back to the first mobile terminal.

[0052] The wearable device receives a first target audio stream in Opus format. After Opus decompression and decoding, it obtains a first target audio stream in PCM format. Then, the first target audio stream in PCM format is transmitted to the first mobile terminal via Bluetooth.

[0053] Optionally, when the first mobile terminal is a mobile phone, the wearable device can send data to the mobile phone using Bluetooth Hands-Free Profile (HFP).

[0054] S306: The first mobile terminal sends the first target audio stream to the second mobile terminal for playback.

[0055] After receiving the first target audio stream in PCM format, the first terminal sends it to the second mobile terminal via a call protocol. The second mobile terminal then plays it for the second user to listen to. Since the first target audio stream has been translated into the language used by the second user, the second user can understand what the first user is saying.

[0056] See Figure 4 The following is a timing interaction diagram of downlink voice translation provided in this application embodiment, which mainly includes the following steps: S401: The second mobile terminal collects the second voice stream when the second object is making a call in the second language, and sends the second voice stream to the first mobile terminal.

[0057] When the second user uses the second mobile terminal to talk to the first user, the second mobile terminal collects the analog signal of the second user talking in the second language, encodes it using PCM to obtain the second voice stream, and sends the second voice stream to the first mobile terminal.

[0058] Optionally, in order to improve the network transmission speed of the second voice stream, the second mobile terminal may send the second voice stream to the first mobile terminal after Opus compression encoding.

[0059] S402: The first mobile terminal sends the second voice stream to the wearable device.

[0060] The first mobile terminal does not have permission to transmit data into the call link. Therefore, it sends the second voice stream to the wearable device, which then sends the second voice stream into the call.

[0061] Optionally, the second voice stream received by the first mobile terminal is in Opus format, but the Bluetooth module of the wearable device only understands SBC or AAC format audio. Therefore, the first mobile terminal can decode the second voice stream and encode it into an SBC or AAC format second voice stream according to the Bluetooth protocol before sending it to the wearable device.

[0062] S403: The wearable device compresses and encodes the second voice stream and then sends it to the first mobile terminal.

[0063] To reduce Bluetooth transmission bandwidth usage and latency while ensuring voice quality, the second voice stream can be efficiently compressed.

[0064] Opus encoding, with its extremely low algorithm latency, excellent compression ratio, and adaptability to network jitter, has become an ideal choice for voice compression. Therefore, after receiving a second voice stream in SBC or AAC format, the wearable device performs PCM encoding on the second voice stream to obtain a second digital audio stream, then performs Opus compression encoding on the second digital audio stream to obtain a second compressed audio stream, and sends the second compressed audio stream to the first mobile terminal via an SPP / BLE link.

[0065] By converting the second audio stream from PCM format to Opus format, transmission efficiency and system stability can be significantly improved in real-time translation scenarios.

[0066] S404: The first mobile terminal translates the second voice stream into a second target audio stream in the first language, and compresses and encodes the second target audio stream.

[0067] The second voice stream received by the first mobile terminal is a second compressed audio stream after Opus compression encoding. Therefore, before translation, the second compressed audio stream can be decompressed and decoded by Opus to obtain a second digital audio stream after PCM encoding. Then, the language translation model is called to translate the second digital audio stream into a second target audio stream in the first language. At this time, the second target audio stream is a digital audio signal in PCM format.

[0068] After the first mobile terminal translates the audio, it performs Opus compression encoding on the PCM format second target audio stream to obtain the Opus format second target audio stream.

[0069] S405: The first mobile terminal sends the compressed and encoded second target audio stream to the wearable device via Bluetooth connection.

[0070] The compressed second target audio stream is sent to the wearable device using the SPP / BLE protocol.

[0071] S406: The wearable device decodes and plays the second target audio stream.

[0072] After receiving the second target audio stream in Opus format, the wearable device decodes it into PCM format and plays it to the first user through a speaker or bone conduction device. Since the first mobile terminal has already translated the second speech stream in the second language into the first language, the first user can understand what the second user is saying.

[0073] During real-time translation of two-way voice calls, the wearable device compresses the PCM format voice into Opus format before sending it to the first mobile terminal. The first mobile terminal then translates the speech and recompresses it into Opus format before returning it to the wearable device. This significantly reduces the amount of data transmitted over the Bluetooth link, saving Bluetooth bandwidth and reducing power consumption. At the same time, Opus encoding features low latency, high compression rate, and good packet loss resistance, effectively ensuring the fluency and clarity of real-time translation.

[0074] See Figure 5This diagram illustrates a complete uplink and downlink bidirectional voice call translation architecture. In both uplink and downlink, the wearable device, regardless of whether the voice stream is sent to its own user or received from the other user, compresses it into Opus format after encoding it in PCM format. This Opus format audio stream is then transmitted to the first mobile terminal connected via Bluetooth via the SPP / BLE protocol. The first mobile terminal decompresses and decodes the Opus audio stream back into PCM format, inputs it into a language translation model for translation, compresses the translated text into Opus format, and then sends it back to the Bluetooth-connected wearable device via the SPP / BLE protocol. The wearable device then uses this translation for call or playback. Throughout the entire bidirectional voice translation process, the wearable device acts as a relay, transmitting the translated text and voice over the existing Bluetooth connection, thus achieving real-time translation for bidirectional voice calls.

[0075] During data transmission between the wearable device and the first mobile terminal, in order to reduce network bandwidth and transmission latency, the PCM format digital audio stream is compressed into the Opus format. This compression process involves multiple stages of audio processing and encoding technology.

[0076] Phase 1: Preprocessing Phase

[0077] Before the Opus encoder performs compression encoding, the PCM format digital audio stream needs to undergo the following series of preparation steps: Step 1, Resampling The Opus encoder operates internally at a fixed sampling rate (typically 48kHz). If the sampling rate of the PCM format digital audio stream differs from that of the Opus encoder (e.g., 44.1kHz, 16kHz, or 8kHz), the Opus encoder will first "resample" the digital audio stream to its own sampling rate. This is a crucial step that enables the Opus encoder to handle various audio inputs.

[0078] Step 2, Channel Processing

[0079] The Opus encoder supports both mono and stereo. If the PCM format digital audio stream is stereo, the Opus encoder analyzes the signal correlation between multiple channels. When the signal correlation is below a set correlation threshold, each channel of the digital audio stream can be encoded independently. Otherwise, the multiple channels of the digital audio stream are mixed to process the digital audio stream into joint stereo, and then the mixed result is encoded to save bitrate.

[0080] Step 3: Frame Segmentation

[0081] The Opus encoder divides PCM format digital audio streams into "frames" of a set duration. The Opus encoder offers a very flexible setting for the duration of each "frame," including but not limited to 2.5ms, 5ms, 10ms, 20ms, 40ms, and 60ms. Typically, 20ms is used as one frame, i.e., one audio frame.

[0082] Phase Two: Coding Technology Decisions

[0083] The Opus encoder is known for its high flexibility and excellent sound quality because it combines two core coding technologies: Speech-optimized Linear Predictive Coding (SILK) and Constrained Energy Lapped Transform (CELT). SILK is mainly suitable for speech coding, with a lower bit rate and slightly higher latency, while CELT is mainly suitable for audio coding of music, mixed content, and low-latency scenarios.

[0084] After the digital audio stream is framed, the Opus encoder determines which encoding technique to use for each speech frame based on the audio characteristics of that speech frame.

[0085] Optional audio features include, but are not limited to, audio bandwidth, bit rate limitations, and signal energy distribution.

[0086] Phase 3, Frame Coding

[0087] (1) SILK encoding

[0088] SILK coding involves Linear Predictive Coding (LPC) and Long-Term Prediction (LTP) coding techniques, primarily used for low-bitrate speech transmission (e.g., 6-40 kbps). LPC is applied to each frame using SILK coding to model the spectral envelope of multiple sampling points within each frame, removing short-term correlations. LTP is typically applied to frames belonging to voiced speech segments, utilizing gene periodicity to remove long-term correlations.

[0089] The core idea of ​​LPC technology is that each sampled value can be predicted using a linear combination of several past sampled values. Specifically, for the sampled values ​​in each speech frame, a set of linear prediction coefficients is solved by minimizing the prediction error. This set of prediction coefficients can characterize the spectral envelope characteristics of the vocal tract system, that is, reflect the modulation effect of the vocal tract shape on the sound source signal. Therefore, by using multiple sets of prediction coefficients, the predicted speech signal of the current speech frame can be obtained.

[0090] The core idea of ​​LPC technology is that when the waveform of a speech frame is highly similar to a previous frame or a segment of historical waveforms, it is not necessary to transmit the complete waveform signal of that frame. In this case, the Opus encoder only needs to calculate and transmit the shift parameter and gain parameter of that frame. The shift parameter represents the time interval (i.e., pitch delay) between the current speech frame and similar waveforms in historical frames, and the gain parameter represents the amplitude scaling factor of the similar waveforms. The Opus decoder extracts the corresponding waveform segment from the historical frame based on the shift parameter and multiplies it by the gain parameter to reconstruct the predicted speech signal of the current speech frame, thereby restoring the periodic structure of the speech.

[0091] Typically, to achieve finer control in the time domain, speech frames can be further subdivided into multiple subframes, thereby more accurately modeling pitch variations, energy fluctuations, and non-stationary characteristics, while controlling latency and bit rate.

[0092] After obtaining the predicted speech signal of the speech frame, residual quantization encoding can be performed on the original speech signal based on the speech frame. Specifically, the predicted speech signal is subtracted from the original speech signal to obtain the residual, which is then quantized, and the quantized parameters are losslessly compressed. The residual is usually very small, and quantizing and encoding it can greatly save the amount of data.

[0093] Optionally, lossless compression can be achieved using entropy coding.

[0094] It should be noted that before encoding the speech frame using SILK, the speech frame can be high-pass filtered to remove DC offset and low-frequency noise from the original speech signal.

[0095] (2) CELT encoding

[0096] CELT encoding uses a modified discrete cosine transform (MDCT) as a time-frequency conversion tool, similar to MP3 or AAC, but optimized for extremely low latency.

[0097] Because digital audio streams are segmented into independent speech frames for processing, discontinuous blocking artifacts appear at the boundaries between frames. To eliminate these blocking artifacts, MDCT uses a special "temporal aliasing cancellation" technique, adding a smooth window function (such as a sine window) to each speech frame. The window length is twice the frame length (e.g., 480 points for a frame and 960 points for a window), and the windows overlap by 50%, thereby reducing boundary noise caused by segmentation.

[0098] In CELT coding, the speech frame is decomposed into a combination of sine waves of different frequencies using MDCT (Multi-Frequency Transform), obtaining MDCT transform coefficients. These coefficients, as frequency domain data, represent the energy distribution of the signal at different frequencies. Furthermore, these coefficients are combined into several wider and unequal frequency groups based on a psychoacoustic model (frequencies inaudible to the human ear are discarded or have reduced precision). Each group is called a band, allowing for more efficient coding that better matches auditory characteristics. At low bit rates, some high-frequency coefficients are coarsely quantized or even discarded. In this case, the CELT encoder inserts specific noise to fill these gaps, preventing the sound from sounding too dull or producing a "metallic" tone.

[0099] After quantization, the quantized spectral data is compressed using a range encoder. Optionally, an entropy encoder can be used as the range encoder.

[0100] (3) Hybrid coding

[0101] In addition to SILK encoding and CELT encoding, this application also provides a hybrid encoding technique, which combines SILK encoding and CELT encoding at a medium code rate, wherein the low-frequency part is encoded using SILK encoding and the high-frequency part is encoded using CELT encoding.

[0102] Phase 4, Bitrate Control

[0103] After quantization is completed, the Opus encoder estimates the number of bits required to encode the quantization result, compares the estimated number of bits with the target bit rate, dynamically adjusts the quantization parameters based on the comparison result, and requantizes, thereby achieving dynamic adjustment of SILK and CELT quantization precision.

[0104] When the target bit rate is very low, the Opus encoder will sacrifice high-frequency details to ensure basic intelligibility.

[0105] The target bitrate can be set by the user or determined adaptively by the network.

[0106] Phase 5, Packaging

[0107] After the quantization result that meets the target bit rate is encoded, it is packaged into Opus data packets. Each data packet contains a frame header that indicates the encoding technology used, frame size, and other information.

[0108] like Figure 6 The diagram shows the complete process for converting the PCM format to the Opus format.

[0109] During real-time translation of two-way voice calls, the PCM format digital audio stream is converted to Opus format to ensure high speech clarity and music fidelity at extremely low bit rates. Furthermore, thanks to the Opus encoder's dynamic coding technology switching (SILK / CELT), psychoacoustic band optimization, and low-latency frame structure, transmission bandwidth usage and power consumption can be effectively reduced, while resisting wireless interference to achieve stable and smooth real-time voice communication.

[0110] In Bluetooth communication, after receiving a compressed audio stream in Opus format, the wearable device or the first mobile terminal needs to decode it back into PCM format. If encoding is like "subtraction" (i.e., removing redundancy), then decoding is like "addition" (i.e., reconstructing the signal).

[0111] Decoding Opus format back to PCM format also requires multiple stages of processing and decoding technology: Phase 1: Unpacking and Frame Parsing The Opus decoder receives an Opus-formatted data packet, which typically consists of a Toc (Table of Contents) byte and encoded data. Since a data packet may contain multiple audio frames, the decoder divides the bitstream into several independent frame encoded data according to the frame length flag.

[0112] The Toc byte is the first byte of the data packet. The encoder uses the Toc byte to identify whether the encoding technology is SILK, CELT or hybrid encoding, whether the bandwidth is narrow bandwidth, wide bandwidth or full bandwidth, and the frame size, etc., in order to determine the boundaries of each frame and complete frame parsing.

[0113] Phase 2: Coding Technology Diversion

[0114] Based on the knowledge of the Toc bytes, the Opus decoder sends each decoded audio frame to the corresponding decoder, such as the SILK decoder, CELT decoder, and hybrid decoder.

[0115] Phase 3, Decoding

[0116] (1) SILK decoding

[0117] First, a range decoder is used to restore the compressed bitstream to quantized parameters (LPC prediction coefficients, LTP gain and shift, etc.).

[0118] Then, based on these parameters, the quantization index of the residual signal is parsed using a probability model, and the residual signal is reconstructed.

[0119] Next, based on the residual signal and the gain of the subframe, the "excitation signal" of the sound source is reconstructed, which is equivalent to restoring the pulse or noise generated by the vocal cords.

[0120] Finally, a complete speech waveform is synthesized based on the excitation signal. During encoding, SILK uses LPC analysis to "subtract" vocal tract features, while during decoding, these features are "added." The SILK decoder uses LPC coefficients to construct a synthesis filter, passing the excitation signal through this filter to synthesize the speech waveform. This is analogous to shaping a flat pulse signal to make it sound like a human voice. Furthermore, using the pitch period parameter of LTP, the periodicity of the speech is recovered, further synthesizing the complete speech waveform.

[0121] (2) CELT decoding

[0122] First, a range decoder is used to decode the quantized spectral coefficients and energy information.

[0123] Then, the spectral coefficients are restored to their normal frequency domain values, and the energy information is used to scale the spectral coefficients.

[0124] Next, at low bit rates, when the CELT decoder detects that high-frequency details have been dropped, it actively injects specific pseudo-random noise to fill the gaps and prevent the sound from sounding muffled or "hollow".

[0125] Finally, the frequency domain coefficients are converted back to time domain signals to obtain discrete semantic frames after framing.

[0126] Phase 3, Post-processing

[0127] To reduce block boundary artifacts, CELT encoding uses temporal aliasing. Therefore, during decoding, adjacent speech frames must be "overlapped and added" to ensure a smooth transition and eliminate block artifacts.

[0128] When the Opus decoder detects a data packet loss in the current speech frame, it does not mute it directly. Instead, it predicts and fills the gap in the current frame by stretching or copying the waveform of the previous frame based on the pitch period and energy of the previous frame. This makes the speech still coherent and without obvious interruptions even when there is severe network jitter.

[0129] Phase 4, Resampling

[0130] Typically, Opus processes samples at 48kHz. If the user needs to output a 44.1kHz or 16kHz PCM format digital audio stream, the Opus decoder will perform high-quality resampling in the final step.

[0131] Phase 5, Output

[0132] The Opus decoder outputs a PCM format digital audio stream, typically encapsulated in int16 or floating-point format, for playback by audio playback devices (sound cards).

[0133] See Figure 7 This describes the process of decoding the Opus format back to PCM format.

[0134] The Opus decoder, with its hybrid architecture of SILK and CELT and packet loss concealment mechanism, maintains natural and intelligible sound quality under limited bandwidth, exhibits extremely high robustness, and ensures smooth and stable calls within the acceptable range of low-power embedded platforms.

[0135] When making voice calls using wearable devices, two-way real-time voice communication and translation are achieved via Bluetooth, breaking down language barriers and allowing users with different native languages ​​to communicate seamlessly and instantly as if speaking the same language. Furthermore, during calls, the wearable device and the first mobile terminal connected via Bluetooth use Opus format compressed audio streams. Through Opus's intelligent compression algorithm, extremely low bit rate and latency characteristics, and powerful error concealment capabilities, a perfect balance of high sound quality, low bandwidth, low latency, and strong robustness is achieved. This enables high-quality, low-latency, and highly reliable two-way real-time voice communication and translation even in the limited bandwidth and potentially unstable environment of Bluetooth transmission, effectively breaking down language barriers and significantly improving the user experience of real-time communication and streaming media services.

[0136] It should be noted that the PCM to Opus format conversion method used for transmission between the wearable device and the first mobile terminal is only an example, and other format transmission schemes, such as PCM to LC3plus or aptX Adaptive, are not excluded.

[0137] Based on the same technical concept, embodiments of this application provide a wearable device, including but not limited to Bluetooth headsets, AR glasses, smartwatches, etc., which can realize the above-mentioned method for translating voice calls on the wearable device side and achieve the same technical effect.

[0138] See Figure 8 The wearable device is connected to the first mobile terminal via Bluetooth, and includes a voice acquisition module 801, an encoding / decoding module 802, and a transceiver module 803, wherein: The voice acquisition module 801 is used to acquire the first voice stream when the first user is speaking in the first language. The encoding / decoding module 802 is used to compress and encode the first speech stream; The transceiver module 803 is configured to send a compressed and encoded first voice stream to a first mobile terminal so that the first mobile terminal can translate the first voice stream into a first target audio stream in a second language; and to receive a compressed and encoded first target audio stream returned by the first mobile terminal. The codec module 802 is also used to decode the first target audio stream; The transceiver module 803 is also used to send the decoded first target audio stream to the first mobile terminal, so that the first mobile terminal sends the decoded first target audio stream to the second user's second mobile terminal.

[0139] Optionally, the codec module 802 is specifically used for: The first speech stream is PCM encoded to obtain a first digital audio stream, and the first digital audio stream is Opus compressed to obtain a first compressed audio stream; and the first target audio stream is Opus decompressed and decoded to obtain a first target audio stream in PCM format.

[0140] The transceiver module 803 is specifically configured to: send a first compressed audio stream to a first mobile terminal so that the first mobile terminal can translate the first speech stream into a first target audio stream in a second language; receive the first target audio stream after Opus compression encoding returned by the first mobile terminal; and send the first target audio stream in PCM format to the first mobile terminal.

[0141] Optionally, the voice acquisition module 801 is also used to receive a second voice stream sent by the first mobile terminal when the second user is talking in a second language. The second voice stream is acquired by the second mobile terminal and sent to the first mobile terminal. The encoding / decoding module 802 is also used to compress and encode the second speech stream; The transceiver module 803 is further configured to send the compressed and encoded second voice stream to the first mobile terminal so that the first mobile terminal can translate the second voice stream into a second target audio stream in the first language; and to receive the compressed and encoded second target audio stream returned by the first mobile terminal and decode and play it.

[0142] Optionally, the codec module 802 is also used for: The second speech stream is PCM encoded to obtain a second digital audio stream, and the second digital audio stream is Opus compressed to obtain a second compressed audio stream; and the second target audio stream is Opus decompressed and decoded to obtain a second target audio stream in PCM format for playback. The transceiver module 803 is further configured to send a second compressed audio stream to a first mobile terminal so that the first mobile terminal translates the second speech stream into a second target audio stream in a first language; and to receive the second target audio stream after Opus compression encoding returned by the first mobile terminal.

[0143] Based on the same technical concept, this application provides a mobile terminal, including but not limited to smartphones, televisions, tablets, etc., which can implement the above-mentioned method for translating voice calls on the mobile terminal side and achieve the same technical effect.

[0144] See Figure 9 The mobile terminal connects to the wearable device via Bluetooth, including a transceiver module 901, a translation module 902, and an encoding / decoding module 903, wherein: The transceiver module 901 is used to receive the compressed and encoded first voice stream sent by the wearable device. The first voice stream is collected by the wearable device when the first user is making a call in the first language. Translation module 902 is used to translate the first speech stream into a first target audio stream in a second language. The encoding / decoding module 903 is used to compress and encode the first target audio stream and send it to the wearable device for decoding. The transceiver module 901 is also used to receive the decoded first target audio stream sent by the wearable device and send the decoded first audio stream to the second user's second mobile terminal.

[0145] Optionally, when the first voice stream is a first compressed audio stream after Opus compression encoding, the encoding / decoding module 903 is specifically used to: perform Opus decompression and decoding on the first compressed audio stream to obtain a first digital audio stream; the first digital audio stream is obtained by the wearable device performing PCM encoding on the first voice stream; Translation module 902 is specifically used to: translate the PCM-encoded first digital audio stream into a first target audio stream in a second language; The encoding / decoding module 903 is specifically used to: perform Opus compression encoding on the first target audio stream to obtain the first target audio stream; The transceiver module 901 is specifically used to send the first target audio stream after Opus compression encoding to the wearable device, so that the wearable device can convert the first target audio stream into PCM format.

[0146] Optionally, the transceiver module 901 is further configured to: receive a second voice stream sent by the second mobile terminal when the second user is making a call in a second language, and send the second voice stream to the wearable device for compression encoding; and receive the compressed and encoded second voice stream sent by the wearable device. Translation module 902 is also used to: translate the second speech stream into a second target audio stream in the first language; The codec module 903 is also used to compress and encode the second target audio stream; The transceiver module 901 is also used to send the compressed and encoded second target audio stream to the wearable device for decoding and playback.

[0147] Optionally, when the second voice stream is Opus compressed and encoded into a second compressed audio stream, the codec module 903 is specifically used to: translate the second voice stream into a second target audio stream in the first language, and perform Opus decompression and decoding on the second compressed audio stream to obtain a second digital audio stream; the second digital audio stream is obtained by PCM encoding the second voice stream of the second user when making a call in the second language by the wearable device. Translation module 902 is specifically used to: translate the second digital audio stream into a second target audio stream in the first language; The encoding / decoding module 903 is specifically used to: perform Opus compression encoding on the second target audio stream to obtain the second target audio stream; The transceiver module 901 is specifically used to send the second target audio stream after Opus compression encoding to the wearable device, so that the wearable device can decompress and decode the second target audio in Opus and play it in PCM format.

[0148] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0149] Based on the same technical concept, this application provides a wearable device that can implement the steps of the above-described voice translation method and achieve the same technical effect.

[0150] See Figure 10 The wearable device includes a processor 1001, a memory 1002, a Bluetooth module 1003, a microphone 1004, and a playback device 1005. The playback device 1005, microphone 1004, Bluetooth module 1003, memory 1002, and processor 1001 are connected via a bus 1006. The playback device 1005 is used to play audio; Speaker 1004 is used for voice acquisition; Bluetooth module 1003 is used to communicate with mobile terminals to send and receive data; The memory 1002 stores a computer program, and the processor 1001 executes the program according to the computer program. Figure 3 and Figure 4 The steps of translating spoken audio in a Chinese phone call.

[0151] It should be noted that, Figure 10 This is just one example; wearable devices may also include components such as power supplies, buttons, and displays, which are not shown.

[0152] Based on the same technical concept, this application provides a mobile terminal that can implement the steps of the above-described voice translation method and achieve the same technical effect.

[0153] See Figure 11 The mobile terminal includes a processor 1101, a memory 1102 and a Bluetooth module 1103, which can be connected to the processor 1101 via a bus 1104. Bluetooth module 1103 is used to communicate with wearable devices to send and receive data; The memory 1102 stores a computer program, and the processor 1101 executes the program according to the computer program. Figure 3 and Figure 4 The steps of the method for translating voice call audio are shown below.

[0154] It should be noted that, Figure 11 This is just one example; other components not shown may include a power supply, buttons, a display screen, a speaker, a playback device, etc.

[0155] In this embodiment, the memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system and programs required for running face-to-face communication functions; the data storage area may store various instant messaging information and operation instruction sets. The memory may be volatile memory, such as random-access memory (RAM); it may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it may be any other medium capable of carrying or storing a desired computer program having an instruction or data structure form and accessible by a computer, but is not limited thereto. The memory may be a combination of the above-mentioned memories. The processor may include one or more central processing units (CPUs), GPUs, or digital processing units, etc.

[0156] This application does not limit the specific connection medium between the devices in the above-described device. The buses connecting the above-described devices are depicted with thick lines in the figures. The connection methods between other components are only for illustrative purposes and are not intended to be limiting. Buses can be divided into address buses, data buses, control buses, etc. For ease of description, only one thick line is used to describe them in the figures, but this does not mean that there is only one bus or one type of bus.

[0157] This application also provides a computer-readable storage medium for storing instructions that, when executed, can complete the steps of the voice translation method in the foregoing embodiments.

[0158] This application also provides a computer program product for storing a computer program that performs the steps of any of the voice translation methods described in the foregoing embodiments.

[0159] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0161] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0162] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0163] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for translating voice messages, characterized in that, Applied to wearable devices, wherein the wearable device is connected to a first mobile terminal via Bluetooth, the method includes: The system collects a first voice stream from a first user speaking in a first language, compresses and encodes the first voice stream, and sends it to the first mobile terminal so that the first mobile terminal can translate the first voice stream into a first target audio stream in a second language. The system receives the compressed and encoded first target audio stream returned by the first mobile terminal, decodes the first target audio stream, and sends it back to the first mobile terminal, so that the first mobile terminal sends the decoded first target audio stream to the second user's second mobile terminal.

2. The method as described in claim 1, characterized in that, The step of compressing and encoding the first speech stream and sending it to the first mobile terminal so that the first mobile terminal can translate the first speech stream into a first target audio stream in a second language includes: The first speech stream is PCM encoded to obtain a first digital audio stream, and the first digital audio stream is Opus compressed to obtain a first compressed audio stream. The first compressed audio stream is sent to the first mobile terminal so that the first mobile terminal can translate the first speech stream into a first target audio stream in a second language. The step of receiving the compressed and encoded first target audio stream returned by the first mobile terminal, and decoding the first target audio stream and sending it to the first mobile terminal includes: The system receives the first target audio stream after Opus compression encoding returned by the first mobile terminal, performs Opus decompression and decoding on the first target audio stream to obtain the first target audio stream in PCM format, and sends the first target audio stream in PCM format to the first mobile terminal.

3. The method as described in claim 1, characterized in that, The method further includes: The system receives a second voice stream from the first mobile terminal when the second user is making a call in the second language. The second voice stream is collected by the second mobile terminal and sent to the first mobile terminal. The second speech stream is compressed and encoded and then sent to the first mobile terminal so that the first mobile terminal can translate the second speech stream into a second target audio stream in the first language. The system receives the compressed and encoded second target audio stream returned by the first mobile terminal and decodes and plays it.

4. The method as described in claim 3, characterized in that, The step of compressing and encoding the second speech stream and sending it to the first mobile terminal so that the first mobile terminal can translate the second speech stream into a second target audio stream in the first language includes: The second speech stream is PCM encoded to obtain a second digital audio stream, and the second digital audio stream is Opus compressed to obtain a second compressed audio stream. The second compressed audio stream is sent to the first mobile terminal so that the first mobile terminal can translate the second speech stream into a second target audio stream in the first language. The step of receiving the compressed and encoded second target audio stream returned by the first mobile terminal and decoding and playing it includes: The system receives the second target audio stream after Opus compression encoding returned by the first mobile terminal, and performs Opus decompression and decoding on the second target audio stream to obtain a second target audio stream in PCM format for playback.

5. A method for translating voice messages, characterized in that, Applied to a first mobile terminal, the first mobile terminal being connected to a wearable device via Bluetooth, the method includes: The wearable device receives a compressed and encoded first voice stream, which is collected by the wearable device when the first user is making a call in a first language. The first speech stream is translated into a first target audio stream in a second language, and the first target audio stream is compressed and encoded before being sent to the wearable device for decoding. The device receives the decoded first target audio stream sent by the wearable device and sends the decoded first audio stream to the second user's second mobile terminal.

6. The method as described in claim 5, characterized in that, When the first speech stream is a first compressed audio stream encoded with Opus, the step of translating the first speech stream into a first target audio stream in a second language, and then compressing and encoding the first target audio stream before sending it to the wearable device for decoding includes: The first compressed audio stream is decompressed and decoded using Opus to obtain a first digital audio stream; the first digital audio stream is obtained by the wearable device through PCM encoding of the first speech stream. The first digital audio stream encoded by PCM is translated into a first target audio stream in a second language, and the first target audio stream is compressed and encoded by Opus to obtain the first target audio stream. The first target audio stream, after Opus compression encoding, is sent to the wearable device so that the wearable device converts the first target audio stream into PCM format.

7. The method as described in claim 5, characterized in that, The method further includes: Receive a second voice stream from a second mobile terminal when a second user is making a call in a second language, and send the second voice stream to the wearable device for compression encoding; Receive the compressed and encoded second voice stream sent by the wearable device; The second speech stream is translated into a second target audio stream in the first language, and the second target audio stream is compressed and encoded before being sent to the wearable device for decoding and playback.

8. The method as described in claim 7, characterized in that, When the second speech stream is Opus compressed and encoded into a second compressed audio stream, the step of translating the second speech stream into a second target audio stream in the first language, and then compressing and encoding the second target audio stream before sending it to the wearable device for decoding and playback includes: The second compressed audio stream is decompressed and decoded using Opus to obtain a second digital audio stream; the second digital audio stream is obtained by PCM encoding the second voice stream of the second user when making a call in the second language by the wearable device. The second digital audio stream is translated into a second target audio stream in the first language, and the second target audio stream is compressed and encoded using Opus to obtain the second target audio stream. The second target audio stream, after Opus compression encoding, is sent to the wearable device so that the wearable device can decompress and decode the second target audio using Opus and play it in PCM format.

9. A wearable device, characterized in that, It includes a processor, a memory, a Bluetooth module, a microphone, and a playback device, wherein the playback device, the microphone, the Bluetooth module, the memory, and the processor are connected via a bus; The playback device is used to play call audio; The microphone is used to capture call audio; The Bluetooth module is used to communicate with the mobile terminal to send and receive audio data. The memory stores a computer program, and the processor executes the steps of the method according to any one of claims 1 to 4.

10. A mobile terminal, characterized in that, It includes a processor, a memory, and a Bluetooth module, wherein the Bluetooth module, the memory, and the processor are connected via a bus; The Bluetooth module is used to communicate with wearable devices for audio data transmission and reception; The memory stores a computer program, and the processor executes the steps of the method according to any one of claims 5 to 8 based on the computer program.

Images