Concurrent reception of multiple user voice inputs for translation

Abstract

A method is provided, comprising: receiving, by a mobile device for providing translation, a sound input; determining that the sound input is from a foreign language speaker; in response to determining that the sound input is from the foreign language speaker, detecting an endpoint in the sound input; and in response to detecting an endpoint in the sound input: buffering the sound input up to the endpoint; inputting, by the mobile device or by the remote computing device, the buffered sound up to an endpoint for translation to another language; and providing the translated sound input for output.

Description

[0001] This application is a divisional application of an invention patent application with an application date of February 6, 2019, application number 201980011421.5, and invention name “Concurrent reception of multiple user voice inputs for translation”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 628,421, filed on February 9, 2018, the disclosure of which is incorporated herein by reference. Technical Field Background Art

[0004] Mobile computing devices such as mobile phones have available auxiliary devices that can provide additional functionality. For example, some earbuds provide a translation experience in which a user can touch one earbud, speak in a first language, and the phrase spoken is translated into a second language. The user can hear the translated phrase, for example, from a speaker in the mobile computing device. Similarly, a phrase spoken to a mobile phone in a second language can be translated into the first language and output through the earbud. This exchange is performed serially, so that each participant must wait for the other party's speech to be completed, translated, and output through the speaker of the mobile computing device or auxiliary device before speaking a response. The serial nature of the exchange significantly increases the delay of the conversational mode and introduces uncomfortable social cues. The interaction flow requires both parties to carefully pay attention to which person is speaking and whether the translation is being output on each device, and is therefore non-intuitive. Summary of the invention

[0005] The present disclosure provides a more natural dialogue exchange between a user and a foreign language speaker using the translation features of a mobile device and an auxiliary device. In particular, the microphones of both the mobile device and the auxiliary device are always listening for voice input. The mobile device and the auxiliary device can also determine whether the voice input is from the user or from the foreign language speaker. The mobile device and the auxiliary device can automatically determine when the spoken phrase is complete and ready for translation, send the spoken phrase for translation, and immediately start listening again. In this regard, even if another user is still speaking, in response to the translated voice of the other user, acknowledgments such as "OK", "Yes", "Oh No" or other social prompts can be captured, translated, and output through dialogue.

[0006] One aspect of the present disclosure provides a system for providing translation, which includes an auxiliary device and a mobile computing device coupled to the auxiliary device. For example, the mobile computing device can be wirelessly coupled to the auxiliary device. The auxiliary device includes: a microphone configured to continuously listen to a first input from a first user; and a communication interface adapted to send the first input. The microphone of the auxiliary device can be configured to continuously listen to the first input from the first user by continuing to listen when an endpoint in speech is detected. The mobile computing device includes: a mobile computing device microphone configured to continuously listen to a second input from a second user while the auxiliary device microphone listens to the first input from the first user; a communication interface adapted to receive the first input from the auxiliary device and send the information to the auxiliary device; one or more processors adapted to process the first input and the second input; and an output adapted to provide at least one of a processed first input or a processed second input. The microphone of the mobile computing device can be configured to continuously listen to the second input from the second user by continuing to listen when an endpoint in speech is detected. The processing of the first input and the second input by the one or more processors may include providing a translation of at least one of the first input and the second input. This may include the one or more processors of the mobile computing device performing the translation, or the one or more processors of the mobile computing device sending at least one of the first input and the second input to the remote device for translation.

[0007] The auxiliary device and the mobile device may be wirelessly coupled using two independent wireless communication channels including a first channel and a second channel, the first channel being used for transmission from the auxiliary device to the mobile device, and the second channel being used for transmission from the mobile device to the auxiliary device. The first channel and the second channel may be of different types. For example, the first channel may be an RFComm channel and the second channel may be an A2DP channel. According to some examples, any one or both of the auxiliary device and the mobile device may be configured to automatically determine the endpoint in the received voice and continue to listen to the subsequent voice immediately after the endpoint. In some examples, any one or both of the devices may also be configured to determine which of the first user or the second user is providing input. For example, the device may include a speaker recognition unit configured to determine at least one of the sound or the spoken language. In some examples, one or more processors of the mobile device are also configured to cross-reference the volume level of the input received through the auxiliary device microphone with the volume level of the input received through the mobile computing device microphone, such as to determine which of the first user or the second user is providing input. In other examples, one or more processors of the mobile device are also configured to cross-reference the waveform of the voice input received through the auxiliary device microphone with the waveform of the voice input received through the mobile computing device microphone.

[0008] Another aspect of the present disclosure provides an auxiliary device (such as earplugs), which includes: a microphone; a communication interface configured to wirelessly communicate with a computing device; an output source; and one or more processors in communication with the microphone, the communication interface, and the output source. The one or more processors can be configured to receive a first input from a first user through the microphone, detect an endpoint in the first input, receive a subsequent input from the first user after the endpoint, and send the first input to the computing device through the communication interface while receiving the subsequent input until the endpoint. According to some examples, the one or more processors can also be configured to receive a translated speech from a second user from the computing device, and output the translated speech through the output source. The communication interface can be configured to communicate with the computing device using two independent wireless communication channels including a first channel and a second channel having a different type from the first channel, the first channel being used to send information to the computing device, and the second channel being used to receive information from the computing device.

[0009] Another aspect of the present disclosure provides a mobile computing device (such as a mobile phone) comprising a microphone, a communication interface configured to wirelessly communicate with an electronic accessory, an output source, and one or more processors. The one or more processors may be configured to receive a first sound input through a microphone, detect an endpoint in the first sound input, translate the received first input until the endpoint, listen to subsequent inputs while translating, and send the translated first input to the electronic accessory for output. In some examples, the one or more processors are configured to distinguish a first sound input from a first user from a second sound input from a second user. For example, a speaker recognition unit may be configured to determine at least one of a sound or a spoken language. According to other examples, the one or more processors are also configured to receive a second input from an electronic accessory, and cross-reference the volume level of the second input with the volume level of the first input or cross-reference the waveform of the second input with the waveform of the first input.

[0010] Another aspect of the present disclosure provides a method for providing translation using an auxiliary device and a computing device wirelessly connected to the auxiliary device, the method comprising: continuously listening to a first input from a first user at a microphone of the auxiliary device, and sending the first input when the first input is received; while the auxiliary device microphone listens to the first input from the first user, continuously listening to a second input from a second user at a microphone of the computing device; receiving the first input sent from the auxiliary device at a communication interface of the computing device; processing the first input and the second input at the computing device to provide a translation of at least one of the first input and the second input; and outputting the translation of the at least one of the first input and the second input from the computing device. A non-transitory computer-readable storage medium comprising computer program instructions is also provided, which, when executed by a system comprising the auxiliary device and the computing device, causes the system to perform the steps of the method.

[0011] More specifically, a method is provided, including: receiving voice input via a mobile device for providing translation; determining that the voice input is from a foreign language speaker; in response to determining that the voice input is from a foreign language speaker, detecting an endpoint in the voice input; and in response to detecting the endpoint in the voice input: buffering the voice input until the endpoint; translating the buffered voice input until the endpoint into another language via the mobile device or via a remote computing device; and providing the translated voice input for output BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic diagram of an example system in use by two users according to aspects of the present disclosure.

[0013] Figure 2 yes Figure 1 Schematic diagram of an example system.

[0014] Figure 3 It is shown Figure 1 A block diagram of components of an example system.

[0015] Figure 4 is a flow chart illustrating an example method according to aspects of the present disclosure.

[0016] Figure 5 is a flow chart illustrating another example method according to aspects of the present disclosure. DETAILED DESCRIPTION

[0017] The present disclosure provides an improved translation experience between a first user and a second user using a mobile computing device and an auxiliary device such as a pair of earbuds. The first user may be, for example, a person who speaks a foreign language, and the second user may be the owner of the mobile computing device and the auxiliary device. The microphones on both the mobile device and the auxiliary device capture input from the first user and the second user simultaneously, respectively, rather than alternating between the mobile device and the auxiliary device. Each device may determine when to create an endpoint for a voice block or send a voice block for translation, for example, based on a pause in the voice. Each device may send the received voice to an endpoint for translation and output accordingly, so that it is provided in a natural communication flow. Listening by the microphone is automatically resumed immediately after the endpoint is created, so that the voice will not be lost.

[0018] Example System

[0019] Figure 1 is a schematic diagram of an example system in use. A first user 101 provides a voice input 110. The voice input 110 may be in a foreign language for a second user 102. The voice input 110 is received at a mobile computing device 170 of the second user 102. The second user 102 also provides a second voice input 120 in a second language. The second voice input 120 may be received by both the mobile device 170 and an auxiliary device 180 worn by the second user 102. According to some examples, the auxiliary device 180 may be intended to primarily receive the second voice input 120, while the mobile device 170 is intended to primarily receive the first voice input 110. However, even in this example, the mobile device 170 may pick up some of the second voice input 120 as a side effect, and may filter out the second voice input 120. For example, the mobile device 170 may determine which input is received from the first user 101 and which input is received from the second user 102, and translate the voice from the first user 101. The auxiliary device 180 may also determine that the received input is from the second user 102, and send the input to the mobile device for translation. Each of the mobile device 170 and the auxiliary device 180 remains listening during the entire communication between the first user 101 and the second user 102 and provides translated outputs that are relevant to each other from time to time.

[0020] In the example shown, auxiliary device 180 is a pair of wireless earbuds. However, it should be understood that auxiliary device 180 can be any of a variety of different types of auxiliary devices. For example, auxiliary device 180 can be a pair of wired earbuds, headphones, a head-mounted display, a smart watch, a mobile assistant, etc.

[0021] Mobile computing device 170 may be, for example, a mobile phone, a tablet computer, a laptop computer, a gaming system, or any other type of mobile computing device. In some examples, mobile computing device 170 may be coupled to a network such as a cellular network, a wireless Internet network, etc. Translation capabilities may be stored on mobile computing device 170 or accessed from a remote source by mobile computing device 170. For example, mobile device 170 may interface with a cloud computing environment in which speech translation from a first language to a second language is performed and provided back to mobile device 170.

[0022] Figure 2 is a schematic diagram of an example system including auxiliary device 180 and mobile device 170 , and illustrates signal exchanges between mobile device 170 and auxiliary device 180 .

[0023] In this example, auxiliary device 180 is shown as a pair of earplugs, which may include, for example, a speaker portion 187 adjacent to an inner ear engagement surface 188 and an input portion 185 adjacent to an outer surface. In some examples, a user may enter input by pressing input portion 185 while speaking, or by tapping input portion 185 before speaking. In other examples, no manual user input is required, and the user may simply begin speaking. The user's voice may be received by a microphone in an earplug (not shown) or in mobile device 170. The user may hear the translated speech from another person through speaker portion 187.

[0024] Auxiliary device 180 is wirelessly coupled to mobile device 170. The wireless connection between the devices may include, for example, a short-range pairing connection, such as Bluetooth. Other types of wireless connections are also possible.

[0025] In such Figure 2In some examples shown, different wireless protocols may be used for information transmission from earplug 180 to mobile device 170 compared to information transmission in the reverse path from mobile device 170 to earplug 180. Wireless pairing between mobile device and auxiliary device can usually use HFP / SCO channel, which allows lower quality two-way communication. This example allows replacing HFP / SCO channel with two concurrent channels. In some examples, concurrent channels can be of different types. The first channel 240 for transmitting information from mobile device 170 to auxiliary device 180 can be a high-quality one-way audio channel, such as A2DP. The second channel 250 for transmitting information from auxiliary device 180 to mobile device 170 can be, for example, an RFComm channel. This allows mobile device 170 to process two input audio streams at a time. In this regard, microphones on both mobile device 170 and auxiliary device 180 can be used concurrently, with audio being transmitted simultaneously in both directions. It should be understood that A2DP and RFComm are merely examples of different types of channels that can be used. In addition, in some examples, channels 240, 250 can be independent channels of the same type, each of which provides one-way communication.

[0026] Figure 3 An example block diagram of an auxiliary device 180 and a mobile device 170 is provided. Each device includes one or more processors 371, 381, memory 372, 382, ​​and other components typically found in mobile computing devices and auxiliary devices. Although multiple components are shown, it should be understood that such components are merely non-limiting examples and other components may be included additionally or alternatively.

[0027] As described above, auxiliary device 180 can be any of various types of devices, such as earbuds, head-mounted devices, smart watches, etc. Mobile device 170 can also take various forms, such as smartphones, tablets, laptops, game consoles, etc.

[0028] The one or more processors 371, 381 may be any conventional processor, such as a commercially available microprocessor. Alternatively, the one or more processors may be a dedicated device, such as an application specific integrated circuit (ASIC) or other hardware-based processor. Figure 3The processors, memories, and other elements of auxiliary device 180 and mobile computing device 170 are functionally shown as being within the same respective blocks, but one of ordinary skill in the art will appreciate that a processor or memory may actually include multiple processors or memories that may or may not be stored within the same physical housing. Similarly, a memory may be a hard drive or other storage medium located in a housing different from that of auxiliary device 180 or mobile computing device 170. Thus, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0029] Memory 382 may store information accessible to processor 381, including data 384 and instructions 383 executable by processor 381. Memory 382 may be a type of memory operable to store information accessible to processor 381, including non-transitory computer-readable media or other media storing data readable by electronic devices such as hard drives, memory cards, read-only memories ("ROM"), random access memories ("RAM"), optical disks, and other writable read-only memories. The subject matter disclosed herein may include different combinations of the foregoing, whereby different portions of instructions 383 and data 384 are stored on different types of media.

[0030] Data 384 can be retrieved, stored or modified by processor 381 according to instruction 383. For example, although the present disclosure is not limited to a specific data structure, data 384 can be stored in a computer register, in a relational database as a table with multiple different fields and records, an XML document or a flat file. Data 384 can also be formatted in a computer readable format such as, but not limited to, a binary value, ASCII or a Unicode. As a further example only, data 384 can be stored as a bitmap composed of pixels, which are stored in compressed or uncompressed or various image formats (e.g., JPEG), vector-based formats (e.g., SVG) or computer instructions for drawing graphics. In addition, data 384 can include information sufficient to identify relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories (including other network locations) or information used by functions to calculate relevant data.

[0031] Instructions 383 may be executed to facilitate translation performed by the mobile computing device. For example, instructions 383 allow listening and receiving a user's voice, such as through microphone 388. Microphone 388 may be beamformed so that it is directed to receive audio from the direction of the user's mouth. In this regard, auxiliary device 180 may recognize the received voice as the user's voice, rather than the voice of a foreign language speaker who is not wearing auxiliary device 180.

[0032] Instructions 383 may also allow for detecting endpoints in the received speech. For example, endpoints may be automatically determined based on any of pauses, key words, intonation, inflection, or a combination of these or other factors in the speech. Once an endpoint is detected, auxiliary device 180 may buffer the received speech while immediately resuming listening. In other examples, as an alternative or in addition to buffering, auxiliary device 180 may transmit the received speech to mobile device 170 for translation. For example, as described above in conjunction with Figure 2 As discussed, auxiliary device 180 can transmit voice via RFComm or other communication link. In some examples, all endpoint creation can be performed by mobile device 170. In such an example, auxiliary device 180 can transmit all audio streams to mobile device 170 which handles all endpoint creation, voice recognition and translation offline or online.

[0033] While the auxiliary device 180 is executing the instructions 383, the mobile device 170 may also be executing the instructions 373 stored in the memory 372 along with the data 374. For example, similar to the auxiliary device 180, the mobile device 170 may also include a memory 372 that stores data 374 and instructions 373 that may be executed by one or more processors 371. Similar to the memory 382 and data 384 of the auxiliary device 180, the memory 372 may be any of a variety of types and the data 374 may be any of a variety of formats. While the auxiliary device 180 is listening to and receiving speech from the user wearing the auxiliary device 180, the mobile device 170 may also be listening to and receiving speech through the microphone 378. The microphone 378 may not be beamformed and may receive speech from a foreign language speaker (e.g., Figure 1 ) and a user of the auxiliary device 180 (eg Figure 1 Thus, instructions 373 may allow for determining which user is providing voice input so that some voice input (e.g., from a foreign language speaker) may be translated for output via auxiliary device 180 and other voice input (e.g., from a user) may be filtered.

[0034] Any of a variety of voice recognition techniques can be used. As an example, the mobile device 170 can cross-reference the volume level between the auxiliary device microphone 388 and the mobile device microphone 378. If the sound received by the microphone 388 is quiet and the sound received by the microphone 378 is loud, it can be determined that the foreign speaker is providing voice input. On the contrary, if the sound received by the two microphones 388, 378 is loud, it can be determined that the owner / user is speaking. As another example technology, a voice recognition unit can be used. The voice recognition unit can be trained to recognize the voice of the user / owner of the auxiliary device 180 and the mobile device 170. Therefore, if the voice recognition unit detects the voice of the owner, it can ignore the voice. Similarly, the voice recognition unit can be trained to detect the language mainly spoken by the owner / user, and the voice detected in the language can be filtered out. As another example technology, audio echo cancellation technology can be used. For example, the mobile device 170 can listen to the two microphones 388, 378, detect overlapping audio, and recognize that the overlapping audio belongs to the owner. Overlapping audio may be detected by identifying similar waveforms or patterns of acoustic inputs, or detecting similar plosives or transient attacks.In some examples, any combination of the foregoing techniques or other techniques may be used.

[0035] When the detected speech is from a foreign language speaker, instruction 373 may further allow continued listening until an endpoint is detected. As described above, an endpoint may be detected based on pauses, keywords, intonation, or other factors. The received speech from the foreign language speaker is buffered, and microphone 378 resumes listening.

[0036] The mobile device 170 can perform translation of both the foreign language speaker input received by the microphone 378 and the owner input received by the communication from the auxiliary device 180. Such translation can be performed on the mobile device 170 itself, or can be performed using one or more remote computing devices (such as the cloud). For example, the mobile device 170 can upload the voice for translation to the remote computing network that performs the translation, and receive the response including the translated voice from the remote computing network. The translation of the voice from the foreign language speaker can be provided to the auxiliary device 180 to be output through the output 387. The translation from the owner's voice can be output through the output 377 of the mobile device 170. The outputs 377, 387 can each include, for example, one or more speakers adapted to provide audio output. In some examples, the outputs 377, 387 can also include one or more other types, such as a display, tactile feedback, etc.

[0037] It should be understood that the auxiliary device 180 and the mobile device 170 may each include other components not shown, such as charging inputs for batteries, signal processing components, etc. Such components may also be used in the execution of the instructions 383 , 373 .

[0038] Example Method

[0039] In addition to the operations described above and shown in the accompanying drawings, various operations will now be described. It should be understood that the following operations need not be performed in the exact order described below. Instead, the various steps may be processed in a different order or simultaneously, and steps may also be added or omitted.

[0040] Figure 4 The method 400 executed by the mobile device 170 is shown. In block 410, the mobile device listens for and receives voice input. Although the mobile device is intended to capture speech such as a foreign language speaker (e.g. Figure 1 101), but it can also capture the voice of the primary user (e.g. Figure 1 The device can continuously listen and receive input without manual interaction such as pressing a button.

[0041] In block 420, the mobile device determines whether the received sound input is from the mobile device owner or from a foreign language speaker. For example, the mobile device can use voice recognition, language recognition, etc. As another example, the mobile device can cross-reference the volume of the sound received at the mobile device with the volume of the sound received at the auxiliary device and relayed to the mobile device.

[0042] If it is determined in block 430 that the input is from the mobile device owner, the mobile device can ignore the input. Therefore, the method returns to block 410 to continue listening for input from a foreign language speaker. However, if it is determined that the input is from a foreign language speaker, the method proceeds to block 440.

[0043] In block 440, the mobile device determines whether an endpoint in the speech is detected. For example, if there is a pause in the speech input within a predetermined time period such as half a second, one second, two seconds, etc., the mobile device can determine that the speech endpoint has been reached. Other examples of detecting endpoints can include detecting changes in tone or intonation, or detecting keywords. Detecting endpoints helps to ensure the correct translation of complete phrases. For example, the translation of each word is often inaccurate. As an example only, while adjectives are usually said before their related nouns in English, in Spanish adjectives are said after the noun. By creating an endpoint for the speech after the complete statement, the phrase can be translated as a whole, and the words can be rearranged to be suitable for the language translated.

[0044] If no endpoint is detected, the mobile device keeps listening and waiting for the endpoint in block 445. However, if an endpoint is detected, the mobile device buffers the received input in block 450. It should be understood that buffering can be performed while receiving input before the endpoint is detected.

[0045] In block 460, the mobile device translates the buffered voice input to the endpoint. The translation may be performed at the mobile device or by a remote computing device.

[0046] In block 470, the translated input is provided to the auxiliary device for output. Figure 2 The translated input is provided via one of the two communication channels described in .

[0047] Although not shown, the mobile device also receives speech from the auxiliary device. For example, the auxiliary device may be combined with the following Figure 5 The device may receive speech from the owner and provide the received speech to the mobile device for translation. Once the translation is complete, the mobile device may output the translated speech for the foreign language speaker. For example, the mobile device may transmit an audible translation of the speech through a speaker for the foreign language speaker to hear, may provide a text translation of the speech on a display, and the like.

[0048] Figure 5 A method 500 is shown for execution by a secondary device to receive a translation. The method 500 may be executed by the secondary device concurrently with the mobile device executing the method 400.

[0049] In block 510, the auxiliary device listens and receives voice input from the owner. In the case where the microphone in the auxiliary device is beamformed, it can be determined that any input received is from the owner. In other examples, voice recognition technology such as the voice recognition technology described above can be used.

[0050] In block 520, the auxiliary device determines whether an endpoint has been reached. The endpoint can be automatically detected based on a pattern in the voice. In other examples, the endpoint can be manually entered by the owner, such as by pressing a button. If the endpoint is not detected, the device continues to listen in block 525 until the endpoint is detected. During and / or after reception, the received voice can be buffered at the auxiliary device. In some alternatives, instead of the auxiliary device performing endpoint creation, the auxiliary device continuously streams the received audio to the mobile device. The mobile device can run two sound recognizers simultaneously to detect sounds and endpoints accordingly.

[0051] In block 530, the received speech from the owner is transmitted to the mobile device for translation and output. Figure 2As discussed, speech may be sent via the second communication channel. The speech may be sent simultaneously with speech received from the mobile device (block 540) for output by the auxiliary device (block 550).

[0052] By keeping the microphones on both the auxiliary device and the mobile device continuously open, speech from both the owner and the foreign speaker is continuously received. In this regard, the owner and the foreign speaker can have a more natural conversation including interjections, affirmations, acknowledgments, etc., and there are no awkward pauses while waiting for translation. Through automatic endpoint creation, phrases or other speech blocks can be detected and translated without manual input from the user. In addition, sound recognition technology can be used to determine which user is providing input, and therefore determine how the input should be translated. Therefore, less user manipulation of the device is required. Instead, the user can talk naturally, and the auxiliary device and the mobile device can automatically provide assistance, allowing near real-time translation.

[0053] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above may be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be considered as an illustration rather than a limitation of the subject matter defined by the claims. In addition, the provision of the examples described herein and the expressions "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples; rather, these examples are intended to illustrate only one of many possible embodiments. In addition, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A method comprising: receiving voice input via a mobile device for providing translation; Determine that the sound input comes from a foreign language speaker; In response to determining that the voice input is from a foreign language speaker, detecting endpoints in the voice input; as well as In response to detecting an endpoint in sound input: Buffer sound input until the endpoint; translating the buffered sound input into another language through the mobile device or through a remote computing device until the endpoint; and Provides translated sound input for output.

2. The method according to claim 1, wherein: Detecting endpoints in the audio input is based on pauses in the audio input.

3. The method according to claim 1, wherein: Detecting endpoints in the voice input is based on patterns in the voice input or based on at least one of a keyword, tone, or intonation of the voice input.

4. The method according to any one of claims 1 to 3, wherein: Determining that the sound input is from a foreign language speaker is based on a volume level of the sound input or a waveform of the sound input.

5. The method according to claim 4, wherein: Determining that the sound input is from a foreign language speaker includes cross-referencing a volume level of another sound input received through a microphone of an auxiliary device wirelessly coupled to the mobile device with a volume level of the sound input received through the microphone of the mobile device.

6. The method according to claim 4, wherein: Determining that the sound input is from a foreign language speaker includes cross-referencing a waveform of another sound input received through a microphone of an auxiliary device wirelessly coupled to the mobile device with a waveform of the sound input received through the microphone of the mobile device. 7 . The method of claim 1 , wherein determining that the voice input is from a foreign language speaker comprises determining that the voice input is not from a user of the mobile device but is from a foreign language speaker.

8. The method of claim 1 , wherein the mobile device is wirelessly coupled to the auxiliary device, and the method further comprises: listening at a microphone of the secondary device for a first voice input from a user of the mobile device; as well as While listening to a first voice input from a user of the mobile device at the microphone of the auxiliary device, a second voice input from a foreign language speaker is listened to at the microphone of the mobile device.

9. The method of claim 8, wherein the auxiliary device and the mobile device are wirelessly coupled using two independent wireless communication channels, including a first channel for transmissions from the auxiliary device to the mobile device and a second channel for transmissions from the mobile device to the auxiliary device.

10. The method of claim 9, wherein the first channel and the second channel are of different types.

11. The method of claim 1 , further comprising sending the translated voice input to an auxiliary device for output.

12. The method according to any one of claims 8 to 11, wherein: A mobile device is a mobile phone, tablet, laptop, or gaming system; and The secondary device is a pair of wired or wireless earbuds, headphones, a head-mounted display, or a smartwatch.

13. A mobile device, comprising: A memory for storing instructions; as well as A processor is communicatively coupled to the memory and is configured to execute instructions to perform a process including: Receive sound input; Determine that the sound input comes from a foreign language speaker; In response to determining that the voice input is from a foreign language speaker, detecting endpoints in the voice input; as well as In response to detecting an endpoint in sound input: Buffer sound input until the endpoint; Translate the buffered audio input to another language at the endpoint; and Provides translated sound input for output.

14. The mobile device according to claim 13, wherein: Detecting endpoints in the audio input is based on pauses in the audio input.

15. The mobile device according to claim 13, wherein: Detecting endpoints in the voice input is based on patterns in the voice input or based on at least one of a keyword, tone, or intonation of the voice input.

16. The mobile device according to any one of claims 13 to 15, wherein: Determining that the sound input is from a foreign language speaker is based on a volume level of the sound input or a waveform of the sound input.

17. The mobile device according to claim 16, wherein: Determining that the sound input is from a foreign language speaker includes cross-referencing a volume level of another sound input received through a microphone of an auxiliary device wirelessly coupled to the mobile device with a volume level of the sound input received through the microphone of the mobile device.

18. The mobile device according to claim 16, wherein: Determining that the sound input is from a foreign language speaker includes cross-referencing a waveform of another sound input received through a microphone of an auxiliary device wirelessly coupled to the mobile device with a waveform of the sound input received through the microphone of the mobile device.

19. A non-transitory computer-readable medium storing instructions that, when executed, cause a processor of a mobile device to perform a process comprising: Receive sound input; Determine that the sound input comes from a foreign language speaker; In response to determining that the voice input is from a foreign language speaker, detecting endpoints in the voice input; as well as In response to detecting an endpoint in sound input: Buffer sound input until the endpoint; Translate the buffered audio input to another language at the endpoint; and Provides translated sound input for output.

20. The non-transitory computer readable medium of claim 19, wherein: Detecting endpoints in the audio input is based on pauses in the audio input.

Images