Low latency neighboring group translation

Abstract

A local wireless network is used to establish a multi-language translation group between users speaking more than two different languages such that when one user speaks to his or her computing device, the user's computing device can perform automatic speech recognition, in some cases, translation to different languages, to generate non-audio data, and in some cases translate to different languages. And transmitting to computing devices of other users in the group. The computing devices of other users then use the received non-audio data to generate spoken audio outputs suitable for their respective users. Generating spoken audio output on computing devices of other users may also include performing a translation to a respective user's desired language to enable each user to receive spoken audio output in its desired language in response to voice input from another user irrespective of the original language of the voice input.

Description

[0001] This application is a divisional application of the PCT invention patent application with application date of April 11, 2018, application number 201880094315.3, and invention name “Low-latency neighboring group translation”. Technical Field

[0002] The present disclosure relates to group translation, and in particular, to methods, apparatus, and computer-readable media for establishing a multi-language translation group among three or more users who speak different languages ​​using local wireless networking. Background Art

[0003] Real-time translation is often used to enable people who speak different languages ​​to participate in a conversation. In this regard, real-time translation refers to a translation that is performed substantially simultaneously with the conversation and that is intended to minimize the delay between the time when a person speaks a language and the time when a text or oral representation of the language is provided to another user. In many cases, translation between languages ​​is performed by an online service that receives an audio stream of the speaker's voice from the listener's device (e.g., captured by the microphone of the listener's mobile device), performs a translation to the language desired by the listener, and then returns an audio stream of the translated speech, or alternatively, returns a text representation of the translated speech to playback and / or display on the listener's device. However, it has been found that in many cases where online connectivity is limited or nonexistent, particularly when capturing and playback audio on a mobile device, translation may be required.

[0004] Furthermore, in many cases, the user is required to explicitly specify the "from" and "to" languages ​​for performing the translation. However, there are many occasions where a group of people who speak more than two languages ​​attempt to converse with one another, which can lead to additional difficulties associated with selecting and switching between the correct languages ​​in which to perform the translation. As an example, when a tour guide leads a group of people from many different countries, even if the guide's native language is specifically selected in which to perform the translation, others in the group may ask questions in a variety of different languages ​​that that person cannot understand. Summary of the invention

[0005] The present disclosure generally relates to a method, apparatus and computer-readable medium (temporary and non-temporary) for establishing a multilingual translation group between three or more users who speak different languages ​​using local wireless networking, so that when a user speaks to his corresponding computing device (e.g., mobile device), the computing device of the user can perform automatic speech recognition, and in some cases, perform translation to different languages ​​to generate non-audio data (e.g., text data) of the computing devices of other users that can then be transmitted to the computing devices of other users in the multilingual translation group. The computing devices of other users can then use the non-audio data of the computing devices transmitted to other users to generate oral audio output suitable for their corresponding users. Generating oral audio output on the computing devices of other users can also include performing translation to the desired language of the corresponding user, so that each user can receive oral audio output in the language desired in response to the voice input from another user, regardless of the original language of the voice input.

[0006] Therefore, consistent with some embodiments, the group translation method may include: in a first computing device of the first user and configured to output spoken audio in a first language, joining a multilingual translation group additionally including a second user and a third user by establishing a local wireless network connection to one or both of a second computing device of the second user and a third computing device of the third user, wherein the second computing device and the third computing device are respectively configured to output spoken audio in a second language and a third language, and wherein the first language, the second language, and the third language are different from each other; and in the first computing device, and in response to voice input in a second language received at the second computing device and for the multilingual translation group: receiving non-audio data associated with the voice input; and generating a spoken audio output in the first language and for the first user from the non-audio data associated with the voice input.

[0007] Some embodiments may further include: in a third computing device, and in response to a voice input in a second language received at a second computing device and directed to a multilingual translation group: receiving non-audio data associated with the voice input; and generating a second spoken audio output in a third language and directed to a third user from the non-audio data associated with the voice input. Some embodiments may also include: in a second computing device: receiving voice input; performing automatic voice recognition on the voice input to generate non-audio data associated with the voice input; and sending the non-audio data associated with the voice input to the first computing device and the third computing device through the multilingual translation group. In addition, some embodiments may also include: in a second computing device: receiving voice input; performing automatic voice recognition on the voice input, and translating the output of the automatic voice recognition from the second language locally into a different language to generate non-audio data associated with the voice input; and sending the non-audio data associated with the voice input to the first computing device and the third computing device through the multilingual translation group.

[0008] In addition, in some embodiments, generating a spoken audio output in the first computing device includes locally translating non-audio data associated with the voice input into the first language. Some embodiments may also include: in the first computing device: receiving a second voice input from the first user; performing automatic speech recognition on the second voice input to generate second non-audio data associated with the second voice input; and sending the second non-audio data associated with the second voice input to the second computing device and the third computing device through a multilingual translation group. Some implementations may further include, in the first computing device: receiving a second voice input from the first user; performing automatic speech recognition on the second voice input, and translating the output of the automatic speech recognition from the first language locally into a different language to generate second non-audio data associated with the second voice input; and sending the second non-audio data associated with the second voice input to the second computing device and the third computing device through a multilingual translation group.

[0009] In addition, in some embodiments, the non-audio data associated with the voice input includes text data associated with the voice input. In addition, in some embodiments, the non-audio data associated with the voice input includes one or more feature vectors associated with the voice input. In addition, in some embodiments, the local wireless network connection includes a Bluetooth connection, a personal area network connection, a wireless mesh network connection, a near field connection (NFC) or a wireless local area network (WLAN) connection. In addition, in some embodiments, the local wireless connection uses a first wireless network, wherein joining the multilingual translation group is performed through the first wireless network and receiving the non-audio data associated with the voice input is performed through the second wireless network.

[0010] Some embodiments may also include, in the first computing device, communicating non-audio data associated with the voice input and received from the second computing device to a third computing device. Additionally, in some embodiments, the non-audio data associated with the voice input is received in a broadcast message generated by the second computing device. Additionally, in some embodiments, the non-audio data associated with the voice input is received in a point-to-point message generated by the second computing device and addressed to the first computing device.

[0011] In addition, some embodiments may also include initiating a multilingual translation group by the following steps: broadcasting an announcement message from one of the first computing device and the second computing device; and receiving a response to the announcement message from the other of the first computing device and the second computing device at the one of the first computing device and the second computing device. In some embodiments, the first computing device includes a mobile device, and wherein joining the multilingual translation group, receiving non-audio data associated with the voice input, and generating oral audio output are performed by an app residing on the mobile device. In addition, in some embodiments, the first computing device also includes a wireless headset that wirelessly communicates with the mobile device, and the method also includes: playing back the oral audio output through the wireless headset. In some embodiments, the app generates a chat room display on a display of the mobile device, and the method also includes: displaying a text representation of the oral audio output in the first language in the chat room display.

[0012] Additionally, in some embodiments, the voice input is a pre-stored message selected by the second user. Additionally, in some embodiments, the voice input is a message generated by a virtual assistant.

[0013] Consistent with some embodiments, a group translation method may include: in a first computing device of a first user and configured to output oral audio in a first language, joining a multilingual translation group additionally including a second user and a third user by establishing a local wireless network connection to one or both of a second computing device of a second user and a third computing device of a third user, wherein the second computing device and the third computing device are respectively configured to output oral audio in a second language and a third language, and wherein the first language, the second language, and the third language are different from each other; and in the first computing device: receiving voice input in a first language from the first user; performing automatic speech recognition on the voice input to generate speech recognition data; using the speech recognition data to generate non-audio data associated with the voice input; and sending the non-audio data associated with the voice input to the second computing device and the third computing device through the multilingual translation group, so that the second computing device and the third computing device generate corresponding oral audio outputs in the second language and the third language, respectively, and for the second user and the third user, respectively.

[0014] In some embodiments, generating non-audio data associated with the voice input includes locally translating the voice recognition data into a language different from the first language. In addition, in some embodiments, generating non-audio data associated with the voice input includes locally translating the voice recognition data into each of a second and a third language. In addition, in some embodiments, the non-audio data associated with the voice input includes text data associated with the voice input or one or more feature vectors associated with the voice input. In addition, in some embodiments, sending the non-audio data associated with the voice input includes sending a broadcast message to the second computing device and the third computing device.

[0015] In some embodiments, sending non-audio data associated with the voice input includes sending separate messages addressed to the second computing device and the third computing device, respectively. Some embodiments may also include initiating a multilingual translation group by the following steps: broadcasting an announcement message from one of the first computing device and the second computing device; and receiving a response to the announcement message from the other of the first computing device and the second computing device at the one of the first computing device and the second computing device. In some embodiments, the first computing device includes a mobile device, and wherein joining the multilingual translation group, performing automatic speech recognition on the voice input, and generating non-audio data associated with the voice input are performed by an app residing on the mobile device. In addition, in some embodiments, the app generates a chat room display on a display of the mobile device, and the method also includes: displaying a text representation of the spoken audio output in the first language in the chat room display.

[0016] Consistent with some embodiments, a first computing device may be configured to output oral audio in a first language to a first user, and include one or more processors and a memory operably connected to the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to: join a multilingual translation group that additionally includes a second user and a third user by establishing a local wireless network connection to one or both of a second computing device of the second user and a third computing device of the third user, wherein the second computing device and the third computing device are configured to output oral audio in a second language and a third language, respectively, and wherein the first language, the second language, and the third language are different from each other; and in response to voice input in a second language received at the second computing device and directed to the multilingual translation group: receive non-audio data associated with the voice input; and generate oral audio output in the first language and directed to the first user from the non-audio data associated with the voice input.

[0017] In some embodiments, the first computing device includes a mobile device, and wherein the instructions define a plurality of software components, the plurality of software components including: a local translation component configured to generate text in a first language from non-audio data associated with a speech input; and a text-to-speech component configured to generate a spoken audio output from the text generated by the local translation component. In addition, in some embodiments, the plurality of software components further include: a multilingual group translation app that communicates with each of the local translation component and the text-to-speech component; an automatic speech recognition component that communicates with the multilingual group translation app and is configured to recognize the spoken input; and a chat room component that communicates with the multilingual group translation app and is configured to generate a chat room display on a display of the mobile device and display a text representation of the spoken audio output in the chat room display in the first language. In addition, in some embodiments, the plurality of software components further include a networking component that is configured to establish and maintain a wireless network connection with a second computing device.

[0018] Consistent with some embodiments, a first computing device may be configured to output spoken audio in a first language to a first user and include one or more processors and a memory operably connected to the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to: join a multilingual translation group that additionally includes a second user and a third user by establishing a local wireless network connection to one or both of a second computing device of the second user and a third computing device of the third user, wherein the second computing device and the third computing device are configured to output spoken audio in a second language and a third language, respectively, and wherein the first language, the second language, and the third language are different from each other; receive speech input in the first language from the first user; perform automatic speech recognition on the speech input to generate speech recognition data; generate non-audio data associated with the speech input using the speech recognition data; and send the non-audio data associated with the speech input to the second computing device and the third computing device through the multilingual translation group so that the second computing device and the third computing device generate corresponding spoken audio output in the second language and the third language, respectively, and for the second user and the third user, respectively.

[0019] Consistent with some embodiments, at least one non-transitory computer-readable medium includes instructions that, in response to execution of the instructions by one or more processors of a first computing device configured to output spoken audio in a first language to a first user, cause the one or more processors to: join a multilingual translation group that additionally includes a second user and a third user by establishing a local wireless network connection to one or both of a second computing device of the second user and a third computing device of the third user, wherein the second computing device and the third computing device are configured to output spoken audio in a second language and a third language, respectively, and wherein the first language, the second language, and the third language are different from each other; and in response to voice input in a second language received at the second computing device and directed to the multilingual translation group: receive non-audio data associated with the voice input; and generate spoken audio output in the first language and directed to the first user from the non-audio data associated with the voice input.

[0020] Consistent with some embodiments, at least one non-transitory computer-readable medium includes instructions that, in response to execution of the instructions by one or more processors of a first computing device configured to output spoken audio in a first language to a first user, cause the one or more processors to: join a multilingual translation group that additionally includes a second user and a third user by establishing a local wireless network connection to one or both of a second computing device of the second user and a third computing device of the third user, wherein the second computing device and the third computing device are configured to output spoken audio in a second language and a third language, respectively, and wherein the first language, the second language, and the third language are different from each other; receive speech input in the first language from the first user; perform automatic speech recognition on the speech input to generate speech recognition data; generate non-audio data associated with the speech input using the speech recognition data; and send the non-audio data associated with the speech input to the second computing device and the third computing device through the multilingual translation group so that the second computing device and the third computing device generate corresponding spoken audio output in the second language and the third language, respectively, and for the second user and the third user, respectively.

[0021] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by a processor to perform a method such as one or more of the methods described above. Yet another embodiment may include a system comprising a memory and one or more processors operable to execute instructions stored in the memory to implement one or more modules or engines that individually or collectively perform a method such as one or more of the methods described above.

[0022] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 An environment is shown in which low-latency proximity group translation may be utilized in accordance with various embodiments.

[0024] Figure 2A and 2B A flow chart illustrating an example method in accordance with various implementations is depicted.

[0025] Figure 3A , 3B , 3C, and 3D depict examples of how group member devices may present a graphical user interface according to various embodiments.

[0026] Figure 4 One example of a translation group data structure is depicted in accordance with various implementations.

[0027] Figure 5 A flow chart illustrating another example method in accordance with various implementations is depicted.

[0028] Figure 6 A flow chart illustrating yet another example method in accordance with various implementations is depicted.

[0029] Figure 7 An example architecture of a computing device is schematically depicted. DETAILED DESCRIPTION

[0030] In various embodiments, low-delay neighboring group translation is performed to promote the communication between users who speak three or more languages ​​using the associated computing devices of different users. In various embodiments, low-delay neighboring group translation can set up a multilingual translation group between three or more users, and can support the communication of non-audio data (e.g., text data, features or nested vectors, etc.) by wireless network between the computing devices of the users of the group, to provide low-delay communication between user computing devices. The computing device can then perform local translation to or from the corresponding language of the user, and local automatic speech recognition and text-to-speech conversion to convert between audio and non-audio data, thereby minimizing the dependence on the online connectivity of the computing device, so that when online connectivity is limited or does not exist, low-delay neighboring group translation can be utilized. Other variations and modifications will be understood from the discussion below.

[0031] Figure 1An example environment 10 is shown in which, in some implementations, low latency proximity group translation may be performed. The example environment 10 includes a plurality of computing devices 12 connected to one another via a local wireless network 14. Each computing device 12 is associated with a particular user (e.g., user A, B, and C), and may include various types of computers and other programmable electronic devices, including mobile devices, smart phones, tablet computers, wearable computers, laptop computers, desktop computers, server computers, controllers, set-top boxes, dedicated computing devices, and virtually any other type of device that includes one or more processors for executing instructions.

[0032] In various embodiments, the computing device 12 may be portable or may be installed or maintained in a fixed location. For example, when used as an announcement system in a public area such as an airplane, train, bus, airport, train or bus station, or when used for a sports or gaming event or for a conference, lecture or presentation, the computing device used by the primary user (e.g., announcer) may not be portable. For example, in a vehicle such as an airplane, train or bus, the computing device may be hardwired and integrated into the vehicle's electronic system or docked with the vehicle's electronic system. Otherwise, in some embodiments, all computing devices 12 may be portable in nature. In various embodiments, the computing device 12 and its user are located in the same approximate proximity, for example, in the same approximate physical location, and within the range of one or more local wireless networks to which the computing device is connected.

[0033] The local wireless network 14 to which each computing device 12 is wirelessly connected may represent various types of short-range wireless networks, including, for example, a Bluetooth network, a personal area network (PAN), a wireless mesh network, a near field communication (NFC) network, or a wireless local area network (WLAN or Wi-Fi), among others, wherein connectivity to the network is limited to computing devices that are relatively proximal to one another.

[0034] Each computing device 12 executes an operating system 16 and various applications, such as a group translation application (or app) 18, in which various operations associated with low-latency proximity group translation can be performed. The application 18 can additionally include various software components resident therein, including, for example, a networking component 20, a chat room or chat room component 22, a translation component 24, a text-to-speech (TTS) component 26, and an automatic speech recognition (ASR) component 28. In some embodiments, each component 20-28 can be implemented separately from the application 18, such as within the operating system 16 or in a separate framework, so the present invention is not limited to the specific architecture described herein.

[0035] The networking component 20 may be used, for example, to establish a network connection with one or more other computing devices 12 via the local wireless network 14 and to manage communications via the local wireless network 14. As will become more apparent below, in some embodiments, the networking component 20 may, among other operations, initiate or respond to connection requests or advertisements, maintain network topology information, and send, receive, or confirm network communications via the local wireless network 14. In the illustrated embodiment, the networking component 20 may also be dedicated to communications via one or more specific network protocols, such as Bluetooth Low Energy (BLE).

[0036] The chat component 22 may be used to create and maintain a "chat room" type user interface for the computing device 12, presenting, for example, transcriptions of text communications within a multi-language translation group and identifying the user or speaker of each communication. Various operations may be supported, such as initiating a communication with the group, initiating a private communication with selected members of the group, recording audio, playing audio, etc. In some implementations, the component 22 may also support other operations utilized in other types of chat rooms (e.g., SMS or text messaging chat rooms, instant messaging (IM) chat rooms, etc.).

[0037] The translation component 24 may be used to perform language translation between two or more languages. Although in some implementations, the translation component 24 may access an online service to perform some or all translations, in the illustrated implementation, the translation component 24 is a local translation component that performs translations locally on the computing device 12. In this regard, in some implementations, one or more language models 30 may reside locally on each computing device 12, and it will be appreciated that in the illustrated implementation, translations may be performed regardless of any online connectivity of the computing device.

[0038] Furthermore, in the illustrated embodiment, communications between computing devices are all performed in a single common language (e.g., English in some embodiments), whereby each translation component 24 is used to perform translations between the single common language and the desired or selected language of the user of the computing device 12. In other words, each computing device 12 may be configured with a primary language by its associated user, and the translation component 24 may be used to translate between the primary language and the common language used by the group. In some embodiments, the primary language of the computing device may be the same as the single common language used by the group, whereby the translation component 24 may not be used or may be omitted entirely. In some embodiments, the translation component 24 and the associated language model 30 may also be limited to translating between two languages, while in other embodiments, the translation component 24 may be coupled with multiple language models 30 to support translations between more than two languages.

[0039] The text-to-speech (TTS) component 26 may be used to convert non-audio data (e.g., text data or features / embedded vectors) into an audio representation of a spoken utterance or its translation. Conversely, the automatic speech recognition (ASR) component 28 may be used to convert an audio representation of a spoken utterance into non-audio data. In some embodiments, the ASR component 28 may generate text corresponding to the spoken utterance, while in other embodiments, the ASR component 28 may generate another form of non-audio data, such as one or more features / embedded vectors describing phonemes, words, or phrases in the spoken utterance.

[0040] It will be appreciated that in some embodiments, any of the components 20-28 may be combined with one another, or alternatively, the functionality of any of the components 20-28 may be split into multiple components. Additionally, as described above, in the illustrated embodiment, none of the components 20-28 rely on online connectivity. However, in other embodiments, for example, the local wireless network 14 or another network (e.g., Figure 1 Online connectivity supported by the Internet 32 ​​as shown can be used to provide access to remote information or operations, such as through one or more cloud services 34. For example, in some embodiments, operations such as translation, speech recognition, and / or text-to-speech conversion can be provided by the cloud service 34.

[0041] In addition, while in some embodiments each computing device 12 may provide integrated audio input and output capabilities (e.g., via an integrated microphone and speaker), in other embodiments, an external audio device may be used to provide audio input and / or output functionality. In some embodiments, for example, the computing device 12 may be connected to a headset 36 including an integrated microphone. Other external devices, including wired or wireless devices such as headsets, headphones, earphones, earplugs, microphones, speakers, etc., may also be used in other embodiments. In some embodiments, for example, wireless headsets, headphones, earplugs, or earphones (collectively referred to herein as wireless headsets) may be used. In addition, the external audio device may also transmit commands for use by the computing device 12, such as via a personal assistant (voice) interface, a button or other hard user input, a touch or gesture sensor, etc.

[0042] Other variations will be apparent to those having the benefit of this disclosure, and therefore, the invention is not limited to the particular environments discussed herein.

[0043] Now go to Figure 2A-2B , showing the Figure 1An example routine 40 for implementing low-latency proximity group translation in an environment of FIG. The routine 40 illustrates operations performed by computing devices of three users (users A, B, and C). Furthermore, for the purpose of placing the routine in context, an example tour guide application is discussed, wherein user A is Margaux, who speaks French, user B is Jim, a tour participant who speaks English, and user C is Diego, a tour participant who speaks Spanish. It should be understood that the routine 40 can be used for other applications, and thus the present invention is not limited to the specific applications discussed below.

[0044] Routine 40 begins in box 42 by user A requesting to create a multilingual translation group on the user's computing device, resulting in the creation of the group by user A's computing device. In this regard, a multilingual translation group can be considered a logical entity that associates at least one group of user computing devices with each other for the purpose of group communication between users who speak three or more different languages, so that communications sent by one user in one language can be transmitted to the computing devices of other users in the group and translated into multiple different languages ​​for these different users. Therefore, in some embodiments, communication through a multilingual translation group can include data communication between computing devices associated with the multilingual translation group using information stored for the multilingual translation group to help route communications between computing devices associated with the group.

[0045] Next, in block 44, the user A computing device broadcasts an announcement message, which is received by each of the user B and user C computing devices, as shown in blocks 46 and 48. For example, the broadcast of the announcement may be, for example, a Bluetooth LE pairing request, wherein the computing devices interact over the Bluetooth network, but it is understood that other forms of announcements may be used for other network protocols. It should also be understood that other computing devices may initiate announcement messages, and in some embodiments, announcements may not be used.

[0046] Next, as shown in frame 50-60, user A computing device adds each of user B and user C to the multilingual translation group.It should be understood that the order of adding users is arbitrary, and additional users can also be added to the multilingual translation group in a similar manner.For user B, user A and user B computing devices negotiate network connection between each other in frames 50 and 52, and user A computing device adds user B to the multilingual translation group in frame 54.For user C, user A and user C computing devices negotiate network connection between each other in frames 56 and 58, and user A computing device adds user C to the multilingual translation group in frame 60.The negotiation network connection in frames 50-52 and 56-58 can be performed in various ways suitable for the specific local wireless network utilized, for example, using the Bluetooth negotiation protocol for Bluetooth network, and it should be understood that various information can be exchanged during such negotiation, such as user name, network information (such as IP address, Bluetooth address, MAC address, etc.), and optionally, the user's native language, selected language or desired language. Adding the user to the multi-language translation group in blocks 54 and 60 may include adding an entry to a data structure for the multi-language translation group (e.g., as described below in conjunction with Figure 4 discussed), so that future communications can be directed to all members of the group.

[0047] Blocks 62-92 then represent processing of a group communication from user A to a multi-language translation group including at least users B and C. It will be assumed for purposes of this example that English is the common language used to communicate non-audio spoken utterances between users, but the invention is not so limited.

[0048] In box 62, the user A computing device receives voice input (e.g., spoken utterance) from user A, such as captured by a microphone from the user A computing device, or receives the voice input from a headset, earbuds, headphone, earphone, etc. that communicates with the user A computing device. In box 64, the user A computing device performs automatic speech recognition on the voice input, for example, in the user A language. In box 66, the user A computing device then determines whether the user A language is a common language (e.g., English), and if not, passes control to box 68 to perform translation from the user A language to the common language, and to box 70 to generate a set of non-audio data in the common language and representing the voice input. However, if the user A language is a common language, the user A computing device bypasses box 68 in box 66 and passes control to box 70 to generate a set of non-audio data in the common language.

[0049] For example, in some embodiments, non-audio data can be formatted as text, while in other embodiments, other formats can be used, for example, machine learning components can be used to perform nested or feature vectors of any or all speech recognition, language translation, and text-to-speech conversion. It should also be understood that various data formats can be used to pass various representations of speech input through automatic speech recognition and language translation. In one embodiment, for example, automatic speech recognition can generate text in the language of user A, and language translation can convert the generated text into translated text in a common language.

[0050] Now go to Figure 2B , control passes to block 72, where the user A computing device broadcasts a message containing non-audio data to each of the user B and user C computing devices. The user A computing device may then display a text representation of the speech input in the user A language in the chat room displayed on the user A computing device (block 76), and the user A computing device may then wait in block 78 for user A to speak a new utterance or for a message to be received representing a new utterance spoken by another member of the group, and process the utterance in the manner discussed herein.

[0051] In each of the user B and user C computing devices, the message broadcast by the user A computing device in box 72 is processed in the manner shown in boxes 80-92. In box 80, the message from the user A computing device is received, and in box 82, the user B or user C computing device determines whether the selected language of the user of the computing device is a common language. If not, the user B or user C computing device passes control to box 84 in box 82 to convert the non-audio data in the message from the common language to the user's selected language, and passes it to box 86 to generate a spoken audio output in the user's selected language (e.g., using a text-to-speech component). However, if the two languages ​​are the same, the user B or user C computing device bypasses box 84 in box 82 and passes control directly to box 86.

[0052] Next, in box 88, the user B or user C computing device plays back the voice output, for example, through a speaker of the computing device or a speaker of earbuds, external speakers, headphones, headsets, earphones, etc., coupled to the user's computing device. Then, in box 90, the user B or user C computing device displays a text representation of the voice input in the user's selected language in the chat room displayed on the user's computing device. The user B or user C computing device can then wait in box 92 for a new utterance to be spoken by the user of the device (which is processed by the user's computing device in the manner described above in conjunction with boxes 62-76), or wait to receive a message representing a new utterance spoken by another member of the group (which is processed by the user's computing device in the manner described above in conjunction with boxes 80-90).

[0053] Continuing with reference to boxes 76 and 90, Figures 3A-3D A chat room display showing example interactions between users A, B, and C. For example, Figure 3A An example user computing device 100 of user B (Jim, who speaks English) is shown, including a display showing a Group Translate application window 104. Field 106 shows a list of group members, Margaux, Diego, and Jim, and in elements 108, 110, and 112, transcriptions of English utterances are shown, along with an indication of the speaker of each utterance, and optionally, an indication of the speaker's selected language.

[0054] The user interface 102 also includes a "speak" control 114 that user B can activate to create a spoken utterance for communicating with others of the group. Additionally, in some embodiments, private communications with individual group members may also be supported, for example, by selecting a "private" control 116.

[0055] For purposes of this example, assume that user B wishes to communicate with the group. The user selects control 114 and speaks to the device the words "In what year did he die?". Figure 3B As shown, as a result of speaking the utterance, user B's device displays a textual representation of the spoken utterance in element 118 .

[0056] Figure 3C An example computing device 120 of user A (Margaux, a tour guide who speaks French) is shown, the example computing device including a display in which a group translation (Tradire le group) application window 124 is displayed. Field 126 displays a list of group members, Margaux, Diego and Jim, and the transcription of the French utterance is displayed in elements 130, 132 and 138, respectively. Figure 3A-3B , and an indication of the speaker of each utterance, and optionally, an indication of the speaker's selected language.

[0057] The user interface 122 also includes a "speak" control 134 that user A can activate to create a spoken utterance for communicating with others in the group. In addition, in some embodiments, a respective "playback" control 136 may be displayed in association with each communication 130, 132, 138 to enable the user to re-listen to the spoken audio output when desired. Figure 3CAlso shown is another optional control 140 that can be displayed in the user interface 122 to enable user A to create one or more pre-stored messages and / or transmit one or more pre-stored messages to the group. Selection of the control 140 may, for example, result in a message similar to Figure 3D As shown, pop-up window 142 can display controls 144, 146 and 148, which user A can select to transmit non-audio data associated with each pre-stored message to other members of the group, thereby enabling each user to translate (if necessary) and play back the spoken audio output corresponding to the pre-stored message in his or her selected language.

[0058] It will be understood that although Figures 3A-3D In the example display of , the user initiates voice input by selecting a “speak” control, but in other implementations, voice input can be captured in other ways, for example, using an “always on” mode in which the computing device captures all of the user's verbal input based on pressing a hard button on the computing device, or based on speaking a voice activation command phrase (similar to a phrase used to activate a personal assistant on certain types of computing devices).

[0059] Now go to Figure 4 In various implementations, many different data structures may be used to maintain group information for a multilingual translation group. In particular, Figure 4 A translation group data structure 150 is shown for the group "Margaux Tour Group", which may include multiple records or entries 152 for each member of the group and which may be maintained on each user computing device. When a new member is added to the group, a new entry 152 is added to the data structure 150. In addition, a transcription 154 may be maintained for the group, including each communication or spoken utterance uttered to the group.

[0060] Each entry 152 may include a username field 156 or other user identification information for its respective member, and optionally a language field 158 indicating the associated user's selected language. In some implementations, language information may not be stored because each user may receive all communications in a common language.

[0061] Each entry 152 may also include a network field 160 that stores each user's network information to enable communications to be transmitted to other user computing devices.

[0062] However, it should be understood that in other embodiments, Figure 4 The present invention may also include data structures other than those shown, and thus the present invention is not limited thereto.

[0063] Next, Figure 5An alternative embodiment is shown in which, instead of sending a broadcast message including non-audio data in a common language to all other group members, the sending computing device can send individual messages including non-audio data in each user's selected language to other group members. In particular, Figure 5 Routine 170 is shown, which starts in box 172 by receiving voice input from a computing device user in a sending computing device. In box 174, automatic speech recognition is performed in the sending computing device to generate a non-audio (e.g., text) representation of the voice input in the user's selected language. Then, in box 176, the sending computing device initiates a loop that sends a message to each other group member. For each such member, in box 178, the sending computing device determines whether the member has selected a language different from the language of the sending user (e.g., based on the language data stored in data structure 150), and if not, control is passed to box 180 to perform translation to the language selected by the member. Then, in box 182, the sending computing device generates and sends a message including non-audio data and addresses the computing device of the member. However, if the language is not different, box 180 is bypassed, and box 178 passes control directly to box 182 to generate and send the message to the member. Once the message has been sent to all other members, routine 170 is completed.

[0064] In other embodiments, routine 170 may also be varied. For example, instead of sending a separate message to each member, a message may be generated for each language represented in the group and sent to members who select that language, thereby reducing bandwidth and reducing translation overhead for the sending device. In other embodiments, a single broadcast message may be generated that includes a non-audio representation of each represented language, so that each member computing device that receives the message may select the non-audio data in the language selected by the member.

[0065] In still other embodiments, one or more of automatic speech recognition, translation, and text-to-speech conversion may be performed by a cloud service rather than locally within the computing device. On the other hand, in certain embodiments, it may be desirable to perform these actions locally and use local computing resources of the computing device to reduce latency and bandwidth consumption of the computing device, as well as to reduce the computing device's reliance on online connectivity to the cloud service, which may otherwise limit the use of group translation in locations with little or no online connectivity.

[0066] In addition, in some embodiments, it may be desirable to use multiple wireless networks to implement group translation. For example, in the case of supporting multiple network types, different computing devices can be connected to each other using different network connections. In addition, in some embodiments, it may be desirable to use one type of network to establish a multilingual translation group, and then use different types of networks to transmit data associated with voice input between different computing devices. Therefore, for example, in an example embodiment, a multilingual translation group can be established using Bluetooth, NFC or other short-range networks, and in the process of establishing the group, network information associated with each network can be collected and distributed between group members, such as the IP address of each computing device, so that future communications (e.g., non-audio data associated with spoken utterances) between group members can be performed through other types of networks (e.g., through Wi-Fi).

[0067] Next, Figure 6 Another embodiment is shown, in particular, routine 200, which can be executed by each computing device to receive and process non-audio data messages sent by other members of a multilingual translation group. In this embodiment, a mesh network topology is used, whereby, instead of establishing a point-to-point connection between all member computing devices, when there is no point-to-point connection between the sending and receiving devices, a member computing device is used to pass the message to other member computing devices. Therefore, routine 200 can start in frame 202 by receiving a non-audio data message from a sending computing device at a receiving computing device. Then, in frame 204, the receiving computing device determines whether the received message is addressed to the computing device, and if so, control is passed to frames 206-210 to process the received message, including translating non-audio data into the user's selected language (frame 206) by the receiving computing device, generating a spoken audio output (frame 208) corresponding to the speech input of the user's selected language and playing the spoken audio output (frame 210).

[0068] On the other hand, if the non-audio data message is not addressed to the user's computing device, then in box 204, the receiving computing device passes control to box 212 to pass the message to another computing device, causing the message to propagate through the mesh network until the message is received by its intended destination computing device.

[0069] Therefore, it is understood that various embodiments can provide various technical advantages compared with existing methods.By using a common language, and the transmission (communication) of the non-audio data of the common language between the speaker and the listener computing device, the translation operation is distributed between the speaker and the listener computing device, and the computing device is usually not required to perform multiple translations for a given spoken utterance.Doing so can reduce the delay, processing overhead and power consumption in each computing device.In addition, by transmitting non-audio data instead of audio data, the amount of data that needs to be transmitted between devices can be reduced.This can help reduce the delay associated with the operation of the multilingual translation group and / or can avoid any quality loss associated with the wireless transmission of audio data usually.In addition, by locally performing automatic speech recognition, translation and text-to-speech conversion in each computing device, delay and dependence on online connectivity can be reduced.In addition, the proximity of the local wireless network that can be used to establish a multilingual translation group can promote the group to be established when multiple individuals in the same location wish to participate in such a group, and in addition, because in many embodiments, each computing device is only responsible for the translation between the single language selected by the user and the common language at most (and if the user happens to say the common language, then do not translate at all), so the scalability is very high in terms of the number of users and the number of supported languages. In many implementations, adding another group member who speaks a different language than every other group member generally does not change the processing and bandwidth overhead or latency of any single computing device participating in the multi-language translation group.

[0070] Furthermore, while the embodiments discussed herein focus on tour guide applications, the invention has applicability in many other situations, for example, to support announcements in areas where multiple individuals who speak different languages ​​may be in the same location, such as on an airplane, bus, ship, or train, at a train terminal or airport, at a presentation or conference, at a large gathering such as the United Nations, at a sporting or video game event, etc. Furthermore, the embodiments discussed herein can facilitate communication between individuals who speak more than two languages ​​in the same location, such as at a cocktail party, reception, business meeting, restaurant, etc.

[0071] In some embodiments, it may be desirable to incorporate virtual assistant functionality into a group translation app as well, for example, in a chat room for a multi-language translation group. Thus, for example, rather than the initial voice input being transmitted as non-audio data and translated on demand by each computing device, either spoken by the user of the sending computing device or provided as a pre-stored message, the initial voice input may be an audio or non-audio message generated by the virtual assistant. The voice input may be automatically generated, for example, in response to the virtual assistant detecting a message that raises a question that can be answered by the virtual assistant, or the voice input may be manually generated in response to a dedicated request to the virtual assistant. Thus, for example, in response to the above in conjunction with Figure 3B In the example discussed, the message "In what year did he die?" created by Jim and shown in element 118, the virtual assistant in Jim's computing device or another computing device (e.g., the computing device of the tour guide Margaux) can detect that a group member has asked a question and can generate a response such as "Leonardo da Vinci died in 1519" in text or spoken audio form, and can cause a non-audio message similar to that generated in box 70 of routine 40 (and any of boxes 62-68 are executed as needed to generate a non-audio representation of the response), which can be processed in boxes 72-92 in the same manner as any other voice input. Thus, each member of the group can receive a response in audio and / or text format and in a language of his or her choice. As another alternative, the group can be prompted to respond, for example, by selecting a response similar to Figure 3C The initiation of the virtual assistant response can be manually initiated by a control of a button 136 on an element 138 of the virtual assistant. As yet another alternative, a separate control can be provided on a computing device (e.g., on the tour guide's computing device) to enable a user of the computing device to ask the virtual assistant for a response.

[0072] Figure 7 300. The computing device 300 generally includes at least one processor 302 that communicates with a number of peripheral devices via a bus subsystem 304. These peripheral devices may include a storage subsystem 306, including, for example, a memory subsystem 308 and a file storage subsystem 310; a user interface input device 312; a user interface output device 314; and a network interface subsystem 316. The input and output devices allow a user to interact with the computing device 300. The network interface subsystem 316 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0073] The user interface input devices 312 may include a keyboard, a pointing device such as a mouse, trackball, touchpad or graphic tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 300 or over a communication network.

[0074] User interface output device 314 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or other mechanisms for creating a visible image. The display subsystem may also provide a non-visual display, such as through an audio output device. Typically, the use of the term "output device" is intended to include devices and methods for outputting information from computing device 300 to a user or another machine or computing device of all possible types.

[0075] The storage subsystem 306 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 306 may include a Figure 2A-2B Selected aspects and / or implementations of routine 40 Figure 1 The logic of one or more of the group translation app 18, the networking component 20, the chat component 22, the translation component 24, the automatic speech recognition component 26, and the text-to-speech conversion component 28.

[0076] These software modules are typically executed by the processor 302 alone or in combination with other processors. The memory subsystem 308 used in the storage subsystem 306 may include a variety of memories, including a main random access memory (RAM) 318 for storing instructions and data during program execution and a read-only memory (ROM) 320 in which fixed instructions are stored. The file storage subsystem 310 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments may be stored by the file storage subsystem 310 in the storage subsystem 306, or in other machines accessible to the processor(s) 302.

[0077] The bus subsystem 304 provides a mechanism that enables the various components and subsystems of the computing device 300 to communicate with each other as intended. Although the bus subsystem 304 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0078] The computing device 300 may be of various types, including a mobile device, a smart phone, a tablet computer, a laptop computer, a desktop computer, a wearable computer, a programmable electronic device, a set-top box, a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, for purposes of illustrating some embodiments, only a single device 300 is described. Figure 7 The description of computing device 300 depicted in FIG. 1 is intended only as a specific example. Figure 7 Many other configurations of computing device 300 having more or fewer components than the computing device 300 depicted in FIG. 3 are possible.

[0079] In situations where the systems described herein collect personal information about a user or may make use of personal information, the user may be provided with an opportunity to control whether an application or feature collects user information (e.g., information about the user's social network, social actions or behaviors, occupation, the user's preferences, or the user's current location) or to control whether and / or how content that may be more relevant to the user is received. In addition, certain data may be processed in one or more ways before being stored or used so as to remove personally identifiable information. For example, the identity of a user may be processed so that personally identifiable information about the user cannot be determined, or the geographic location of a user for which location information is obtained may be generalized (e.g., to a city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user may have control over how information about the user is collected and how the information is used by the content server.

[0080] Although several embodiments have been described and shown herein, a variety of other means and / or structures for performing functions and / or obtaining results and / or one or more advantages described herein may be utilized, and each such variation and / or modification is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein should be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications of the teachings used. Using only routine experiments, those skilled in the art will recognize or be able to determine many equivalents of the specific embodiments described herein. Therefore, it should be understood that the foregoing embodiments are given only by way of example, and within the scope of the attached claims and their equivalents, the embodiments may be practiced in a manner different from that specifically described and claimed. The embodiments of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods are not mutually contradictory, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A computer-implemented method comprising: In a first computing device of a first user in communication with a second computing device of a second user and a third computing device of a third user, performing automatic speech recognition on speech input received from the first user in a first language and locally translating output of the automatic speech recognition from the first language of the first user into a different language to generate non-audio data associated with the speech input; as well as In a first computing device, non-audio data associated with a speech input is sent to a second computing device and a third computing device over one or more networks so that the second computing device locally generates at least one of a translation of the non-audio data into a second language of a second user and the third computing device locally generates a translation of the non-audio data into a third language of a third user.

2. The method according to claim 1, further comprising: In the first computing device, and in response to second speech input in a second language received at the second computing device: receiving second non-audio data associated with a second speech input; and Generating a spoken audio output in a first language and for a first user from second non-audio data associated with a second voice input, wherein generating the spoken audio output in the first computing device includes locally translating the second non-audio data associated with the second voice input into the first language.

3. The method according to claim 2, further comprising: In a third computing device, and in response to a second speech input in a second language received at the second computing device: receiving second non-audio data associated with a second speech input; and A second spoken audio output in a third language and intended for a third user is generated from second non-audio data associated with the second speech input.

4. The method according to claim 2, further comprising: In the second computing device: receiving a second voice input; performing automatic speech recognition on the second speech input to generate second non-audio data associated with the second speech input; and Second non-audio data associated with the second voice input is sent to the first computing device and the third computing device.

5. The method according to claim 2, further comprising, in the second computing device: receiving a second voice input; performing automatic speech recognition on the second speech input and locally translating the output of the automatic speech recognition performed on the second speech input from the second language into a different language to generate second non-audio data associated with the second speech input; and Second non-audio data associated with the second voice input is sent to the first computing device and the third computing device.

6. The method according to claim 1, wherein: The non-audio data includes one or more feature vectors that describe phonemes, words, or phrases in the speech input.

7. The method according to claim 1, wherein: The non-audio data includes text data in a common language other than the first language and at least one of the second language and the third language.

8. The method according to claim 1, wherein: The first computing device communicates with the second computing device and the third computing device via a local wireless network connection.

9. The method according to claim 8, wherein: The local wireless network connection includes a Bluetooth connection, a personal area network connection, a wireless mesh network connection, a near field connection (NFC), or a wireless local area network (WLAN) connection.

10. The method according to claim 1, further comprising: In the second computing device, non-audio data associated with the speech input received from the first computing device is communicated to the third computing device.

11. The method according to claim 1, wherein: The non-audio data is sent to the second computing device and the third computing device using a broadcast message.

12. The method according to claim 1, wherein: Sending the non-audio data to the second computing device and the third computing device is performed using a point-to-point message addressed to each of the second computing device and the third computing device.

13. The method according to claim 1, further comprising: In the first computing device, a multi-language translation group including a second user and a third user is joined by establishing a local wireless network connection with one or both of the second computing device and the third computing device, wherein the non-audio data is for the multi-language translation group.

14. The method according to claim 13, further comprising: Initiate a multilingual translation group in the following ways: broadcasting an announcement message from one of the first computing device and the second computing device; as well as A response to the announcement message is received at one of the first computing device and the second computing device from the other of the first computing device and the second computing device.

15. A first computing device configured to output spoken audio in a first language to a first user, comprising one or more processors and a memory operably coupled to the one or more processors, wherein: The memory stores instructions, and in response to one or more processors executing the instructions, the one or more processors: performing automatic speech recognition on speech input in a first language received from a first user, and locally translating output of the automatic speech recognition from the first language of the first user into a different language to generate non-audio data associated with the speech input; and Non-audio data associated with the speech input is sent via one or more networks to a second computing device and a third computing device associated with a second user and a third user, respectively, so that the second computing device locally generates at least one of a translation of the non-audio data into a second language of the second user and the third computing device locally generates a translation of the non-audio data into a third language of the third user.

16. The first computing device of claim 15, wherein: The memory further stores instructions that, in response to the one or more processors executing the instructions, cause the one or more processors to: receiving second non-audio data associated with a second speech input in a second language received at a second computing device; and Generating a spoken audio output in a first language and for a first user from second non-audio data associated with a second voice input, wherein generating the spoken audio output in the first computing device includes locally translating the second non-audio data associated with the second voice input into the first language.

17. The first computing device of claim 15, wherein: The non-audio data includes one or more feature vectors that describe phonemes, words, or phrases in the speech input.

18. The first computing device of claim 15, wherein: The non-audio data includes text data in a common language other than the first language and at least one of the second language and the third language.

19. The first computing device of claim 15, wherein: The memory also stores instructions that, in response to one or more processors executing the instructions, cause the one or more processors to join a multilingual translation group including the second user and the third user by establishing a local wireless network connection with one or both of the second computing device and the third computing device, wherein the non-audio data is for the multilingual translation group.

20. The first computing device of claim 19, wherein: The memory further stores instructions, which, in response to the one or more processors executing the instructions, cause the one or more processors to initiate a multi-language translation group in the following manner: Broadcast announcement messages; as well as A response to the announcement message is received from at least one of the second computing device and the third computing device.

Images