Microphone switching for translation
By dynamically controlling microphone signal switching and sensor information, the quality issues of translation applications in cases of overlapping speech and occlusion are resolved, achieving high-quality translation and a natural dialogue experience.
Patent Information
- Application Number
- CN202610662427.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2026-03-13
- Filing Date
- 2026-05-14
- Publication Date
- 2026-11-17
AI Technical Summary
Existing translation applications struggle to provide high-quality audio signals and a natural conversational experience when dealing with overlapping speech and occlusion.
By dynamically controlling the switching of microphone signals, the microphone that is physically closest to the speaker and/or along an unobstructed acoustic path is selected. Combined with machine learning models and sensor information, intelligent switching and translation of audio input can be achieved.
It improves translation quality, saves processing resources and battery power, and provides users with a more natural conversational experience.
Smart Images

Figure CN123123771A_ABST
Abstract
Description
Cross-references to related applications
[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 806,596, filed May 15, 2025, entitled “MICROPHONE SWITCHING FORTRANSLATIONS,” the entire contents of which are incorporated herein by reference. Technical Field
[0002] This manual relates in general to audio devices, and more specifically to microphone switching for purposes such as translation. Background Technology
[0003] Electronic devices sometimes have built-in translation apps that can translate written text in one language into written text in another language. Some translation apps can also translate spoken language into written text in another language. Attached Figure Description
[0004] The specific features of this subject matter are set forth in the appended claims. However, for purposes of explanation, several embodiments of this subject matter are illustrated in the following figures.
[0005] Figure 1 Example system architectures are illustrated based on one or more specific implementations, including various electronic devices that can implement the systems of this subject.
[0006] Figure 2 This is a schematic diagram illustrating a use case of performing translation operations involving multiple microphones from multiple corresponding devices, based on a specific implementation of the technology in this subject matter.
[0007] Figure 3 This is a schematic diagram illustrating another use case of performing translation operations involving multiple microphones from multiple corresponding devices, according to a specific implementation of the technology of this subject.
[0008] Figure 4 A schematic diagram illustrating an electronic device performing a translation operation according to one or more specific embodiments of the art described herein is provided.
[0009] Figure 5A This is a timing diagram illustrating example input to a translation application in a first use case with overlapping speech, based on one or more specific implementations of the techniques described in this subject matter.
[0010] Figure 5B This is another timing diagram illustrating example input to a translation application in a first use case with overlapping speech, based on one or more specific implementations of the techniques of this subject.
[0011] Figure 6This is a timing diagram illustrating the input to a translation application in a second use case with overlapping speech, based on one or more specific implementations of the techniques described in this subject matter.
[0012] Figure 7 A schematic diagram illustrating an electronic device performing bidirectional translation operations according to one or more specific embodiments of the subject matter is shown.
[0013] Figure 8 This is a timing diagram illustrating the input from electronic device control to a translation application in a bidirectional translation use case with overlapping speech, according to one or more specific implementations of the present invention.
[0014] Figure 9 Example electronic devices that provide notifications according to one or more specific implementations of the technology of this subject are illustrated.
[0015] Figure 10 This is a flowchart of an example process for microphone switching for translation, based on one or more specific implementations of the subject matter.
[0016] Figure 11 This is a flowchart illustrating an example process of performance translation based on one or more specific implementations of the subject matter technology.
[0017] Figure 12A and Figure 12B Examples of dynamic control of the field of view of a beam used for processing microphone signals are illustrated in one or more specific embodiments of the subject matter art.
[0018] Figure 13A and Figure 13B This illustrates aspects of another example of dynamic control of the field of view of a beam used for processing microphone signals, according to one or more specific implementations of the subject matter.
[0019] Figure 14 This is a flowchart illustrating an example process for dynamic control of the field of view of a beam used to process microphone signals, according to one or more specific embodiments of the subject matter.
[0020] Figure 15 An example is an electronic system that can be used to implement one or more specific embodiments of the subject matter. Detailed Implementation
[0021] The detailed description listed below is intended to describe various configurations of the subject matter and is not intended to represent the only configuration in which the subject matter can be practiced. The accompanying drawings are incorporated herein and form part of the detailed description. The detailed description includes specific details intended to provide a thorough understanding of the subject matter. However, the subject matter is not limited to the specific details listed herein, but can be practiced in one or more other specific embodiments. In one or more specific embodiments, structures and components are shown in block diagram form to avoid obscuring the concepts of the subject matter.
[0022] The specific implementations of the subject matter techniques described herein provide dynamic control over audio input to a translator application. For example, a translator application may run on an electronic device and can be configured to generate translated audio output in response to received audio input. Dynamic control over audio input can help provide the highest quality audio signal for translation and / or contribute to a more natural, conversational translation experience for the user.
[0023] For example, the audio input to the translator at a first device (e.g., a portable electronic device, such as a smartphone) can be switched between a first microphone signal from a first microphone of the first device, a second microphone signal from a second microphone of a second device (e.g., an audio output device, such as earbuds or headphones or a headband-compatible device), and / or no input. As discussed in further detail below, switching between the first and second microphone signals can help provide the highest quality audio signal for translation by selecting the microphone that is physically closest to the speaker and / or positioned to receive the speaker's speech along an unobstructed acoustic path. As discussed in further detail below, switching between microphone signals and no signal can help save processing and / or battery power while providing the user with a more natural conversational translation experience.
[0024] The switching can be based on a first microphone signal, a second microphone signal, and / or other signals (such as contextual signals), and can be performed at various times throughout the translation session using the translation application. As an example, the switching can be based on the signal-to-noise ratio of one or both microphone signals, the noise level detected using one or both microphones, and / or the presence of the user's own voice in one or both microphone signals. The switching can also be based on one or more other input selection criteria and / or signals, such as the state of the translator application on the first device, the physical state of the first device, the activation trigger of the translator application, and / or other criteria, which may be based on information from the processor and / or one or more sensors of the first device.
[0025] Figure 1An example system architecture 100 comprising various electronic devices capable of implementing the system of this subject matter is illustrated according to one or more embodiments. However, not all depicted components are usable in all embodiments, and one or more embodiments may include additional or different components compared to those shown in the figures. Variations in the arrangement and type of these components may be made without departing from the spirit or scope of the claims set forth herein. Additional components, different components, or fewer components may be provided.
[0026] System architecture 100 includes audio devices 150, electronic devices 104 (e.g., handheld electronic devices, such as smartphones or tablets), electronic devices 110, electronic devices 115, and a server 120, all communicatively coupled via a network 106 (e.g., a local area network or a wide area network). For illustrative purposes, system architecture 100 is described in detail below. Figure 1 The system is exemplified as including audio device 150, electronic device 104, electronic device 110, electronic device 115, and server 120; however, system architecture 100 may include any number of electronic or / or audio devices and any number of servers or a data center including multiple servers.
[0027] Audio device 150 may be implemented as a headset (e.g., a pair of speakers mounted in speaker housings coupled together by a headband) or an earbud (e.g., one of a pair of earbuds, each having a speaker disposed in a housing conforming to a portion of the user's ear) configured to be worn by a user (also referred to as a wearer when the audio device is worn by the user). Each audio device 150 may include one or more speakers (such as speaker 152 configured to project sound into the ear of user 101) and one or more sensors (such as sensor 151). Sensor 151 may include one or more microphones configured to receive audio input (e.g., based on the user's own voice, ambient sound, and / or other external voice input). Sensor 151 may include other sensors, such as inertial measurement unit (IMU) sensors (e.g., accelerometers, magnetometers, and / or gyroscopes). In one or more embodiments, the audio device 150 may be integrated and / or communicatively coupled (e.g., and may collaborate with) another device, such as a wearable device, such as a smartwatch, smart ring, or head-mounted device (HMD), which includes one or more displays for displaying visual content (e.g., movies, shows, other video content, and / or game content) corresponding to the audio content output by the audio device 150. In one or more embodiments, when the audio device 150 is implemented as a head-mounted device, the head-mounted device may include one or more interfaces for mechanically attaching its housing to one or more structures (e.g., straps, arms, or other attachment mechanisms) configured to secure the housing to a wearer (e.g., to the wearer's head). In one or more embodiments, the head-mounted device may be implemented as smart glasses (e.g., a device in which one or more lenses or other transparent elements may be provided in a frame (such as an eyeglass frame or sunglasses frame) through which a user can directly view their physical environment, and display content may be displayed on the one or more lenses or other transparent elements to be presented in a combined manner with the direct view of the physical environment seen by the user through the lenses or other transparent elements (e.g., to overlay on and / or be integrated with the direct view). For example, in one embodiment, an audio device may be configured as a pair of smart glasses having one or more speakers 152 and one or more microphones 206, wherein lenses mounted in a frame with temples are configured as transparent displays, or may be arranged such that display content (e.g., virtual content) can be displayed and / or projected onto them.
[0028] Despite Figure 1 Invisible in (see example) Figure 2However, each audio device may include processing circuitry (e.g., including memory and / or one or more processors) and communication circuitry (e.g., one or more antennas, etc.) for receiving and / or processing audio content and / or control signals from one or more of electronic devices 104, 110, 115, and / or server 120 and / or providing audio data and / or other signals to electronic devices 104, 110, 115, and / or server 120. The processing circuitry of audio device 150 is operable to operate speaker 152 to generate sound. The memory may store one or more machine learning models and / or one or more rule-based processes for performing one or more of the operations described herein.
[0029] Audio device 150 may include communication circuitry for communicating (e.g., directly or via network 106) with electronic devices 104, 110, 115, and / or server 120. This communication circuitry includes, for example, one or more wireless interfaces, such as WLAN wireless devices, cellular wireless devices, Bluetooth wireless devices, Zigbee wireless devices, near field communication (NFC) wireless devices, and / or other wireless devices. Electronic devices 104, 110, 115, and / or server 120 may include communication circuitry for communicating (e.g., directly or via network 106) with audio device 150 and / or with other devices in electronic devices 104, 110, 115, and / or server 120. This communication circuitry includes, for example, one or more wireless interfaces, such as WLAN wireless devices, cellular wireless devices, Bluetooth wireless devices, Zigbee wireless devices, near field communication (NFC) wireless devices, and / or other wireless devices.
[0030] In one or more embodiments, the audio device 150 may also include one or more machine learning models trained to identify the voice of user 101, any spoken commands from various spoken commands from user 101, and / or any standardized sounds from various standardized sounds (such as police or ambulance sirens, smoke or fire alarms, and / or travel announcements (e.g., station announcements, bus stop announcements)).
[0031] Audio device 150 is communicatively coupled to ancillary devices, such as electronic devices 104, 110, and / or 115. Generally, such ancillary devices may include more computing resources and / or available power compared to audio device 150. In one or more embodiments, the memory of electronic devices 104, 110, and / or 115 may store code and / or instructions for one or more applications, such as a translation application. In one or more embodiments, the memory of electronic devices 104, 110, and / or 115 may store one or more machine learning models trained to perform one or more portions of the translation operations described herein. Electronic devices 104, 110, and / or 115 may also include one or more machine learning models trained to identify the voice of user 101, any spoken commands from various spoken commands received from user 101, and / or any standardized sounds (such as police or ambulance sirens, smoke or fire alarms, travel announcements (e.g., station announcements, bus stop announcements)) based on audio data received from audio device 150.
[0032] Audio device 150 can operate in various modes. For example, audio device 150 can operate in a first operating mode (e.g., a transparency operating mode), in which ambient sounds are delivered to the ears of user 101 by the speakers of audio device 150. In this first operating mode, audio content (such as translated audio and / or media content) (e.g., received from electronic device 104) can be output by the speakers 152 of audio device 150 without using noise cancellation content, and external sounds are allowed and / or amplified for delivery to the user while the audio content is playing. Audio device 150 can operate in a second operating mode (e.g., a noise cancellation operating mode, such as active noise cancellation (ANC) operation, which may be an adaptive ANC operation that adaptively eliminates residual ambient noise using an error signal from an error microphone), in which ambient sounds received by one or more microphones of audio device 150 are canceled or suppressed to prevent ambient sounds from reaching the ears of user 101. In this second operating mode, audio content (e.g., translated audio content and / or media content, such as music, podcasts, application-related audio, games, videos, etc.) can be played by the audio device in combination with noise cancellation content (e.g., noise cancellation content generated based on ambient sounds received by one or more microphones of the audio device 150 to cancel out ambient sounds by cancelling interference).
[0033] Audio device 150 may also include one or more sensors, such as touch sensors and / or force sensors, for receiving user input. For example, a user / wearer of audio device 150 may briefly tap a touch sensor or pinch force sensor to control the audio content being played, control the playback volume, start or end a translation session, and / or switch between a transparency operation mode and a noise cancellation operation mode. In one or more embodiments, a user may press and hold a force sensor while the audio device is operating in noise cancellation operation mode to temporarily switch to transparency operation mode until the force sensor is released. In one or more embodiments, user 101 may provide input to sensor 151 to enable or disable translation operation at audio device 150 and / or accompanying devices (such as electronics 104, electronics 110, and / or electronics 115).
[0034] Electronic device 104 may be, for example, a portable computing device (such as a smartphone, laptop computer), a peripheral device (e.g., a digital camera, headphones), a tablet device, a wearable device (such as a smartwatch, smart wristband, etc.), including any other suitable device, such as processing circuitry and / or communication circuitry for providing audio content to audio device 150. Figure 1 In this example, electronic device 104 is depicted as a mobile smartphone device with a touchscreen. In one or more embodiments, electronic device 104 and / or audio device 150 may be and / or include the following relative to the following. Figure 11 The electronic system discussed refers to all or part of the electronic equipment discussed.
[0035] Electronic device 110 may be, for example, a portable computing device (such as a smartphone, tablet, or laptop computer), a peripheral device (e.g., a digital camera, headphones), a tablet device, a wearable device (such as a smartwatch, smart wristband, etc.), including, for example, any other suitable device for providing audio content to audio device 150, such as processing circuitry and / or communication circuitry. Figure 1 In this embodiment, by way of example, electronic device 110 is depicted as a tablet device with a touchscreen. In one or more embodiments, electronic device 104 and / or audio device 150 may be and / or may include the following relative to the following. Figure 11 The electronic system discussed refers to all or part of the electronic equipment discussed.
[0036] Electronic device 115 may be, for example, a desktop computer, a portable computing device (such as a laptop computer), a smartphone, a peripheral device (e.g., a digital camera, a headset), a tablet device, a home hub device, or a wearable device (such as a watch, a wristband, etc.). Figure 1In this embodiment, electronic device 115 is depicted as a desktop computer by way of example. However, this is merely illustrative. In some specific implementations, electronic device 115 may be implemented as a device communicatively coupled to one or more connected networks in a home or building and configured to generate control signals for operating devices on one or more connected networks (e.g., a central device or other device). Electronic device 115 may be and / or may include the following relative to... Figure 11 All or part of the electronic system under discussion.
[0037] In one or more specific implementations, Figure 1 Two or more devices may be owned by and / or registered to the same user's account. For example, audio device 150 and electronic device 104 (e.g., and / or electronic device 110 and / or electronic device 115) may be owned by user 101's account, registered to that account (e.g., registered with server 120), and / or logged into that account.
[0038] Server 120 may form all or part of a computer network or server group 130, such as in a cloud computing or data center implementation. For example, server 120 stores data and software and includes specific hardware (e.g., processors, graphics processors, and other dedicated or custom processors such as those for user authentication and / or authorization, account services, and / or streaming media content to electronic devices 104, 110, and / or 115). In one implementation, server 120 may function as a cloud storage server. In one or more implementations, server 120 may store a machine learning model trained to translate audio data of speech including one or more first languages (e.g., English, French, Spanish, Dutch, German, Swedish, Hindi, one or more Chinese languages, etc.) into audio data of computer-generated speech including one or more second languages (e.g., English, French, Spanish, Dutch, German, Swedish, Hindi, one or more Chinese languages, etc.). The translation model may be configured to translate one language into another language, multiple different languages into one language, or multiple different languages into multiple other different languages. In one or more embodiments, audio device 150, electronic device 104, electronic device 110, and / or electronic device 115 may acquire one or more translation models among those translation models to be executed at audio device 150, electronic device 104, electronic device 110, and / or electronic device 115. For example, user 101 may use the user interface of a translation application at audio device 150, electronic device 104, electronic device 110, and / or electronic device 115 to select from one or more target translation languages, which may cause the device to download the corresponding translation model in those one or more selected target translation applications. For example, before or during an English speaker's travel to France or another French-speaking country, an English-to-French translation model may be downloaded to audio device 150, electronic device 104, electronic device 110, and / or electronic device 115.
[0039] Figure 2 Example use cases are illustrated according to various aspects disclosed in this subject, wherein electronic device 104 operates as an accessory to audio device 150 of user 101 during translation operations. For example... Figure 2 As shown, the audio device 150 may include processing circuitry 202, a speaker 152, one or more sensors 204, and / or one or more microphones 206. For example, the sensors 204 and microphones 206 may be... Figure 1Specific implementations of sensor 151. As shown, audio device 150 may include a plurality of microphones 206. The plurality of microphones 206 may include a microphone 206 positioned (e.g., behind speaker 152) to receive ambient sound, a microphone 206 positioned (e.g., closest to the wearer's mouth) to receive voice input from the wearer, and / or a microphone 206 positioned (e.g., in front of speaker 152) as an error microphone for detecting sound (e.g., residual or uncancelled ambient sound) in or near the wearer's ear canal. Sensor 204 may include inertial measurement unit (IMU) sensors (e.g., accelerometers, magnetometers, and / or gyroscopes), proximity sensors, touch sensors, force sensors, and / or other sensors. Figure 2 In the example, the audio device 150 is implemented as an earbud, and may be one of a pair of earbuds having the same or similar components and configured for wired or wireless communication between the pair of earbuds.
[0040] Electronic device 104 may include processing circuitry 210, memory 212, communication circuitry 215, one or more speakers 216, one or more microphones 214, and / or one or more sensors 218. As shown, electronic device 104 may include multiple microphones 214 located in various positions. Sensors 218 may include light sensors, inertial measurement unit (IMU) sensors (e.g., accelerometers, magnetometers, and / or gyroscopes), proximity sensors, touch sensors, force sensors, and / or other sensors.
[0041] exist Figure 2 In the example, user 101 may be attempting to converse with person 201, who is not user 101. In one example use case, user 101 may be able to speak and understand one language, such as English, and person 201 may be able to speak and understand a different language, such as Spanish. User 101 may not be able to speak or understand another language of person 201 (e.g., or may have limited ability to speak and understand that other language). To facilitate this conversation, for example, electronic device 104 may perform a translation operation that translates speech (e.g., including words, phrases, sentences, etc.) from person 201's second language (e.g., Spanish) into audio output, which includes a computer-generated translation of the speech from person 201's second language into a first language (e.g., English).
[0042] For example, the speech 222 of person 201 may be received by one or more microphones in microphones 206 of audio device 150 and / or by one or more microphones 214 of electronic device 104. Input audio data (such as microphone signals including a representation of the speech 222 of person 201) may be transmitted from audio device 150 to electronic device 104 (e.g., an accessory device). A translation application running at electronic device 104 may generate translated audio data in response to receiving input audio data from audio device 150 and / or receiving microphone signals from microphones 214 of electronic device 104. As shown, the translated audio data may be transmitted from electronic device 104 to audio device 150. Then, speaker 152 of audio device 150 may generate audio output 225 corresponding to the translated audio data, which includes a computer-generated translation of the speech 222 of person 201. In this way, electronic device 104 and audio device 150 allow user 101 to understand human 201's speech substantially in real time (e.g., with a delay of one or two seconds or less than one second) while human 201 is speaking, even if human 201 is speaking a language that user 101 does not understand.
[0043] exist Figure 2 In the example, to simplify the initial discussion, the only sound depicted as received by the microphone 206 of the audio device 150 and the microphone 214 of the electronic device 104 is the speech of person 201. In this example, both the microphone 206 of the audio device 150 and the microphone 214 of the electronic device 104 receive the speech 222 of person 201. However, as Figure 3 As illustrated, the translation operation using electronic device 104 and audio device 150 may not always occur under the condition that the voice 222 of person 201 is the only sound received at the microphone or under the condition that the voice 222 of both devices is received at the microphone.
[0044] For example, at some point while person 201 is speaking with their voice 222, user 101 may also speak (e.g., there may be overlapping speech). In this use case, user 101's voice 324 may also be received at microphone 206 of audio device 150 (e.g., it overlaps or partially overlaps with person 201's voice 222). Voice 324 may also be received at one or more microphones in microphone 214 of electronic device 104.
[0045] Figure 3Examples are also illustrated of how, in some use cases, an obstruction 300 (e.g., a container or other object) could block or suppress the reception of a person 201's voice 222 to the microphone 214 of the electronic device 104. For example, the electronic device 104 may be located in a container, such as a pocket, bag, wallet, backpack, or other container, once or multiple times. In these example use cases, the microphone 214 of the electronic device 104 may be physically blocked from receiving some or all of the person 201's voice 222.
[0046] Because, as Figure 3 As illustrated, for example, the voice 222 of person 201 may be received in combination with other sounds (e.g., including the voice 324 of user 101), and / or the microphone 214 of electronic device 104 may be obstructed or blocked to varying degrees. Therefore, it may be advantageous to be able to variably and / or dynamically control which microphones provide audio input to the translation application at electronic device 104 (e.g., based on conditions in the physical environment surrounding electronic device 104 and audio device 150). Aspects of the subject matter can adaptively provide dynamic control of input to the translation application in response to changing states of electronic device 104 and / or changing states of the sound environment surrounding audio device 150 and / or electronic device 104.
[0047] For example, Figure 4 This is a block diagram of electronic device 104, which illustrates various operations that can be performed by electronic device 104 during translation operations. Figure 4 The various parts of the architecture can be implemented in software, firmware, and / or hardware, including by one or more processors and a memory device containing instructions that, when executed by the processor, cause the processor to perform the operations described herein. For example, in Figure 4 In the diagram, the trapezoidal frame may indicate that the microphone 214, speaker 216, sensor 218, communication circuitry 215, and IMU 411 can be hardware components, and the rectangular frame may indicate that one or more signal analyzers (SA) 407, one or more voice activity detectors (VAD) 408, condition detector 406, input controller 400, and translator application 402 can be implemented in software, including via one or more processors (e.g., Figure 2 The processing circuitry 210 executes instructions and a memory device (e.g., memory 212) containing the instructions, which, when executed by the processor, cause the processor to perform the operations described herein.
[0048] like Figure 4 As shown, one or more microphones 214 of the electronic device 104 can receive direct voice input, such as... Figure 2 and / or Figure 3The voice 222 of person 201. As shown in the figure, the communication circuit 215 of the electronic device can also receive incoming audio data, such as from... Figure 2 and Figure 3 The input audio data of the audio device 150 shown. For example, the input audio data may be encoded microphone signals from one or more microphones 206 of the audio device 150. Direct voice input may include the voice 222 of person 201 when person 201 is speaking, and may also include the voice 324 of user 101 when person 201 is speaking. Incoming audio data from the audio device 150 may include the voice 222 of person 201 when person 201 is speaking, and may also (e.g., primarily in the use case where user 101 is wearing the audio device 150) include the voice 324 of user 101 when user 101 is speaking.
[0049] As shown in the figure, electronic device 104 may include an input controller 400 that controls the inputs to translator application 402. Figure 4 In the example, input controller 400 is depicted as separate from translator application 402. In one or more other embodiments, some or all of the functionality of input controller 400 as described herein may be performed by the translator application (e.g., to internally control input to translation block 452 of the translator application). Input controller 400 may dynamically control input to the translator application continuously, periodically, and / or in response to one or more decision point triggers during translation operations. As an example, decision point triggers may include the start of a translation session, changes in the state of translator application 402, changes in the quality and / or content of microphone signals from various microphones, and / or changes in the state of electronic device 104.
[0050] In one or more embodiments, the translator application 402 may include one or more machine learning models and / or one or more rule-based processes trained and / or programmed to: (i) receive audio input comprising speech content in a first language, and (ii) generate a corresponding audio output comprising a computer-generated version of the speech content translated into a second language different from the first language. For example, the translator application 402 may include an automatic speech recognizer (ASR) 450 that converts speech represented by microphone signals into text, a translation block 452 that translates text in the first language into translated text in the second language, and a speech generator 454 that generates audio signals (e.g., translated audio data) based on the translated text in the second language. However, this example of the functionality of the translator application is merely illustrative, and in other embodiments, the translator application may generate translated audio data in different ways (e.g., using machine learning models that have been trained to generate translated audio data directly from input audio data without first converting the input audio data into text). In various specific implementations, translation block 452 may be configured (e.g., trained) to perform direct translation of words and / or phrases into a second language, or may include a large language model (e.g., located locally on the device, or remotely on the device with the explicit permission of the person whose speech is to be translated) that translates the meaning of speech in the first language into corresponding utterances in the second language. As shown, the translated audio data generated by translator application 402 may be provided to communication circuitry 215 for (e.g., encoding and) transmission to audio device 150. For example, when (e.g., decoded and) provided to speaker 152 of audio device 150, the translated audio data may cause speaker 152 to generate a sound that mimics the speech of a person speaking the translated text.
[0051] like Figure 4 As shown, the input controller 400 may include a switch 404 (e.g., a digital switch or a software switch). The switch 404 may be controlled by a control signal generated by a condition detector 406 of the input controller 400. As indicated in the figure, the control signal may control the switch 404 to selectively provide one of the following to the translator application 402 (e.g., to the translator application's translation block 452): a local microphone signal from the microphone 214 of the electronic device 104, a remote microphone signal derived from incoming audio data from the audio device 150, or an empty input 410 to the translator application.
[0052] As shown, the condition detector 406 can receive various input signals, which it can use to control the switch 404 (e.g., to select input to the translator application 402). As shown, the input signals to the condition detector 406 may include signal analysis information from the signal analyzer 407 (e.g., one or more noise levels (sound pressure level or SPL) and / or one or more signal-to-noise ratios (SNR) of one or more of the local microphone signal and / or remote microphone signal), one or more voice activity detection signals from VAD 408 (e.g., VAD1, VAD2), one or more sensor signals from sensor 218, one or more IMU signals from IMU 411 (e.g., orientation and / or motion signals), and / or application status information from the translator application.
[0053] In one or more embodiments, condition detector 406 may determine device status information of electronic device 104 based on sensor signals from sensor 218 and / or IMU signals from IMU 411. In one or more other embodiments, device status information may be determined and provided to condition detector 406 through a separate process at electronic device 104.
[0054] For example, condition detector 406 may use sensor signals and / or IMU signals to determine whether electronic device 104 (e.g., and / or one or more of its microphones 214) is in an obstructed state (e.g., in a pocket or bag that may obstruct one or more of the microphones 214) or an unobstructed state (e.g., in the hand of user 101 or person 201, or on a table or surface near or between user 101 and person 201). For example, sensor 218 may include one or more proximity sensors. Proximity sensor signals may be used to determine whether an object (e.g., the inside of a container such as a pocket or bag) is touching or very close to electronic device 104 (e.g., within one centimeter), thereby indicating that electronic device 104 is in an obstructed state. In one or more embodiments, condition detector 406 may use other information to determine and / or confirm the obstructed / unobstructed state of electronic device 104. For example, electronic device 104 may perform one or more acoustic tests that compare microphone signals from a plurality of microphones 214 (e.g., in response to predetermined outputs from one or more speakers 216) to determine whether one or more microphones of the electronic device 214 are covered or obstructed. As another example, if an IMU signal indicates that electronic device 104 is face down, condition detector 406 may determine that electronic device 104 is unobstructed (e.g., placed face down on a table or surface) even if a proximity sensor signal indicates that the face of the electronic device is close to an object (e.g., a table or surface). As another example, if an IMU signal indicates that electronic device 104 is moving in accordance with characteristic movements of user 101's hand (e.g., walking or talking hand movements), condition detector 406 may determine that electronic device 104 is unobstructed (e.g., being held by user 101 or person 201) even if a proximity sensor signal indicates that an object (e.g., a hand) is close to one or more parts of electronic device 104 (e.g., because the user's hand or fingers cover the proximity sensor, leaving microphone 214 unobstructed).
[0055] In one or more specific implementations, the condition detector 406 may use other input signals to determine and / or confirm whether the electronic device 104 is in an obstructed or unobstructed state. For example, application status information received from a translator application may indicate the state of the translator application's user interface. For example, the application status information may indicate that the translator application's user interface is active and / or displayed in the foreground of the electronic device 104's display, thereby indicating that the electronic device 104 is in an unobstructed state (e.g., in conjunction with other signals described above indicating that the device is in an unobstructed state, and / or even if one or more of the other signals indicate that the device may be in an obstructed state). In use cases where the application status information indicates that the translator application's user interface is minimized, locked, or running in the background, the user interface status information may help indicate or confirm other input signals indicating that the electronic device 104 is in an obstructed state. As another example, the condition detector 406 may receive an indication that the electronic device 104 is unlocked, thereby helping to indicate or confirm that the electronic device 104 is in an unobstructed state (e.g., in conjunction with other signals described above indicating that the device is in an unobstructed state, and / or even if one or more of the other signals indicate that the device may be in an obstructed state).
[0056] In some examples, when condition detector 406 determines that electronic device 104 is unobstructed, condition detector 406 may control switch 404 to provide a local microphone signal from microphone 214 of electronic device 104 to translator application 402. In some examples, when condition detector 406 determines that electronic device 104 is obstructed, condition detector 406 may control switch 404 to provide a remote microphone signal from audio device 150 to translator application 402. However, these decisions made by condition detector 406 may also depend on other input signals received by condition detector (e.g., as supplement to device status), such as one or more VAD signals received from VAD 408.
[0057] For example, a VAD signal can indicate when one or more voices (e.g., voice 222 of person 201 and / or voice 324 of user 101) are active (e.g., present in local microphone signals and / or incoming audio data). For example, VAD 408 can receive both local microphone signals and remote microphone signals, and can determine the presence of one or more voices from the local microphone signals and / or remote microphone signals. For example, VAD 408 may include a self-voice activity detector (OVAD) and a target voice activity detector (TVAD). The OVAD can be configured to detect voice 324 of user 101. The TVAD can be configured to detect voice 222 of person 201 other than user 101. As an example, the OVAD can be trained based on the user's own voice, trained based on the speech of the user's own language, and / or configured to detect the user's own voice based on the (e.g., approximate) known position of the user's mouth relative to the audio device 150 (e.g., based on near-field characteristics of the user's voice being close to the microphone 206 of the audio device 150, such as the different times at which voice 324 arrives at different microphones of the audio device 150). As an example, TVAD can be trained to detect speech in another language and / or to detect a speaker other than the user based on the location information of the target speaker (e.g., in response to detecting far-field features of human speech in the microphone signal from the microphone 206 of audio device 150 and / or the microphone 214 of electronic device 104, such as the time when speech 222 arrives at the microphone of audio device 150 is equal or the same).
[0058] In one or more embodiments, the OVAD of VAD 408 can generate a first VAD signal (e.g., VAD1) indicating the presence of the user's own speech in the local microphone signal and / or the remote microphone signal, and the TVAD of VAD 408 can generate a second VAD signal (e.g., VAD2) indicating the presence of the target person's (e.g., person 201) speech in the local microphone signal and / or the remote microphone signal. In one or more embodiments, the first VAD signal and the second VAD signal can each include binary values indicating the speech and non-speech of the corresponding user / person. Using the first VAD signal from the OVAD and the second VAD signal from the TVAD, the electronic device 104 can automatically switch between translating the target person's speech and (i) pausing translation during the user's own speech and / or (ii) translating the user's own speech.
[0059] exist Figure 4In the example, VAD 408 operates at electronics device 104 to detect the user's own speech and target speech in a local microphone signal and / or a remote microphone signal. In one or more other embodiments, one or more VADs (e.g., OVAD and / or TVAD) in VAD 408 may operate on the remote microphone signal at audio device 150 before the remote microphone signal is provided from audio device 150 to electronics device 104. In one or more embodiments, the remote microphone signal may comprise a single stream, and VAD 408 (at electronics device 104 or at audio device 150) may mark a time window or time interval of that single stream as including only the user's own speech (e.g., OVAD=1 and TVAD!=1), including only the target speech (TVAD=1 and OVAD!=1), not detecting any speech (e.g., OVAD=0 and TVAD=0), or including both the user's own speech and the target speech (e.g., OVAD=1 and TVAD=1).
[0060] In one or more other embodiments, the remote microphone signal may be a multi-stream signal comprising multiple streams, such as a self-speech stream and a target speech stream, some or all of which may be provided to the translator application 452. For example, the audio device 150 may include an OVAD that detects the user's own speech in the microphone signal from the microphone 206 of the audio device 150, and a TVAD that detects the target person's speech in the microphone signal from the microphone 206 of the audio device 150. Audio device 150 can generate a first audio stream (e.g., its own speech stream) and a second audio stream (e.g., a target speech stream) based on the output of OVAD and TVAD. The first audio stream emphasizes the user's own speech in the microphone signal from microphone 206 of audio device 150 (e.g., by capturing the microphone signal using beamforming operations targeting the expected position of the user's mouth, and / or by canceling ambient noise and / or the target person's speech in the first audio stream). The second audio stream emphasizes the target person's speech in the microphone signal from microphone 206 of audio device 150 (e.g., by capturing the microphone signal using beamforming operations targeting the expected and / or detected position of the target person, and / or by canceling ambient noise and / or the user's own speech in the first audio stream). The first and second audio streams can then be provided from audio device 150 to electronic device 104 as remote microphone signals (e.g., simultaneously as multi-stream signals). Providing multi-stream audio in this way facilitates continued translation of the target person's speech even while the user is speaking (e.g., in contrast to pausing translation, such as providing empty input to the translator application when both the target person and the user are speaking).
[0061] In one or more embodiments, a self-voice stream may be included in the remote microphone signal only if the user's own voice is detected (e.g., using an OVAD at audio device 150). In one or more other embodiments, a self-voice stream may be provided regardless of whether the user's voice is present in the self-voice stream (e.g., and this self-voice stream may be configured (e.g., using user-targeted beamforming and / or ambient noise cancellation) to enhance the user's voice). For example, when buffers with or without the user's own voice are included in the self-voice stream, the time window or time interval of the buffered self-voice stream may be marked to include or exclude the user's own voice. In one or more embodiments, a target voice stream may be included in the remote microphone signal only if the voice of the target person is detected (e.g., using a TVAD at audio device 150). In one or more other embodiments, a target voice stream may be provided regardless of whether the target person's voice is present in the target voice stream (e.g., and this target voice stream may be configured (e.g., using remote-targeted beamforming and / or ambient noise cancellation) to enhance the target person's voice). For example, when buffers containing or excluding the target person's speech are included in the target speech stream, the time window or time interval of the target speech stream buffer can be marked as including or excluding the target person's speech.
[0062] In one or more specific implementations, the remote microphone signal may be provided with or may include metadata for one or more audio streams included therein. As an example, the metadata may include OVAD and / or TVAD markers for time windows and / or time intervals for the one or more audio streams included therein, and / or may include spatial information, such as direction of arrival (DOA) information for time windows and / or time intervals for the one or more audio streams included therein (e.g., determined based on multiple microphone signals from multiple microphones 206 of audio device 150).
[0063] In one or more embodiments, spatial information may be used by electronic device 104 (e.g., by speech generator 454 and / or subsequent spatializer) to generate spatialized translated audio data to be provided to audio device 150. For example, when output by speaker 152 of audio device 150, the spatialized audio output may be perceived by user 101 as originating from the same spatial location (e.g., direction) as the target person (e.g., person 201) (e.g., emanating from or emanating from that spatial location). In some embodiments, the spatialization of the audio output may take into account the movement of the user's head and / or audio device 150 during the delay time between the target speech 324 occurring in the physical environment and the output of the spatialized translation of that speech from the device's speaker (e.g., by tracking the positioning and / or movement of the user's head and modifying the spatialization to remove detected head movements, such that the perceived location of the spatialized audio output remains static and independent of any movement of the user's head). In one or more embodiments, the spatialized audio data may be speech-matched with the speech of person 201. For example, the metadata for the one or more audio streams may include speech feature information (e.g., pitch, timbre, prosody, rhythm, speech rate, etc.) determined by the audio device 150 for the target person, and speech generation may generate spatialized audio output with the same or similar speech features. This spatialization and / or speech matching can help the user naturally perceive their environment during a translation session, such as when multiple people are speaking around the user, when the user's head is moving, and / or when the target person is moving relative to the user.
[0064] In one or more specific implementations, audio device 150 and / or electronic device 104 may use the corresponding determined location of one or more corresponding persons speaking in the physical environment surrounding user 101 (e.g., where the corresponding location is determined using DOA information acquired based on differences between multiple microphone signals from multiple microphones and / or based on other signals such as images of the one or more persons) to filter out environmental speech from local microphone signals and / or remote microphone signals and / or otherwise suppress such environmental speech (e.g., to amplify the speech of one or more target speakers). For example, user 101 may be having a conversation with person 201 in a physical environment where others besides user 101 and person 201 are also speaking (e.g., others are talking on their phones or talking to other people in the physical environment). As another example, user 101 may be listening to conversations between two (or more) other people, whether actively or passively participating in the conversation (e.g., during periods when the user is only temporarily silent, or during periods when two (or more) other people are speaking in another language and may not understand the user speaking in their own language). In these example use cases, it may be beneficial to enhance the speech of a speaker within the dynamically adjustable “field of view” (FOV) of the microphone array and filter out or otherwise suppress the speech of people outside that dynamic FOV. For example, in a use case where user 101 is having a conversation with person 201 in a physical environment where people other than user 101 and person 201 are also speaking, audio device 150 and / or electronic device 104 may dynamically adjust the beam generated by the collaboration of multiple microphones of audio device 150 and / or electronic device 104 so that it moves and / or expands or shrinks and / or changes shape as person 201 moves relative to user 101 (e.g., due to user 101 moving or turning their head and / or due to the movement of person 201 itself) to exclude the speech of people other than person 201. In some examples, person 201 may be identified as the target person based on the fact that person 201’s speech has a higher signal-to-noise ratio than that of other people’s speech and / or based on other signals such as the orientation of the user’s head toward person 201 (e.g., as determined by the image sensor and / or motion / or orientation sensor of audio device 150 and / or electronic device 104), the user’s interaction with person 201 (e.g., speaking toward person 201, looking at the person and / or facing the person), the depth or distance of person 201 from the user, etc.As another example, in a use case where user 101 may be listening to a conversation between two (or more) other people, the beam (e.g., the FOV of the beam) may be enlarged in size and / or shape relative to the beam used when user 101 is speaking to a single other person (e.g., to amplify the speech of the two (or more) people the user is listening to), and may expand or shrink (and / or change shape) as the two (or more) people move away from or toward each other (e.g., to include the position of another person joining the conversation between the two (or more) other people that user 101 is listening to). In this way, whether the speech of one, two, or more people that user 101 is listening to is computer-generated speech translated from another language into the user's language, or the speech is only delivered to the user's ears and not translated by the speakers 206 of audio device 150, dynamic beaming (e.g., dynamic FOV of the beam, including the direction, shape, and / or size of the beam) can help ensure (e.g., to the user's ears and / or to the translation application) that the incoming audio signal includes the speech of one or more people that the user is interested in.
[0065] In one or more embodiments, time windows or time intervals for the local microphone signal and the remote microphone signal may each be buffered at the electronic device 104. A first VAD signal and a second VAD signal may be generated at the electronic device 104 and / or at the audio device 150 for each window or time interval. The condition detector 406 may (e.g., in part) control the input to the translator application 402 (e.g., control switch 404) based on the first VAD signal and the second VAD signal.
[0066] For example, Figure 5A This is a timing diagram illustrating how input controller 400 can (e.g., in part) control inputs to translator application 402 (e.g., control switch 404) based on a first VAD signal and a second VAD signal. In this example, the first VAD signal and the second VAD signal are generated locally at electronic device 104. For example, Figure 5A The timeline 500 shows the voice 324 of user 101, the voice 222 of person 201, the local microphone signal of microphone 214 of electronic device 104, the remote microphone signal of microphone 206 of audio device 150, and the timeline 508 of input buffer 509 from which the translator application can read input audio data for translation.
[0067] exist Figure 5AIn the example, box 507 indicates that the microphone 214 of electronic device 104 is blocked (e.g., electronic device 104 has been determined by condition detector 406 to be in an obstructed condition or state). For this reason, in Figure 5A In the example, none of the buffers in the local audio data buffer 511 from microphone 214 are provided to the input buffer 509 of the translator application. Instead, the buffer 513 of remote audio data from microphone 206 (e.g., audio device 150) is provided to the input buffer 509 of the translator application (e.g., after being transferred from audio device 150 to electronic device 104). However, Figure 5A It is also exemplified that a single stream is provided from microphone 206 of audio device 150. Figure 5A In the example, at some point during the translation operation, no data is available to the translator application's input buffer 509 (e.g., Figure 4 Switch 404 is switched to empty input 410.
[0068] For example, a translation operation using electronic device 104 and audio device 150 may begin at time T1. For example, the translation operation may begin at time T1 in response to a user launching translator application 402, starting a translation session in the system control center of electronic device 104, pressing the “Start” option in the user interface of translator application 402 or otherwise interacting with it, providing user input (e.g., tapping, pressing, pressing and holding, pressing multiple times, providing voice input to the virtual assistant application to start a translation session, or other inputs predefined for initiating a translation session and / or launching translator application 402), and / or in response to electronic device 104 (e.g., VAD 408) automatically detecting speech in a language other than the primary language of user 101 near electronic device 104.
[0069] exist Figure 5A In the example, the timeline includes a portion 510 where no speech is present in the remote microphone signal (e.g., from time T1 to time T2 and from time T5 to time T6), a portion 512 where only the target speech (e.g., speech from person 201 for translation) is present in the remote microphone signal (e.g., from time T2 to time T3 and from time T6 to time T7), an overlapping speech portion 514 where both the user's speech (e.g., speech from user 101) and the target speech (e.g., speech from person 201 for translation) are present in the remote microphone signal (e.g., from time T3 to time T4 and from time T7 to time T8), and a portion 516 where only the user's speech (e.g., speech from user 101) is present in the remote microphone signal (e.g., from time T4 to time T5 and after time T8).
[0070] As shown, during the portion 512 (e.g., from time T2 to time T3 and from time T6 to time T7) in the remote microphone signal where only the target speech (e.g., speech from person 201 for translation) exists, the remote microphone signal is provided (e.g., from buffer 513 storing remote microphone data) to input buffer 509 for translation. Figure 5A As shown, during the period when there is an overlapping speech portion 514 of both the user's speech (e.g., speech from user 101) and the target speech (e.g., speech from person 201 for translation) in the remote microphone signal, a buffer containing the speech 222 of person 201 may not be provided to the input buffer 509 (e.g., by not providing audio data to the input buffer 509, such as by...). Figure 4 In some implementations, translation can be paused when user 101 is speaking simultaneously with person 201. This allows the user to experience less disruptive output from audio device 150 while user 101 is speaking with person 201, thus enabling a more natural conversational interaction. (Examples include providing an empty signal to input buffer 509, or providing only buffer 513 containing user speech 324 to input buffer 509.) As discussed in further detail below, in other specific implementations, while user 101 is speaking, the translation of speech 222 of person 201 can continue (e.g., by continuing to provide a local microphone signal or a remote microphone signal to the translator application for the translation of speech 222 of person 201), and the audio output of the translation from audio device 150 can be changed to be ducked (e.g., reduced in volume relative to other audio content) or blurred (e.g., temporally aliasing or trailing the speech, low-pass filtering, and / or adding reverb and / or echo) to help avoid interrupting user 101 while they are speaking with that person (e.g., while still allowing a reduced amount of translated speech from another person to be delivered to the user). For example, in conjunction with the following Figure 5B As discussed in further detail, multi-stream signals with their own speech stream and target speech stream can facilitate continued translation of speech 222 of person 201 during overlapping speech periods.
[0071] Figure 5AAlso illustrated are some specific implementations and / or use cases of providing a single stream in the removed microphone signal from microphone 206 (e.g., from audio device 150), during the period when the user's own speech is detected (e.g., VAD1=1 or OVAD=1, such as during overlapping speech segments 514 and / or 516), how a remote microphone signal (e.g., to the input buffer 509 of the translator application or a separate input buffer) can be provided to translator application 402 (e.g., to the input buffer 509 of the translator application or a separate input buffer). In these cases, a remote microphone signal (e.g., for translating the user's own speech from the user's language to the language of human 201, such as for output by speaker 216 of electronic device 104) is provided. Figure 5A In this example, because a single stream is provided from audio device 150 (e.g., microphone 206), buffer 513 is used only by the translator application to translate either the user's own speech or the target speech, and translation of the target speech is paused while the user's speech is present (e.g., regardless of whether buffer 513 containing the user's own speech is being translated). In other specific implementations where multiple streams are provided from audio device 150, translation of the target speech can continue even when the user's own speech is present, as discussed in further detail below.
[0072] For example, Figure 5B An example is illustrated where two streams, buffers 587 and 589, are included in a remote microphone signal from microphone 206. In this example, buffer 587 is a self-speech buffer generated from a time window or time interval of its own speech stream from the remote microphone signal, and buffer 589 is a target speech buffer of the target speech stream of the remote microphone signal. Figure 5A compared to, Figure 5B This illustrates how, when a multi-stream signal is provided from audio device 150, a target speech buffer 589, including a representation of the target person's speech (e.g., when the speech 222 of person 201 is present), can be provided to input buffer 509 for translation, even when user speech 324 is present (e.g., even during overlapping speech periods). As shown, a buffer 589 including the user's own speech can also optionally be provided to a translator application for translating the user's own speech into the language of person 201. Figure 5B In the example, for the sake of simplicity in the accompanying drawings, the timeline 504 showing the buffer 511 of the microphone 214 from the electronic device 104 is omitted. However, it should be understood that the buffer 511 of the microphone 214 from the electronic device 104 may be... Figure 5B The example generated, as it is in Figure 5A The same as in the example.
[0073] Figure 6This is a timing diagram illustrating how the input to the translator application 402 (e.g., via the input to the input buffer 509) can be controlled in different ways in different use cases. For example, in Figure 6 middle, Figure 5A The box 507 is removed, indicating that the electronic device 104 has been determined (e.g., by the condition detector 406) to be in an unobstructed state. As an example, the electronic device 104 may be on a table or other surface between user 101 and person 201, where user 101 may be holding the electronic device 104 near person 201, or person 201 may be holding the electronic device 104. Figure 6 In the example, timeline segments 510, 512, 514, and 516 are... Figure 5A Those parts are the same, and when the user's own voice is detected, a buffer containing the user's own voice or an empty input is provided to the input buffer 509, such as... Figure 5A As shown. However, in Figure 6 In the portion 512 (e.g., from time T2 to time T3 and from time T6 to time T7) during which only the target speech (e.g., speech for translation from person 201) exists in the remote microphone signal and / or the local microphone signal, the local microphone signal (e.g., but not) Figure 5A (Example: remote microphone signal) Figure 6 The middle is provided (e.g., by combination) Figure 4 The described input controller 400 is provided to the input buffer 509 for translation. Figure 6 In the example, during portions 514 and 516 where the user's own speech is present in the remote microphone signal and / or the local microphone signal, the remote microphone signal (e.g., the entire remote microphone signal in a single-stream embodiment, or the speech stream of the remote microphone signal in a multi-stream embodiment) can still be provided to the input buffer 509 or other input buffers (e.g., to translate the user's own speech into the language of person 201, such as for output by the speaker 216 of the electronic device 104).
[0074] For example, Figure 7A block diagram illustrates a use case where, in addition to the speech 222 of person 201 being translated into the language of user 101 and sent as translated audio data for output by audio device 150, the user's own speech is also translated into the language of person 201 for output by speaker 216 of electronic device 104. In this example use case, speaker 216 can output a computer-generated translated version of the speech of user 101, and speaker 152 of audio device 150 can output a computer-generated translated version of the speech of person 201, enabling user 101 and person 201 to have (e.g., substantially) real-time conversations with each other, speaking in their respective languages. However, as Figure 7 As shown, a portion of the translated user speech output by the speaker 216 of the electronic device 104 can be received by the microphone 214 of the electronic device 104. In one or more embodiments, the electronic device 104 may include an echo canceller 700. For example, the echo canceller 700 may use known translated user audio data (e.g., and the known output time of the speaker 216 outputting this data) to suppress or cancel the portion of the local microphone signal corresponding to the translated user speech output by the speaker 216. Figure 7 In the example, echo cancellation is performed before the local microphone signal is provided to signal analyzer 407, VAD 408, and switch 404. In one or more other embodiments, echo cancellation may be performed at other states of the processing flow, such as after VAD 408 and before signal analyzer 407 and switch 404 (e.g., making the output of VAD 408 available for selective control of when echo cancellation is performed). For example, in some embodiments, echo cancellation may be performed only during the time period during which the output from speaker 216 overlaps with the translation of the speech of person 201.
[0075] For example, Figure 8 This is a timing diagram illustrating an example use case where the speaker 216 of electronic device 104 outputs translated user speech. Figure 8 The example includes a timeline 800, which shows the speaker 216 of the electronic device 104 outputting translated user speech 802 between time T4 and time T6', where time T6' is... Figure 5A and Figure 6 Between time T6 and time T7. Figure 8 In the example, timeline segments 510, 512, 514, and 516 are... Figure 6 Those parts are the same, and when the user's own speech is detected, only the buffer containing the user's speech 324 or an empty input is provided to the input buffer 509, such as... Figure 5A and Figure 6As shown. However, as shown, echo cancellation can be performed during echo cancellation period 804 (e.g., via...). Figure 7 The echo canceller 700 (as shown in the figure) can perform echo cancellation at all times during the period when the speaker is operated to output the translated user speech. Figure 8 In the example, echo cancellation is performed during echo cancellation period 804, during which the translated user speech output by speaker 216 overlaps with the speech 222 of person 201, which is provided as input (e.g., via input buffer 509) to translator application 402. In this way, power and / or computational resources can be saved by avoiding performing echo cancellation when microphone 214 of electronic device 104 is not providing input to translator application 402.
[0076] exist Figure 8 In the example, such as in Figure 5A and Figure 6 In the example, the remote microphone signal is a single-stream signal, and the translator application only translates the user's own speech 324 or the speech 222 of person 201. However, it should be understood that, in combination Figure 7 and Figure 8 The described echo cancellation operation can also be performed in Figure 5B In the specific implementation, the remote microphone signal is a multi-stream signal, and both the user's own speech 324 and the speech 222 of person 201 are translated during the overlapping speech period.
[0077] In the various examples described herein, the output of the translator application is computer-generated speech spoken in a second language by person 201 in the first language, without any other content. However, in some use cases, the microphone signal may include other content. For example, user 101 may be watching a play, opera, or film in which characters speak in a language the user does not understand, and the play, opera, or film may include other original audio content, such as music or background sounds. In these use cases, the translator application and / or one or more other processes at electronic device 104 may filter out or otherwise suppress other original audio content, suppress or obfuscate other original audio content, or transmit other original audio content unchanged (e.g., so that the user receives only or primarily the full audio of the play, opera, or film in which the spoken words being translated are modified). In these use cases, as discussed herein, the translated speech of one or more characters or performers in the play, opera, film, etc., may be spatialized and / or speech-matched according to the corresponding position and / or speech of the speaker among those characters or performers in the play, opera, film, etc. In other examples, the speech to be translated may be derived from a recording of a person’s previous speech or other speech received through a loudspeaker (e.g., live speech, such as announcements at train or bus stations, airport announcements, sporting event announcements, concert sounds, etc. while traveling).
[0078] As described herein, electronic device 104 may selectively provide a local microphone signal, a remote microphone signal, or no input to translator application 402, including providing a remote microphone signal to the translator application when the device status indicates that the electronic device is in an obstructed state. However, in some use cases, such as in noisy environments, or when person 201 speaks softly or moves away from audio device 150 and / or electronic device 104 (e.g., causing signal analyzer 407 to determine high SPL or low SNR), translation may be limited or unavailable. Therefore, in some use cases, electronic device 104 (e.g., translator application 402) may provide notification to user 101 to move electronic device 104 closer to person 201 who needs translation (e.g., in some use cases, including removing electronic device 104 from a container), making the local microphone signal available for translation.
[0079] For example, Figure 9 An example is a notification 902 that can be provided by the user interface 900 of the translator application 402 (e.g., a notification including text such as "Move this device closer to the person who needs translation"). Figure 9In this example, notification 902 is provided by display 904 of electronic device 104. However, this is merely illustrative, and in various specific implementations, notification may be provided by electronic device 104 and / or one or more other output components of audio device 150 (e.g., audio notification, haptic notification, or any other notification output).
[0080] In response to notification 902, user 101 may move electronic device 104 closer to person 201 to improve reception of speech 222 for translation by person 201. In one or more other embodiments, user 101 may move one or more microphones closer to another device (e.g., a smartwatch, tablet, one of a pair of earbuds corresponding to audio device 150, or any other device communicatively coupled to the device performing the translation operation and / or associated with the same account as that device), and input controller 400 may provide microphone signals from one or more microphones of that other device to translator application 402 for translating the speech represented therein.
[0081] In various specific implementations, the user interface 900 may provide additional functionality for users interacting with the translator application 402. For example, the user interface 900 may include selectable options for starting a translation session, selectable options for choosing the language of the speech to be translated and / or the language the speech is to be translated into. As discussed herein, the translator application 402 may manage the downloading and / or storage of one or more translation models for one or more selected languages at the electronic device 104. In other examples, the language to be translated may be automatically detected by the electronic device 104, and the corresponding language model may be automatically downloaded (e.g., without user input) (or automatically accessed if previously downloaded). Automatic detection of the language to be translated may include: automatically detecting speech in languages other than the user's language (e.g., as previously selected by the user or as automatically determined by the electronic device 104 based on the user's own speech), and determining the language of that speech. Automatic detection of the language to be translated may allow conversations with multiple people in multiple languages (e.g., in a group or roundtable discussion where multiple people are speaking multiple different corresponding languages). In this example, translator application 402 may translate each of the different languages detected in speech around user 101 into user 101's language when electronic device 104 performs the microphone switching (e.g., dynamic input control) described herein. In one or more specific implementations, regardless of whether multiple speakers are speaking in multiple different languages or in a common language, the audio output of the translated speech may be spatialized (e.g., by the coordinated operation of multiple speakers 152 of audio device 150, using the location and / or direction of arrival determined by multiple microphones of audio device 150 and / or electronic device 104), such that each person's translated speech is perceived by user 101 as originating from that person's direction. As discussed herein, this spatialization may take into account any movement of the user's head during the possible delay time between each person's speech and the output of the translation of that speech (e.g., such that the speech is output at the location where the speech occurred).
[0082] Figure 10 A flowchart illustrating an example process 1000 for microphone switching for translation, implemented according to a specific embodiment of the subject matter, is provided. For illustrative purposes, this document primarily references and incorporates... Figures 1 to 9 The process 1000 is described using the audio device 150 and electronic device 104. However, the process 1000 is not limited to the combination of... Figures 1 to 9The described audio device 150 and electronic device 104, and one or more frames (or operations) of process 1000, may be performed by one or more other components of other suitable devices (including electronic device 110, electronic device 115, and / or server 120). Further for illustrative purposes, some frames of process 1000 are described herein as occurring sequentially or linearly. However, multiple frames of process 1000 may occur in parallel. Furthermore, the frames of process 1000 need not be performed in the order shown, and / or one or more frames of process 1000 need not be performed and / or may be replaced by other operations.
[0083] like Figure 10 As illustrated, at block 1002, a first microphone signal (e.g., a local microphone signal) can be acquired (e.g., at input controller 400) from a first microphone (e.g., microphone 214) of a first device (e.g., electronic device 104). In one or more embodiments, multiple first microphone signals can be acquired from multiple first microphones at the first device.
[0084] At block 1004, the first device (e.g., input controller 400) may acquire (e.g., via wired or wireless connection) a second microphone signal (e.g., a remote microphone signal) from the second microphone (e.g., microphone 206) of the second device (e.g., audio device 150, which in some examples may be implemented as earbuds, a pair of earbuds, headphones, or head-mountable devices such as smart glasses). For example, the first device may include communication circuitry (e.g., communication circuitry 215) for receiving wired or wireless transmissions from and / or sending wired or wireless transmissions to the second device (e.g., control signals, translated audio data, and / or media content output by the speakers of the second device).
[0085] At box 1006, a first device (e.g., input controller 400 and / or condition detector 406) may control audio input to a translator application (e.g., translator application 402) at the first device based on a first microphone signal and a second microphone signal. The translator application is configured to generate audio translation (e.g., translated audio data) of speech content (e.g., speech 222) in the audio input from a first language (e.g., that of a person 201 other than user 101 of the first and second devices) to a second language (e.g., that of user 101).
[0086] For example, at least one of the first microphone signal or the second microphone signal may include a representation of the speech (e.g., speech 222) of a person speaking the first language (e.g., person 201) other than the user of the first device (e.g., user 101), and the process 1000 may also include generating an audio output (e.g., translated audio data) in response to a controlled audio input using a translator application, the audio output including a representation of speech (e.g., computer-generated speech) spoken in a second language. For example, the speech spoken in the second language may be computer-generated speech, and the first device may select a computer-generated speech from a plurality of computer-generated speech stored at the first device. For example, the first device may select a computer-generated speech from the plurality of computer-generated speech based on the speech of the person speaking the first language (e.g., speech 222). For example, the first device may determine that the speech of the person speaking the first language is a low-pitched speech, a high-pitched speech, and / or a speech with a specific accent (e.g., corresponding to the first language). The first device may select from the plurality of computer-generated voices a computer-generated voice that sounds similar to the voice of a person speaking the first language (e.g., a high-pitched voice, a low-pitched voice, and / or has the same or similar accent).
[0087] In one or more use cases, controlling audio input at block 1006 may include: selecting either a first microphone signal or a second microphone signal as audio input based at least in part on the signal-to-noise ratio (SNR) of the first microphone signal (e.g., via condition detector 406, such as via operation switch 404). (e.g., if the SNR of one of the microphone signals is low, the device may select the other microphone signal for input to the translator application). In one or more other use cases, controlling audio input may include: determining, at least in part on whether any audio input is provided to the translator application based on the determination of whether the second microphone signal contains the voice (e.g., voice 324) of a user (e.g., user 101) of the first device (e.g., whether VAD1=1). For example, determining whether to provide any audio input to the translator application based at least in part on the determination of whether the second microphone signal contains the voice of a user of the first device (e.g., user 101). Figure 5A , Figure 6 and / or Figure 8 The first determination of the absence of the user's voice in the second microphone signal (between T2 and T3 or between T6 and T7) involves providing the translator application with at least one of the first microphone signal or the second microphone signal as audio input, and based on a second time (e.g., at... Figure 5A , Figure 6 and / or Figure 8A second determination of the presence of the user's speech from the first device in the second microphone signal (between T3 and T5 or between T7 and T8) determines that no audio input will be provided to the translator application (e.g., providing empty input 410). In this example, no translation is generated while the user is speaking. However, in other examples, translation may continue even while the user is speaking (e.g., by providing the translator application with at least one of the first or second microphone signals as audio input), and the translated audio output may be ducked or obscured so that the user can hear a weakened version of the translation while the user is speaking.
[0088] For example, in one or more embodiments, the second microphone signal from the second device may include a multi-stream signal comprising: a self-speech stream configured to enhance the representation of the speech of a user of the first device (e.g., user 101); and a target speech stream configured to enhance the representation of the speech (e.g., speech 222) of a person other than the user of the first device (e.g., person 201). In this example, selecting either the first microphone signal or the second microphone signal as audio input based at least in part on the signal-to-noise ratio (SNR) of the first microphone signal may include selecting either the target speech stream of the second microphone signal or the first microphone signal as audio input based at least in part on the SNR of the first microphone signal, regardless of whether the user's speech is represented in the first microphone signal or in the self-speech stream of the second microphone signal. For example, at least one of the target speech stream of the second microphone signal or the first microphone signal may include a representation of the speech of a person speaking in the first language other than the user of the first device, and process 1000 may also include using the first device to generate a spatialized audio output based on the audio input, the spatialized audio output including speech speaking in the second language and configured to be perceived as coming from the location of a person other than the user of the first device.
[0089] In one or more use cases, controlling the audio input may include selecting (e.g., via condition detector 406 and / or switch 404) a first microphone signal or a second microphone signal as the audio input based at least in part on the condition of the first device (e.g., an obstructed or unobstructed condition). For example, the first device (e.g., condition detector 406) may use sensor signals from at least one sensor (e.g., sensor 218) at the first device to determine the condition of the first device. For example, the at least one sensor may include a proximity sensor or orientation sensor (e.g., an IMU sensor of IMU 411, such as a gyroscope, accelerometer, or magnetometer). For example, determining the condition of the first device may include determining whether the first microphone of the first device is obstructed. In one or more embodiments, the first device (e.g., condition detector 406) may determine the condition of the first device based at least in part on user input that initiates a translator application and / or a translation session (e.g., if a user provides input to the first device to initiate a translator application, the first device may determine that the first device is likely unobstructed because the user is interacting with the first device). In one or more use cases, controlling the audio input may include selecting (e.g., status detector 406 and / or switch 404) a first microphone signal or a second microphone signal as the audio input, based at least in part on the state of the translator application at the first device. For example, the state of the translator application may include the state of the translator application's user interface (e.g., user interface 900) (e.g., foreground state, active state, background state, minimized state). In one or more use cases, the first device may provide (e.g., towards the person requiring translation) a prompt (e.g., notification 902, or audio or haptic notification) for moving the first device, based at least in part on a second microphone signal from the second device.
[0090] In one or more use cases, the translator application may also generate a translation (e.g., translated user audio signal) of the speech (e.g., speech 324) of a user (e.g., user 101) of the first device from the second language to the first language based on a second microphone signal from the second device, for output by a speaker (e.g., speaker 216) of the first device. The first device (e.g., echo canceller 700) may also perform echo cancellation on the first microphone signal before providing the translator application with the first microphone signal from the first device for translating the speech (e.g., speech 222) of a person other than the user (e.g., person 201) from the first language to the second language.
[0091] According to various aspects of the present subject matter, an electronic device (e.g., electronic device 104) is provided, comprising: a first microphone (e.g., microphone 214); a memory (e.g., memory 212) storing a translator application (e.g., translator application 402) configured to generate audio translations of speech content in audio input from a first language to a second language; and one or more processors configured to: acquire a first microphone signal from the first microphone; acquire a second microphone signal from a second microphone (e.g., microphone 206) of an audio device (e.g., audio device 150) separate from the electronic device; and control audio input to the translator application based on the first and second microphone signals. The electronic device may also include communication circuitry (e.g., communication circuitry 215). At least one of the first microphone signal or the second microphone signal may include a representation of the speech (e.g., speech 324) of a person speaking in a first language (e.g., person 201) other than the user of the electronic device (e.g., user 101), and the one or more processors may be further configured to: wirelessly receive the first microphone signal from the audio device using communication circuitry; generate an audio output in response to a controlled audio input using a translator application, the audio output including computer-generated speech speaking in a second language; and provide the audio output to the audio device using communication circuitry so that the computer-generated speech is output by the speaker (e.g., speaker 152) of the audio device.
[0092] In one or more embodiments, the memory stores speech features (e.g., pitch, tone, accent, etc.) of a plurality of computer-generated speech, and the one or more processors may be further configured to select computer-generated speech from the plurality of computer-generated speech based at least in part on human speech in a first microphone signal or a second microphone signal (e.g., by selecting computer-generated speech that has speech features similar to those of a person speaking a first language).
[0093] In one or more embodiments, the processors may be configured to control the audio input by selecting either a first microphone signal or a second microphone signal as the audio input, based at least in part on the sound pressure level (SPL) in the first microphone signal. As another example, the processors may be configured to control the audio input by determining, at least in part, whether to provide any audio input (e.g., or empty input 410) to the translator application based on the determination of whether the second microphone signal contains the voice (e.g., voice 324) of a user (e.g., user 101) of an electronic device.
[0094] According to various aspects of the subject matter, a system is provided that includes an audio device (e.g., audio device 150) including a first microphone (e.g., microphone 206); and an electronic device (e.g., electronic device 104) including a second microphone (e.g., microphone 214). The audio device may be configured to send a first microphone signal from the first microphone to the electronic device, and the electronic device may be configured to: acquire a second microphone signal from the second microphone; and control audio input to a translator application at the electronic device based on the first and second microphone signals, the translator application being configured to generate an audio translation of speech content in the audio input from a first language to a second language.
[0095] As discussed in this paper, in some use cases, the translation operations described herein can be used to translate performances (e.g., operas, plays, musicals, concerts, films, etc.). For example, Figure 11 This is a flowchart of an example process 1100 for performance translation based on one or more specific implementations of the subject matter technology. For illustrative purposes, this document primarily refers to and incorporates... Figures 1 to 9 The process 1100 is described using the audio device 150 and electronic device 104. However, the process 1100 is not limited to the combination of... Figures 1 to 9 The described audio device 150 and electronic device 104, and one or more frames (or operations) of process 1100, may be performed by one or more other components of other suitable devices (including electronic device 110, electronic device 115, and / or server 120). Further for illustrative purposes, some frames of process 1100 are described herein as occurring sequentially or linearly. However, multiple frames of process 1100 may occur in parallel. Furthermore, the frames of process 1100 need not be performed in the order shown, and / or one or more frames of process 1100 need not be performed and / or may be replaced with other operations.
[0096] At box 1102, audio input corresponding to the performance (e.g., one or more microphones 206 and / or one or more microphones 214) can be acquired during the performance using at least one microphone of a device (e.g., audio device 150 and / or electronic device 104). For example, user 101 may be an audience member of the performance and wear and / or possess audio device 150 and / or electronic device 104. The audio input may include speech in a first language and ambient sounds. For example, speech may include speech from characters in the performance, and ambient sounds may include at least one of the following: other speech from other characters in the performance, or music.
[0097] At box 1104, during a performance, a translation of the speech in a second language can be obtained, for example, from the translator application 402 described herein (e.g., generated locally on the device or obtained from another device). For example, the first device may include an audio device, such as in-ear headphones, and obtaining the translation of the speech in a second language may include: providing at least a portion of audio input from the in-ear headphones (e.g., a single stream of a target speech buffer or multiple streams of multiple speech buffers) to an accompanying device; and receiving computer-generated speech from the accompanying device, the computer-generated speech stating the translation of the speech in a second language. In another example, the audio device may generate the translation locally on the audio device.
[0098] In one or more embodiments, process 1100 may further include removing (e.g., filtering, suppressing, or eliminating) speech in the first language from the audio output (e.g., speech in the first language may be removed from the audio output). In one or more other embodiments, the audio output may include a modified version of speech in the first language (e.g., process 1100 may include modifying speech in the first language in the audio output). For example, process 1100 may also include generating (e.g., at this device or other devices) a modified version of speech in the first language by reducing the volume of speech in the first language and / or by obfuscating speech in the first language. For example, the modified version of speech in the first language may include a reduced-volume version and / or an obfuscated version of speech in the first language.
[0099] At box 1106, during a performance and using at least one speaker of the device (e.g., one or more speakers 152), audio output may be output. The audio output may include ambient sounds and computer-generated speech that speaks a translation of the speech in a second language (e.g., obtained from a translator application as shown at box 1104).
[0100] In one or more embodiments, the speech in the first language may be spoken by a character in a performance located at a position in the physical environment of the device (e.g., a position on a stage or a movie screen), and process 1100 may further include determining the position of the character in the physical environment of the device prior to output; and spatializing the computer-generated speech uttering the translation of the speech based on that position. For example, outputting audio output using the at least one speaker at block 1106 may make the spatialized computer-generated speech uttering the translation of the speech perceptible as emanating from or originating from the character's position. In one or more embodiments, process 1100 may further include: determining one or more features of the character's speech prior to output; and performing speech matching of the computer-generated speech uttering the translation of the speech with the one or more features of the character's speech. For example, the computer-generated speech uttering the translation of the speech may be spatialized based on position, and / or the computer-generated speech uttering the translation of the speech may be speech-matched with one or more characteristics of the character's speech.
[0101] As discussed in this paper, among various use cases, it may be beneficial to enhance the speech of one or more people speaking within a dynamically adjustable “field of view” (FOV) of a microphone array. This could be useful for generating audio input for translator applications and / or for electronically transmitting speech (e.g., untranslated speech) to a user’s ears via headphones or in-ear headphones. Figure 12A and Figure 12B Examples of dynamic control of the field of view of a beam used for processing microphone signals are illustrated in one or more specific embodiments of the subject matter. Figure 12A and Figure 12B The example illustrates a top-down schematic of user 101 who is having a conversation with person 201 (e.g., this might be determined by audio device 150 and / or electronic device 104 based on user 101's attention vector 1200 and person 201's attention vector 1202 pointing or approximately pointing at each other, such as by using a camera with a visual FOV 1208 and / or using the microphone of audio device 150 to detect the direction of arrival of person 201's speech). In this example, user 101's attention vector 1200 and person 201's attention vector 1202 may indicate the corresponding direction that user 101 and person 201 are looking or facing and / or the corresponding direction that user 101 and person 201 are speaking. In this example, another person 1204 is present in the physical environment of user 101 and person 201 and is speaking and / or facing in a direction 1205 away from user 101. For example, the other person 1204 may be having a conversation with one or more additional people.
[0102] like Figure 12A As shown, when user 101 is looking in the direction of person 201 and / or speaking in that direction, the field of view 1206 of microphone 206 of audio device 150 (e.g., defined by angular boundaries, such as extending between a first angular limit θ1 and a second angular limit θ2) may include the position of person 201. Figure 12B An example is shown of how the FOV 1206 can be dynamically adjusted based on the position of person 201, the position of another person 1204, and the movement of the audio device 150 (e.g., as indicated by changes in the attention vector 1200). Figure 12B In the example, in response to device movement, the size of the field of view (FOV) 1206 (e.g., the angular size Δθ between the first angular constraint θ1 and the second angular constraint θ2) and the center of the FOV 1206 are adjusted such that, throughout the entire time period (e.g., when user 101 is conversing with person 201), person 201 remains within the FOV 1206, while the position of person 1204 is excluded from the field of view 1206. Figure 12B In the example, due to the rotation of the device in the direction of the angle limit θ2 and due to the presence of a person 1204 who is not participating in the conversation with user 101, the second angle limit θ2 increases more than the first angle limit θ1. In this way, the beam with FOV 1206 can be used to process the microphone signal to amplify the speech of person 201 and suppress the speech of person 1204.
[0103] Figure 13A and Figure 13B This illustrates aspects of another example of dynamic control of the field of view of a beam used for processing microphone signals, according to one or more specific implementations of the subject matter. Figure 13A and Figure 13B In the example, person 201 and person 1304 are having a conversation, and user 101 is participating in the conversation (e.g., this could be detected by audio device 150 and / or electronic device 104 based on user 101's attention vector 1200, person 201's attention vector 1202, and / or person 1304's attention vector 1305, such as by using a camera with a visual FOV 1208 and / or using the microphone of audio device 150 to detect the direction of arrival of the speech of person 201 and person 1304). User 101 can be an active participant in the conversation between person 201 and person 1304, or can be a passive listener in the conversation (with the explicit permission of person 201 and person 1304).
[0104] like Figure 13A As shown, when user 101 is looking in a direction generally toward person 201 and person 1304 and / or is speaking in that direction, the field of view 1206 of the microphone 206 of the audio device 150 may include the positions of person 201 and person 1304. Figure 13B An example is shown of how the FOV 1206 can be dynamically adjusted based on the position of person 201, the position of another person 1304, and the movement of the audio device 150 (e.g., as indicated by changes in the attention vector 1200). Figure 13B In the example, in response to device movement, the size (e.g., the angular size Δθ between the first angular limit θ1 and the second angular limit θ2) and center of FOV 1206 are adjusted such that people 201 and 1304 remain within FOV 1206 throughout the entire time period (e.g., when user 101 attends a conversation between people 201 and 1304). Figure 13B In the example, due to the rotation of the device in the direction of the angle limit θ2 and the operation of keeping person 1304 within FOV 1206, the second angle limit θ2 increases more than the first angle limit θ1. In this way, the beam with FOV 1206 can be used to amplify the speech of person 201 and person 1304 and suppress sounds and / or other speech outside FOV 1206.
[0105] Figure 14 This is a flowchart of an example process 1400 for dynamic control of the field of view of a beam used for processing microphone signals, according to one or more specific embodiments of the subject matter. For illustrative purposes, this document primarily refers to the combined... Figures 1 to 9 and Figures 12A to 13B The process 1400 is described using the audio device 150 and electronic device 104. However, the process 1400 is not limited to the combination of... Figures 1 to 9 and Figures 12A to 13B The described audio device 150 and electronic device 104, and one or more frames (or operations) of process 1400, may be performed by one or more other components of other suitable devices (including electronic device 110, electronic device 115, and / or server 120). Further for illustrative purposes, some frames of process 1400 are described herein as occurring sequentially or linearly. However, multiple frames of process 1400 may occur in parallel. Furthermore, the frames of process 1400 need not be performed in the order shown, and / or one or more frames of process 1400 need not be performed and / or may be replaced by other operations.
[0106] At box 1402, a first position of a first person (e.g., person 201) and a second position of a second person (e.g., person 1204 or person 1304) may be determined relative to a device (e.g., audio device 150 and / or electronic device 104) having multiple microphones (e.g., microphone 206 and / or microphone 214). Determining the first and / or second positions may include: determining the first and / or second positions based on the direction of arrival of the voices of the first and / or second persons at the device and / or based on one or more images of the physical environment of the first and second persons (e.g., captured by one or more cameras of the device and / or another device).
[0107] At box 1404, multiple corresponding microphone signals can be acquired from the plurality of microphones. Microphone signals can be acquired when one or both of the first and second persons are speaking in one or more languages.
[0108] At block 1406, during the time period in which the plurality of corresponding microphone signals are acquired and the first voice of a first person is present among the plurality of corresponding microphone signals, the size (e.g., FOV 1206) of the beam used to process the plurality of corresponding microphone signals can be dynamically determined (e.g., by the device having microphones or by an accessory device to which the microphone signals are provided) based on the first position, the second position, and the motion of the device (e.g., as determined by one or more inertial sensors of the device, such as accelerometers, gyroscopes, and / or magnetometers). In one or more embodiments, the orientation or direction of the beam can also be dynamically determined.
[0109] At box 1408, during this time period, a beam with a dynamically determined field of view can be used to process (e.g., by a device with a microphone and / or by an accessory device to which the microphone signal is provided) the plurality of corresponding microphone signals to generate audio output data including a version of the first person's speech.
[0110] At box 1410, audio output data, including a first voice of a first person, may be provided for output by a speaker (e.g., a speaker of a device with a microphone) during that time period. The audio output data may be generated at the device with the speaker and / or by another device (e.g., a companion device) and provided to the device with the speaker. In one or more embodiments, the audio output data may be spatialized for spatial output by multiple speakers.
[0111] In some examples (e.g., in) Figure 12A and Figure 12BIn the example, the field of view includes the first person's first position throughout the entire time period, and dynamically determining the size of the beam's field of view based on the first position, the second position, and the device's motion includes: expanding the size of the field of view in response to the device's motion while excluding the second position from the field of view. For example, dynamically determining the size of the beam's field of view based on the first position, the second position, and the device's motion may include: dynamically determining the size of the beam's field of view based on the first position, the second position, the device's motion, and indications that the device's user is conversing with the first person (e.g., attention vector 1200, attention vector 1202, images of the user and / or the first person, and / or the direction of arrival of the first person's speech). In some examples, the second person is conversing with a third person during the time period.
[0112] In some examples (e.g., in) Figure 13A and Figure 13B In the example, dynamically determining the size of the beam's field of view based on the first position, the second position, and the device's movement may include: expanding the field of view in response to the device's movement while keeping the first and second positions within the field of view. For example, the first and second people may be conversing with each other and / or with the device's user.
[0113] In some examples, the first person's first speech version includes a direct reproduction of the first person's first speech (e.g., untranslated). In other examples, the first person's first speech version may include computer-generated speech uttering a translation of the first person's first speech.
[0114] Various aspects of this subject matter can improve the efficiency, accuracy, and / or latency of electronically generated speech translation, which is generated substantially in real time with the speech itself. For example, switching the input of a translator application between multiple microphones on one or more devices located at different physical locations relative to the speaker can help provide a more accurate translation. As another example, pausing the translation operation (e.g., by pausing the input signal to the translator application while the user of the electronic device is speaking) can help reduce the processing and / or battery power consumed by the electronic device. Therefore, these and other aspects of this subject matter, when implemented by one or more electronic devices (such as smartphones and one or more audio devices), can provide improvements to the functionality of other technologies (e.g., electronic translation) and the computing devices on which those other technologies are implemented.
[0115] As described above, one aspect of the present invention is the collection and use of data available from specific and legitimate sources for performing microphone switching for translation. This disclosure contemplates that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data may include voice samples, voice profiles, demographic data, location-based data, online identifiers, telephone numbers, email addresses, home addresses, biometric data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information, physical activity information, respiratory information, heart rate information, etc.), date of birth, or any other personal information.
[0116] This disclosure recognizes that the use of such personal information data in the techniques of this invention can benefit users. For example, personal information data can be used to perform microphone switching for translation.
[0117] This disclosure anticipates that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, it is expected that such entities will implement and consistently apply privacy practices generally recognized as meeting or exceeding industry or governmental requirements for protecting user privacy. Such information regarding the use of personal data should be prominently displayed and readily accessible to the user, and should be updated as the collection and / or use of data changes. Users' personal information should be collected only for lawful use. Furthermore, such collection / sharing should only occur after receiving user consent or other lawful grounds provided for in applicable law. Additionally, such entities should consider taking any necessary steps to protect and safeguard the right to access such personal information data and to ensure that other entities with access to personal information data comply with the privacy policies and procedures of other entities. Furthermore, such entities may be subject to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. Moreover, policies and practices should be tailored to the specific types of personal information data collected and / or accessed, and made applicable to applicable laws and standards, including specific jurisdictional considerations that may be used to impose higher standards. For example, in the United States, the collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly.
[0118] Regardless of the foregoing, this disclosure also anticipates implementation schemes that allow users to selectively block the use or access to personal information data. That is, this disclosure anticipates hardware and / or software components to prevent or block access to such personal information data. For example, in the case of microphone switching for translation, the subject matter technology can be configured to allow users to opt-in or opt-out during or at any time after registering for the service to participate in the collection and / or sharing of personal information data. In addition to providing opt-in and opt-out options, this disclosure also anticipates providing notifications related to access to or use of personal information. For example, users may be notified when downloading an application that their personal information data will be accessed, and then reminded again just before the application accesses the personal information data.
[0119] Furthermore, the intent of this disclosure is that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by restricting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level or at a level insufficient for facial recognition), controlling how data is stored (e.g., aggregating data among users), and / or other methods such as differentiated privacy.
[0120] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it is also contemplated that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present invention will not become inoperable due to the absence of all or part of such personal information data.
[0121] Figure 15 An electronic system 1500 is illustrated, which can be used to implement one or more specific embodiments of the subject matter. The electronic system 1500 may be as follows: Figure 1 The audio device 150, electronic device 104, electronic device 110, electronic device 115, and / or server 120 shown are included, and / or may be part of them. Electronic system 1500 may include various types of computer-readable media and interfaces for various other types of computer-readable media. Electronic system 1500 includes a bus 1508, one or more processing units 1512, system memory 1504 (and / or buffers), ROM 1510, persistent storage device 1502, input device interface 1514, output device interface 1506, and one or more network interfaces 1516, or subsets and variations thereof.
[0122] Bus 1508 generally represents all system buses, peripheral buses, and chipset buses that communicatively connect multiple internal devices of electronic system 1500. In one or more embodiments, bus 1508 communicatively connects one or more processing units 1512 to ROM 1510, system memory 1504, and persistent storage device 1502. One or more processing units 1512 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed in this subject matter. In different embodiments, one or more processing units 1512 may be a single processor or a multi-core processor.
[0123] ROM 1510 stores static data and instructions required by one or more processing units 1512 and other modules of electronic system 1500. On the other hand, persistent storage device 1502 can be a read-write memory device. Persistent storage device 1502 can be a non-volatile memory cell that stores instructions and data even when electronic system 1500 is powered off. In one or more embodiments, mass storage devices (such as magnetic disks or optical disks and their corresponding disk drives) can be used as persistent storage device 1502.
[0124] In one or more embodiments, a removable storage device (such as a floppy disk, flash drive, and its corresponding disk drive) may be used as persistent storage device 1502. Like persistent storage device 1502, system memory 1504 may be a read-write memory device. However, unlike persistent storage device 1502, system memory 1504 may be volatile read-write memory, such as random access memory. System memory 1504 may store any instructions and data that one or more processing units 1512 may need during operation. In one or more embodiments, the processes disclosed in this subject matter are stored in system memory 1504, persistent storage device 1502, and / or ROM 1510 (each implemented as a non-transitory computer-readable medium). One or more processing units 1512 retrieve instructions to be executed and data to be processed from these various memory units to execute the processes of one or more embodiments.
[0125] Bus 1508 is also connected to input device interface 1514 and output device interface 1506. Input device interface 1514 enables a user to communicate information to electronic system 1500 and select commands. Input devices that can be used with input device interface 1514 may include, for example, an alphanumeric keypad and pointing devices (also known as "cursor control devices"). Output device interface 1506 may, for example, enable the display of images generated by electronic system 1500. Output devices that can be used with output device interface 1506 may include, for example, printers and display devices such as liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic light-emitting diode (OLED) displays, flexible displays, flat panel displays, solid-state displays, projectors, or any other device for outputting information. One or more embodiments may include a device that functions as both an input device and an output device, such as a touchscreen. In these embodiments, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, verbal, or tactile input.
[0126] Finally, as Figure 15 As shown, bus 1508 also couples electronic system 1500 to one or more networks and / or one or more network nodes via one or more network interfaces 1516, such as Figure 1 The electronic device 150 is shown. Thus, the electronic system 1500 may be part of a computer network (such as a LAN, wide area network (“WAN”), or intranet), or may be part of a network within a network (such as the Internet). Any or all components of the electronic system 1500 may be used in conjunction with the subject matter disclosed herein.
[0127] These functions can be implemented in computer software, firmware, or hardware. The technology can be implemented using one or more computer program products. Programmable processors and computers can be included in or packaged as mobile devices. These processes and logic flows can be executed by one or more programmable processors and one or more programmable logic circuits. General-purpose and special-purpose computing devices, as well as storage devices, can be interconnected via communication networks.
[0128] Some specific implementations include electronic components such as microprocessors, storage devices, and memories that store computer program instructions in machine-readable or computer-readable media (also referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Examples of such computer-readable media include RAM, ROM, read-only optical discs (CD-ROM), recordable optical discs (CD-R), rewritable optical discs (CD-RW), read-only digital versatile optical discs (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and / or solid-state hard disk drives, read-only and recordable Blu-ray drives. ® Disks, high-density optical discs, any other optical or magnetic media, and floppy disks. Computer-readable media may store computer programs that can be executed by at least one processing unit and include a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as machine code generated by a compiler, and files that include higher-level code that can be executed by a computer, electronic component, or microprocessor using an interpreter.
[0129] While the above discussion primarily concerns microprocessors or multi-core processors that execute software, some implementations are performed by one or more integrated circuits such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some implementations, such integrated circuits execute instructions stored on the circuit itself.
[0130] As used in this specification and any claim of this application, the terms "computer," "server," "processor," and "memory" refer to electronic or other technical devices. These terms exclude persons or groups of persons. For the purposes of this specification, the terms "displayed" or "being displayed" mean displayed on an electronic device. As used in this specification and any claim of this application, the terms "computer-readable medium" and "computer-readable medium" are strictly limited to tangible, touchable objects that store information in a form readable by a computer. These terms do not include any wireless signals, wired download signals, or any other transient signals.
[0131] To provide interaction with the user, specific embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, verbal, or tactile input. Furthermore, the computer can interact with the user by transmitting documents to and receiving documents from the device used by the user; for example, by transmitting a webpage to a web browser in response to a request received from a web browser on the user's client device.
[0132] Implementations of the subject matter described in this specification can be implemented in a computing system that includes backend components, such as a data server, or middleware components, such as an application server, or frontend components, such as a client computer with a graphical user interface or web browser through which a user can interact with a specific implementation of the subject matter described in this specification, or any combination of one or more such backend components, middleware components, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), interconnected networks (e.g., the Internet) and peer-to-peer networks (e.g., self-organizing peer-to-peer networks).
[0133] A computing system may include clients and servers. Clients and servers are generally geographically isolated but can interact via a communication network. The client-server relationship is established by computer programs running on respective computers and having a client-server relationship with each other. In some implementations, the server sends data (e.g., HTML pages) to the client device (e.g., to display data to a user interacting with the client device and to receive user input from the user interacting with the client device). Data generated at the client device (e.g., the result of user interaction) can be received from the client device at the server.
[0134] Those skilled in the art will recognize that the various exemplary blocks, modules, elements, components, methods, and algorithms described herein can be implemented as electronic hardware, computer software, or a combination of both. To illustrate this interchangeability between hardware and software, the various exemplary blocks, modules, elements, components, methods, and algorithms have been generally described above in terms of functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The functionality may be implemented differently for each specific application. Various components and blocks may be arranged differently (e.g., in different orders or partitioned in different ways), all without departing from the scope of the subject matter.
[0135] It should be understood that the specific order or hierarchical structure of the steps in the process disclosed in this invention is an example of the exemplary method. Based on design preferences, it should be understood that the specific order or hierarchical structure of the steps in the process can be rearranged. Some steps in the process can be performed simultaneously. The appended method claims present elements of various steps in the sample order and are not intended to limit one to the presented specific order or hierarchical structure.
[0136] The preceding description is provided to enable any person skilled in the art to practice the various aspects described herein. The preceding description provides various examples of the subject matter, and the subject matter is not limited to these examples. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, this claim is not intended to be limited to the aspects shown herein, but rather to be consistent with the language of the claim, wherein reference to the singular form of an element is not intended to mean “only one,” but rather “one or more,” unless specifically indicated. Unless otherwise specifically stated, the term “some” means one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are for convenience only and do not limit the invention described herein.
[0137] As used herein, the term "website" can include any aspect of a website, including one or more web pages, one or more servers used to host or store web-related content, etc. Therefore, the term "website" is used interchangeably with the terms "web page" and "server." The predicates "configured to," "operable to," and "programmed to" do not imply any specific tangible or intangible modification to a particular subject but are intended to be used interchangeably. For example, a component or a processor configured to monitor and control operations may also mean that the processor is programmed to monitor and control operations or that the processor is operable to monitor and control operations. Similarly, a processor configured to execute code can be interpreted as a processor programmed to execute code or operable to execute code.
[0138] As used herein, the term "automatic" can include execution by a computer or machine without user intervention; for example, by instructions in response to a predicate action of a computer or machine or other initiating mechanism. The word "example" is used herein to mean "serving as an example or illustration." No aspect or design described herein as an "example" is necessarily to be construed as preferred or advantageous over other aspects or designs.
[0139] Phrases such as "aspect" do not imply that this aspect is essential to the subject matter or that this aspect applies to all configurations of the subject matter. Disclosures relating to an aspect may apply to all configurations, or one or more configurations. An aspect may provide one or more examples. Phrases such as "aspect" may refer to one or more aspects, or vice versa. Phrases such as "implementation" do not imply that this implementation is essential to the subject matter or that this implementation applies to all configurations of the subject matter. Disclosures relating to an implementation may apply to all implementations, or one or more implementations. An implementation may provide one or more examples. Phrases such as "implementation" may refer to one or more implementations, or vice versa. Phrases such as "configuration" do not imply that this configuration is essential to the subject matter or that this configuration applies to all configurations of the subject matter. Disclosures relating to a configuration may apply to all configurations or one or more configurations. A configuration may provide one or more examples. Phrases such as "configuration" may refer to one or more configurations, or vice versa.
Claims
1. A method, the method comprising: Obtain the first microphone signal from the first microphone of the first device; The first device acquires the second microphone signal from the second microphone of the second device; as well as The first device controls the audio input to a translator application at the first device based on the first microphone signal and the second microphone signal. The translator application is configured to generate an audio translation of the speech content in the audio input from a first language to a second language.
2. The method of claim 1, wherein at least one of the first microphone signal or the second microphone signal includes a representation of the speech of a person speaking the first language other than the user of the first device, and wherein the method further comprises: The translator application generates audio output in response to controlled audio input, the audio output including speech spoken in the second language.
3. The method of claim 1, wherein controlling the audio input includes selecting either the first microphone signal or the second microphone signal as the audio input based at least in part on the signal-to-noise ratio of the first microphone signal.
4. The method of claim 3, wherein the second microphone signal from the second device comprises a multi-stream signal, the multi-stream signal comprising: An auto-voice stream, configured to enhance the representation of the user's voice on the first device, and A target speech stream, configured to enhance the representation of speech by people other than the user of the first device, and Selecting either the first microphone signal or the second microphone signal as the audio input based at least in part on the signal-to-noise ratio of the first microphone signal includes: selecting either the target speech stream of the second microphone signal or the first microphone signal as the audio input based at least in part on the signal-to-noise ratio of the first microphone signal, regardless of whether the user's speech of the first device is present in either the first microphone signal or the second microphone signal's own speech stream.
5. The method of claim 4, wherein at least one of the target speech stream of the second microphone signal or the first microphone signal includes the representation of the speech of the person speaking the first language other than the user of the first device, and wherein the method further comprises: The first device generates a spatialized audio output based on the audio input, the spatialized audio output including speech spoken in the second language and configured to be perceived as coming from the location of a person other than the user of the first device.
6. The method of claim 1, wherein controlling the audio input comprises: The determination of whether to provide any audio input to the translator application is based at least in part on the determination of whether the user's voice from the first device is present in the second microphone signal.
7. The method of claim 6, wherein determining whether to provide any audio input to the translator application based at least in part on the determination of whether the user's voice from the first device is present in the second microphone signal comprises: Based on a first determination that the user's voice, which is not present in the second microphone signal, is not present in the first device at a first time, the translator application is provided with at least one of the first microphone signal or the second microphone signal as the audio input; as well as Based on a second determination that the user's voice from the first device is present in the second microphone signal at a second time, it is determined that no audio input will be provided to the translator application.
8. The method of claim 1, wherein controlling the audio input comprises: The first microphone signal or the second microphone signal is selected as the audio input, at least in part, based on the condition of the first device.
9. The method according to claim 8, further comprising: The condition of the first device is determined using sensor signals from at least one sensor at the first device, wherein the at least one sensor includes a proximity sensor or an orientation sensor, and determining the condition of the first device further includes determining whether the first microphone of the first device is blocked.
10. The method according to claim 8, further comprising: The condition of the first device is determined at least in part based on user input that initiates the translator application.
11. The method of claim 1, wherein controlling the audio input comprises: The first microphone signal or the second microphone signal is selected as the audio input based at least in part on the state of the user interface of the translator application at the first device.
12. The method of claim 11, wherein the second device is a head-mounted device.
13. The method according to claim 1, further comprising: The first device provides a prompt for moving the first device based at least in part on a second microphone signal from the second device.
14. The method according to claim 1, further comprising: The translator application uses the second microphone signal from the second device to generate a translation of the user's speech from the second language to the first language for output by the speaker of the first device; as well as Before providing the first microphone signal from the first device to the translator application for translating the speech of a person other than the user from the first language to the second language, an echo cancellation operation is performed on the first microphone signal.
15. An electronic device, the electronic device comprising: First microphone; A memory that stores a translator application configured to generate an audio translation of speech content from a first language to a second language in an audio input. and One or more processors, said one or more processors being configured to: Obtain the first microphone signal from the first microphone; Acquire a second microphone signal from a second microphone of an audio device separate from the electronic device; and The audio input to the translator application is controlled based on the first microphone signal and the second microphone signal.
16. The electronic device of claim 15, further comprising communication circuitry, wherein at least one of the first microphone signal or the second microphone signal includes a representation of the speech of a person speaking the first language other than a user of the electronic device, and wherein the one or more processors are further configured to: The communication circuit is used to wirelessly receive the first microphone signal from the audio device; The translator application generates audio output in response to controlled audio input, the audio output including computer-generated speech in the second language; and The communication circuitry is used to provide the audio output to the audio device so that the computer-generated speech is output by the speaker of the audio device.
17. The electronic device of claim 16, wherein the memory stores speech features of a plurality of computer-generated speech, and wherein the one or more processors are further configured to: select the computer-generated speech from the plurality of computer-generated speech based at least in part on the speech of the person in the first microphone signal or the second microphone signal.
18. The electronic device of claim 15, wherein the one or more processors are configured to control the audio input by selecting either the first microphone signal or the second microphone signal as the audio input based at least in part on the sound pressure level in the first microphone signal.
19. The electronic device of claim 15, wherein the one or more processors are configured to control the audio input by determining, at least in part, whether to provide any audio input to the translator application based on the determination of whether the user's voice of the electronic device is present in the second microphone signal.
20. A system comprising: An audio device, the audio device including a first microphone; and An electronic device, the electronic device including a second microphone, wherein the audio device is configured to send a first microphone signal from the first microphone to the electronic device, and wherein the electronic device is configured to: Obtain the second microphone signal from the second microphone; as well as The audio input to a translator application at the electronic device is controlled based on the first microphone signal and the second microphone signal. The translator application is configured to generate an audio translation of the speech content in the audio input from a first language to a second language.
21. An electronic device, the electronic device comprising: One or more microphones; One or more speakers; and One or more processors, said one or more processors being configured to: During the performance, one or more microphones are used to acquire audio input corresponding to the performance, the audio input including speech in the first language and ambient sounds; During the performance, a translation of the speech into a second language is obtained; as well as During the performance, audio output is output by operating one or more speakers, the audio output including the ambient sounds and computer-generated speech, the computer-generated speech speaking the translation of the speech in a second language.
22. The electronic device of claim 21, wherein the speech of the first language is removed from the audio output.
23. The electronic device of claim 21, wherein the audio output further comprises a modified version of the speech in the first language.
24. The electronic device of claim 23, wherein the modified version of the speech in the first language includes a reduced-volume version of the speech in the first language.
25. The electronic device of claim 23, wherein the modified version of the speech in the first language includes an obfuscated version of the speech in the first language.
26. The electronic device of claim 21, wherein the speech comprises speech from a character in the performance, and wherein the ambient sound comprises at least one of: other speech from other characters in the performance, or music.
27. The electronic device of claim 21, wherein the speech in the first language is spoken by a character in the performance at a location in the physical environment of the electronic device, and wherein the computer-generated speech translating the speech is spatialized according to the location.
28. The electronic device of claim 27, wherein the audio output is output using the one or more speakers such that the spatialized computer-generated speech uttering the translation of the speech can be perceived as emanating from the location of the character.
29. The electronic device of claim 27, wherein the computer-generated speech uttering the translation of the speech is voice-matched with one or more features of the speech of the character.
30. The electronic device of claim 21, wherein the electronic device comprises an earphone, and wherein acquiring the translation of the speech in the second language comprises: Provide at least a portion of the audio input from the earbuds to the accompanying device; as well as The device receives computer-generated speech in which the speech is translated into the second language.
31. A method, the method comprising: To determine the first position of the first person and the second position of the second person relative to a device with multiple microphones; Acquire multiple corresponding microphone signals from the multiple microphones; During the time period in which the first person's first voice is present among the multiple corresponding microphone signals, the size of the field of view of the beam used to process the multiple corresponding microphone signals is dynamically determined based on the first position, the second position, and the movement of the device. During the time period, the beam with the field of view of the dynamically determined size is used to process the signals from the plurality of corresponding microphones to generate audio output data including a version of the first person's speech; as well as Audio output data, including the version of the first person's first voice, is provided for output by a speaker during the time period.
32. The method of claim 31, wherein the field of view includes the first position of the first person throughout the entire time period, and wherein dynamically determining the size of the field of view of the beam based on the first position, the second position, and the motion of the device comprises: The size of the field of view is expanded in response to the movement of the device, while excluding the second position from the field of view.
33. The method of claim 32, wherein dynamically determining the size of the field of view of the beam based on the first position, the second position, and the motion of the device comprises: The size of the field of view of the beam is dynamically determined based on the first position, the second position, the movement of the device, and an indication that the user of the device is conversing with the first person.
34. The method of claim 33, wherein the second person is having a conversation with a third person during the time period.
35. The method of claim 31, wherein dynamically determining the size of the field of view of the beam based on the first position, the second position, and the motion of the device comprises: The size of the field of view is expanded in response to the movement of the device, while the first position and the second position are maintained within the field of view.
36. The method of claim 35, wherein the first and second persons are in a conversation with a user of the device.
37. The method of claim 31, wherein the version of the first person's first speech comprises a pass-through reproduction of the first person's first speech.
38. The method of claim 31, wherein the version of the first person's first speech comprises speaking a computer-generated speech that translates the first person's first speech.
39. A method comprising: During the performance, at least one microphone of the device is used to acquire audio input corresponding to the performance, the audio input including speech in the first language and ambient sounds; During the performance, a translation of the speech into a second language is obtained; as well as During the performance, at least one speaker of the device outputs audio output, which includes the ambient sound and computer-generated speech, the computer-generated speech speaking the translation of the speech in the second language.
40. The method according to claim 39, further comprising: Modify the speech of the first language in the audio output.
