Voice Processing for Wearable Systems

Abstract

A method for processing an acoustic signal is disclosed. According to one or more embodiments, a first acoustic signal is received via a first microphone. The first acoustic signal is associated with a first speech of a user of a wearable headband unit. A first sensor input is received via a sensor, and a control parameter is determined based on the sensor input. The control parameter is applied to one or more of the first acoustic signal, the wearable headband unit, and the first microphone. Determining the control parameter includes: determining a relationship between the first speech and the first acoustic signal based on the first sensor input.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 687,987, filed on June 21, 2018, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to systems and methods for processing acoustic speech signals, and more particularly to systems and methods for processing acoustic speech signals generated by a user of a wearable device. Background Art

[0004] The task of tools for speech processing is to receive audio input representing human speech, usually via a microphone, and process the audio input to determine words, logical structures, or other outputs corresponding to the audio input. For example, automatic speech recognition (ASR) tools can generate text output based on human speech corresponding to the audio input signal; and natural language processing (NLP) tools can generate logical structures or computer data corresponding to the meaning of the human speech. It is desirable that this process be performed accurately and quickly, and some applications require real-time results.

[0005] Computer speech processing systems have a history of producing questionable results. In general, the accuracy of a speech processing system may be highly dependent on the quality of the input audio signal, with the highest accuracy being obtained from input provided under controlled conditions. For example, a speech processing system may perform well when the audio input is clearly pronounced speech picked up by a microphone at right angles and close distance, without ambient noise, with a high signal-to-noise ratio and a constant volume. However, speech processing systems may have difficulty adapting to the many variables that can be introduced into the input audio signal in real-world situations. For example, when there are multiple speech sources (e.g., multiple people speaking in the same space at the same time); when ambient noise (e.g., wind, rain, electrical interference, ambient noise) is mixed with the source speech signal; when the speaker is not clearly pronounced or speaks with a peculiar or inconsistent tone, accent, or pitch variation; when the speaker moves or rotates relative to the microphone; when the speaker is in an acoustically reflective environment (e.g., a tiled bathroom or a large cathedral); when the speaker is a long distance from the microphone; when the speaker is facing away from the microphone; or when any other number of variables are present and compromise the fidelity of the input audio signal, the speech processing signal may exhibit limited accuracy. These problems may be magnified in mobile or outdoor applications, where unpredictable noise sources may be present; and where attempts to control or understand the environment near the speaker may become difficult or impossible.

[0006] It is desirable to use wearable systems equipped with sensors, such as those in conjunction with head-mounted units, to correct for the effects of such variables on the input audio signals of speech processing systems. By providing more predictable, higher fidelity inputs to speech processing systems, the outputs of these systems can generate more accurate and reliable results. Additionally, wearable systems are well suited for mobile, outdoor applications, precisely the type of applications in which many traditional speech processing systems may perform particularly poorly. Summary of the invention

[0007] Examples of the present disclosure describe systems and methods for processing acoustic signals. According to one or more embodiments, a first acoustic signal is received via a first microphone. The first acoustic signal is associated with a first voice of a user of a wearable headgear unit. A first sensor input is received via a sensor. A control parameter is determined based on the sensor input. The control parameter is applied to one or more of the first acoustic signal, the wearable headband unit, and the first microphone. Determining the control parameter includes: determining a relationship between the first voice and the first acoustic signal based on the first sensor input. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 An example wearable head device is shown that can be used as part of a wearable system in accordance with some embodiments.

[0009] Figure 2 An example handheld controller is shown that may be used as part of a wearable system in accordance with some embodiments.

[0010] Figure 3 An example auxiliary unit that may be used as part of a wearable system according to some embodiments is shown.

[0011] Figure 4 An example functional block diagram for an example wearable system is shown in accordance with some embodiments.

[0012] Figure 5 A flow diagram of an example speech processing system is shown in accordance with some embodiments.

[0013] Figure 6 A flow chart illustrating an example system for processing an acoustic speech signal in accordance with some embodiments. DETAILED DESCRIPTION

[0014] In the following example description, reference is made to the accompanying drawings which form a part thereof and in which are shown by way of illustration specific examples that can be practiced. It will be appreciated that other examples can be used and structural changes can be made without departing from the scope of the disclosed examples.

[0015] Example Wearable System

[0016] Figure 1An example wearable head device 100 configured to be worn on a user's head is shown. The wearable head device 100 can be part of a broader wearable system that includes one or more components, such as a head device (e.g., wearable head device 100), a handheld controller (e.g., handheld controller 200 described below), and / or an auxiliary unit (e.g., auxiliary unit 300 described below). In some examples, the wearable head device 100 can be used in virtual reality, augmented reality, or mixed reality systems or applications. The wearable head device 100 may include one or more displays, such as displays 110A and 110B (which may include left and right transmissive displays, and associated components for coupling light from the displays to the user's eyes, such as orthogonal pupil expansion (OPE) grating groups 112A / 112B and exit pupil expansion (EPE) grating groups 114A / 114B); left and right acoustic structures, such as speakers 120A and 120B (which may be mounted on temples 122A and 122B and positioned adjacent to the user's left and right ears); one or more sensors, such as an infrared sensor, an accelerometer, a GPS unit, an inertial measurement unit (IMU) (e.g., an IMU 126), an acoustic sensor (e.g., a microphone 150); an orthogonal coil electromagnetic receiver (e.g., a receiver 127 shown mounted to the left temple 122A); a left camera and a right camera oriented away from the user (e.g., depth (time of flight) cameras 130A and 130B); and a left eye camera and a right eye camera oriented toward the user (e.g., for detecting the user's eye movement) (e.g., eye cameras 128 and 128B). However, the wearable head device 100 can be combined with any suitable display technology, and any suitable number, type, or combination of sensors or other components without departing from the scope of the present invention. In some examples, the wearable head device 100 can be combined with one or more microphones 150, which are configured to detect audio signals generated by the user's voice; the microphone can be located near the user's mouth in the wearable head device. In some examples, the wearable head device 100 can be combined with networking features (e.g., Wi-Fi functionality) to communicate with other devices and systems including other wearable systems. The wearable head device 100 may further include components such as a battery, a processor, a memory, a storage unit, or various input devices (e.g., buttons, touch pads); or may be coupled to a handheld controller (e.g., handheld controller 200) or an auxiliary unit (e.g., auxiliary unit 300) including one or more such components. In some examples, the sensor may be configured to output a set of coordinates of the head-mounted unit relative to the user's environment and may provide input to a processor that performs a simultaneous localization and mapping (SLAM) process and / or a visual odometry algorithm.In some examples, as described further below, the wearable head device 100 can be coupled to a handheld controller 200 and / or an auxiliary unit 300.

[0017] Figure 2 An example mobile handheld controller assembly 200 of an example wearable system is shown. In some examples, the handheld controller 200 can communicate wired or wirelessly with the wearable head device 100 and / or the auxiliary unit 300 described below. In some examples, the handheld controller 200 includes a handle portion 220 to be held by a user, and one or more buttons 240 disposed along a top surface 210. In some examples, the handheld controller 200 can be configured to be used as an optical tracking target; for example, a sensor (e.g., a camera or other optical sensor) of the wearable head device 100 can be configured to detect the position and / or orientation of the handheld controller 200, which by extension can indicate the position and / or orientation of the hand of the user holding the handheld controller 200. In some examples, the handheld controller 200 may include a processor, a memory, a storage unit, a display, or one or more input devices, such as described above. In some examples, the handheld controller 200 includes one or more sensors (e.g., any sensor or tracking component described above with respect to the wearable head device 100). In some examples, the sensor can detect the position or orientation of the handheld controller 200 relative to the wearable head device 100 or relative to another component of the wearable system. In some examples, the sensor can be located in the handle portion 220 of the handheld controller 200, and / or can be mechanically coupled to the handheld controller. The handheld controller 200 can be configured to provide one or more output signals, such as corresponding to the pressed state of the button 240, or the position, orientation and / or movement of the handheld controller 200 (e.g., via an IMU). The output signal can be used as an input to a processor of the wearable head device 100, the auxiliary unit 300, or another component of the wearable system. In some examples, the handheld controller 200 can include one or more microphones to detect sounds (e.g., the user's voice, ambient sounds), and in some cases, provide a signal corresponding to the detected sound to a processor (e.g., a processor of the wearable head device 100).

[0018] Figure 3An example auxiliary unit 300 of an example wearable system is shown. In some examples, the auxiliary unit 300 can communicate with the wearable head device 100 and / or the handheld controller 200 in wired or wireless communication. The auxiliary unit 300 may include a battery to provide energy for operating one or more components of the wearable system, such as the wearable head device 100 and / or the handheld controller 200 (including a display, a sensor, an acoustic structure, a processor, a microphone, and / or other components of the wearable head device 100 or the handheld controller 200). In some examples, the auxiliary unit 300 may include a processor, a memory, a storage unit, a display, one or more input devices, and / or one or more sensors, such as described above. In some examples, the auxiliary unit 300 includes a clip 310 for attaching the auxiliary unit to a user (e.g., a belt worn by the user). An advantage of using auxiliary unit 300 to house one or more components of the wearable system is that doing so can allow large or heavy components to be carried by the user's waist, chest, or back (which are relatively well suited to supporting large and heavy objects), rather than mounted to the user's head (e.g., placed in wearable head device 100) or carried by the user's hand (e.g., if placed in handheld controller 200). This can be particularly advantageous for relatively heavy or bulky components (such as batteries).

[0019] Figure 4 An example functional block diagram that may correspond to an example wearable system 400 (such as may include the example wearable head device 100, handheld controller 200, and auxiliary unit 300 described above) is shown. In some examples, the wearable system 400 may be used for virtual reality, augmented reality, or mixed reality applications. Figure 4As shown in , the wearable system 400 may include an example handheld controller 400B, referred to herein as a "totem" (and may correspond to the handheld controller 200 described above); the handheld controller 400B may include a totem-to-headband six degrees of freedom (6DOF) totem subsystem 404A. The wearable system 400 may also include an example wearable head device 400A (which may correspond to the wearable headband device 100 described above); the wearable head device 400A includes a totem-to-headband 6DOF headband subsystem 404B. In the example, the 6DOF totem subsystem 404A and the 6DOF headband subsystem 404B cooperate to determine six coordinates of the handheld controller 400B relative to the wearable head device 400A (e.g., offsets in three translation directions and rotations along three axes). The six degrees of freedom can be expressed relative to the coordinate system of the wearable head device 400A. In this coordinate system, the three translation offsets can be expressed as X, Y, and Z offsets, a translation matrix, or some other representation. The rotational degrees of freedom may be represented as a sequence of yaw, pitch, and roll rotations; a vector; a rotation matrix; a quaternion; or some other representation. In some examples, one or more depth cameras 444 (and / or one or more non-depth cameras) included in the wearable head device 400A; and / or one or more optical targets (e.g., the button 240 of the handheld controller 200 described above or a dedicated optical target included in the handheld controller) may be used for 6DOF tracking. In some examples, as described above, the handheld controller 400B may include a camera; and the headband 400A may include an optical target for optical tracking with the camera. In some examples, the wearable head device 400A and the handheld controller 400B each include a set of three orthogonally oriented solenoids for wirelessly transmitting and receiving three distinguishable signals. By measuring the relative magnitude of the three distinguishable signals received in each coil for receiving, the 6DOF of the handheld controller 400B relative to the wearable head device 400A can be determined. In some examples, the 6DOF totem subsystem 404A may include an inertial measurement unit (IMU) that may be used to provide improved accuracy and / or more timely information regarding rapid movements of the handheld controller 400B.

[0020] In some examples involving augmented reality or mixed reality applications, it may be desirable to transform coordinates from a local coordinate space (e.g., a coordinate space that is fixed relative to the wearable head device 400A) to an inertial coordinate space or to an ambient coordinate space. For example, such a transformation may be necessary for the display of the wearable head device 400A to present a virtual object (e.g., a virtual person sitting in a real chair facing forward, regardless of the position and orientation of the wearable head device 400A) at an expected position and orientation relative to the real environment, rather than at a fixed position and orientation on the display (e.g., at the same position in the display of the wearable head device 400A). This can maintain the illusion that the virtual object exists in the real environment (and, for example, does not appear to be placed unnaturally in the real environment as the wearable head device 400A moves and rotates). In some examples, compensating transformations between coordinate spaces can be determined by processing images from the depth camera 444 (e.g., using simultaneous localization and mapping (SLAM) and / or visual odometry processes) to determine a transformation of the wearable head device 400A relative to an inertial or ambient coordinate system. Figure 4 In the example shown in , the depth camera 444 can be coupled to the SLAM / visual odometry module 406 and can provide images to the frame 406. An implementation of the SLAM / visual odometry module 406 may include a processor configured to process the image and determine the position and orientation of the user's head, which can then be used to identify the transformation between the head coordinate space and the real coordinate space. Similarly, in some examples, an additional source of information about the user's head posture and position is obtained from the IMU 409 of the wearable head device 400A. Information from the IMU 409 can be integrated with information from the SLAM / visual odometry module 406 to provide improved accuracy and / or more timely information for rapid adjustment of the user's head posture and position.

[0021] In some examples, depth camera 444 can provide 3D images to gesture tracker 411, which can be implemented in a processor of wearable head device 400A. Gesture tracker 411 can recognize the user's gestures, for example, by matching the 3D images received from depth camera 444 with stored patterns representing gestures. Other suitable techniques for recognizing user gestures will be apparent.

[0022] In some examples, one or more processors 416 may be configured to receive data from the headband subsystem 404B, the IMU 409, the SLAM / visual odometry module 406, the depth camera 444, the microphone 450; and / or the gesture tracker 411. The processor 416 may also send and receive control signals from the 6DOF totem system 404A. In an example where the handheld controller 400B is not limited, the processor 416 may be wirelessly coupled to the 6DOF totem system 404A. The processor 416 may further communicate with additional components such as an audio-visual content memory 418, a graphics processing unit (GPU) 420, and / or a digital signal processor (DSP) audio sound field localizer 422. The DSP audio sound field localizer 422 may be coupled to a head-related transfer function (HRTF) memory 425. The GPU 420 may include a left channel output coupled to a left source 424 of light modulated per image and a right channel output coupled to a right source 426 of light modulated per image. GPU 420 can output stereoscopic image data to the source 424, 426 of the light modulated by the image. DSP audio sound field locator 422 can output audio to the left speaker 412 and / or the right speaker 414. DSP audio sound field locator 422 can receive the input of the direction vector indicating from the user to the virtual sound source (the virtual sound source can be moved by the user, for example, via the handheld controller 400B) from the processor 419. Based on the direction vector, DSP audio sound field locator 422 can determine the corresponding HRTF (for example, by accessing the HRTF, or by interpolating multiple HRTFs). Then, DSP audio sound field locator 422 can apply the determined HRTF to the audio signal, such as the audio signal corresponding to the virtual sound generated by the virtual object. This can be combined with the relative position and orientation of the user relative to the virtual sound in the mixed reality environment (that is, by presenting a virtual sound that matches the user's expectation of the virtual sound that sounds like the real sound in the real environment) to enhance the credibility and authenticity of the virtual sound.

[0023] In some examples, such as Figure 4 As shown in , one or more of the processor 416, GPU 420, DSP audio sound field localizer 422, HRTF memory 425, and audio / visual content memory 418 may be included in an auxiliary unit 400C (which may correspond to the auxiliary unit 300 described above). The auxiliary unit 400C may include a battery 427 to power its components and / or to power the wearable head device 400A and / or the handheld controller 400B. Including such components in an auxiliary unit that can be mounted to the user's waist can limit the size and weight of the wearable head device 400A, which in turn can reduce fatigue on the user's head and neck.

[0024] exist Figure 4While elements corresponding to the various components of the example wearable system 400 are presented, various other suitable arrangements of these components will become apparent to those skilled in the art. For example, Figure 4 Elements presented as being associated with the auxiliary unit 400C may instead be associated with the wearable head device 400A or the handheld controller 400B. In addition, some wearable systems may completely abandon the handheld controller 400B or the auxiliary unit 400C. Such changes and modifications should be understood to be included within the scope of the disclosed examples.

[0025] Voice processing engine

[0026] A speech processing system typically includes a system that accepts an input audio signal (source signal) corresponding to human speech; processes and analyzes the input audio signal; and, as a result of the analysis, generates an output corresponding to human speech. The process or module used to perform these tasks can be considered a speech processing engine. In the case of an automatic speech recognition system, the output of the speech processing engine can be a text transcription of human speech. In the case of a natural language processing system, the output can be one or more commands or instructions indicated by human speech; or some representation of the semantic meaning of human speech (e.g., a logical representation or data structure). Other types of speech processing systems (e.g., automatic translation systems) are also contemplated and are within the scope of this disclosure.

[0027] Speech processing systems are found in a wide variety of products and applications: traditional telephone systems; automated voice messaging systems; voice assistants (including stand-alone and smartphone-based voice assistants); vehicles and aircraft; desktop and document processing software; data entry; home appliances; medical devices; language translation software; closed captioning systems; and so on. An advantage of speech processing systems is that they can allow a user to provide input to a computer system using natural spoken language (such as presented to a microphone), replacing traditional computer input devices such as keyboards or touch panels; therefore, speech processing systems are particularly useful in environments where traditional input devices (such as keyboards) may be unavailable or impractical. In addition, by allowing the user to provide intuitive speech-based input, speech recognition systems can enhance immersion. As such, speech recognition can be naturally applicable to wearable systems, and is particularly suitable for virtual reality, augmented reality, and / or mixed reality applications of wearable systems, in which user immersion is a primary goal and in which it may be desirable to limit the use of traditional computer input devices, whose presence may reduce immersion.

[0028] Figure 5An automatic speech recognition engine 500 according to some embodiments is shown. The engine 500 is intended to generally illustrate an automatic speech recognition system; other specific systems are also possible and within the scope of the present disclosure. The engine 500 can be implemented using one or more processors (e.g., CPU, GPU and / or DSP), memory, input devices (e.g., microphones), output devices (e.g., displays, speakers), networks, databases, and / or other suitable components. In the engine 500, an audio signal 510 corresponding to a source human speech signal is presented to a signal preprocessing stage 520. In some examples, the signal preprocessing stage 520 can apply one or more signal processing functions to the audio signal 510. For example, the preprocessing functions may include audio processing functions such as peak compression; noise reduction; bandwidth limiting; equalization; signal attenuation or other suitable functions. These preprocessing functions can simplify the task of later processing and analyzing the audio signal 510. For example, the feature extraction algorithm can be calibrated so that it performs optimally on an input signal with certain audio characteristics (such as gain and frequency characteristics) and the signal-to-noise ratio of the input signal is maximized. In some examples, preprocessing the audio signal 510 can condition the signal so that it can be more reliably analyzed elsewhere in the engine 500. For example, the signal preprocessing stage 520 can re-encode the audio signal 510 (e.g., re-encode at a particular bit rate); or convert the audio signal 510 from a first form (e.g., a time domain signal) to a second form (e.g., a frequency domain signal; or a parametric representation), which can simplify subsequent processing of the audio signal 510. In some examples, one or more functions of the preprocessing stage 520 can be performed by a DSP.

[0029] At stage 530, a feature extraction process may be applied to the audio signal 510 (as preprocessed at stage 520). The purpose of feature extraction is to identify individual speech features of the audio signal 510 and to reduce or eliminate variations in those features so that the features can be processed efficiently and consistently (e.g., compared to patterns stored in a database). For example, feature extraction may reduce, eliminate, or control variations in the pitch, gender, accent, pronunciation, and rhythm of a speaker. Feature extraction may also reduce, eliminate, or control variations in the recording device (e.g., type of microphone), signal transmission (e.g., via land-based telephone lines or cellular telephone networks), or recording environment (e.g., room acoustics; background noise level; distance of the speaker from the microphone; angle of the speaker from the microphone). Various suitable techniques for feature extraction are known in the art.

[0030] At stage 530, speech features extracted from audio signal 510 may be presented to decoder stage 540. The goal of decoder stage 540 is to determine a text output 570 that corresponds to the source human speech from which audio signal 510 was generated. In some examples, text output 570 need not be text, but may be another data representation of the source speech. There are a variety of techniques for decoding speech features into text; for example, hidden Markov models; Viterbi decoding; beam search; dynamic search; multi-pass search; weighted finite state sensor (WFST); or any suitable combination of the above. Other suitable techniques will be familiar to those skilled in the art.

[0031] In some examples, decoder 540 can utilize acoustic modeling stage 550 to facilitate generating text output 570. Acoustic modeling stage 550 can use a model of the relationship between speech signals and language units (e.g., phonemes) to identify one or more language units from audio signal 510 (including one or more features extracted at stage 530). Various suitable acoustic modeling techniques that can be applied at acoustic modeling stage 550 will be familiar to those skilled in the art.

[0032] In some examples, the decoder 540 can utilize a language modeling stage 560 to facilitate generating a text output 570. The language modeling stage 560 can use models of the grammar, vocabulary, and other features of the language to determine the language units (e.g., phonemes) that are most likely to correspond to features of the audio signal 510. For example, the language model applied by the stage 560 can conclude that a particular extracted feature is more likely to correspond to a word that occurs with high frequency in the speaker's language than to a word that occurs with low frequency in the language. Various suitable language modeling techniques that can be applied at the language modeling stage 560 will be familiar to those skilled in the art. In addition, the decoder 540 can utilize other suitable techniques or models to facilitate generating a text output 570; the present disclosure is not limited to any particular technique or group of techniques.

[0033] Typically, the text output 570 does not completely correspond to the source human speech; on the contrary, the possibility that the text output 570 correctly corresponds to the source human speech can be expressed as some probability or confidence interval. Due to the many variables that may affect the audio signal 510, even a complex speech recognition system cannot always produce perfect text output for all speakers. For example, the reliability of a speech recognition system such as engine 500 may be highly dependent on the quality of the input audio signal 510. When the audio signal 510 is recorded under ideal conditions (e.g., in an acoustically controlled environment), the human speaker speaks clearly and directly to the microphone from a close distance, and the source speech can be more easily determined from the audio signal. For example, at stage 530, features can be more reliably extracted from the audio signal 510; and the decoder 540 can more effectively determine which text output 570 corresponds to these features (e.g., acoustic modeling can be more reliably applied to the feature at stage 550, and / or language modeling can be more reliably applied to the feature at stage 560).

[0034] However, in real-world applications, the audio signal 510 may deviate from ideal conditions, making it more difficult to determine the source human speech. For example, the audio signal 510 may incorporate ambient noise, such as may be introduced by an outdoor environment or a large distance between the human speaker and the microphone; electrical noise, such as from electrical interference (e.g., a battery charger for a smartphone); natural reverberation, such as from nearby surfaces (e.g., concrete, bathroom tiles) or acoustic spaces (e.g., caves, cathedrals); or other unwanted effects. In addition, the audio signal 510 may suffer from attenuation of certain frequencies, such as may be produced when the human speaker turns his or her back to the microphone; this is particularly problematic when the attenuated frequencies are those that carry a lot of information related to speech (e.g., formant frequencies that can be used to distinguish vowels). Similarly, the audio signal 510 may suffer from overall low amplitude or low signal-to-noise ratio, which may be caused by excessive distance between the human speaker and the microphone. In addition, if the human speaker moves and reorients himself or herself while speaking, the audio signal 510 may change characteristics during the course of the signal, further complicating efforts to determine the underlying speech.

[0035] Although the example system 500 shows an example speech recognition engine, other types of speech processing engines may also follow a similar structure: for example, a natural language processing engine may perform a signal preprocessing stage when accepting an input audio signal corresponding to human speech; extract components from the signal (e.g., via a segmentation and / or tokenization stage); and in some cases perform detection / analysis of the components with the aid of one or more language modeling subsystems. In addition, in some examples, the output of an automatic speech recognition engine such as that shown in the example system 500 may be used as input to another language processing engine; for example, a natural language processing engine may accept as input the text output 570 of the example system 500. The system may encounter challenges similar to those faced by the example system 500. For example, changes in the input audio signal such as described above (which make it more difficult to recover the source speech signal underlying the audio signal) may also make it more difficult to provide other forms of output (e.g., a logical representation or data structure in the case of a natural language processing engine). Therefore, such as described below, the system will also benefit from the present invention.

[0036] Improving speech processing using wearable systems

[0037] The present disclosure is directed to a method for using an input from a sensor (e.g., a wearable device (e.g., such as the one described above) to Figure 1A system and method for improving the accuracy of a speech processing system by using input from sensors associated with a head mounted device (such as a head mounted device) described above to reduce, eliminate or control variations in an input audio signal, such as the variations described above with respect to audio signal 510. The variations may be particularly noticeable in mobile applications of speech processing or applications of speech processing in an uncontrolled environment (such as an outdoor environment). Wearable systems are generally intended for such applications and may encounter such variations. A wearable system may generally refer to any combination of a head device (e.g., wearable head device 100), a handheld controller (e.g., handheld controller 200), an auxiliary unit (e.g., auxiliary unit 300), and / or an environment of a head device. In some embodiments, the sensors of the wearable system may be on the head device, the handheld controller, the auxiliary unit, and / or in the environment of the head device. For example, because the wearable system may be designed to be mobile, the audio signal of the user of the wearable system recorded at a single fixed microphone may suffer from a low signal-to-noise ratio if the user is away from a single fixed microphone (e.g., at a stand-alone voice assistant device), or the audio signal may suffer from "acoustic shadowing" or an unnecessary frequency response if the user turns away from the microphone. In addition, as expected by a mobile user of a wearable system, the audio signal may change characteristics over time as the user moves and rotates about a single fixed microphone. In addition, because some wearable systems are intended for use in uncontrolled environments, there is a high probability that ambient noise (or other people's voices) will be recorded along with the target person's voice. Similarly, this uncontrolled environment may introduce undesirable echoes and reverberations in the audio signal 510, which may mask the underlying speech.

[0038] As mentioned above about Figure 1 As described in the example wearable head device 100 in FIG. 1 , the wearable system may include one or more sensors that may provide input about a user and / or environment of the wearable system. For example, the wearable head device 100 may include a camera (e.g., Figure 4444 described in ) to output visual signals corresponding to the environment; in some examples, the camera can be a forward-facing camera on a head-mounted unit that shows what is currently in front of the user of the wearable system. In some examples, the wearable head device 100 may include a LIDAR unit, a radar unit, and / or an acoustic sensor that can output signals corresponding to the physical geometry of the user's environment (e.g., a wall, a physical object). In some examples, the wearable head device 100 may include a GPS unit that can indicate geographic coordinates corresponding to the current location of the wearable system. In some examples, the wearable head device 100 may include an accelerometer, a gyroscope; and / or an inertial measurement unit (IMU) to indicate the orientation of the wearable head device 100. In some examples, the wearable head device 100 may include an environmental sensor, such as a temperature or pressure sensor. In some examples, the wearable head device 100 may include a biometric sensor, such as an iris camera, a fingerprint sensor, an eye tracking sensor, or a sensor that measures the user's vital signs. In examples where the wearable head device 100 includes a head-mounted unit, the orientation can correspond to the orientation of the user's head (and, by extension, the direction of the user's mouth and the user's voice). Other suitable sensors may also be included. In some embodiments, the handheld controller 200, the auxiliary unit 300, and / or the environment of the wearable head device 100 may include any suitable one or more sensors mentioned above with respect to the wearable head device 100. In addition, in some cases, one or more sensors may be placed in the environment with which the wearable system interacts. For example, the wearable system may be designed to be worn by the driver of a vehicle, and appropriate sensors (e.g., depth cameras, accelerometers, etc.) may be placed inside the vehicle. One advantage of this approach is that the sensors can occupy known positions within the environment; this configuration can simplify the interpretation of data provided by these sensors compared to sensors that can be attached to wearable devices that move within the environment.

[0039] Signals provided by such sensors of the wearable system (e.g., the wearable head device 100, the handheld controller 200, the auxiliary unit 300, and / or the environment of the wearable head device 100) can be used to provide information about the characteristics of the audio signal recorded by the wearable system, and / or information about the relationship between the audio signal and the underlying source speech signal. This information can then be used to more effectively determine the source speech underlying the audio signal.

[0040] To illustrate, Figure 6 An example speech recognition system 600 is shown that incorporates a wearable system to improve speech recognition of audio signals recorded by one or more microphones. Figure 6A user of a wearable system 601 is shown, which may correspond to the wearable system 400 described above and may include one or more of the example wearable head device 100, the handheld controller 200, and the auxiliary unit 300. The user of the wearable system 601 provides a verbal expression 602 ("source speech"), which is detected at one or more microphones 604, which output a corresponding audio signal 606. The wearable system 601 may include one or more sensors described above (including one or more cameras), a LIDAR unit, a radar unit, an acoustic sensor, a GPS unit, an accelerometer, a gyroscope, an IMU, a microphone (which may be one of the microphones 604), a temperature sensor, a biometric sensor, or any other suitable sensor or sensor combination. The wearable system 601 may also include one or more processors (e.g., CPU, GPU, and / or DSP), memory, input devices (e.g., microphones), output devices (e.g., displays, speakers), networks, and / or databases. These components can be implemented using a combination of the wearable head device 100, the handheld controller 200, and the auxiliary unit 300. In some examples, the sensors of the wearable system 601 provide sensor data 608 (which can be multi-channel sensor data, such as in examples where one or more sensors (including sensors from more than two sensor types) are presented in parallel). The sensor data 608 can be provided simultaneously with the microphone 604 that detects the source voice; that is, the sensors of the wearable system 601 can provide sensor data 608 corresponding to the conditions when the source voice is provided. In some examples, one or more microphones 604 can be included in the wearable head device, handheld controller, and / or auxiliary unit of the wearable system 601 and / or its environment.

[0041] In the example system 600, the subsystem 610 can accept sensor data 608 and audio signal 606 as input; determine control parameters to apply to process the audio signal 606; and provide the processed audio signal as input to a signal processing engine (e.g., speech recognition engine 650 and / or natural language processing (NLP) engine 670), which can generate output (e.g., text output 660 and / or natural language processing output 680, respectively). The subsystem 610 includes the following description and is described in Figure 6One or more processes, levels and / or modules shown in . Subsystem 610 can be implemented using any combination of one or more processors (e.g., CPU, GPU and / or DSP), memory, network, database and / or other suitable components. In some examples, some or all of subsystem 610 can be implemented on wearable system 601: for example, on one or more of a head device (e.g., wearable head device 100), a handheld controller (e.g., handheld controller 200), and / or an auxiliary unit (e.g., auxiliary unit 300). In some examples, some or all of subsystem 610 can be implemented on a device (e.g., a smart phone or a stand-alone voice assistant device) that includes microphone 604. In some examples, some or all of subsystem 610 can be implemented on a cloud server or another network-enabled computing device. For example, a local device may perform latency-sensitive functions of subsystem 610, such as functions related to processing audio signals and / or sensor data, while a cloud server or other network-enabled device may perform functions of subsystem 610 that require significant computing or memory resources (e.g., training or applying complex artificial neural networks) and transmit outputs to the local device. Other implementations of subsystem 610 will be apparent to those skilled in the art and are within the scope of the present disclosure.

[0042] In the example system 600, the subsystem 610 includes a sensor data analysis stage 620. The sensor data analysis stage 620 can process and analyze the sensor data 608, for example, to determine information about the environment of the wearable system 601. In examples where the sensor data 608 includes sensor data from different sources (e.g., camera data and GPS data), the sensor data analysis stage 620 can combine the sensor data ("sensor fusion") according to techniques known to those skilled in the art (e.g., Kalman filters). In some examples, in addition to the sensor data 608, the sensor data analysis stage 620 can also combine data from other sources. For example, the sensor data analysis stage 620 can combine the sensor data 608 from a GPS unit with map data and / or satellite data (e.g., from a memory or database storing the data) to determine location information based on the output of the GPS unit. As an example, the GPS unit can output GPS coordinates corresponding to the latitude and longitude of the wearable system 601 as part of the sensor data 608. The sensor data analysis stage 620 can use the latitude and longitude with the map data to identify the country, town, street, unit (e.g., commercial unit or residential unit), or room where the wearable system 601 is located; or to identify nearby businesses or points of interest. Similarly, building data (e.g., data from public structural records) can be combined with the sensor data 608 to identify the building where the wearable system 601 is located; or weather data (e.g., a real-time feed from satellite data) can be combined with the sensor data 608 to identify the current weather conditions at the location. Other example applications will be apparent and are within the scope of the present disclosure.

[0043] On a smaller scale, the sensor data analysis stage 620 can analyze the sensor data 608 to generate information about objects and geometry in the immediate vicinity of the wearable system 601, or about the user of the wearable system 601. For example, using techniques known in the art, data from a LIDAR sensor or radar unit of the wearable system 601 can indicate that the wearable system 601 is facing a wall eight feet away and at an angle θ to the normal of the wall; and image data from a camera of the wearable system 601 can identify that the wall may be made of tile (an acoustically reflective material). In some examples, the acoustic sensor of the wearable system 601 can be used to measure the acoustic effects that a surface may have on an acoustic signal (e.g., by comparing a reflected signal from a surface to a source signal transmitted to the surface). In some examples, the sensor data analysis stage 620 can use the sensor data 608 to determine the position and / or orientation of the user of the wearable system 601, such as using an accelerometer, gyroscope, or IMU associated with the wearable system 601. In some examples, such as in augmented reality or mixed reality applications, stage 620 can incorporate a map or other representation of the user's current environment; for example, where the sensors of wearable system 601 have been used to construct a 3D representation of the geometry of the room, that 3D representation data can be used in conjunction with sensor data 608. Similarly, stage 620 can incorporate information such as the materials of nearby surfaces and the acoustic properties of those surfaces, information about other users in the environment (e.g., their position and orientation and / or the acoustic properties of their voices), and / or information about the user of wearable system 601 (e.g., the user's age group, gender, native language, and / or voice characteristics).

[0044] In some examples, the sensor data analysis stage 620 can analyze the sensor data 608 to generate information related to the microphone 604. For example, the sensor data 608 can provide the position and / or orientation of the one or more microphones 604, such as the position and / or orientation relative to the wearable system 601. In examples where the wearable system 601 includes one or more microphones 604, the position and orientation of the one or more microphones 604 can be directly linked to the position and orientation of the wearable system 601. In some examples, the wearable system 601 can include one or more additional microphones that are not the one or more microphones 604; the additional one or more microphones can be used to provide a baseline audio signal, for example, corresponding to a user's voice detected from a known position and orientation and from a short distance. For example, the additional one or more microphones may be in a known position with a known orientation in the user's environment. The amplitude, phase, and frequency characteristics of the baseline audio signal can be compared with the audio signal 606 to identify the relationship between the source voice 602 and the audio signal 606. For example, if the first audio signal detected the first time has an amplitude of half that of the baseline audio signal, and the second audio signal detected the second time has an amplitude of one quarter that of the baseline audio signal, it can be inferred that the user has moved away from the microphone 604 during the interval between the first and second times. This can be extended to any suitable number of microphones (e.g., an initial microphone, and two or more additional microphones).

[0045] Based on the sensor data 608 and / or other data as described above, the information output by the sensor data analysis stage 620 can identify a relationship between the source speech 602 and the corresponding audio signal 606. The information describing the relationship can be used at stage 630 to calculate one or more control parameters that can be applied to the audio signal 606, one or more microphones 604, and / or the wearable system 601. The application of these control parameters can improve the accuracy with which the system 600 can recover the underlying source speech 602 from the audio signal 606. In some examples, the control parameters calculated at stage 630 can include digital signal processing (DSP) parameters that can be applied to process the audio signal 606. For example, the control parameters can include parameters for a digital signal processing (DSP) noise reduction process (e.g., a signal threshold below which gated noise reduction will be applied to the audio signal 606, or a noise frequency of the audio signal 606 will be attenuated); parameters for a DSP echo cancellation or dereverberation process (e.g., a time value corresponding to a delay between the audio signal 606 and an echo of the signal); or parameters for other audio DSP processing (e.g., phase correction, limiting, pitch correction).

[0046] In some examples, the control parameters may define a DSP filter to be applied to the audio signal 606. For example, the sensor data 608 (e.g., from a microphone of a head-mounted unit of the wearable system 601) may indicate a characteristic frequency curve corresponding to the speech of the user of the wearable system 601 (i.e., the user generating the source speech 602). The frequency curve may be used to determine control parameters defining a digital bandpass filter to be applied to the audio signal 606; the bandpass filter may isolate frequencies that closely correspond to the source speech 602 so as to make the source speech 602 more prominent in the audio signal 606. In some examples, the sensor data 608 (e.g., from a microphone of a head-mounted unit of the wearable system 601) may indicate a characteristic frequency curve corresponding to the speech of a different user (other than the user of the wearable system 601) near the wearable system 601. The frequency curve may be used to determine control parameters defining a digital notch filter to be applied to the audio signal 606; the notch filter may remove unwanted speech from the audio signal 606 so as to make the source speech 602 more prominent in the audio signal 606. Similarly, sensor data 608 (e.g., a camera of wearable system 601) can identify specific other individuals nearby, as well as their positions relative to wearable system 601; this information can determine the level of the notch filter (e.g., the closer the individual is, the louder his or her voice may be in audio signal 606, and the greater the level of attenuation that may need to be applied). As another example, the presence of certain surfaces and / or materials near a user may affect the frequency characteristics of the user's voice as detected by microphone 604; for example, if the user is standing in the corner of a room, certain low frequencies of the user's voice may be emphasized in audio signal 606; this information can be used to generate parameters (e.g., cutoff frequency) of a high-pass filter to be applied to audio signal 606. These control parameters can be applied to audio signal 606 at stage 640; or as part of speech recognition engine 650 and / or natural language processing engine 670 (e.g., at a stage corresponding to signal preprocessing stage 520 or feature extraction stage 530 described above).

[0047] In some examples, the control parameters calculated at stage 630 can be used to configure microphone 604. For example, the control parameters can include hardware configuration parameters (such as the gain level for a hardware amplifier coupled to microphone 604), beamforming parameters to adjust the directivity of microphone 604 (e.g., the vector that microphone 602 should point to), parameters that determine which of multiple microphones 604 should be enabled or disabled, or parameters that control where or how microphone 604 should be located (e.g., in an example where microphone 604 is attached to a mobile platform). In some examples, microphone 604 can be a component of a smartphone or another mobile device; and the control parameters calculated at stage 630 can be used to control the mobile device (e.g., enable various components of the mobile device, or configure or operate software on the mobile device).

[0048] In some examples, the control parameters calculated at stage 630 can be used to control the wearable system 601 itself. For example, such control parameters can include parameters for presenting a message to a user of the wearable system 601, such as via the display 110A / 110B or the speaker 120A / 120B (e.g., an audio or video message that the user should move away from a nearby wall to improve voice recognition accuracy); or parameters for enabling, disabling, or reconfiguring one or more sensors of the wearable system 601 (e.g., redirecting a camera mounted in a servo system to obtain more useful camera data). In examples where the wearable system 601 includes a microphone 604, the control parameters can be sent to the wearable system 601 to control the microphone 604 as described above.

[0049] In some examples, the control parameters calculated at stage 630 can be used to influence the decoding process (e.g., the decoding process 540 described above) of the speech processing system (e.g., the speech recognition engine 650 and / or the natural language processing engine). For example, the sensor data 608 can indicate characteristics of the user's environment, behavior, or mental state, which can affect the user's use of language. For example, the sensor data 608 (e.g., data from a camera and / or a GPS unit) can indicate that the user is attending a football game; because when the user is attending a football game, the user's speech (i.e., the source speech 602) is more likely than usual to contain football-related words (e.g., "coach", "quarterback", "touchdown"), so the control parameters of the speech processing system (e.g., language modeling stage 560) can be temporarily set to reflect the higher probability that the audio signal corresponds to a word related to football.

[0050] In some examples, such as Figure 6As shown in , the various update modules 632, 634, and 636 can determine the control parameters calculated at stage 630, or can apply the control parameters and / or sensor data 608 calculated at stage 630 to the audio signal 606, the microphone 604, the wearable system 601, or any hardware or software subsystem described above. For example, the beamformer update module 632 can determine how to update the beamforming module (e.g., the beamforming module of the microphone 604) to improve the recognition or natural language processing of the audio signal 606 based on the sensor data 608 or one or more control parameters calculated at stage 630. In some examples, the beamforming update module 632 can control the directivity of the sensor array (e.g., the array of microphones 604) to maximize the signal-to-noise ratio of the signal detected by the sensor array. For example, the beamforming update module 632 can adjust the directivity of the microphone 604 so that the source speech 602 is detected with minimal noise and distortion; for example, in a room with multiple voices, the software of the adaptive beamforming module can guide the microphone 604 to maximize the signal power corresponding to the sound of interest (e.g., the sound corresponding to the source speech 602). For example, a sensor of the wearable system 601 can output data indicating that the wearable system 601 is located at a certain distance from the microphone 604, from which a time-of-flight value of the source speech 602 to the microphone 604 can be determined. The time-of-flight value can be used to calibrate the beamforming using techniques familiar to those skilled in the art to maximize the ability of the speech recognition engine 650 to recognize the source speech 602 from the audio signal 606.

[0051] In some examples, the noise reduction update module 634 can determine how to update the noise reduction process to improve the recognition or natural language processing of the audio signal 606 based on the sensor data 608 or one or more control parameters calculated at the stage 630. In some examples, the noise reduction update module 634 can control the parameters of the noise reduction process applied to the audio signal 606 so as to maximize the signal-to-noise ratio of the audio signal 606; this in turn can facilitate the automatic speech recognition performed by the speech recognition engine 650. For example, the noise reduction update module 634 can selectively apply signal attenuation to the frequencies of the audio signal 606 where noise may be present, while enhancing (or decreasing to attenuate) the frequencies of the audio signal 606 that carry information about the source speech 602. The sensors of the wearable system 601 can provide data to help the noise reduction update module 634 identify which frequencies of the audio signal 606 may correspond to noise and which frequencies may carry information about the source speech 602. For example, the sensors of the wearable system 601 (e.g., GPS, LIDAR, etc.) can identify that the wearable system 601 is located on an airplane. Airplanes may be associated with background noise having certain characteristic frequencies; for example, airplane engine noise may be primarily concentrated around a known frequency f0. Based on this information from the sensor, the noise reduction update module 634 may attenuate frequency f0 of the audio signal 606. Similarly, a sensor of the wearable system 601 (e.g., a microphone mounted to the wearable system 601) may identify a frequency signature corresponding to the speech of the user of the wearable system 601; the noise reduction update module 634 may apply a bandpass filter to the frequency range corresponding to the frequency signature, or may ensure that noise reduction is not applied to the frequency range.

[0052] In some examples, the echo cancellation (or dereverberation) update module 636 can determine how to update the echo cancellation unit to improve the recognition or natural language processing of the audio signal 606 based on the sensor data 608 or one or more control parameters calculated at the stage 630. In some examples, the echo cancellation update module 636 can control the parameters of the echo cancellation unit applied to the audio signal 606 so as to maximize the ability of the speech recognition engine 650 to determine the source speech 602 from the audio signal 606. For example, the echo cancellation update module 636 can instruct the echo cancellation unit to detect and correct (e.g., via a comb filter) an echo in the audio signal 606 that is 100 milliseconds after the source speech. Because the echo can interfere with the ability of the speech processing system (e.g., the speech recognition engine 650, the natural language processing engine 670) to determine the source speech (e.g., by such as the above description of the source speech). Figure 5The removal of these echoes can result in higher accuracy for speech recognition, natural language processing, and other speech processing tasks, as described at stage 530 affecting the ability to extract features from the audio signal 606. In some examples, sensors of the wearable system 601 can provide sensor data 608, which can be used to determine control parameters for echo cancellation. For example, the sensor (e.g., camera, LIDAR, radar, acoustic sensor) can determine that the user of the wearable system 601 is located ten feet from the surface and facing the surface at an angle θ1 to the normal of the surface, and further, the microphone 604 is located twenty feet from the surface and facing the surface at an angle θ2. From the sensor data, it can be calculated that the surface is likely to generate an echo at some time after the source signal 602 (i.e., the flight time from the user to the surface plus the flight time from the surface to the microphone 604). Similarly, it can be determined from the sensor data that the surface corresponds to a bathroom tile surface, or another surface with known acoustic reflection properties; and these reflection properties can be used to calculate control parameters of a dereverberation unit that will attenuate the resulting acoustic reflections in the audio signal 606.

[0053] Similarly, in some examples, signal conditioning can be applied to the speech audio signal 606 to account for the equalization imposed on the speech audio signal 606 by the acoustic environment. For example, a room can increase or attenuate certain frequencies of the speech audio signal 606, for example by virtue of the room's geometry (e.g., size, cubic volume), materials (e.g., concrete, bathroom tiles), or other characteristics that can affect the signal as detected by a microphone (e.g., microphone 604) in the acoustic environment. These effects can complicate the ability of the speech recognition engine 650 to continue to perform consistently in different acoustic environments. Sensors of the wearable system 601 can provide sensor data 608 that can be used to counteract the effects. For example, as described above, the sensor data 608 can indicate the size or shape of the room or the presence of acoustically significant materials, from which one or more filters can be determined and applied to the audio signal to counteract the effects of the room. In some examples, the sensor data 608 can be provided by a sensor (e.g., camera, LIDAR, radar, acoustic sensor) that indicates the cubic volume of the room in which the user is located; the acoustic effects of the room can be modeled as a filter; and the inversion of the filter can be applied to the audio signal to compensate for the acoustic effects.

[0054] In some examples, modules 632, 634, or 636, or other suitable elements of the example system 600 (e.g., stage 620 or stage 640) can utilize a predetermined mapping to determine control parameters based on the sensor data 608. In some of the above examples, the control parameters are calculated directly based on the sensor data 608. For example, as described above, the echo cancellation update module 636 can determine control parameters for an echo cancellation unit that can be applied to attenuate echoes present in the audio signal 606. As described above, the control parameters can be calculated by geometrically determining the distance between the wearable system 601 and the microphone 604, calculating the flight time that an audio signal propagating at the speed of sound in the air needs to propagate from the wearable system 601 to the microphone, and setting the echo period of the echo cancellation unit to correspond to the flight time. However, in some cases, the control parameters can be determined by comparing the sensor data 608 with a mapping of sensor data to control parameters. The mapping can be stored in a database, such as on a cloud server or another networked device. In some examples, one or more elements of the example system 601 (e.g., beamformer update module 632, noise reduction update module 634, echo cancellation update module 636, stage 630, stage 640, speech recognition engine 650, and / or natural language processing engine 670) may query a database to obtain one or more control parameters corresponding to the sensor data 608. This process may be performed in place of or as an additional process in addition to direct calculation of control parameters such as described above. In this process, the sensor data 608 may be provided to a predetermined mapping; and one or more control parameters that are closest to the sensor data 608 in the predetermined mapping may be returned. Using a predetermined mapping of sensor data to parameters may have multiple advantages. For example, performing a lookup from a predetermined mapping may be computationally cheaper than processing sensor data in real time, particularly when the calculation may involve complex geometric data, or where the sensors of the wearable system 601 may suffer from significant latency or bandwidth limitations. In addition, the predetermined mapping may capture relationships between sensor data and control parameters that may be difficult to calculate (e.g., through mathematical modeling) from the sensor data alone.

[0055] In some examples, machine learning techniques can be used to generate or improve the mapping of sensor data to control parameters, or otherwise determine control parameters from sensor data 608. For example, a neural network (or other appropriate machine learning technique) can be trained to identify desired control parameters based on sensor data input according to techniques familiar to those skilled in the art. Through user feedback, desired control parameters can be identified and the neural network can be further improved. For example, a user of system 600 can be prompted to rate the quality of speech recognition output (e.g., text output 660) and / or natural language processing output (e.g., natural language processing output 680). The user rating can be used to adjust the likelihood that a given set of sensor data will result in a specific set of control parameters. For example, if a user reports a high rating for text output 660 for a specific set of control parameters and a specific set of sensor data, a mapping of the sensor data to these control parameters (or a link between them) can be created. Conversely, a low rating can result in the mapping of sensor data to control parameters being weakened or deleted.

[0056] Similarly, machine learning techniques can be used to improve the ability of a speech recognition engine (e.g., 650) to distinguish between speech belonging to a user and speech belonging to other entities (e.g., speech emanating from a television or stereo system). As described above, a neural network (or other appropriate machine learning techniques) can be trained to perform this distinction (i.e., determine whether the input audio belongs to the user or to another source). In some cases, a calibration routine (in which a user provides a set of input audio known to belong to the user) can be used to train the neural network. Other suitable machine learning techniques can also be used for the same purpose.

[0057] Although update modules 632, 634, and 636 are described as including a beamforming update module, a noise reduction update module, and an echo cancellation module, respectively, other suitable modules may be included in any combination. For example, in some examples, the EQ update module may determine how to update a filtering process (such as described above) based on sensor data 608 and / or one or more control parameters calculated at stage 630. In addition, the functionality described above with respect to modules 632, 634, and 636 may be performed as part of other elements of the example system 600 (such as stage 630 and / or stage 640, or speech recognition engine 650 or natural language processing engine 670).

[0058] Although the disclosed examples have been fully described with reference to the accompanying drawings, it is noted that various changes and modifications will become apparent to those skilled in the art. For example, elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. Such changes and modifications should be understood to be included within the scope of the disclosed examples as defined by the appended claims.

Claims

1. A method for processing an acoustic signal, the method comprising: include: receiving, via the first microphone, a first acoustic signal associated with a first speech of a user of the wearable headband unit; receiving a first sensor input via a sensor; determining a characteristic frequency curve associated with the first speech of the user; determining a control parameter based on the sensor input and further based on the characteristic frequency curve associated with the first speech of the user; and applying the control parameter to one or more of the first acoustic signal, the wearable headband unit, and the first microphone, Wherein, determining the control parameter includes: determining a relationship between the first speech and the first acoustic signal based on the first sensor input.

2. The method according to claim 1, in, The control parameters are applied to the first acoustic signal to generate a second acoustic signal, and the method further includes providing the second acoustic signal to a speech recognition engine to generate a text output corresponding to the first speech.

3. The method according to claim 1, in, The control parameter is applied to the first acoustic signal to generate a second acoustic signal, and the method further includes providing the second acoustic signal to a natural language processing engine to generate natural language data corresponding to the first speech.

4. The method according to claim 1, in, The wearable headband unit includes the first microphone.

5. The method according to claim 1, in, Determining the control parameter based on the sensor input includes: detecting a surface based on the sensor input; and determining an effect of the surface on a relationship between the first speech and the first acoustic signal, wherein the control parameter, when applied to the one or more of the first acoustic signal, the wearable headband unit, and the first microphone, reduces the effect of the surface on the relationship between the first speech and the first acoustic signal.

6. The method of claim 5, further comprising determining acoustic properties of the surface, in, The influence of the surface on the relationship between the first speech and the first acoustic signal is determined based on the acoustic characteristic.

7. The method according to claim 1, in, Determining the control parameter based on the sensor input includes: detecting a person different from the user based on the sensor input; and determining an effect of the person's speech on a relationship between the first speech and the first acoustic signal, wherein the control parameter, when applied to the one or more of the first acoustic signal, the wearable headband unit, and the first microphone, reduces the effect of the speech on the relationship between the first speech and the first acoustic signal.

8. The method according to claim 1, in, Determining the control parameter based on the sensor input includes applying the sensor input to an input of an artificial neural network.

9. The method according to claim 1, in, The control parameters include control parameters for an echo cancellation module, and determining the control parameters based on the sensor input includes: detecting a surface based on the sensor input; and A time of flight between the surface and the first microphone is determined.

10. The method according to claim 1, in, The control parameters include control parameters for a beamforming module, and determining the control parameters based on the sensor input includes determining a time of flight between the user and the first microphone.

11. The method according to claim 1, in, The control parameters include control parameters for a noise reduction module, and determining the control parameters based on the sensor input includes determining frequencies to be attenuated in the first acoustic signal.

12. The method according to claim 1, in, The wearable headband unit includes a second microphone, The sensor input comprises a second acoustic signal detected via the second microphone, and The control parameter is determined based on a difference between the first acoustic signal and the second acoustic signal.

13. The method according to claim 1, in, The wearable headband unit includes a plurality of microphones excluding the first microphone, The method further includes: receiving, via the plurality of microphones, a plurality of acoustic signals associated with the first speech; and The control parameter is determined based on a difference between the first acoustic signal and the plurality of acoustic signals.

14. The method according to claim 1, in, The sensor is coupled to the wearable headband unit.

15. The method according to claim 1, in, The sensor is located in the user's environment.

16. The method according to claim 1, in: the first sensor input being indicative of a location of the user, The control parameters include a filter having a cutoff frequency, and The cutoff frequency is determined based on the location of the user.

17. A system, include: A wearable headband unit comprising: a display for displaying a mixed reality environment to a user; Speakers; and One or more processors configured to perform a method comprising: Receiving, via a first microphone, a first acoustic signal associated with a first voice of the user; determining a characteristic frequency curve associated with the first speech of the user; receiving a first sensor input via a sensor; determining a control parameter based on the sensor input and further based on the characteristic frequency curve associated with the first speech of the user; and applying the control parameter to one or more of the first acoustic signal, the wearable headband unit, and the first microphone, Wherein, determining the control parameter includes: determining a relationship between the first speech and the first acoustic signal based on the first sensor input.

18. The system according to claim 17, in, The control parameters are applied to the first acoustic signal to generate a second acoustic signal, and the method further includes providing the second acoustic signal to a speech recognition engine to generate a text output corresponding to the first speech.

19. The system according to claim 17, in, The control parameter is applied to the first acoustic signal to generate a second acoustic signal, and the method further includes providing the second acoustic signal to a natural language processing engine to generate natural language data corresponding to the first speech.

20. The system according to claim 17, in, The wearable headband unit further includes the first microphone.

21. The system according to claim 17, in, Determining the control parameter based on the sensor input includes: detecting a surface based on the sensor input; and determining an effect of the surface on a relationship between the first speech and the first acoustic signal, wherein the control parameter, when applied to the one or more of the first acoustic signal, the wearable headband unit, and the first microphone, reduces the effect of the surface on the relationship between the first speech and the first acoustic signal.

22. The system according to claim 21, in, The method further comprises determining an acoustic property of the surface, and wherein the influence of the surface on the relationship between the first speech and the first acoustic signal is determined based on the acoustic property.

23. The system according to claim 17, in, Determining the control parameter based on the sensor input includes: detecting a person different from the user based on the sensor input; and determining an effect of the person's speech on a relationship between the first speech and the first acoustic signal, wherein the control parameter, when applied to the one or more of the first acoustic signal, the wearable headband unit, and the first microphone, reduces the effect of the speech on the relationship between the first speech and the first acoustic signal.

24. The system according to claim 17, in, Determining the control parameter based on the sensor input includes applying the sensor input to an input of an artificial neural network.

25. The system according to claim 17, in, The control parameters include control parameters for an echo cancellation module, and determining the control parameters based on the sensor input includes: detecting a surface based on the sensor input; and A time of flight between the surface and the first microphone is determined.

26. The system according to claim 17, in, The control parameters include control parameters for a beamforming module, and determining the control parameters based on the sensor input includes determining a time of flight between the user and the first microphone.

27. The system according to claim 17, in, The control parameters include control parameters for a noise reduction module, and determining the control parameters based on the sensor input includes determining frequencies to be attenuated in the first acoustic signal.

28. The system according to claim 17, in, The wearable headband unit includes a second microphone, The sensor input comprises a second acoustic signal detected via the second microphone, and The control parameter is determined based on a difference between the first acoustic signal and the second acoustic signal.

29. The system according to claim 17, in: The wearable headband unit includes a plurality of microphones excluding the first microphone, The method further includes: receiving, via the plurality of microphones, a plurality of acoustic signals associated with the first speech; and The control parameter is determined based on a difference between the first acoustic signal and the plurality of acoustic signals.

30. The system according to claim 17, in, The sensor is coupled to the wearable headband unit.

31. The system according to claim 17, in, The sensor is located in the user's environment.

Images