Determining input for a speech processing engine

The system uses wearable sensors to filter non-intended speech inputs by analyzing audio and sensor data, improving speech processing accuracy in uncontrolled environments.

JP7580383B2Active Publication Date: 2024-11-11MAGIC LEAP INC
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2021551538
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-03-01
Filing Date
2020-02-28
Publication Date
2024-11-11
Estimated Expiration
2040-02-28

AI Technical Summary

Technical Problem

Conventional speech processing engines struggle with accurately identifying user-intended speech inputs in uncontrolled environments due to ambient noise and non-directed audio signals, leading to computational resource waste and reduced accuracy.

Method used

A system that utilizes wearable sensors to determine the probability of user-directed speech by comparing audio signals with sensor data, filtering out non-intended inputs using probabilistic models and thresholds to enhance accuracy.

Benefits of technology

Improves speech processing accuracy by isolating intended speech inputs, reducing computational overhead, and enhancing reliability in uncontrolled environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007580383000001
    Figure 0007580383000001
  • Figure 0007580383000002
    Figure 0007580383000002
  • Figure 0007580383000003
    Figure 0007580383000003
Patent Text Reader

Abstract

A method for presenting a signal to a speech processing engine is disclosed. According to an embodiment of the method, an audio signal is received via a microphone. A portion of the audio signal is identified and a probability that the portion comprises speech directed by a user of the speech processing engine as input to the speech processing engine is determined. In accordance with a determination that the probability exceeds a threshold, the portion of the audio signal is presented as input to the speech processing engine. In accordance with a determination that the probability does not exceed the threshold, the portion of the audio signal is not presented as input to the speech processing engine.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Application No. 62 / 812,959, filed March 1, 2019, the contents of which are incorporated herein by reference in their entirety.

[0002] The present disclosure relates generally to systems and methods for processing speech signals, and more particularly, to systems and methods for processing speech signals for presentation to a speech processing engine. [Background technology]

[0003] A system for speech recognition is typically tasked with receiving audio input representing human speech via one or more microphones, processing the audio input, and determining words, logical structures, or other output that correspond to the audio input. For example, an automatic speech recognition (ASR) system may generate text output based on a human speech that corresponds to an audio input signal, and a natural language processing (NLP) tool may generate logical structures or computer data that correspond to the meaning of the human speech. While such systems may contain any number of components, at the core of such systems is a speech processing engine, which is a component that receives an audio signal as an input, performs some recognition logic on the input, and outputs some text that corresponds to the input. (Although reference is made herein to a speech processing engine, other forms of speech processing in addition to speech recognition should also be considered within the scope of this disclosure.)

[0004] Historically, audio input, as detected via a microphone, was provided to the speech processing engine in a structured and predictable manner. For example, in response to a first prompt (e.g., "Please start speaking"), a user may speak directly into a desktop computer's microphone immediately after pressing a first button input (e.g., a "Start" or "Record" button or a microphone icon in a software interface) or after a significant period of silence. Similarly, in response to a second prompt (e.g., "Please stop speaking"), a user may stop providing microphone input immediately before pressing a second button input (e.g., a "Stop" or "Pause" button) or by remaining silent for a period of time. Such structured input sequences leave little room for doubt as to when the user was providing input to the speech processing engine (e.g., between the first and second prompts or between pressing the start and stop buttons). Moreover, such systems typically required intentional action on the part of the user, so that it could generally be assumed that the user's speech input was directed to the speech processing engine and not to some other listener (e.g., a person in an adjacent room). Thus, many speech processing engines at the time may not have had any particular need to distinguish, from the microphone input, those portions of the input that were intended to be directed to the speech processing engine to provide speech recognition input, and vice versa, the unintended portions.

[0005] The manner in which users provide speech recognition input is changing as speech processing engines become more prevalent and more fully integrated into users' daily lives. For example, several automated voice assistants are now stored in or otherwise integrated with home appliances, automobile dashboards, smartphones, wearable devices, "living room" devices (e.g., devices with integrated "smart" voice assistants), and other environments quite different from traditional desktop computers. In many cases, this level of integration into daily life makes speech processing engines more widely usable. However, these systems would be cluttered with system prompts, button inputs, and other conventional mechanisms for demarcating microphone input to the speech processing engine. Instead, some such systems place one or more microphones in an "always on" state, where the microphone listens for a "wake-up word" (e.g., the "name" of the device or any other predefined word or phrase) that indicates the start of a speech recognition input sequence. In response to detection of the wake-up word, the speech processing engine can process the ensuing sequence of microphone inputs as input to the speech processing engine.

[0006] Wake-up word systems replace the need for a discrete prompt or button input for the speech processing engine, but they do run the risk of false positives, such as when the wake-up word is spoken by a user without the intention of activating the speech processing engine, or inadvertently "spoken" by a television or music speaker in the vicinity of the speech processing engine. It may be desirable to replace the wake-up word system with a more intelligent method that determines, based on speech input, whether the user intends to provide input to the speech processing engine with respect to that utterance, i.e., whether the user's utterance is "input speech" as opposed to "non-input speech". This would allow users to interact with the speech processing engine more naturally (e.g., without the need to invoke a dedicated wake-up word) and encourage the use of such a system in everyday environments and situations.

[0007] A related problem with speech processing engines occupying everyday spaces is that these engines may detect a large amount of ambient noise, speech not directed to the engine, or other audio signals that are not intended as input. For example, a speech processing engine in a living room will detect not only the user's intended speech processing engine input (e.g., "What's the weather forecast for Tuesday?"), but also sounds and speech from pets, devices (e.g., TV speakers), or other people, ambient noise, or parts of the user's speech that are directed to other people in the vicinity. Processing these non-input parts of the audio signal can waste computing resources and impair the accuracy of the speech processing engine, which may already be limited in some uncontrolled environments (e.g., outdoors). It is desirable for the speech processing engine to identify from the microphone input the parts of the microphone input (input speech) that represent input intended for the speech processing engine, and not consider the unintended parts (non-input speech).

[0008] Additionally, it may be desirable to use sensor-equipped systems, including those incorporating wearable head-mounted units, to improve the speech processing engine's ability to identify and not consider audio inputs that are not intended for the speech processing engine. Sensor data (e.g., data from individual sensors or from multiple sensors fused together by an inertial measurement unit) can be used by the speech processing engine to help identify and process only those portions of the audio input signal that are input speech, allowing the output of those systems to produce more accurate and more reliable results. In particular, sensors on wearable devices can be particularly useful. For example, such sensors can indicate the user's position and orientation, the user's eye movements and eye gaze targets, the user's hand movements, and biometric data (e.g., vital signs such as heart rate and respiration rate). In many cases, these indications can provide the speech recognition system with the same kinds of non-verbal cues (such as the user's movements and body language) that humans use to intuitively understand another person speaking. In addition, wearable systems are well suited for mobile outdoor applications, i.e. precisely the type of applications where many conventional speech processing engines can perform particularly poorly. Summary of the Invention [Means for solving the problem]

[0009] An embodiment of the present disclosure describes a system and method for presenting a signal to a speech processing engine. According to an exemplary method, an audio signal is received via one or more microphones. A portion of the audio signal is identified and a probability is determined that the portion comprises speech directed by a user of the speech processing engine as an input to the speech processing engine. In accordance with a determination that the probability exceeds a threshold, the portion of the audio signal is presented as an input to the speech processing engine. In accordance with a determination that the probability does not exceed a threshold, the portion of the audio signal is not presented as an input to the speech processing engine. The present specification also provides, for example, the following items: (Item 1) 1. A method of presenting a signal to a speech processing engine, the method comprising: receiving a first audio signal via a first microphone; identifying a first portion of the first audio signal; determining, for a first portion of the first audio signal, a first probability that the first portion comprises speech directed by a user of the speech processing engine as input to the speech processing engine; presenting a first portion of the first audio signal as an input to the speech processing engine in response to a determination that the first probability exceeds a first threshold; in response to a determination that the first probability does not exceed the first threshold, not presenting a first portion of the first audio signal as an input to the speech processing engine. A method comprising: (Item 2) 2. The method of claim 1, wherein the first probability is determined based on a comparison of a first portion of the first audio signal with a plurality of audio signals in a database, each audio signal of the plurality of audio signals being associated with a probability that that individual audio signal comprises speech directed as input to the speech processing engine. (Item 3) 2. The method of claim 1, further comprising receiving first sensor data from a first sensor associated with the user, the first probability being determined based on a comparison of the first sensor data with a plurality of sensor data in a database, each sensor data of the plurality of sensor data in the database being associated with an audio signal and further associated with a probability that the respective audio signal comprises speech directed as input to the speech processing engine. (Item 4) 4. The method of claim 3, wherein the first sensor is associated with a wearable head unit configured to be worn by a user. (Item 5) 5. The method of claim 4, wherein the first sensor data indicates one or more of the user's position, orientation, eye movement, eye gaze target, or vital signs. (Item 6) 6. The method of claim 5, wherein the first portion of the first audio signal is identified based on the first sensor data. (Item 7) identifying a second portion of the first audio signal; determining, for a second portion of the first audio signal, a second probability that the second portion comprises speech directed by a user as input to the speech processing engine; presenting a second portion of the first audio signal as an input to the speech processing engine in response to a determination that the second probability exceeds the first threshold; in response to a determination that the second probability does not exceed the first threshold, not presenting a second portion of the first audio signal as an input to the speech processing engine. The method of claim 1, further comprising: (Item 8) 8. The method of claim 7, wherein the second probability is determined based, at least in part, on the first probability. (Item 9) 1. A system for providing input to a speech processing engine, the system comprising: A microphone; A circuitry comprising: receiving a first audio signal via the microphone; identifying a first portion of the first audio signal; determining, for a first portion of the first audio signal, a first probability that the first portion comprises speech directed by a user as input to the speech processing engine; presenting a first portion of the first audio signal as an input to the speech processing engine in response to a determination that the first probability exceeds a first threshold; in response to a determination that the first probability does not exceed the first threshold, not presenting a first portion of the first audio signal as an input to the speech processing engine. and a circuitry configured to implement the Including, the system. (Item 10) 10. The system of claim 9, wherein the first probability is determined based on a comparison of the first portion of the first audio signal with a plurality of audio signals in a database, each audio signal of the plurality of audio signals being associated with a probability that that individual audio signal comprises speech directed as input to the speech processing engine. (Item 11) The system further includes a sensor, and the circuitry further comprises: 10. The system of claim 9, further comprising: receiving first sensor data from the user via the sensor; and the first probability is determined based on a comparison of the first sensor data with a plurality of sensor data in a database, each sensor data of the plurality of sensor data in the database being associated with an audio signal and further associated with a probability that the respective audio signal comprises speech directed as input to the speech processing engine. (Item 12) Item 12. The method of item 11, wherein the first sensor data indicates one or more of the user's position, orientation, eye movement, eye gaze target, or vital signs. (Item 13) Item 13. The system of item 12, wherein the first portion of the first audio signal is identified based on the first sensor data. (Item 14) The circuitry further comprises: identifying a second portion of the first audio signal; determining, for a second portion of the first audio signal, a second probability that the second portion comprises speech directed by a user as input to the speech processing engine; presenting a second portion of the first audio signal as an input to the speech processing engine in response to a determination that the second probability exceeds the first threshold; in response to a determination that the second probability does not exceed the first threshold, not presenting a second portion of the first audio signal as an input to the speech processing engine. 10. The system of claim 9, configured to perform the steps of: (Item 15) Item 15. The system of item 14, wherein the second probability is determined based, at least in part, on the first probability. (Item 16) 10. The system of claim 9, wherein the system includes a wearable head unit including the microphone and the circuitry. (Item 17) 10. The system of claim 9, wherein the system includes a vehicle including the microphone and the circuitry. (Item 18) 10. The system of claim 9, wherein the system includes an electronic voice assistant including the microphone and the circuitry. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an exemplary wearable system according to some embodiments of the present disclosure.

[0011] [Diagram 2] FIG. 2 illustrates an example handheld controller that may be used with an example wearable system according to some embodiments of the present disclosure.

[0012] [Diagram 3] FIG. 3 illustrates an example auxiliary unit that may be used with an example wearable system according to some embodiments of the present disclosure.

[0013] [Figure 4] FIG. 4 illustrates an example functional block diagram for an example wearable system according to some embodiments of the present disclosure.

[0014] [Diagram 5] FIG. 5 illustrates a flow chart of an example system for processing an acoustic speech signal according to some embodiments of the present disclosure.

[0015] [Figure 6A] 6A-6D illustrate examples of processing an acoustic speech signal according to some embodiments of the present disclosure. [Figure 6B] 6A-6D illustrate examples of processing an acoustic speech signal according to some embodiments of the present disclosure. [Figure 6C] 6A-6D illustrate examples of processing an acoustic speech signal according to some embodiments of the present disclosure. [Figure 6D] 6A-6D illustrate examples of processing an acoustic speech signal according to some embodiments of the present disclosure.

[0016] [Figure 7A] 7A-7C illustrate a flowchart of an example system for processing an acoustic speech signal according to some embodiments of the present disclosure. [Figure 7B] 7A-7C illustrate a flowchart of an example system for processing an acoustic speech signal according to some embodiments of the present disclosure. [Figure 7C] 7A-7C illustrate a flowchart of an example system for processing an acoustic speech signal according to some embodiments of the present disclosure.

[0017] [Figure 8] FIG. 8 illustrates a flowchart showing portions of an example system for processing acoustic speech signals according to some embodiments of the present disclosure.

[0018] [Figure 9] FIG. 9 illustrates a flowchart showing portions of an example system for processing acoustic speech signals according to some embodiments of the present disclosure.

[0019] [Figure 10] FIG. 10 illustrates a user interacting with one or more recipients, according to some embodiments of the present disclosure.

[0020] [Figure 11] FIG. 11 illustrates an example process for capturing audio and non-audio classifier training data according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] In the following description of the embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown, by way of illustration, specific embodiments which may be practiced. It is to be understood that other embodiments may be used and structural changes may be made without departing from the scope of the disclosed embodiments.

[0022] Exemplary Wearable System

[0023] 1 illustrates an exemplary wearable head device 100 configured to be worn on a user's head. The wearable head device 100 may be part of a broader wearable system that includes one or more components, such as a head device (e.g., wearable head device 100), a handheld controller (e.g., handheld controller 200 described below), and / or an auxiliary unit (e.g., auxiliary unit 300 described below). In some examples, the wearable head device 100 can be used for virtual reality, augmented reality, or mixed reality systems or applications. The wearable head device 100 includes one or more displays, such as displays 110A and 110B (which may comprise left and right transmissive displays and associated components for coupling light from the displays to the user's eyes, such as orthogonal pupil expansion (OPE) grating set 112A / 112B and exit pupil expansion (EPE) grating set 114A / 114B), left and right acoustic structures, such as speakers 120A and 120B (which may be mounted on temple arms 122A and 122B, respectively, and positioned adjacent the user's left and right ears), and an infrared sensor. The wearable head device 100 may include one or more sensors, such as a GPS unit, an accelerometer, a GPS unit, an inertial measurement unit (IMU, e.g., IMU 126), an acoustic sensor (e.g., microphone 150), a quadrature coil electromagnetic receiver (e.g., receiver 127 shown mounted on left temple arm 122A), left and right cameras oriented away from the user (e.g., depth (time of flight) cameras 130A and 130B), and left and right eye cameras oriented toward the user (e.g., for detecting the user's eye movements) (e.g., eye cameras 128A and 128B). However, the wearable head device 100 may incorporate any suitable display technology and any suitable number, type, or combination of sensors or other components without departing from the scope of the invention.In some embodiments, the wearable head device 100 may include one or more microphones 150 configured to detect audio signals generated by the user's voice, and such microphones may be positioned adjacent to the user's mouth. In some embodiments, the wearable head device 100 may incorporate networking features (e.g., Wi-Fi capabilities) to communicate with other devices and systems, including other wearable systems. The wearable head device 100 may further include components such as a battery, a processor, memory, a storage unit, or various input devices (e.g., buttons, touchpads), or may be coupled to a handheld controller (e.g., handheld controller 200) or auxiliary unit (e.g., auxiliary unit 300) that includes one or more such components. In some embodiments, the sensors may be configured to output a set of coordinates of the head-mounted unit relative to the user's environment and may provide input to a processor to implement a simultaneous localization and mapping (SLAM) procedure and / or a visual odometry algorithm. In some embodiments, the wearable head device 100 may be coupled to a handheld controller 200 and / or an auxiliary unit 300, as described further below.

[0024] FIG. 2 illustrates an exemplary mobile handheld controller component 200 of an exemplary wearable system. In some examples, the handheld controller 200 may communicate wired or wirelessly with the wearable head device 100 and / or the auxiliary unit 300 described below. In some examples, the handheld controller 200 includes a handle portion 220 to be held by a user and one or more buttons 240 disposed along a top surface 210. In some examples, the handheld controller 200 may be configured for use as an optical tracking target, e.g., a sensor (e.g., a camera or other optical sensor) of the wearable head device 100 may be configured to detect the position and / or orientation of the handheld controller 200, which in turn may indicate the position and / or orientation of a user's hand holding the handheld controller 200. In some examples, the handheld controller 200 may include a processor, a memory, a storage unit, a display, or one or more input devices such as those described above. In some examples, the handheld controller 200 includes one or more sensors (e.g., any of the sensors or tracking components described above with respect to the wearable head device 100). In some examples, the sensors can detect a position or orientation of the handheld controller 200 relative to the wearable head device 100 or relative to another component of the wearable system. In some examples, the sensors may be located in the handle portion 220 of the handheld controller 200 and / or may be mechanically coupled to the handheld controller. The handheld controller 200 can be configured to provide one or more output signals corresponding, for example, to a press state of the button 240, or to a position, orientation, and / or movement of the handheld controller 200 (e.g., via an IMU). Such output signals may be used as inputs to a processor of the wearable head device 100, to the auxiliary unit 300, or to another component of the wearable system.In some embodiments, the handheld controller 200 may include one or more microphones to detect sounds (e.g., a user's speech, environmental sounds) and, in some cases, provide signals corresponding to the detected sounds to a processor (e.g., a processor of the wearable head device 100).

[0025] FIG. 3 illustrates an exemplary auxiliary unit 300 of an exemplary wearable system. In some examples, the auxiliary unit 300 may communicate wired or wirelessly with the wearable head device 100 and / or the handheld controller 200. The auxiliary unit 300 may include a battery to provide energy to operate one or more components of the wearable system, such as the wearable head device 100 and / or the handheld controller 200 (including a display, a sensor, an acoustic structure, a processor, a microphone, and / or other components of the wearable head device 100 or the handheld controller 200). In some examples, the auxiliary unit 300 may include a processor, a memory, a storage unit, a display, one or more input devices, and / or one or more sensors, such as those described above. In some examples, the auxiliary unit 300 includes a clip 310 for attaching the auxiliary unit to a user (e.g., a belt worn by the user). An advantage of using the auxiliary unit 300 to store one or more components of a wearable system is that doing so may allow a large or heavy component to be carried on the user's waist, chest, or back, which are relatively better suited for supporting large and heavy objects, rather than being mounted on the user's head (e.g., when stored in the wearable head device 100) or carried by the user's hands (e.g., when stored in the handheld controller 200). This may be particularly advantageous with respect to relatively heavy or bulky components, such as batteries.

[0026] FIG. 4 illustrates an example functional block diagram that may correspond to an example wearable system 400, such as may include the example wearable head device 100, handheld controller 200, and auxiliary unit 300 described above. In some examples, the wearable system 400 may be used for virtual reality, augmented reality, or mixed reality applications. As shown in FIG. 4, the wearable system 400 may include an example handheld controller 400B, referred to herein as a "totem" (and that may correspond to the handheld controller 200 described above), which may include a totem / headgear six degree of freedom (6DOF) totem subsystem 404A. The wearable system 400 may also include an example headgear device 400A (which may correspond to the wearable headgear device 100 described above), which includes a totem / headgear 6DOF headgear subsystem 404B. In an embodiment, the 6DOF totem subsystem 404A and the 6DOF headgear subsystem 404B cooperate to determine six coordinates (e.g., offsets in three translation directions and rotations along three axes) of the handheld controller 400B relative to the headgear device 400A. The six degrees of freedom may be expressed relative to the coordinate system of the headgear device 400A. The three translation offsets may be expressed as X, Y, and Z offsets in such coordinate system, a translation matrix, or some other representation. The rotational degrees of freedom may be expressed as a sequence of yaw, pitch, and roll rotations, a vector, a rotation matrix, a quaternion, or some other representation. In some embodiments, one or more depth cameras 444 (and / or one or more non-depth cameras) and / or one or more optical targets (e.g., buttons 240 of the handheld controller 200 as described above or dedicated optical targets included in the handheld controller) included in the headgear device 400A can be used for 6DOF tracking.In some embodiments, the handheld controller 400B can include a camera as described above, and the headgear device 400A can include an optical target for optical tracking in conjunction with the camera. In some embodiments, the headgear device 400A and the handheld controller 400B each include a set of three orthogonally oriented solenoids that are used to wirelessly transmit and receive three distinguishable signals. By measuring the relative magnitudes of the three distinguishable signals received in each of the coils used to receive, the 6DOF of the handheld controller 400B relative to the headgear device 400A can be determined. In some embodiments, the 6DOF totem subsystem 404A can include an inertial measurement unit (IMU), which is useful for providing improved accuracy and / or more timely information regarding high speed movements of the handheld controller 400B.

[0027] In some implementations involving augmented or mixed reality applications, it may be desirable to transform coordinates from a local coordinate space (e.g., a coordinate space fixed relative to the headgear device 400A) to an inertial coordinate space or to an environmental coordinate space. For example, such a transformation may be necessary for the display of the headgear device 400A to present virtual objects in an expected position and orientation relative to the real environment (e.g., a virtual person sitting in a real chair facing forward, regardless of the position and orientation of the headgear device 400A), rather than in a fixed position and orientation on the display (e.g., in the same position on the display of the headgear device 400A). This can maintain the illusion that the virtual objects exist in the real environment (and do not appear unnaturally positioned in the real environment, for example, as the headgear device 400A shifts and rotates). In some embodiments, a compensation transformation between coordinate spaces can be determined by processing images from the depth camera 444 (e.g., using simultaneous localization and mapping (SLAM) and / or visual odometry procedures) to determine a transformation of the headgear device 400A relative to an inertial or environmental coordinate system. In the embodiment shown in FIG. 4, the depth camera 444 can be coupled to the SLAM / visual odometry block 406 and can provide images to the block 406. The SLAM / visual odometry block 406 implementation can include a processor configured to process the images and then determine the position and orientation of the user's head, which can be used to identify a transformation between the head coordinate space and the real coordinate space. Similarly, in some embodiments, an additional source of information regarding the user's head pose and location is obtained from the IMU 409 of the headgear device 400A. Information from the IMU 409 can be integrated with information from the SLAM / visual odometry block 406 to provide improved accuracy and / or more timely information regarding fast adjustments of the user's head pose and position.

[0028] In some embodiments, the depth camera 444 can provide 3D images to a hand gesture tracker 411, which can be implemented within a processor of the headgear device 400A. The hand gesture tracker 411 can identify the user's hand gestures, for example, by matching the 3D images received from the depth camera 444 to stored patterns representing hand gestures. Other suitable techniques for identifying the user's hand gestures will also be apparent.

[0029] In some embodiments, one or more processors 416 may be configured to receive data from the headgear subsystem 404B, the IMU 409, the SLAM / visual odometry block 406, the depth camera 444, the microphone 450, and / or the hand gesture tracker 411. The processor 416 may also send and receive control signals to the 6DOF totem system 404A. The processor 416 may be wirelessly coupled to the 6DOF totem system 404A, such as in embodiments where the handheld controller 400B is not tethered. The processor 416 may further communicate with additional components, such as an audiovisual content memory 418, a graphical processing unit (GPU) 420, and / or a digital signal processor (DSP) audio spatializer 422. The DSP audio spatializer 422 may be coupled to a head-related transfer function (HRTF) memory 425. The GPU 420 may include a left channel output coupled to a left source of imagewise modulated light 424 and a right channel output coupled to a right source of imagewise modulated light 426. The GPU 420 may output stereoscopic image data to the imagewise modulated light sources 424, 426. The DSP audio spatializer 422 may output audio to the left speaker 412 and / or the right speaker 414. The DSP audio spatializer 422 may receive an input from the processor 419 indicating a direction vector from the user to a virtual sound source (which may be moved by the user, e.g., via the handheld controller 400B). Based on the direction vector, the DSP audio spatializer 422 may determine a corresponding HRTF (e.g., by accessing an HRTF or by interpolating multiple HRTFs). The DSP audio spatializer 422 may then apply the determined HRTF to an audio signal, such as an audio signal corresponding to a virtual sound generated by a virtual object.This can improve the believability and realism of virtual sounds by incorporating the user's relative position and orientation to the virtual sounds in the mixed reality environment, i.e., by presenting virtual sounds that match the user's expectations of what they would hear if the virtual sounds were real sounds in a real environment.

[0030] 4, one or more of the processor 416, GPU 420, DSP audio spatializer 422, HRTF memory 425, and audio / visual content memory 418 may be included within an auxiliary unit 400C (which may correspond to auxiliary unit 300 described above). The auxiliary unit 400C may include a battery 427 to power its components and / or provide power to the headgear device 400A and / or the handheld controller 400B. Including such components within an auxiliary unit, which may be mounted on the user's waist, can limit the size and weight of the headgear device 400A, which in turn can reduce fatigue in the user's head and neck.

[0031] 4 presents elements corresponding to various components of the exemplary wearable system 400, various other suitable arrangements of these components will be apparent to those skilled in the art. For example, elements presented in FIG. 4 as being associated with the auxiliary unit 400C may instead be associated with the headgear device 400A or the handheld controller 400B. Furthermore, some wearable systems may dispense with the handheld controller 400B or the auxiliary unit 400C entirely. Such variations and modifications are to be understood as falling within the scope of the disclosed embodiments.

[0032] Speech Processing Engine

[0033] A speech recognition system generally includes a speech processing engine that may receive an input audio signal corresponding to human speech (a source signal), process and analyze the input audio signal, and produce, as a result of the analysis, an output corresponding to the human speech. In the case of an automatic speech recognition (ASR) system, for example, the output of the speech processing engine may be a text transcript of the human speech. In the case of a natural language processing system, the output may be one or more commands or instructions indicated by the human speech, or a semantically meaningful representation (e.g., a logical representation or data structure) of the human speech. Other types of speech processing systems (e.g., automatic translation systems), including those that do not necessarily "recognize" speech, are also contemplated and are within the scope of this disclosure.

[0034] Speech recognition systems are found in a wide variety of products and applications, namely, traditional telephone systems, automated voice messaging systems, voice assistants (including stand-alone and smartphone-based voice assistants), vehicles and aircraft, desktop and document processing software, data entry, home appliances, medical devices, language translation software, closed captioning systems, and others. An advantage of speech recognition systems is that they can enable users to provide input to computer systems using naturally spoken language as presented to one or more microphones instead of traditional computer input devices such as keyboards or touch panels. Thus, speech recognition systems can be particularly useful in environments where traditional input devices (e.g., keyboards) may be unavailable or impractical. Furthermore, by allowing users to provide intuitive voice-based input, speech processing engines can enhance the sense of immersion. Speech recognition may therefore be a natural fit for wearable systems, particularly for virtual reality, augmented reality, and / or mixed reality applications of wearable systems where user immersion is a primary goal and it may be desirable to limit the use of traditional computer input devices, whose presence may detract from the sense of immersion.

[0035] Typically, the output of any speech processing engine will not correspond with perfect certainty to the source human speech. Even advanced speech processing engines will not consistently produce perfect text output for all speakers due to the many variables that may affect the audio signal provided as input. For example, the reliability of a speech processing engine may depend heavily on the quality of the input audio signal. If the input audio signal is recorded under ideal conditions, e.g., in an acoustically controlled environment with a single human speaker speaking clearly, directly, and at close range into a microphone, the source speech may be more easily determined from the audio signal. However, in real-world applications, the input audio signal may deviate from ideal conditions such that it becomes more difficult to determine the source human speech. For example, the input audio signal may contain significant ambient noise, or speech from multiple speakers in addition to the user. For example, speech from other people, pets, or electronic devices (e.g., a television) may be mixed with the user's speech in the input signal. In addition, even the user's speech may include not only speech intended for the speech processing engine (input speech), but also speech directed to other listeners (other people, pets, or other devices, etc.) By isolating the input speech from the broader input audio signal, the fidelity of the input processed by the speech processing engine can be improved, and therefore the accuracy of the speech processing engine's output can be improved.

[0036] Identifying and segmenting input utterances

[0037] The present disclosure is directed to systems and methods for improving the accuracy of speech processing systems by removing from raw speech signals those portions of the signal that are not directed to the speech processing system by a user. As described herein, such non-input portions can be identified (e.g., classified) based on audio characteristics of the speech signal itself (e.g., sudden changes in the vocabulary, semantics, or grammar of the speech) and / or by using input from a sensor associated with a wearable device (e.g., a head-mounted device as described above with respect to FIG. 1). Such non-input portions may be particularly noticeable in mobile applications of speech processing, in-home use of speech processing systems, or applications of speech processing in uncontrolled environments, such as outdoor environments, where other audio or ambient noise may be present. Wearable systems are frequently intended for use in such applications and therefore may be particularly susceptible to non-directed speech. For example, when some wearable systems are intended for use in uncontrolled environments, there may be a high potential that environmental noise (or other human speech) will be recorded along with the target human speech. Sensors in a wearable system (such as those described above with respect to FIGS. 1-4) are well suited to solving this problem, as described herein, however, in some embodiments, directionality can be determined based solely on the speech signal, without benefit of sensor input, as described herein.

[0038] 5 illustrates an example system 500, according to some embodiments, in which a speech processing engine 550 produces a text output 552 (as described above) based on a raw speech signal 510 provided as input. In some examples, the raw speech signal 510 may be provided as detected by one or more microphones, while in some examples it may be provided from a data file (e.g., an audio waveform file), from an audio stream (e.g., provided over a network), or from any other suitable source. In the system 500, improved accuracy of the text output 552 may be achieved by presenting as input to the speech processing engine 550 a “directed” speech signal 540 that includes only those portions of the raw input speech signal 510 that are determined to constitute input speech that is directed to the speech processing engine 550 (as opposed to, e.g., irrelevant speech as described above). The directed speech signal 540 can be determined in stage 530 from the raw input speech signal 510 and / or from sensor data 520, which may correspond to data from sensors as described above with respect to the exemplary wearable head device 100 in FIG. 1.

[0039] In stage 530, the raw speech signal 510 can be divided into individual speech segments, and then for each segment, a probability can be determined that the segment corresponds to an input utterance intended as input for the speech processing engine 550. In some cases, probabilistic modeling or machine learning techniques can indicate this probability for each segment of the raw speech signal 510. The directed speech signal 540 can then be generated from the raw speech signal 510 by filtering segments of the raw speech signal 510 that do not meet a threshold probability of corresponding to input speech (as opposed to non-input speech). (As used herein, input speech can include input audio provided by a particular user and directed by the user towards a speech recognition system.)

[0040] 6A-6D respectively illustrate examples of a raw speech signal, a segmented version of the speech signal, a probabilistic model of the raw speech signal (although in some embodiments machine learning techniques may be used), and a directed speech signal generated from the raw speech signal. FIG. 6A shows an example audio waveform 600 (which may correspond to the raw speech signal 510) represented as amplitude (e.g., in voltage) V(t) as a function of time, as may be detected by one or more microphones and / or represented in a waveform audio file. In the example, waveform 600 corresponds to a user's utterance of the example sequence "How's the weather?... Not now, Charlie... Tomorrow." In an embodiment, the speech sequence includes at least a portion ("How's the weather?") that is intended as a query to a speech processing engine (e.g., speech processing engine 550), at least a portion ("Not now, Charlie") that is intended for another listener (perhaps Charlie) and not as input to the speech processing engine, and at least a portion ("Tomorrow") that could semantically reasonably belong to either a speech recognition input portion ("How's the weather?...tomorrow") or a non-input portion ("Not now, Charlie...tomorrow"). In addition, raw speech signal 510 includes non-oral noise between spoken word portions. If raw speech signal 510 were applied directly as input to speech processing engine 550, the system could produce unexpected results because the presence of the non-input utterances ("Not now, Charlie" and possibly "Tomorrow") could interfere with the system's ability to meaningfully respond to the input utterance ("How's the weather?", possibly with the modifier "Tomorrow"). Higher quality results can be achieved by filtering the raw speech signal 600 prior to providing input to the speech processing engine 550 to filter out non-input speech that is not directed to the speech processing engine 550 (e.g., "Not now, Charlie") and generating a directed audio signal that includes speech that is directed to the speech processing engine 550 (e.g., "How's the weather?...tomorrow").(As used herein, non-input speech may include input audio that is not provided by a particular user and / or is not directed towards a speech processing system.)

[0041] The segmentation process can divide the raw speech signal into individual segments of audio that can be individually evaluated as corresponding to input speech or non-input speech. FIG. 6B illustrates an example segmentation of the audio of the raw speech signal 600 into segments. The segments can include phonemes, words, phrases, sentences, utterances, or any combination of the above. For each segment, the example system 500 can determine whether the segment corresponds to input speech or non-input speech, and the result of the determination is used to determine whether the segment should be included or excluded from the directed speech signal 540. As shown in FIG. 6B, a segment of the signal 600 can be represented as a region of the signal 600 that is between two time points (e.g., along the invariant t-axis). For example, in the figure, a first segment 601 (e.g., corresponding to "What's the weather like?") is between points t0 and t1, a second segment 602 (e.g., corresponding to non-speech such as background noise) is between points t1 and t2, a third segment 603 (e.g., corresponding to "Not now") is between points t2 and t3, a fourth segment 604 (e.g., corresponding to "Charlie") is between points t3 and t4, a fifth segment 605 (e.g., corresponding to non-speech such as background noise) is between points t4 and t5, a sixth segment 606 (e.g., corresponding to "tomorrow") is between points t5 and t6, and a seventh segment 607 (e.g., corresponding to non-speech such as background noise) is between points t6 and t7.

[0042] Such segment boundaries can be determined according to one or more suitable techniques. For example, various techniques known in the art can be used to determine boundaries of spoken words or phrases. According to some such techniques, boundaries between segments can be determined based on, for example, periods of relative silence (indicating gaps between "chunks" of speech), changes in pitch or intonation (which may indicate the beginning or end of a word, phrase, or utterance), changes in speech frequency (which may indicate the beginning or end of a word, phrase, or utterance, or a transition from one word, phrase, or utterance to another), breathing patterns (which may indicate that the speaker is about to start a new word, phrase, or utterance), and the like. In some examples, statistical analysis of the speech signal can be useful for identifying segment boundaries. For example, portions of the speech signal that represent statistical outliers in the signal (e.g., portions of the speech signal that comprise frequency components that are not commonly found anywhere in the signal) may indicate the beginning or end of a word, phrase, or utterance. Various machine learning techniques can also be used to identify segment boundaries.

[0043] In some embodiments, the sensor data 520 can be used to segment the speech signal (e.g., raw speech signal 510) by indicating potential separation points where a user may be likely to change the target of their speech (e.g., transition from speaking to a speech processing engine to speaking to another person in the room). For example, the sensor data may indicate when a user turns their head, changes the focus of their eye gaze, or moves to a different location in the room. Such sudden changes in sensor data can be used to indicate boundaries between speech segments.

[0044] The length (e.g., average duration, or number of syllables) of a speech segment may vary. In some examples, a segment may be on the order of a few words, such that it may generally constitute a spoken phrase. In some examples, a segment may be longer (e.g., constituting one or more complete sentences or utterances) or shorter (e.g., constituting individual words or even individual syllables). As described herein, speech can be included or excluded from the directed speech signal 540 on a segment-by-segment basis, such that, on a segment-by-segment basis, either the entire segment is included or the entire segment is excluded. Utilizing longer segments may increase the risk that a single segment will include both input speech and non-input speech, which may cause undesirable results. Excluding such a segment from the directed speech signal 540 would result in failing to present the user's input speech to the speech processing engine 550, while including it would present the non-input speech to the speech processing engine 550, contrary to the goal of generating the directed speech signal 540. Using shorter segments can reduce this problem, but presents a possible tradeoff in the computational overhead (and attendant latency) required to process additional segments for a single speech signal. A desirable balance in segment size may be to group single-related words or thoughts into a single segment as much as possible, such that the entire segment is either directed or not directed to the speech processing engine 550. For example, in the exemplary signal 600, "How's the weather?" and "Not right now" constitute a single chunk of speech that rises or falls together, respectively, and therefore may be beneficial to group as a single segment. However, segments may be arbitrarily large or small (including segments as small as a single digital audio sample), and this disclosure is not limited to any particular segmentation size.

[0045] In some examples, segmentation may be performed on a pre-recorded speech signal, where the entire speech signal is captured before being segmented. Segmentation may be relatively more accurate and / or efficient in such examples, since knowledge of the entire speech signal may be used to generate more meaningful speech segments. That is, it may be easier to determine which parts of the speech signal should be segmented together when the entire signal is known. However, in some examples, "live" speech may be segmented as it is detected. Techniques for segmenting pre-recorded speech signals may also be used to segment live speech signals (e.g., by applying such techniques to buffered chunks of live speech). In some cases, segmentation decisions regarding live speech may need to be revised periodically as new speech clarifies the intent of previous speech. In addition, parts of speech may be flagged for manual scrutiny, where they can later be manually evaluated and corrected.

[0046] 6C demonstrates an example probabilistic model 610 corresponding to speech signal 600. In an embodiment, probabilistic model 610 may represent a probability p(t) as a function of time t that a segment of corresponding audio signal 600 at time t is user speech directed to speech processing engine 550. (Alternatively, in some embodiments, p(t) may describe the probability that the segment is not user speech directed to the speech processing engine.) For example, in an embodiment, for a time t that falls between t0 and t1, k1 So, p(t k1 ) is equal to 0.9, and at time t k1 The portion of the speech signal 600 in (V(t k1 ), e.g., “weather”) has a 90% probability of being a user utterance directed to the speech processing engine 550. Similarly, time t k2 So, p(t k2 ) is equal to 0.1, and at time t k2 The portion of the speech signal 600 in (V(t k2), e.g., “Charlie”) has a 10% probability of being a user utterance directed to the speech processing engine 550.

[0047] As shown in the figure, probability p(t) can be determined on a segment-by-segment basis such that for a segment starting at time t0 and ending at time t1, p(t) remains constant between p(t0) and p(t1) (i.e., the entire segment will have the same probability value). Thus, in the probability model 610, segment 601 ("How's the weather?") has a corresponding probability value 611 of 0.9, segment 603 ("Not right now") has a corresponding probability value 613 of 0.3, segment 604 ("Charlie") has a corresponding probability value 614 of 0.1, and segment 606 ("Tomorrow") has a corresponding probability value 616 of 0.6. In the figure, the remaining segments (i.e., segments 602, 605, and 607, which may correspond to background noise or other non-speech audio) have corresponding probability values ​​of zero (i.e., 612, 615, and 617, respectively).

[0048] Classifying input utterances

[0049] Determining a probability value for a speech segment may be referred to as "classifying" the speech segment, and a module or process for making this determination (e.g., 562, 568, 574) may be referred to as a "classifier." Figures 7A, 7B, and 7C illustrate an example classifier of example system 500 for determining a probability value for a segment of a speech signal (e.g., segment 610 of speech signal 600 described above). This determination can be made using the speech signal itself (e.g., as shown in Figure 7A), using sensor data associated with the user (e.g., as shown in Figure 7B), or using some combination of the speech signal and sensor data (e.g., as shown in Figure 7C).

[0050] In the example shown in FIG. 7A, the speech segment 516, the statistical data 512 about the speech signal, and / or the speech data repository 527 are used by the classifier 562 to determine a probability value 566 that the speech segment 516 corresponds to an input utterance (e.g., a user utterance directed to a speech recognition system). In stage 563, the speech segment 516 can be parameterized / characterized according to one or more parameters, such as by using the statistical data 512 of the speech signal. This can facilitate classifying the speech segment based on the speech data repository 527. The speech data repository 527 may be stored in a database. A Fourier transform of the time-based speech segment 516 can be performed to provide a spectral representation of the speech segment (e.g., a function of frequency indicating the relative distribution of various frequency parameters within the speech segment 516). In some cases, the speech segment 516 can be compared against the statistical data 512 to determine the extent to which the speech segment 516 deviates from the larger speech signal of which it is a part. For example, this may indicate the level of volume (or changes therein) or component frequencies of the speech segment, which may be used to characterize the speech segment in stage 564. In some examples, aspects of the speaker, such as the speaker's age, gender, and / or native language, may be used as parameters to characterize the speech segment 516. Other ways in which the speech segment 516 may be parameterized with such parameters used to characterize the speech segment in stage 564 will be apparent to those skilled in the art. As examples, the speech segment 516 may be pre-processed using pre-emphasis, spectral analysis, sound pressure analysis, DCT / MFCC / LPC / MQ analysis, Mel filter bank filtering, noise reduction, band pass filtering of the signal to the most useful speech range (e.g., 85-8,000 Hz), and dynamic range compression.The remaining signal can then be parameterized into a set of time-invariant features (e.g., speaker identification / biometric, gender identification, average fundamental frequency, average sound pressure) and time-varying feature vectors (e.g., formant center frequencies and bandwidths, fundamental frequency, DCT / MFCC / LPC / MQ coefficients, phoneme identification, consonant identification, pitch contour, sound pressure contour).

[0051] In an embodiment, at step 564, a probability value 566 is determined that the speech segment 516 corresponds to the input utterance. The probability value 566 may be determined using the speech data repository 527. For example, a database including the speech data repository 527 may identify, for elements of utterances in the database, whether those elements correspond to the input utterance. Various types of data may be represented in the speech data repository 527. In some embodiments, the speech data repository 527 may include a set of audio waveforms corresponding to the speech segments, and for each waveform, may indicate whether the corresponding speech segment belongs to the input utterance. In some embodiments, instead of or in addition to the audio waveforms, the speech data repository 527 may include audio parameters corresponding to the speech segments. Speech segment 516 may be compared to speech segments in speech data repository 527, for example, by comparing an audio waveform of speech segment 516 with an audio waveform of speech data repository 527, or by comparing parameters of speech segment 516 (as may be characterized in stage 563) with similar parameters of speech data repository 527. Based on such comparison, a probability 566 for speech segment 516 may be determined. (Methods for creating the data in speech data repository 527 are described below.)

[0052] Techniques for determining the probability 566 will be familiar to those skilled in the art. For example, in some embodiments, nearest neighbor interpolation may be used in stage 564 to compare the speech segment 516 to similar speech segments in an N-dimensional space (wherein the N dimensions may comprise, for example, audio parameters and / or audio waveform data as described above) and determine a probability value 566 based on the relative distance between the speech segment 516 and its neighbors in the N-dimensional space. As another example, a support vector machine may be used in stage 564 to determine a principle for classifying a speech segment as either an input speech segment or a non-input speech segment based on the speech data repository 527 and classify the speech segment 516 (e.g., determine a probability value 566 that the speech segment is an input utterance) according to the principle. Other suitable techniques for analyzing the speech segments 516 and / or the speech data repository 527 to determine the probabilities 566, comparing the speech segments 516 to the speech data repository 527, and / or classifying the speech segments 516 based on the speech data repository 527 will also become apparent. This disclosure is not limited to any particular technique or combination of techniques.

[0053] In some examples, machine learning techniques, alone or in combination with other techniques described herein, may be used to determine the probability values ​​566. For example, a neural network may be trained on the speech data repository 527 and applied to the speech segments 516 to determine the probability values ​​566 for the speech segments 516. As another example, a genetic algorithm may be used to determine a function for determining the probability 566 for the speech segments 516 based on the speech data repository 527. Other suitable machine learning techniques, as would be familiar to one of ordinary skill in the art, will also be apparent. This disclosure is not limited to any particular technique or combination of techniques.

[0054] In some examples, the probability value 566 for the speech segment 516 may be influenced by other speech segments of the same speech signal. For example, a user may be less likely to provide input in short bursts surrounded by non-input speech (or vice versa). Instead, a user may be more likely to provide speech recognition input primarily in continuous sequences. That is, all other factors being equal, a speech segment 516 is more likely to be an input speech segment if the segment immediately preceding or following it is also an input speech segment, and vice versa. In such examples, probabilistic techniques (e.g., Bayesian networks, hidden Markov models) may be used to determine the probability 566 in stage 564, either alone or in combination with other techniques described herein. A variety of probabilistic techniques may be suitable for this purpose, and the present disclosure is not limited to any particular technique or combination of techniques.

[0055] In some implementations, the speech data repository 527 can be generated by recording a set of speech signals of various speech sources and, for each portion of each speech signal, identifying the speech target for that portion. For example, a user can be observed interacting with a group of people with a speech recognition system present in the same room as the user's speech (and / or other audio) is recorded. For each region of recorded speech, the observer can identify whether that region of speech was directed to the speech recognition system as input from the user (and not some other source) or to some other target. This information can be obvious to the observer by observing the context in which the user is speaking. In general, it is easy and intuitive for a human (unlike a machine) to determine whether the user is speaking to the speech recognition system or something else based on the user's observations. This process can be repeated for multiple users, and in some cases, for non-human speakers (e.g., pets, TV speakers, home appliances), until a sufficiently large and diverse set of speech data (e.g., audio waveform data and / or parameters associated with speech as described above) is generated. From this speech data, individual speech segments can be determined and these speech segments can be associated with an observer's determination of whether the corresponding utterance was directed by the user to the speech recognition system.

[0056] In the embodiment shown in FIG. 7A, as described above, probability value 566 is determined based on the user's own speech as detected by one or more microphones. Thus, the system's predictive value for probability value 566, i.e., the extent to which the embodiment of FIG. 7A allows probability value 566 to be determined more accurately than otherwise, is limited by the degree of correlation between the audio characteristics of the speech signal and whether the speech signal is input speech. The greater the degree of correlation, the more useful the speech signal is in determining the portions of the signal that are input speech. While at least some such correlation may exist between the speech audio and the intended target, correlation may also exist between the intended target of the speech and sensor data associated with the speaker, such as sensor data 520. Thus, the system's overall predictive value may be improved by incorporating sensor data 520, either alone or in addition to raw speech signal 510, as described below with respect to FIGS. 7B and 7C.

[0057] FIG. 7B illustrates an example portion of an example system 500 in which sensor data 520 is used by a classifier 568 to determine a probability value 572 that a speech segment 516 is an input utterance. In some examples, as described above, the sensor data 520 may correspond to data from sensors as described above with respect to the example wearable head device 100 in FIG. 1. As described above, such a wearable system may include one or more sensors that may provide input about a user and / or the environment of the wearable system. For example, the wearable head device 100 may include a camera (e.g., camera 444 illustrated in FIG. 4) and output a visual signal corresponding to the environment. In some examples, the camera may be a forward-facing camera on a head-mounted unit that shows what is currently in front of the user of the wearable system. In some examples, the wearable head device 100 may include a LIDAR unit, a radar unit, and / or an acoustic sensor that may output a signal corresponding to the physical geometry of the user's environment (e.g., walls, physical objects). In some examples, the wearable head device 100 may include a GPS unit, which may indicate geographic coordinates corresponding to the current location of the wearable system. In some examples, the wearable head device 100 may include an accelerometer, a gyroscope, and / or an inertial measurement unit (IMU) to indicate the orientation of the wearable head device 100. In some examples, the wearable head device 100 may include an environmental sensor, such as a temperature or pressure sensor. In some examples, the wearable head device 100 may include a biometric sensor, such as an iris camera, a fingerprint sensor, an eye tracking sensor for measuring the movement or gaze of the user's eyes (e.g., an electro-oculography (EOG) sensor), or a sensor for measuring the user's vital signs. In examples where the wearable head device 100 includes a head-mounted unit, such orientation may correspond to the orientation of the user's head (and, for that matter, the direction of the user's mouth and the user's speech).Other suitable sensors may also be included and may provide sensor data 520. Additionally, in some embodiments, sensors other than those of the wearable system may be utilized, if desired. For example, sensors associated with one or more microphones of the speech recognition system (e.g., GPS, IMU) may be used in conjunction with the sensors of the wearable system to determine the relative distance and orientation between the user and the speech recognition system.

[0058] In an example shown in FIG. 7B, stage 569 can parameterize / characterize speech segments 516 according to one or more parameters as described above with respect to stage 563 with respect to aspects of sensor data 520. This can facilitate classifying speech segments based on sensor data 520. For example, stage 569 can perform a Fourier transform of sensor data 520 signals (e.g., signals (e.g., from GPS, acoustic, radar, or IMU sensors) that describe the user's position or orientation as a function of time elapsed during the speech segment) to determine a spectral representation of those signals. As an example, speech segments 516 can be characterized according to the user's eye movements (e.g., from an EOG sensor), eye gaze targets (e.g., from a camera or EOG sensor), and / or visual targets (e.g., from an RGB camera or LIDAR unit). In some embodiments, the sensor data 520 can be compared to a broader range of sensor data (e.g., sensor data captured over a period of several minutes prior to the onset of the speech signal) to determine the extent to which the sensor data 520 deviates from the broader range of sensor data. Other ways in which the sensor data 520 can be parameterized, with such parameters being used to characterize the speech segment in stage 564, will be apparent to those skilled in the art. As described above with respect to the speech segment 516, the speech segment 564 can also be pre-processed with pre-emphasis, spectral analysis, sound pressure analysis, DCT / MFCC / LPC / MQ analysis, Mel filter bank filtering, noise reduction, band-pass filtering of the signal to the most useful speech range (e.g., 85-8,000 Hz), and dynamic range compression. The remaining signal can then be parameterized into a set of time-invariant features (e.g., speaker identification / biometric, gender identification, average fundamental frequency, average sound pressure) and time-varying feature vectors (e.g., formant center frequencies and bandwidths, fundamental frequency, DCT / MFCC / LPC / MQ coefficients, phoneme identification, consonant identification, pitch contour, sound pressure contour).

[0059] In an embodiment, at step 570, a probability value 572 is determined that the speech segment 516 corresponds to the input utterance. In some approaches, the probability value 572 can be determined using a sensor data repository 528, which may include a database that identifies, for elements of utterances in the database, whether those elements correspond to the input utterance. In some embodiments, the sensor data repository 528 can include data sets representing sensor measurements (e.g., a sequence over time of a user's head position, orientation, and / or eye gaze) corresponding to the speech segment, and for each data set, can indicate whether the corresponding speech segment belongs to the input utterance. In some embodiments, instead of or in addition to a sensor data set, the sensor data repository 528 can include parameters corresponding to the speech segment. The speech segment 516 can be compared to the sensor data repository 528, for example, by comparing corresponding signals in the raw sensor data 520 and the sensor data repository 528, or by comparing parameters of the speech segment 516 (as may be characterized in step 569) with similar parameters in the sensor data repository 528. Based on such a comparison, a probability 572 can be determined for the speech segment 516 .

[0060] Techniques for determining the probabilities 572 will be familiar to those of skill in the art. For example, the techniques described above with respect to determining the probability values ​​566, such as nearest neighbor interpolation, support vector machines, neural networks, genetic algorithms, probabilistic techniques such as Bayesian or Markov networks, or any combination of the above, can be applied in a similar manner to the sensor data repository 528 and the sensor data 520. Other techniques will be apparent, and the present disclosure is not limited to any particular technique or combination of techniques.

[0061] In some embodiments, the sensor data repository 528 need not be accessed by the classifier 568 directly to classify the speech segment 516 in stage 570. For example, stage 570 can apply one or more rules to determine a probability value 572 that the speech segment 516 corresponds to an input utterance based on the sensor data 520. For example, stage 570 can determine based on the sensor data 520 (e.g., data from a position and orientation sensor) that the user is facing the microphone (or has turned to face the microphone just prior to uttering the speech segment 516), and from this information, it can then be determined that the speech segment 516 is likely to be an input utterance. Conversely, stage 570 can determine that the user is facing away from the speech processing engine microphone (or has recently turned to face away from the microphone), and that the speech segment 516 is unlikely to be an input utterance. This is because humans generally tend to look toward the object to which their speech is directed, regardless of whether the object is a person or a device. Similarly, in stage 570, based on the sensor data 520 (e.g., data from a camera or EOG sensor), it can be determined that the user is looking at the microphone (or has recently shifted their eye gaze toward the microphone) and the speech segment 516 is likely to be an input utterance. Conversely, it can be determined that the user is not looking at the microphone and the speech segment is unlikely to be an input utterance. As another example, if the sensor data 520 (e.g., camera data) indicates that the user is looking directly at another person while uttering the speech segment 516, it can be determined that the speech segment 516 is unlikely to be an input utterance (i.e., the speech is instead directed toward the person the user is looking at).Rules for determining how to classify the probability value 572 based on the sensor data can be determined using machine learning techniques familiar to those skilled in the art, such as neural networks or genetic algorithms, using the sensor data repository 528 as a training set.

[0062] In some examples, the sensor data repository 528 can be generated similarly to the speech data repository 527, as described above. For example, the data in the sensor data repository 528 can be generated by recording a set of speech signals of various sources along with accompanying sensor data generated contemporaneously with the speech signals, and for each portion of each speech signal, identifying a speech target for that portion. For example, a user can be observed interacting with a group of people, with a speech recognition system present in the same room as the user's speech is recorded. For each region of the recorded speech, the observer can identify whether that region of speech was directed as input from the user to the speech recognition system or some other target. From this speech and / or sensor data, individual speech segments can be determined, and these speech segments and their accompanying sensor data can be associated with the observer's determination of whether the corresponding speech was directed by the user to the speech recognition system.

[0063] The sensor data 520 can also be used in stage 570 to identify whether a microphone input belongs to a particular user. For example, the amplitude of a user's speech as detected by one or more microphones can be expected to fall within a predictable range that falls off as a function of the distance between the microphone and the user and varies as a function of the user's relative orientation to the microphone (e.g., falls off as the user points the microphone away). (In some cases, this range can be determined empirically for a particular user.) If the sensor data 520 (e.g., GPS data, camera data, acoustic data, radar data) indicates that a user is a particular distance from a microphone, a range of expected amplitudes of that user's speech for that particular distance can be determined. Microphone input that falls outside that amplitude range can be rejected as belonging to a source other than the user. Similarly, other speech characteristics (e.g., high frequency content) can be predicted based on the user's position, orientation, or other sensor data 520, and microphone input that is inconsistent with the sensor data can be rejected. Similarly, microphone input that changes significantly (e.g., in volume or frequency characteristics) while the user's position and orientation remains constant (or vice versa) may also be disavowed. Conversely, microphone input that matches expected characteristics of a user's speech based on sensor data may confirm that the microphone input belongs to that user. Other techniques for identifying the source of microphone input based on sensor data will be apparent to those skilled in the art.

[0064] In FIG. 7B, a probability value 572 is determined based on the user's own speech as detected by one or more microphones, as described above. Similar to the example shown in FIG. 7A and the probability value 566, the system's predictive value for the probability value 572 is limited by the degree of correlation between the intended target of the speech signal and the accompanying sensor data produced with the speech signal. The greater the correlation, the more useful the sensor data is in determining the portion of the signal that is the input speech. Such correlation reflects that sensor data (such as from sensors of a wearable system such as those described above) can provide many of the same body language cues that humans use to interpret and contextualize the speech of others. For example, humans are accustomed to using the speaker's position (e.g., the speaker's movement and distance from the listener), orientation (e.g., the person the speaker is facing), eye gaze (e.g., the person the speaker is making eye contact with), gestural expressions (e.g., hand and arm movements, facial expressions), and the like to determine the speaker's intended speech target. Many of these body language cues also apply when a speaker addresses a device such as a microphone-enabled speech recognition system. Sensor data may respond to this body language, such as by providing data indicative of the speaker's position, orientation, eye patterns, movements, etc. Thus, using sensor data as described above can provide useful information regarding the intended target of a corresponding utterance.

[0065] In some examples, the predictive value of the system can be improved by utilizing both speech data (e.g., as described with respect to FIG. 7A) and sensor data (e.g., as described above with respect to FIG. 7B) that correspond to the same speech signal. For example, if a speech segment corresponds to both a speech cue (e.g., a user raising their voice) and a sensor cue (e.g., a user rapidly turns their head), the two cues combined can provide strong predictive evidence that the speech segment is intended as input from the user to the speech processing engine.

[0066] 7C illustrates an example portion of an example system 500 in which both analysis data 512 and sensor data 520 regarding a speech signal (e.g., speech signal 510) are used by a classifier 574 to determine a probability value 578 that a speech segment 516 has been directed to a speech processing engine by a user. The stages of the example system shown may proceed as described above with respect to FIGS. 7A and 7B. For example, stage 575 may parameterize / characterize the speech segment 516 based on speech characteristics determined from the speech signal 510 and / or speech signal analysis data 512, as described above with respect to stage 563 of FIG. 7A, and stage 575 may also parameterize / characterize the speech segment 516 based on sensor data 520, as described above with respect to stage 569 of FIG. 7B. At step 576, a probability value 578 can be determined for the speech segment 516 based on its speech characteristics, as described above with respect to step 564 of FIG. 7A, and further based on its corresponding sensor data, as described above with respect to step 570 of FIG. 7B. This probability value determination can utilize speech and / or sensor data, such as in speech / sensor data repository 529. The speech / sensor data repository 529 can include a database that includes information relating the speech data to the intended target of the utterance, as described above with respect to speech data repository 527 of FIG. 7A, and can further include information relating the sensor data to the intended target of its corresponding utterance, as described above with respect to sensor data repository 528 of FIG. 7B. Additionally, the speech / sensor data repository 529 can include information relating a combination of speech data and sensor data to the intended speech target. This can be useful in situations where neither the speech data nor the sensor data by itself is independent of the prediction of the intended speech target, but the combination of the two is strongly correlated to the intended speech target and has better predictive value.

[0067] Probabilistic model generation

[0068] FIG. 8 is a flow chart showing a portion of an exemplary system 500, according to some embodiments, illustrating an example of generating a probabilistic model 586 from a raw speech signal 510. In FIG. 8, stage 560 generates a probabilistic model 586 (which may correspond to the probabilistic model 610 described above with respect to FIG. 6C) from the raw speech signal 510 (which may correspond to the signal 600 described above with respect to FIGS. 6A-6B) and the sensor data 520. In stage 560, statistical data 512 regarding the speech signal (e.g., representing a statistical analysis of the speech signal 510 as described above) may be generated according to techniques familiar to those skilled in the art. In stage 514 of stage 560, the speech signal 510 may be segmented into individual speech segments 516, as described above with respect to FIGS. 6A-6D. For each speech segment 516, one or more classifiers (e.g., 562, 568, 574 described above) can be applied to generate a probability value corresponding to the probability that the segment is an input utterance. In the example shown in Figure 8, three classifiers are applied: a first classifier (562) generates a first probability value 566 based on the speech segment 516 and the utterance data 512 as described above with respect to Figure 7A, a second classifier (568) generates a second probability value 572 based on the speech segment 516 and the sensor data 520 as described above with respect to Figure 7B, and a third classifier (574) generates a third probability value 578 based on the speech segment 516, the utterance data 512, and the sensor data 520 as described above with respect to Figure 7C. However, in some embodiments, only one classifier (e.g., classifier 574) need be used, and in some embodiments, additional classifiers beyond the three described herein may be utilized to generate additional individual probability values. In some cases, different classifiers may be applied to different metrics to determine the individual probability values.

[0069] In some examples, when multiple classifiers are used to determine multiple individual probability values ​​for the speech segment 516, as shown in the example of FIG. 8 where classifiers 562, 568, and 574 are used to generate probability values ​​566, 572, and 578, respectively, it may be necessary to determine an overall probability 582 for the speech segment 516 based on the individual probability values ​​generated by the individual classifiers. In such examples, comparison logic 580 may be used to adjust the individual probability values ​​and determine the overall probability 582. In some examples, comparison logic 580 may calculate the overall probability 582 as an average of the individual probabilities (e.g., 566, 572, 578). In some examples, comparison logic 580 may calculate the overall probability 582 as a weighted average of the individual probabilities, for example, weighted by the fidelity of the input data (e.g., speech data 512, sensor data 520). Other suitable techniques that may be employed by the comparison logic 580 will be familiar to those of skill in the art, and this disclosure is not limited to any such technique or combination of techniques. Exemplary techniques for combining the outputs of multiple classifiers include ensemble learning, Bayesian optimal classifiers, bagging (bootstrap aggregating), boosting techniques (e.g., AdaBoost), bucket models, and stacking.

[0070] Once the probability values ​​for the speech segments 516 have been determined as described above, the process of determining the probability values ​​can be repeated for any remaining speech segments 516 (stage 584). For example, the speech signal 600 described above with respect to Figures 6A-6D can be divided into seven speech segments (601-607) as described above. If this speech signal 600 were provided as input 510 to the system shown in Figure 8, stages 562, 568, and 574 could each be applied to each of the seven speech segments, resulting in a probability value 582 for each segment. Once the probability values ​​have been determined for each speech segment 516, the probability values ​​can be used to generate a probability model 586. As described above, the probability model 586 can represent a probability value for each speech segment of the speech signal. For example, in Figure 6C, the probability model 610 represents a probability value for each speech segment of the speech signal 600. Generating a probabilistic model 586 for the speech signal can include expressing probability values ​​as a function of elapsed time of the speech signal; using such a model, such as shown as model 610 in FIG. 6C, a time t can be applied as an input to the model, and the model will indicate the probability that a portion of the speech signal corresponding to time t (e.g., the portion of speech signal 600 after t seconds have elapsed) will be directed as input to the speech processing engine. However, other suitable implementations of the probabilistic model 586 will be apparent and are within the scope of this disclosure.

[0071] Determining directed speech signals

[0072] 9 illustrates a portion of an example system 500 whereby the system 500 determines a directed speech signal 540 from the raw speech signal 510 and / or the sensor data 520, such as by using the probabilistic model 586 described above. As shown in FIG. 9, in stage 530, the system 500 may generate a directed audio signal 540, which may be an input speech signal to the speech processing engine, including speech directed by a user to the speech processing engine, while excluding speech not directed by the user to the speech processing engine. The directed audio signal 540 may correspond to the signal 620 described above with respect to FIG. 6D. An example of stage 530 of generating the directed audio signal 540 may proceed as follows, with reference to FIG. 9. In stage 560, the raw speech signal 510 and / or the sensor data 520 may be used to determine, for one or more segments of the raw speech signal 510, a probability that the segment corresponds to speech directed by a user as input to the speech processing engine. An exemplary implementation of stage 560 is described above with respect to FIG. 8. As described above, the output of target determination stage 560 may be represented as a probabilistic model 586, which may represent the probability, for example, as a function of elapsed time, that a portion of the speech signal 510 is user speech directed to a speech processing engine. For example, model 586 may be a mathematical function that represents the probability, for each time t of a raw speech signal having one or more segments, that a segment of that raw speech signal corresponding to time t is directed to a speech processing engine. As shown in the example in FIG. 9, stage 560 may also output a pass-through signal 588, which may be a buffered signal corresponding to the raw speech signal 510 provided to target determination stage 560.

[0073] In the example stage 590 in FIG. 9, the raw speech signal (e.g., pass-through signal 588) can be filtered based on the probabilistic model 586 such that, with a sufficiently high probability, segments of the raw speech signal 510 that correspond to the input speech can be included in the directed audio signal 540, and conversely, segments of the raw speech signal 510 that do not correspond to the input speech can be excluded from the directed audio signal 540. Stage 590 can employ a threshold probability value to act as a cutoff for determining what constitutes a sufficiently high probability for an audio segment to be included in the directed audio signal 540. For example, as described above, FIG. 6C illustrates a probabilistic model 610 corresponding to the raw speech signal 600 shown in FIGS. 6A and 6B. As described above with respect to FIG. 6C, the probabilistic model 610 indicates, for each speech segment 601-607 of the speech signal 600, the probability that the speech segment corresponds to the input speech. In FIG. 6C, threshold 618 is a value of 0.5. However, other thresholds can be used, if desired. In stage 590, speech segments (e.g., speech segments 601 and 606) with corresponding probability values ​​that meet or exceed threshold 618 may be included in directed audio waveform 540, and segments (e.g., speech segments 602, 603, 604, 605, and 607) whose corresponding probability values ​​do not meet threshold 618 may be excluded from directed audio waveform 540. The result would be an audio waveform 620 shown in FIG. 6D, where only speech segments with sufficiently high probability ("How's the weather?" and "tomorrow") are included in waveform 620, and the remaining segments are excluded. Compared to providing the raw speech signal 600 to a speech recognition system, providing the audio waveform 620 as an input to the speech recognition system promotes accuracy and computational efficiency because the speech recognition system does not need to waste computational resources on unrelated speech (or other audio) which runs the risk of producing erroneous results.

[0074] Training the classifier

[0075] FIG. 10 illustrates an example process 1000 for capturing audio and non-audio classifier training data, according to one or more embodiments of the present disclosure. The process 1000 can be applied to a human test subject 1012 that interacts (as a user might do) with a speech processing engine (e.g., as included in a device with an integrated voice assistant). One or more microphones and one or more sensors can be configured to capture audio data and non-audio data (e.g., sensor data), respectively, from the test subject 1012. In some embodiments, the non-audio data can be non-microphone sensor data, such as, for example, inertial measurement unit data, visual data, and the like. In step 1010 of the process, raw audio data of the test subject's 592 voice can be captured via one or more microphones. Similarly, in step 1020, the test subject's non-audio data can be captured via one or more sensors. In some cases, the test subject 1012 can be equipped with a single device, such as a wearable head device as described above, that can include one or more microphones and one or more sensors. The microphones and sensors may be configured to capture audio data, in step 1010, and non-audio data, in step 1020, respectively. Steps 1010 and 1020 may be performed simultaneously.

[0076] In step 1030, the audio captured in step 1010 can be segmented and tagged as either input speech or non-input speech. This may be an automated process, a manual process, or some combination thereof. For example, the audio data captured in step 1010 can be presented to a voice activity detector (VAD) or a human "tagger" observing the test subject 1012, and the audio data can be manually separated into individual phrases or portions thereof by the tagger. The tagger can then manually identify each phrase as input speech or non-input speech based on the tagger's observations of the test subject 1012 interacting with a speech recognition engine. In some cases, the tagger can annotate each phrase with various metadata (e.g., the intended recipient for each phrase or audio source of each phrase). Other metadata typed in by the tagger can include aspects about the speaker (e.g., the speaker's age, gender, and / or native language). In some implementations, the tagger can also segment and tag non-speech audio (eg, background noise and / or speech from people other than the speaker).

[0077] Similarly, in step 1040, the non-audio data captured in step 1020 may also be segmented and tagged as either directed or not directed to the speech processing engine. In some examples, a human tagger may identify and / or isolate the non-audio data (e.g., sensor data) associated with individual phrases spoken by the test subject 1012 described above. In some cases, the tagger may manually associate the non-audio data with its corresponding audio data. In some examples, the non-audio data may be automatically associated with each phrase based on the start and end times of the segmented and classified phrases from step 1030. In some examples, the non-audio data may include information about the user's head pose, gaze, gestures, location relative to the target recipient phrase, or any other sensor data captured.

[0078] In step 1050, the audio captured in step 1010, the segmented and tagged phrases from step 1030 (e.g., input speech and non-input speech, including background noise or non-speech audio), the non-audio data captured in step 1020, and / or the segmented and tagged non-audio data from step 1040 may be stored in a repository for classifier training. For example, speech data repository 527 described above may store the audio from step 1010 and / or the phrases from step 1030, sensor data repository 528 may store the non-audio data from steps 1020 and / or 1040, and speech / sensor data repository 529 may store any of the above. In some embodiments, the audio captured in step 1010 and / or the segmented and tagged phrases from step 1030 are stored separately from the non-audio data captured in step 1020 and / or the segmented and tagged non-audio data from step 1040 (e.g., the audio data and non-audio data are stored in separate databases). The stored audio data and / or non-audio data can be used to train a classifier as described above.

[0079] In some embodiments, audio and / or non-audio characteristics can be extracted from the input speech, non-input speech, or non-speech (e.g., background noise) stored in one or more databases from step 1050 of FIG. 10. Examples of audio characteristics can include volume (or signal amplitude) level (or changes therein), pre-voicing hesitations, intra-voicing hesitations, speech imperfections (e.g., stuttering, repetition), speech rate, syntax, grammar, vocabulary, phrase length (e.g., duration, word count), pitch (e.g., variance and contour), and / or prosody. Examples of non-audio characteristics that can be extracted from non-audio data include gestures, gaze (and changes therein), head pose (and changes therein), and position (e.g., distance and orientation) (and changes therein) relative to physical and / or virtual objects. In some embodiments, a Fourier transform of each speech and / or non-speech segment (e.g., each audio and / or non-audio segment corresponding to the input utterance, non-input speech, and / or non-speech) is stored in step 1050 of FIG. 10 (e.g., for both the input speech and non-input speech) to provide a spectral representation of each speech segment (e.g., a function of frequency indicating the relative distribution of various frequency parameters within the speech segment). Other methods of extracting time, frequency, and combined time-frequency parameter representations of audio and non-audio data will be familiar to those skilled in the art. In some embodiments, the extracted audio and / or non-audio features may be stored along with the corresponding input speech, non-input speech, and non-speech.

[0080] In some embodiments, the segmented and annotated audio and non-audio data (e.g., input speech, non-input speech, and / or non-speech with corresponding metadata) captured through the process 1000 of FIG. 10 can be fed to one or more classifiers for training purposes as described above. By passing sample classes of input speech, non-input speech, and non-speech through the one or more classifiers, the one or more classifiers can be trained to recognize the input speech, non-input speech, and / or non-speech. In some examples, a large subset (e.g., 60%) of the segmented and annotated audio and non-audio data is passed through one or more classifiers, and a small subset or the remainder (e.g., 40%) of the segmented and annotated audio and non-audio data is used to evaluate the one or more classifiers. Evaluation techniques will be familiar to those skilled in the art. In some embodiments, these classifiers can be further trained by allowing a user to confirm or deny the classification.

[0081] As described above, one or more classifiers (e.g., a naive Bayes classifier, a support vector machine, a k-nearest neighbor classifier, an AdaBoost classifier, a decision tree, or an artificial neural network) distinguish between input speech and non-input speech. These classifiers can be trained to recognize audio and non-audio features associated with the input speech and / or non-input speech for improved speech processing. Methods for training classifiers according to the present disclosure can include capturing audio and / or non-audio data, extracting audio and / or non-audio features of the input speech and non-input speech, training one or more classifiers, e.g., using machine learning techniques, and / or updating the classifiers (e.g., by confirming and / or rejecting classifications) for improved input speech identification, as described below.

[0082] FIG. 11 illustrates an example environment that may be used to generate audio and sensor data for classifier training. The figure illustrates a test subject 592 (which may correspond to the test subject 1012 described above) in an environment 591 that includes a voice target (such as a voice assistant device that includes a speech processing engine) and one or more "disturbance" sources 593A-593H. The disturbance sources are configured to present audio or visual "disturbance" stimuli to the test subject 592 to which the test subject 592 may respond. Audio and non-audio data (e.g., sensor data) associated with the test subject 592's response to these disturbance stimuli may be detected, which may describe the test subject 592's response (as detected by microphones and sensors) to external stimuli presented from the location of the corresponding disturbance source. This audio and non-audio data can thus be used to train a classifier (as described above) to distinguish input speech from non-input speech (e.g., speech directed at external stimuli represented by a disturbing source).

[0083] The interference sources 593A-593H can be placed at various distances from and angles relative to the test subject 592 as shown in the figure. The interference sources 593A-593H can be presented as speakers or visuals, or as any other suitable objects (e.g., humans, animals, electronic devices, etc.) that can produce sound and / or visuals. For example, the interference source 593A can represent a smart home device (e.g., a speaker with an integrated "smart" voice assistant ("smart speaker")), and the interference source 593B can represent a human. The audio and non-audio data can reflect differences in the response of the test subject 592 based on the apparent identity of the interference source. The environment 591 can represent a controlled environment (e.g., a soundproof room or a room in which the interference sources 593A-593H produce sound in a controlled manner) or an uncontrolled environment (e.g., the home of the test subject 592 or a public place). For example, in a controlled environment, a test subject 592 can freely interact (e.g., with little or no instructions or scripts) with a wearable device (e.g., wearable head device 100) with an integrated voice assistant and command the device to perform a particular action (e.g., open an app, play music, query information, e.g., from the Internet, enter information into a calendar, read information from a calendar, make a phone call, send a text message, control a smart thermostat, control a smart lock, control one or more smart lights, or any other action). Test personnel (represented by interference sources 593A-593H) can engage in a conversation with the test subject 592. This prompts the test subject 592 to interact with the wearable device and the test personnel. In some examples, the interference sources 593A-593H can be virtual sources. For example, a software application running on the wearable system can produce sounds from one or more virtual sound sources represented by interference sources 593A-593H.In some embodiments, the interference sources 593A-593H may be presented via a wearable head device worn by the test subject 592 (e.g., via speakers and / or a display of the wearable head device), along with audio and non-audio data potentially captured by microphones and sensors of that same wearable device.

[0084] 11 (e.g., spoken phrases 594A-594D spoken in environment 591) can be detected and used to train one or more classifiers in accordance with the present disclosure. For example, spoken phrases 594A-594D can be recorded (e.g., by one or more microphones 150 on wearable head device 100 or by one or more microphones on audio source 594A) into an audio file as a sustained audio stream such as "Hey, Magic Leap! Open.... Mom, can I...? Not now, Charlie.... Open Maps." Similarly, non-audio data of test subject 592 interacting with one or more disturbance sources 593A-593H can also be captured simultaneously with the audio data. In some examples, data from one or more sensors on a wearable system on the test subject 592 (e.g., the wearable head device 100 in FIG. 1 and / or the handheld controller 200 in FIG. 2) can be used to capture information about the head position of the test subject 592 (e.g., as detected by position and orientation sensors on the wearable head device), hand gestures (e.g., as detected by movement of the handheld controller 200 or by one or more cameras 130A and 130B configured on the wearable head device 100), eye line of sight (e.g., as detected by one or more cameras 128A and 102B configured on the wearable head device 100), and / or the distance of the test subject 592 from one or more sources of interference 593A-593H (e.g., as measured from the wearable head device 100 to one or more of the sources of interference 593A-593H by one or more cameras 130A and 130B and / or GPS, acoustic, radar, or IMU sensors).

[0085] With respect to the systems and methods described above, the elements of the systems and methods can be implemented by one or more computer processors (e.g., CPU or DSP) as appropriate. The present disclosure is not limited to any particular configuration of computer hardware, including computer processors, used to implement these elements. In some cases, multiple computer systems can be employed to implement the systems and methods described above. For example, a first computer processor (e.g., a processor of a wearable device coupled to one or more microphones) can be utilized to receive input microphone signals and perform initial processing of those signals (e.g., signal conditioning and / or segmentation as described above). A second (possibly more computationally powerful) processor can then be utilized to perform more computationally intensive processing, such as determining probability values ​​associated with speech segments of those signals. Another computing device, such as a cloud server, can host a speech processing engine to which the input signals are ultimately provided. Other suitable configurations will be apparent and are within the scope of the present disclosure.

[0086] Although the disclosed embodiments have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. For example, elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. Such changes and modifications are to be understood as being included within the scope of the disclosed embodiments as defined by the appended claims.

Claims

1. 1. A method of presenting a signal to a speech processing engine, the method comprising: receiving a first audio signal via a first microphone; identifying a first portion of the first audio signal; determining, for the first portion of the first audio signal, a first probability that the first portion comprises speech directed by a user of the speech processing engine as input to the speech processing engine; identifying a second portion of the first audio signal, the second portion following the first portion in the first audio signal; and determining, for the second portion of the first audio signal, a second probability that the second portion comprises speech directed by the user as input to the speech processing engine; including the first portion of the first audio signal in a first input signal presented to the speech processing engine in accordance with determining that the first probability exceeds a first threshold; removing the first portion of the first audio signal from the first input signal in accordance with determining that the first probability does not exceed the first threshold; including the second portion of the first audio signal in the first input signal in accordance with determining that the second probability exceeds the first threshold; removing the second portion of the first audio signal from the first input signal in accordance with determining that the second probability does not exceed the first threshold; presenting the first input signal to the speech processing engine; Including, 22. The method of claim 21, wherein presenting the first input signal to the speech processing engine includes presenting the first input signal comprising both the first and second portions of the first audio signal as a single instruction to the speech processing engine, wherein the speech processing engine generates an output based on the single instruction.

2. 2. The method of claim 1 , wherein the first probability is determined based on a comparison of the first portion of the first audio signal with a plurality of audio signals in a database, each audio signal of the plurality of audio signals being associated with a probability that that individual audio signal comprises speech directed as input to a speech processing engine.

3. 2. The method of claim 1, further comprising receiving first sensor data from a first sensor associated with the user, the first probability being determined based on a comparison of the first sensor data to a plurality of sensor data in a database, each sensor data of the plurality of sensor data in the database being associated with an audio signal and further associated with a probability that the respective audio signal comprises speech directed as input to a speech processing engine.

4. The method of claim 3 , wherein the first sensor is associated with a wearable head unit configured to be worn by the user.

5. The method of claim 4 , wherein the first sensor data is indicative of one or more of the user's position, orientation, eye movement, eye gaze target, or vital signs.

6. The method of claim 5 , wherein the first portion of the first audio signal is identified based on the first sensor data.

7. The method of claim 1 , wherein the second probability is determined based, at least in part, on the first probability.

8. 1. A system for providing input to a speech processing engine, the system comprising: A microphone; A circuitry comprising: receiving a first audio signal via the microphone; identifying a first portion of the first audio signal; determining, for the first portion of the first audio signal, a first probability that the first portion comprises speech directed by a user as input to the speech processing engine; identifying a second portion of the first audio signal, the second portion following the first portion in the first audio signal; and determining, for the second portion of the first audio signal, a second probability that the second portion comprises speech directed by the user as input to the speech processing engine; including the first portion of the first audio signal in a first input signal presented to the speech processing engine in accordance with determining that the first probability exceeds a first threshold; removing the first portion of the first audio signal from the first input signal in accordance with determining that the first probability does not exceed the first threshold; including the second portion of the first audio signal in the first input signal in accordance with determining that the second probability exceeds the first threshold; removing the second portion of the first audio signal from the first input signal in accordance with determining that the second probability does not exceed the first threshold; presenting the first input signal to the speech processing engine; and a circuitry configured to implement the Including, Presenting the first input signal to the speech processing engine includes presenting the first input signal comprising both the first and second portions of the first audio signal as a single instruction to the speech processing engine, wherein the speech processing engine generates an output based on the single instruction.

9. 9. The system of claim 8, wherein the first probability is determined based on a comparison of the first portion of the first audio signal with a plurality of audio signals in a database, each audio signal of the plurality of audio signals being associated with a probability that that individual audio signal comprises speech directed as input to a speech processing engine.

10. The system further includes a sensor, and the circuitry further comprises:

10. The system of claim 8, further configured to: receive first sensor data from the user via the sensor; and the first probability is determined based on a comparison of the first sensor data with a plurality of sensor data in a database, each sensor data of the plurality of sensor data in the database being associated with an audio signal and further associated with a probability that the respective audio signal comprises speech directed as input to a speech processing engine.

11. The method of claim 10 , wherein the first sensor data is indicative of one or more of the user's position, orientation, eye movement, eye gaze target, or vital signs.

12. The system of claim 11 , wherein the first portion of the first audio signal is identified based on the first sensor data.

13. The system of claim 8 , wherein the second probability is determined based, at least in part, on the first probability.

14. The system of claim 8 , wherein the system includes a wearable head unit that includes the microphone and the circuitry.

15. The system of claim 8 , wherein the system includes a vehicle that includes the microphone and the circuitry.

16. The system of claim 8 , wherein the system includes an electronic voice assistant including the microphone and the circuitry.

Citation Information

Patent Citations

  • Device and method for speech recognition

    JP2007199552A

  • Reducing need for manual start / end-pointing and trigger phrases

    JP2016004270A

  • Method and Apparatus for Activating Applications by Speech Input

    JP2017520012A

  • Control of electronic devices based on the direction of speech

    JP2018512619A

  • Anchored speech detection and speech recognition

    US20170270919A1