Recognize input for speech recognition engines

Abstract

A method for presenting a signal to a speech recognition engine is disclosed. According to an example of the method, an audio signal is received from a user. A portion of the audio signal is identified, the portion having a first time and a second time. A pause in the portion of the audio signal is identified, the pause including the second time. A determination is made as to whether the pause indicates completion of an utterance of the audio signal. Based on a determination that the pause indicates completion of the utterance, the portion of the audio signal is presented as input to the speech recognition engine. Based on a determination that the pause does not indicate completion of the utterance, the portion of the audio signal is not presented as input to the speech recognition engine.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority under 35 U.S.C. §119 to U.S. Provisional Patent Application No. 62 / 836,593, filed on April 19, 2019, the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] The present disclosure relates generally to systems and methods for processing speech signals, and more particularly, to systems and methods for processing speech signals presented to a speech recognition engine. Background Art

[0004] The task of a system for speech recognition is to receive an audio input representing human speech, typically via one or more microphones, and then process the audio input to determine a word, logical structure, or other output corresponding to the audio input. For example, an automatic speech recognition (ASR) system can generate text output based on human speech corresponding to an audio input signal; a natural language processing (NLP) tool can generate a logical structure or computer data corresponding to the meaning of human speech. While some ASR systems may operate on large amounts of pre-recorded speech—for example, a system is tasked with creating a written transcript of speech recorded by a microphone the day before—some ASR systems must respond to speech input provided in real time. Real-time speech processing presents ASR systems with a unique set of challenges. For example, ASR systems typically process speech not as a single block of input, but as a series of individual words or phrases ("utterances") that carry meaning.

[0005] Identifying when an utterance begins and ends can be critical for an ASR system to accurately process a user's input speech and provide the desired results. For example, consider a real-time "voice assistant" ASR system that communicates with a weather reporting service: the ASR system may receive speech input from a user asking about the weather (e.g., "What's the current weather like?"); convert the speech input into a structured query (e.g., a query for data indicating past, current, or predicted future weather for a particular date and time and a particular location); present the structured query to the weather reporting service; receive query results from the service; and present the query results to the user. The user expects the ASR system to process his or her complete question (not just individual fragments of the question) and to provide an accurate response quickly. The user further expects the ASR system to process natural spoken commands that do not need to follow a specific, rigid format. In this example system, the ASR has the responsibility to recognize the user's complete question in real time; and process the question in a timely manner—ideally, immediately after the user has finished asking the question—to produce an accurate response.

[0006] In this example system, the accuracy of the response may depend on when the ASR system determines that the user's question (utterance) is complete. For example, a user may ask, "What will the weather be like tomorrow?" If the ASR system determines that the utterance is complete too early, after "What will the weather be like?", its corresponding query to the weather service will omit the modifier "tomorrow", and the resulting response will be inaccurate (it will not reflect the date / time expected by the user). In contrast, if the ASR system takes a more conservative approach and waits for several seconds before processing the utterance to confirm that the entire utterance is complete, the user may not consider the ASR system to be sufficiently responsive to his or her command (in addition, in some cases, such a long waiting period may cause inaccuracies due to the inclusion of irrelevant subsequent speech in the utterance).

[0007] ASR systems strive to address this problem of determining when a speaker's utterance is complete quickly and in real time. In some systems, a fixed timeout period is used to determine the endpoint of an utterance: if, following speech input, no speech is received within the duration of the timeout period (e.g., 750 ms), the speech input may be considered the end of the utterance. However, the solution of a fixed timeout period is not perfect: for example, the timeout period may expire before the user's utterance is complete in cases where the user pauses to formulate a question; in cases where the user is temporarily interrupted; or in cases where the user's speech is not fluent (e.g., due to anxiety, language barriers, environmental distractions, cognitive load, etc.). Conversely, once the user's utterance is complete, the response is delayed by at least the duration of the timeout period (during which the ASR system confirms that no further input has been received), and the user is unable to provide additional speech input (e.g., pertaining to a new utterance) during this duration. This interaction limits the usefulness of the ASR system and may unhelpfully highlight that the user is communicating with a machine rather than another person.

[0008] It would be desirable for ASR systems to take a more intuitive approach to determining when a user has finished providing an utterance. In ordinary face-to-face communication—and to a lesser extent, in telephone communication—people use a variety of contextual cues to understand when another person has finished talking. For example, when a speaker pauses, people assess the speaker's prosody, facial expression, eye gaze, mannerisms, gestures, and posture to indicate whether the speaker has finished speaking or is simply pausing in the middle of a single thought. An ASR system can use similar cues to identify the beginning and end locations of a user's utterance. As described below, in some examples, an ASR system can identify such contextual cues from microphone input; in some examples, an ASR system that communicates with one or more sensors (e.g., as part of a wearable system) can collect additional speech cues about the speaker from the output of these sensors and use these cues to identify utterance boundaries without the problems associated with conventional solutions, such as those described above. Summary of the invention

[0009] A method for presenting a signal to a speech recognition engine is disclosed. According to an example of the method, an audio signal is received from a user. A portion of the audio signal is identified, the portion having a first time and a second time. A pause in the portion of the audio signal is identified, the pause including the second time. A determination is made as to whether the pause indicates completion of an utterance of the audio signal. Based on determining that the pause indicates completion of the utterance, the portion of the audio signal is presented as input to the speech recognition engine. Based on determining that the pause does not indicate completion of the utterance, the portion of the audio signal is not presented as input to the speech recognition engine. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 An example wearable system according to examples of the present disclosure is shown.

[0011] Figure 2 An example handheld controller that may be used in conjunction with an example wearable system according to examples of the present disclosure is shown.

[0012] Figure 3 An example auxiliary unit that may be used in conjunction with an example wearable system according to examples of the present disclosure is shown.

[0013] Figure 4 An example functional block diagram of an example wearable system according to one or more examples of the present disclosure is shown.

[0014] Figure 5 An example audio waveform for input to an example ASR system is shown according to one or more examples of the present disclosure.

[0015] Figure 6 A flow chart illustrating an example of processing an acoustic speech signal using an ASR system according to one or more examples of the present disclosure is shown.

[0016] Figure 7 A flow chart illustrating an example of processing an acoustic speech signal using an ASR system according to one or more examples of the present disclosure is shown.

[0017] Figure 8A-8B A flow chart illustrating an example of detecting pauses in an input speech signal according to one or more examples of the present disclosure is shown.

[0018] Figure 9A-9B A flowchart illustrating an example of determining whether an input utterance has terminated according to one or more examples of the present disclosure.

[0019] Fig.10 A flow chart illustrating an example of classifying input data to determine a probability associated with the input data according to one or more examples of the present disclosure. DETAILED DESCRIPTION

[0020] In the following description of the examples, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific examples that may be practiced. It should be understood that other examples may be used and structural changes may be made without departing from the scope of the disclosed examples.

[0021] Example Wearable System

[0022] Figure 1An example wearable head device 100 configured to be worn on a user's head is shown. The wearable head device 100 can be part of a broader wearable system that includes one or more components, such as a head device (e.g., the wearable head device 100), a handheld controller (e.g., the handheld controller 200 described below), and / or an auxiliary unit (e.g., the auxiliary unit 300 described below). In some examples, the wearable head device 100 can be used for virtual reality, augmented reality, or mixed reality systems or applications. The wearable head device 100 can include one or more displays, such as displays 110A and 110B (which can include left and right transmissive displays, and associated components for coupling light from the displays to the user's eyes, such as an orthogonal pupil expansion (OPE) grating group 112A / 112B and an exit pupil expansion (EPE) grating group 114A / 114B); left and right acoustic structures, such as speakers 120A and 120B (which can be mounted on temples 122A and 122B and positioned near the user's left and right ears, respectively); a One or more sensors, such as infrared sensors, accelerometers, GPS units, inertial measurement units (IMUs) (e.g., IMUs 126), acoustic sensors (e.g., microphones 150); orthogonal coil electromagnetic receivers (e.g., receivers 127 mounted on the left temple 122A shown); left and right cameras oriented away from the user (e.g., depth (time of flight) cameras 130A and 130B); and left and right eye cameras oriented toward the user (e.g., for detecting eye movements of the user) (e.g., eye cameras 128 and 128B). However, without departing from the scope of the present invention, the wearable head device 100 may be combined with any suitable display technology, and any suitable number, type, or combination of sensors or other components. In some examples, the wearable head device 100 may integrate one or more microphones 150 configured to detect audio signals generated by the user's voice; such microphones may be placed in the wearable head device adjacent to the user's mouth. In some examples, the wearable head device 100 may integrate network features (e.g., Wi-Fi functionality) to communicate with other devices and systems including other wearable systems. The wearable head device 100 may also include components such as a battery, a processor, a memory, a storage unit, or various input devices (e.g., buttons, touch pads); or may be coupled to a handheld controller (e.g., handheld controller 200) or an auxiliary unit (e.g., auxiliary unit 300) that includes one or more such components. In some examples, the sensor may be configured to output a set of coordinates of the head-mounted unit relative to the user's environment and may provide input to a processor that performs a simultaneous localization and mapping (SLAM) process and / or a visual odometry algorithm.In some examples, the wearable head device 100 can be coupled to a handheld controller 200 and / or an auxiliary unit 300, as further described below.

[0023] Figure 2 An example mobile handheld controller assembly 200 of an example wearable system is shown. In some examples, the handheld controller 200 can communicate with the wearable head device 100 and / or the auxiliary unit 300 described below in wired or wireless communication. In some examples, the handheld controller 200 includes a handle portion 220 to be held by a user, and one or more buttons 240 disposed along a top surface 210. In some examples, the handheld controller 200 can be configured to be used as an optical tracking target; for example, a sensor (e.g., a camera or other optical sensor) of the wearable head device 100 can be configured to detect the position and / or orientation of the handheld controller 200—which can be extended to indicate the position and / or orientation of the hand of the user holding the handheld controller 200. In some examples, the handheld controller 200 can include a processor, a memory, a storage unit, a display, or one or more input devices as described above. In some examples, the handheld controller 200 includes one or more sensors (e.g., any sensor or tracking component described above with respect to the wearable head device 100). In some examples, the sensor can detect the position or orientation of the handheld controller 200 relative to the wearable head device 100 or another component of the wearable system. In some examples, the sensor may be positioned in the handle portion 220 of the handheld controller 200, and / or may be mechanically coupled to the handheld controller. The handheld controller 200 may be configured to provide one or more output signals corresponding to, for example, the pressed state of the button 240; or the position, orientation, and / or movement of the handheld controller 200 (e.g., via an IMU). Such output signals may be used as input to a processor of the wearable head device 100, the auxiliary unit 300, or another component of the wearable system. In some examples, the handheld controller 200 may include one or more microphones to detect sounds (e.g., the user's voice, ambient sounds), and in some cases provide signals corresponding to the detected sounds to a processor (e.g., a processor of the wearable head device 100).

[0024] Figure 3An example auxiliary unit 300 of an example wearable system is shown. In some examples, the auxiliary unit 300 can communicate with the wearable head device 100 and / or the handheld controller 200 in wired or wireless communication. The auxiliary unit 300 may include a battery to provide energy to operate one or more components of the wearable system, such as the wearable head device 100 and / or the handheld controller 200 (including a display, a sensor, an acoustic structure, a processor, a microphone, and / or other components of the wearable head device 100 or the handheld controller 200). In some examples, the auxiliary unit 300 may include a processor, a memory, a storage unit, a display, one or more input devices, and / or one or more sensors such as those described above. In some examples, the auxiliary unit 300 includes a clip 310 for attaching the auxiliary unit to a user (e.g., a belt worn by the user). One advantage of using auxiliary unit 300 to house one or more components of a wearable system is that doing so can allow large or heavy components to be carried on the user's waist, chest, or back—parts of the body that are relatively suitable for supporting large and heavy objects—rather than mounted to the user's head (e.g., if mounted in wearable head device 100) or carried by the user's hand (e.g., if mounted in handheld controller 200). This can be particularly advantageous for relatively heavy or large components (such as batteries).

[0025] Figure 4 An example functional block diagram that may correspond to an example wearable system 400 is shown, such as may include the above-described example wearable head device 100, handheld controller 200, and auxiliary unit 300. In some examples, the wearable system 400 may be used for virtual reality, augmented reality, or mixed reality applications. Figure 4As shown, the wearable system 400 may include an example handheld controller 400B, referred to herein as a "totem" (and may correspond to the handheld controller 200 described above); the handheld controller 400B may include a totem to head mounted device six degrees of freedom (6DOF) totem subsystem 404A. The wearable system 400 may also include an example wearable head device 400A (which may correspond to the wearable head device 100 described above); the wearable head device 400A includes a totem to head mounted device 6DOF head mounted device subsystem 404B. In this example, the 6DOF totem subsystem 404A and the 6DOF head mounted device subsystem 404B cooperate to determine six coordinates (e.g., offsets in three translation directions and rotations along three axes) of the handheld controller 400B relative to the wearable head device 400A. The six degrees of freedom may be embodied relative to the coordinate system of the wearable head device 400A. The three translation offsets may be represented as X, Y, and Z offsets in such a coordinate system, as a translation matrix, or as some other representation. The rotational degrees of freedom may be represented as a sequence of yaw, pitch, and roll rotations; as a vector; as a rotation matrix; as a quaternion; or as some other representation. In some examples, one or more depth cameras 444 (and / or one or more non-depth cameras) are included in the wearable head device 400A; and / or one or more optical targets (e.g., the button 240 of the handheld controller 200 described above, or a dedicated optical target included in the handheld controller) may be used for 6DOF tracking. In some examples, the handheld controller 400B may include the above-mentioned camera; and the head-mounted device 400A may include an optical target for optical tracking in conjunction with the camera. In some examples, the wearable head device 400A and the handheld controller 400B each include a set of three orthogonally oriented solenoids for wirelessly transmitting and receiving three distinguishable signals. By measuring the relative amplitudes of the three distinguishable signals received in each coil for receiving, the 6DOF of the handheld controller 400B relative to the wearable head device 400A can be determined. In some examples, the 6DOF totem subsystem 404A may include an inertial measurement unit (IMU) for providing improved accuracy and / or more timely information regarding rapid movements of the handheld controller 400B.

[0026] In some examples involving augmented reality or mixed reality applications, it may be necessary to transform coordinates from a local coordinate space (e.g., a coordinate space fixed relative to the wearable head device 400A) to an inertial coordinate space, or to an environmental coordinate space. For example, for the display of the wearable head device 400A, such a transformation is required to present virtual objects at an expected position and orientation relative to the real environment (e.g., a virtual person sitting in a real chair, facing forward, regardless of the position and orientation of the wearable head device 400A), rather than presenting the virtual objects at a fixed position and orientation on the display (e.g., at the same position in the display of the wearable head device 400A). This can maintain the illusion that the virtual objects exist in the real environment (and, for example, do not appear in unnatural positions in the real environment as the wearable head device 400A moves and rotates). In some examples, the imagery from the depth camera 444 can be processed to determine the transformation of the wearable head device 400A relative to the inertial or environmental coordinate system, thereby determining a compensating transformation between coordinate spaces (e.g., using simultaneous localization and mapping (SLAM) and / or visual ranging programs). Figure 4 In the example shown, the depth camera 444 can be coupled to the SLAM / visual odometry block 406 and can provide images to the block 406. The SLAM / visual odometry block 406 embodiment may include a processor configured to process the image and determine the position and orientation of the user's head, which can then be used to identify the transformation between the head coordinate space and the real coordinate space. Similarly, in some examples, an additional source of information about the user's head posture and position is obtained from the IMU 409 of the wearable head device 400A. Information from the IMU 409 can be integrated with information from the SLAM / visual odometry block 406 to provide improved accuracy and / or more timely information for rapid adjustment of the user's head posture and position.

[0027] In some examples, depth camera 444 can provide 3D images to gesture tracker 411, which can be implemented in a processor of wearable head device 400A. Gesture tracker 411 can recognize the user's gesture by, for example, matching the 3D images received from depth camera 444 with stored patterns representing gestures. Other suitable techniques for recognizing the user's gestures will be apparent.

[0028] In some examples, one or more processors 416 can be configured to receive data from the head mounted device subsystem 404B, the IMU 409, the SLAM / visual odometry block 406, the depth camera 444, the microphone 450; and / or the gesture tracker 411. The processor 416 can also send and receive control signals from the 6DOF totem system 404A. The processor 416 can be wirelessly coupled to the 6DOF totem system 404A, such as in the example of an untethered handheld controller 400B. The processor 416 can also communicate with additional components, such as an audio-visual content storage 418, a graphics processing unit (GPU) 420, and / or a digital signal processor (DSP) audio stereo spatial effector 422. The DSP audio stereo spatial effector 422 can be coupled to a head-related transfer function (HRTF) storage 425. The GPU 420 can include a left channel output coupled to a left image modulated light source 424 and a right channel output coupled to a right image modulated light source 426. The GPU 420 may output stereo image data to the image modulated light sources 424, 426. The DSP audio stereo spatial effector 422 may output audio to the left speaker 412 and / or the right speaker 414. The DSP audio stereo spatial effector 422 may receive an input from the processor 419 indicating a direction vector from the user to the virtual sound source (which may be moved by the user, for example, via the handheld controller 400B). Based on the direction vector, the DSP audio stereo spatial effector 422 may determine the corresponding HRTF (e.g., by accessing the HRTF, or by interpolating multiple HRTFs). The DSP audio stereo spatial effector 422 may then apply the determined HRTF to an audio signal, such as an audio signal corresponding to a virtual sound generated by a virtual object. This may enhance the credibility and realism of the virtual sound by incorporating the user's relative position and orientation relative to the virtual sound in the mixed reality environment—that is, presenting a virtual sound that matches the user's expectations so that the virtual sound sounds like a real sound in a real environment.

[0029] In some examples, such as Figure 4 As shown, one or more of the processor 416, GPU 420, DSP audio stereo spatial effector 422, HRTF memory 425, and audio-visual content memory 418 may be included in an auxiliary unit 400C (which may correspond to the auxiliary unit 300 described above). The auxiliary unit 400C may include a battery 427 to power its components and / or to power the wearable head device 400A and / or the handheld controller 400B. Including such components in an auxiliary unit that can be mounted to the user's waist can limit the size and weight of the wearable head device 400A, thereby alleviating fatigue on the user's head and neck.

[0030] although Figure 4 Elements corresponding to the various components of the example wearable system 400 are presented, but various other suitable arrangements of these components will become apparent to those skilled in the art. For example, Figure 4 Elements presented as being associated with the auxiliary unit 400C may alternatively be associated with the wearable head device 400A or the handheld controller 400B. In addition, some wearable systems may completely abandon the handheld controller 400B or the auxiliary unit 400C. Such changes and modifications should be understood to be included within the scope of the disclosed examples.

[0031] Speech Recognition System

[0032] A speech recognition system typically includes a speech recognition engine that can accept an input audio signal (source signal) corresponding to human speech; process and analyze the input audio signal; and generate an output corresponding to human speech based on the analysis. For example, in the case of an automatic speech recognition (ASR) system, the output of the speech recognition engine can be a text transcription of human speech. In the case of a natural language processing system, the output may be one or more commands or instructions indicated by human speech; or some representation of the semantics of human speech (e.g., a logical expression or data structure). Other types of speech processing systems, including those that do not necessarily "recognize" speech (e.g., automatic translation systems), are also contemplated and within the scope of the present disclosure. In addition, as used herein, a speech recognition engine may include one or more of an automatic speech recognition engine, a natural language understanding engine, and other suitable components.

[0033] ASR systems are found in a variety of products and applications: conventional telephone systems; automated voice messaging systems; voice assistants (including standalone and smartphone-based voice assistants); vehicles and aircraft; desktop and document processing software; data entry; household appliances; medical devices; language translation software; closed captioning systems, etc. One advantage of ASR systems is that they can allow users to provide input to a computer system using natural spoken language, such as presented to a microphone, rather than conventional computer input devices, such as a keyboard or touchpad; therefore, speech recognition systems are particularly useful in environments where conventional input devices (e.g., keyboards) are unavailable or impractical. In addition, by allowing users to provide intuitive voice-based input, speech recognition engines can enhance immersion. Therefore, ASR can be naturally applicable to wearable systems, especially virtual reality, augmented reality, and / or mixed reality applications of wearable systems, in which user immersion is the primary goal and it is desirable to limit the use of conventional computer input devices whose presence affects immersion.

[0034] Identify input speech boundaries

[0035] The effectiveness of an ASR system is limited by its ability to quickly present accurate input data to a speech recognition engine. Presenting accurate input data may require correctly identifying when individual input sequences begin and end. Some ASR systems have difficulty determining when a speaker's utterance is complete in a timely and real-time manner. The present disclosure relates to systems and methods for improving the accuracy of a speech processing system by accurately identifying the end point of an utterance presented as input to the speech processing system. Rapidly and accurately determining the location of the end of an utterance enables the speech processing system to provide correct results in a timely and real-time manner—i.e., in response to a real-time stream of input audio, where the entire input audio signal cannot be known in advance.

[0036] Figure 5 An example audio waveform 500 is shown, such as an audio waveform that may be detected by one or more microphones and presented as input to an ASR system. The waveform 500 represents a user speaking an example utterance "What's the weather tomorrow in Moscow?", with the intent that the ASR system receiving the utterance as input will query a weather service and respond with a forecast for tomorrow's weather in Moscow. The speed and accuracy of the ASR system response will depend on the point at which the ASR system determines the user's utterance ends. The example waveform 500 begins at an initial time t0. If the ASR system prematurely determines that the utterance ends at time t1 (which falls after the phrase "what's the weather" and before the word "tomorrow"), the input utterance (i.e., the input speech falling between time t0 and time t1) will be determined to be "what's the weather". Since this input utterance lacks the qualifiers added by the user (i.e., "tomorrow" and "in Moscow"), the ASR system's response to this utterance will not meet the user's expectations. For example, the system might return the current weather (not tomorrow's weather) for the user's current location (not Moscow). Similarly, if the ASR system determines that the utterance ends at time t2 (which falls after the phrase "what's the weather tomorrow" and before the phrase "in Moscow"), the input utterance (i.e., the input speech falling between time t0 and time t2) will be determined to be "what's the weather tomorrow", and the resulting response will again not meet the user's expectations (Moscow tomorrow's weather). An ideal ASR system can determine that the end time of the utterance is t3, which happens to fall at the end of the entire input query, and will correctly identify the input utterance as "what's the weather in Moscow".

[0037] An ideal ASR system also does not include the end of the input signal that is not part of the input utterance. For example, if the ASR system determines that the utterance ends at time t4, the input utterance will include all of the correct input speech (i.e., "what's the weather in Moscow"), but will also include irrelevant information (the portion of the input speech signal between t3 and t4). This irrelevant information may introduce errors into the input utterance and further delay the response of the ASR system (i.e., at least the signal processing time span between t3 and t4), causing the user to perceive an insufficient response.

[0038] Some ASR systems may incorrectly identify the end point of an input utterance. For example, when presented with the example waveform 500 as input, some ASR systems may incorrectly identify the end point of the utterance as t1, t2, or t4, rather than t3.

[0039] Figure 6An example process 600 that can be performed by an ASR system is shown. The example process 600 uses a timeout interval to determine when an input speech is terminated; that is, when the input speech has not been received for a period of time exceeding the timeout interval, the speech is considered complete and presented to the ASR engine for processing. As shown, in stage 610, input speech is detected in real time from one or more microphones 602. In stage 620, the process 600 determines whether the input speech is still being received; if so, the input speech is considered to be ongoing, and the process 600 returns to stage 610 to continue detecting the input speech. If not, the process 600 queries whether the timeout interval 632 has ended. If not, the process 600 returns to stage 610; if so, the speech is considered complete (stage 640). In stage 650, the process 600 then presents the speech (or some representation of the speech) to the ASR engine 660 for processing. The ASR engine 660 can generate a speech-to-text representation of the input speech. A natural language understanding (NLU) engine 665 can perform additional processing based on the representation. For example, the NLU engine 665 can generate a semantic representation of the speech-to-text representation output from the ASR engine 660; determine that the input utterance represents a request for a weather report; query a weather report service using a structured query generated from the input utterance; and receive a response from the weather report service. In some embodiments, the process 600 can stream the utterance (or some representation of the utterance) for processing (e.g., by the ASR engine 660). In some embodiments, the ASR engine 660 can return or output incremental results. In some embodiments, the NLU engine 665 can receive output from the ASR engine 660 for processing after the utterance is terminated. At stage 670, the process 600 can receive a response from the NLU engine 665 and present the response to the user at stage 680 (e.g., via a text-to-speech engine coupled to an output speaker).

[0040] The above process 600 is prone to errors because by using a simple timeout interval to terminate the input utterance at stages 630 and 640, the process 600 may prematurely terminate the utterance before the user has finished speaking the utterance. Referring to the above waveform 500, this may cause the input utterance to terminate at time t1 or t2 instead of t3. This may occur when the user inadvertently inserts a non-speech gap between two words of a single utterance (e.g., in the example waveform 500, a pause between "weather" and "tomorrow" or between "tomorrow" and "Moscow"). If these gaps exceed the length of the timeout interval 632, the process 600 may prematurely determine that the input utterance has been completed, even if the user is still continuing to speak the utterance (this situation is particularly common in the following situations: that is, complex input queries in which the user needs additional time to formulate their question; or the user has a speech disorder; or the user feels anxious when interacting with the microphone or ASR system).

[0041] This problem cannot be completely solved simply by increasing the length of timeout interval 632 because there is a tradeoff between the duration of this interval and the perceived responsiveness of the ASR system. That is, even though timeout interval 632 can be increased to exceed any possible intra-utterance input gap—preventing process 600 from cutting off the input utterance prematurely—the ASR system will wait for the duration of this extended timeout interval before determining that the utterance has ended. This delay can be annoying to users, who may perceive the delay as unresponsiveness—especially compared to face-to-face human interaction where the listener can quickly and intuitively understand when the speaker has finished speaking. In some embodiments, the delay can cause crosstalk—when the user perceives the ASR system to be unresponsive and begins speaking again (e.g., repeating the initial input)—which can result in a cascade of errors.

[0042] Figure 7 An example process 700 (e.g., which may be performed by an ASR system) is shown that may more quickly and accurately recognize input utterances than process 600. In process 700, as described below, pauses in input speech may be detected and then contextual cues analyzed to indicate whether the pause may represent the completion of the current utterance (in which case the utterance may be terminated and presented to the ASR engine and NLU engine), or whether the pause indicates that the current utterance is ongoing (in which case the ASR system should continue detecting the current utterance).

[0043] In process 700, at stage 710, an audio input presented by a user is detected from one or more microphones 602 (in some examples, the audio input can be received as streaming data or as one or more data files as an alternative or supplement to the microphone output). The audio input can be stored in an input buffer or other memory for access by process 700. At stage 720, process 700 can determine (e.g., based on the input buffer and / or sensor data, as described in more detail below) whether the user pauses when presenting the input speech. If no pause is detected, indicating that the user's current speech is ongoing, the process can return to stage 710 to continue detecting audio input. If a pause is detected at stage 720, process 700 can determine the possibility that the pause indicates the completion of the current speech (rather than the continuation of the current speech) at stage 730. For example, stage 720 can determine a digital confidence value, which represents the possibility that the pause indicates the completion of the current speech. This determination can be made based on the content of the input buffer and / or sensor data, as described in more detail below.

[0044] At stage 732, process 700 may evaluate a determination of whether the pause detected at stage 730 indicates completion of the current utterance. If it has been determined with sufficient confidence (e.g., with a confidence level exceeding a threshold) that the pause indicates completion of the current utterance, process 700 may continue to terminate the utterance (stage 740); present the utterance (stage 750) to an ASR engine (760); receive a response from an NLU engine (765) (stage 770); and present the response to the user (stage 780). These steps may correspond to stages 640, 650, ASR engine 660, NLU engine 665, stage 670, and stage 680, respectively, described above with respect to process 600.

[0045] If the process 700 determines that the pause is unlikely to indicate that the current utterance is complete (e.g., the determined confidence level does not meet the threshold), the process 700 can take various actions in response at stage 732. In some examples, the process 700 can adjust or reset (stage 734) the parameters used to determine whether a pause is detected (such as described herein with respect to stage 720). For example, the process 700 can increase or reset the timeout interval used to detect pauses in the input speech at stage 720 at stage 734. This may be beneficial if the process 700 determines that more time is needed to determine whether the user intends to complete the current utterance. In some examples, the process 700 can present a prompt to the user (stage 736), such as a prompt to make additional input (e.g., a visual and / or auditory prompt asking the user to indicate whether he or she has completed speaking). This is beneficial in situations where it is unclear whether the current utterance is complete—for example, in situations where the process 700 returns a confidence level that is less than but close to a threshold. In some examples, when a pause is detected that does not indicate that the current utterance is complete, process 700 can combine the current utterance with the second utterance (stage 738); for example, the utterance before the pause can be connected to the second utterance after the pause to present the combined utterance to a speech recognition engine (e.g., an ASR engine and / or an NLU engine). In some examples, process 700 can return to stage 710 to continue detecting input speech without taking any additional actions such as those described with respect to stages 734, 736, or 738; this behavior may be preferred in the case where the confidence returned by stage 730 is far below the threshold required to terminate the current utterance.

[0046] Fig. 8A Stage 720 for implementing process 700 (such as described above with respect to Figure 7 Flowchart of an example process of using a method for performing an audio input process (described in the foregoing). In the figure, audio input data 810 (e.g., a speech input signal stored in an input buffer) can be evaluated to determine the presence of a pause in the input. At stage 820, the process can determine whether the value of one or more characteristics of the input signal is above or below a threshold for a period exceeding a timeout interval. In some examples, it can be determined whether the amplitude of the input signal is below a threshold amplitude level for a period exceeding a timeout interval. If so, a pause in the user's input speech can be indicated (stage 860), such as described above with respect to Figure 6 In some embodiments, the length of the timeout interval may vary depending on the characteristics. In some examples, the spectral analysis may identify the speech signal as distinct from other ambient or incidental sounds, and may determine whether the output of the analysis is above or below one or more thresholds for a period exceeding the timeout interval.

[0047] However, even if it is determined at stage 820 that the timeout interval 822 has not yet ended, the process may examine the audio input data 810 (stage 830) to determine whether the speech data includes a language cue indicating a pause in the input speech (rather than relative silence). These language cues may include prosodic features of the user (e.g., rhythm, intonation, timbre, volume), the presence of a trailing word, the presence of a closing word or phrase (e.g., "thank you" when completing a verbal request), etc. These language cues may indicate that the current utterance is complete, even if the timeout interval has not yet ended. At stage 840, the process may evaluate whether there are any such language cues, and if so, whether they indicate that the input speech has paused (stage 860) or has not paused (stage 850). In some cases, stage 840 may make this determination by comparing the confidence generated at stage 830 to a threshold. By evaluating the presence of a language cue to indicate that the utterance has been completed, even before the timeout interval expires, the process may avoid a sense of unresponsiveness caused by waiting for the timeout interval to expire before presenting the utterance to be processed (e.g., by an ASR engine and / or NLU engine), as described above.

[0048] Figure 8B 8. A flowchart of an example process for implementing stage 720 of process 700 is shown, wherein sensor input data 844 is used in place of or in conjunction with audio input data 810 such as described above. In some examples, as described above, sensor input data 844 may be combined with audio input data such as that described above with respect to Figure 1 As described above, such a wearable system may include one or more sensors that may provide input about a user and / or the environment of the wearable system. For example, the wearable head device 100 may include a camera (e.g., Figure 4444 described in ) to output visual signals corresponding to the environment; in some examples, the camera can be a forward-facing camera on a head-mounted unit (for displaying what is currently in front of the wearable system user). In some examples, the wearable head device 100 may include a LIDAR unit, a radar unit, and / or an acoustic sensor that can output signals corresponding to the physical geometry of the user's environment (e.g., walls, physical objects). In some examples, the wearable head device 100 may include a GPS unit that can indicate geographic coordinates corresponding to the current location of the wearable system. In some examples, the wearable head device 100 may include an accelerometer, a gyroscope; and / or an inertial measurement unit (IMU) to indicate the orientation of the wearable head device 100. In some examples, the wearable head device 100 may include an environmental sensor, such as a temperature or pressure sensor. In some examples, the wearable head device 100 may include a biometric sensor, such as an iris camera; a fingerprint sensor; an eye tracking sensor (e.g., an electrooculogram (EOG) sensor) to measure the user's eye movement or eye gaze; or a sensor that measures the user's vital signs. In examples where the wearable head device 100 includes a head mounted unit, such an orientation may correspond to the orientation of the user's head (and, by extension, to the direction of the user's mouth and the user's voice). Other suitable sensors may be included and may provide sensor input data 844. Furthermore, in some examples, sensors other than those of the wearable system may be appropriately utilized. For example, sensors associated with a microphone of a speech recognition system (e.g., GPS, IMU) may be used in conjunction with sensors of the wearable system to determine the relative distance and orientation between the user and the speech recognition system.

[0049] At stage 842, process 800 may examine sensor input data 844 to determine whether the sensor data includes non-verbal cues indicating a pause in the input speech. These non-verbal cues may include, for example, features of the user's eye gaze; head posture; hand gestures; vital signs (e.g., breathing pattern, heart rate); and facial expressions. These non-verbal cues may indicate that the current utterance is complete even if the timeout interval has not expired, and even in the absence of, for example, the above-described non-verbal cues regarding the completion of the utterance. Fig. 8AIn the case of the language prompt described. For example, the pause in the user's voice may correspond to: changes in the user's eye gaze target, changes in the user's head posture, gestures performed by the user, changes in the user's vital signs (e.g., breathing patterns, heart rate), changes in the user's facial expressions, changes in the movement or rotation away from the microphone, changes in the user's posture or other physical features indicated by the sensor input data 844, changes in orientation, and / or any one or more of the above features. In stage 846, the process can evaluate whether there are any such non-verbal prompts, and if so, they indicate whether the input voice has paused (stage 860) or not (stage 850). In some cases, stage 846 can make this determination by comparing the confidence generated in stage 842 with a threshold. As described above, by evaluating the presence of non-verbal prompts to indicate that the speech has been completed, even before the timeout time interval expires, the process can avoid the sense of unresponsiveness caused by waiting for the timeout time interval to terminate before presenting the speech to the speech recognition engine, as described above. Figure 8B While the verbal cues and non-verbal cues are shown analyzed in different stages (ie, stage 840 and stage 846), some examples may combine the verbal cues and non-verbal cues to be analyzed in a single stage.

[0050] Fig.9A Stage 730 for implementing process 700 (such as described above with respect to Figure 7 At stage 730, process 700 determines, for a pause such as that identified at stage 720 described above, whether the pause is likely to correspond to the end of the current utterance. Fig.9A In the process, the audio input data 910 (e.g., a speech signal stored in an input buffer, which may correspond to 810 above) may be evaluated at stage 920 to determine the presence of gap sounds in the audio input data. The gap sounds may be words, phrases, syllables, or other vocalizations that appear in the input audio indicating that the current utterance is not yet complete (such as when the user is thinking). For example, the gap sounds may include hesitation sounds (e.g., "um", "uh"); extended syllables (e.g., "to (ooo)" extended at the end of the phrase "I'mgoing to"); repetitions (e.g., "and, and, and ......"); trailing filler words (e.g., "like", "I mean"), and / or other indications that the user may provide additional input audio belonging to the current utterance. Such gap sounds may be specific to an individual user, a specific language, or a type of verbal input (e.g., a question, an instructional statement). As described below, various classifiers may be used to identify gap sounds.

[0051] At stage 930, the process may determine whether any such gap sounds were detected at stage 920. If not, the process may infer (stage 970) that the current utterance is complete. If gap sounds are present, the process may evaluate at stage 940 whether the gap sounds indicate that the current utterance is ongoing. For example, the presence of hesitation sounds may indicate that the user is organizing a complete utterance (e.g., "What's the weather...uh...tomorrow"). Similarly, extended syllables, repetitions, filler words, and other gap sounds may indicate that the current utterance is not yet complete. In some examples, stage 940 may generate a confidence level indicating the likelihood of the presence of gap sounds and indicating whether the current utterance is complete.

[0052] At stage 950, if it is determined at stage 940 that the current utterance is ongoing, the process may infer (stage 960) that the current utterance is not complete. Figure 7 As described, this may cause the process to perform various responsive actions: for example, process 700 may extend a timeout interval (e.g., 822), in which, in order to detect a pause, the user is prompted for additional input indicating whether the current utterance is complete and / or combining the current utterance with a second utterance; or no action is taken at all. In some examples, which action (if any) is performed depends on the confidence generated at stage 940; for example, in response to a high confidence that the utterance is not yet complete, process 700 simply returns to stage 710 to continue detecting audio input without taking any further action; and in response to a medium confidence (indicating, for example, uncertainty about whether the current utterance is complete), process 700 may explicitly prompt the user for additional input (stage 736). Similarly, if it is determined at stage 940 that the current utterance is complete, the process may indicate so (stage 970), and the process may continue to present the utterance to the speech recognition system, such as described above.

[0053] Fig. 9B A flow chart of an example process for implementing stage 730 of example process 700 is shown, wherein sensor input data 942 is used instead of or in conjunction with audio input data 910 such as described above. Sensor input data 942 may correspond to sensor input data 844 described above: For example, sensor input data 942 may be composed of sensor input data such as described above with respect to Figure 1942. The sensor outputs described in the example wearable head device 100 in FIG. 942. As described above, such sensors may include one or more cameras (e.g., RGB cameras, depth cameras); LIDAR units; radar units; acoustic sensors; GPS units; accelerometers; gyroscopes; IMUs; environmental sensors; biometric sensors (e.g., iris cameras, fingerprint sensors, eye tracking sensors, and / or sensors for measuring user vital signs). Other suitable sensors may be included and may provide sensor input data 942. In addition, in some examples, sensors other than sensors of the wearable system may be appropriately utilized. For example, as described above, sensors associated with a microphone of a speech recognition system (e.g., GPS, IMU) may be used in conjunction with sensors of the wearable system to determine the relative distance and orientation between the user and the speech recognition system.

[0054] about Fig. 9B , sensor input data 942 may be evaluated at stage 944 to determine whether the sensor data indicates that the current utterance is in progress, or whether the current utterance is complete. For example, the completion (or incompleteness) of an utterance may correspond to a change in a user's eye gaze target, a change in the user's head posture, a gesture performed by the user, a change in a user's vital signs (e.g., breathing pattern, heart rate), a change in a user's facial expression, a change in movement or rotation away from a microphone, a change in the user's posture, or a change in other physical characteristics indicated by sensor input data 944, a change in orientation, and / or a change in any one or more of the above characteristics. In some examples, stage 944 may generate a confidence level indicating the likelihood that the current utterance is complete. Based on the determination made at stage 944 (e.g., by comparing the confidence level to a threshold), the process at stage 950 may indicate that the utterance is complete (stage 970) or incomplete (stage 960).

[0055] In the above process 700, the importance of input data (e.g., audio data, sensor data) relative to how the data is presented to a speech recognition engine (e.g., an ASR engine and / or an NLU engine) can be evaluated at one or more stages. For example, at stage 830 of process 720, as described above, the audio input data can be evaluated to determine whether the data includes language cues that the current utterance is complete. At stage 842, as described above, the sensor data can be evaluated for non-verbal cues (e.g., changes in facial expressions) regarding the completion of the current utterance. At stage 920, as described above, the audio input data can be evaluated to identify the presence of gap sounds; and at stage 940, it can be evaluated whether these gap sounds indicate that the current utterance is ongoing. At stage 944, as described above, the sensor input data can be evaluated to determine whether the sensor input data indicates that the current utterance is ongoing.

[0056] In some examples, the audio input data and / or sensor input data used as described above can be classified according to one or more parameters - generating one or more classifiers representing the data. These classifiers can be used (e.g., by the example process 700) to evaluate the importance of the data (e.g., the probability associated with the data). Fig.10 An example process 1000 is shown for classifying input data 1010 to determine probabilities of interest associated with the input data. As used herein, the probabilities of interest can correspond to the probabilities described above with respect to example process 700: the probability that the audio input data and / or sensor input data indicates a pause in the input speech; the probability that a pause indicates that an utterance is complete; and / or the probability that the presence of interstitial sounds indicates that an utterance is ongoing; or other suitable probabilities. Fig.10 , the determination may be performed using the audio input data 1016, alone or in combination with the sensor input data 1020. Determining a probability value for the input data 1010 may be referred to as “classifying” the input data 1010, and a module or process for performing such a determination (e.g., 1074) may be referred to as a “classifier.”

[0057] exist Fig.10 In the illustrated example process, input data 1010 (e.g., audio input data 1016 and / or sensor input data 1020) may be used in conjunction with speech / sensor data 1029 (e.g., from a database) to determine one or more probabilities of interest for the input data 1010. In some examples, the audio input data 1016 and / or sensor input data 1020 may be parameterized / characterized according to one or more parameters at stage 1075 to facilitate classification of speech segments based on the speech / sensor data 1029. A Fourier transform of the input data 1010 may be performed to provide a spectral representation of the input data 1010 (e.g., a frequency function indicating the relative prevalence of various frequency parameters in the audio input data 1016 and / or sensor input data 1020). For example, the process may identify levels (or changes thereto) of amplitude or component frequencies of a user's speech, position, eye gaze, and / or body movement; these values ​​may indicate pauses in the user's speech, the presence of interstitial sounds, or completion of a user's utterance, as described above. In some examples, characteristics of a user—e.g., the user's age, gender, and / or native language—can be used as parameters to characterize input data 1010. Other ways of parameterizing input data 1010, where these parameterizations are used to determine probabilities of interest for the input data, will be apparent to those skilled in the art.

[0058] At stage 1076 of the example, a probability value 1078 is determined for the probability of interest of the input data 1010. In some examples, the probability value 1078 can be determined using the speech / sensor data 1029, such as a database including the speech / sensor data 1029 identifying whether the elements of the speech and / or sensor data in the database correspond to the input speech. In some examples, the audio input data 1016 may include a set of audio waveforms corresponding to speech segments; and the corresponding speech segment may indicate a pause or gap sound for each waveform. In some examples, as an alternative or in addition to the audio waveform, the audio input data 1016 may include audio parameters corresponding to the speech segment. The audio input data 1016 may be compared with the speech segments of the speech / sensor data 1029—for example, by comparing the audio waveform of the audio input data 1016 with a similar waveform of the speech / sensor data 1029, or by comparing the parameters of the audio input data 1016 (such as can be characterized in stage 1075) with similar parameters of the speech / sensor data 1029. Based on these comparisons, a probability value 1078 for the audio input data 1016 may be determined.

[0059] Similar techniques may be applied to sensor input data 1020. For example, sensor input data 1020 may include a sequence of raw sensor data; and may indicate for the raw sensor data whether the data indicates a pause, completion, or continuation of speech. Similarly, sensor input data 1020 may include sensor input parameters corresponding to the sensor data. Sensor input data 1020 may be compared to elements of speech / sensor data 1029, such as described above with respect to audio input data 1016.

[0060] Techniques for determining probabilities 1078 based on input data 1010 are familiar to those skilled in the art. For example, in some examples, nearest neighbor interpolation can be used at stage 1076 to compare elements of input data 1010 with similar data elements in an N-dimensional space (where N dimensions can include, for example, the above-mentioned audio parameters, audio waveform data, sensor parameters, or raw sensor data); and the probability value 1078 is determined based on the relative distance between the elements of the input data 1010 and its adjacent elements in the N-dimensional space. As another example, a support vector machine can be used at stage 1076 to determine a basis for classifying elements of input data 1010 as indicating speech completion or indicating speech incompleteness based on the speech / sensor database 1029; and to classify the input data 1010 based on the basis (e.g., determining a probability value 1078 that the input data 1010 indicates the presence of a completed speech, a pause, or a gap sound). Other suitable techniques for analyzing input data 1010 and / or speech / sensor data 1029, comparing input data 1010 to speech / sensor data 1029, and / or classifying input data 1010 based on speech / sensor data 1029 to determine probability 1078 will be apparent; the present disclosure is not limited to any particular technique or combination of techniques.

[0061] In some examples, machine learning techniques may be used alone or in combination with other techniques described herein to determine probability values ​​1078. For example, a neural network may be trained on speech / sensor data 1029 and applied to input data 1010 to determine probability values ​​1078 for the input data. As another example, a genetic algorithm may be used to determine a function based on speech / sensor data 1029 to determine probability values ​​1078 corresponding to input data 1010. Other suitable machine learning techniques familiar to those skilled in the art will be apparent; the present disclosure is not limited to any particular technique or combination of techniques.

[0062] In some examples, speech / sensor data 1029 may be generated by recording a set of speech data and / or sensor data for various users, and identifying for elements of the data whether the user has completed an utterance; has paused his or her speech; or is providing intermittent sounds. For example, while the user's speech is being recorded, the user may be observed interacting with a group of people, with a speech recognition system located in the same room; the user's sensor data (e.g., the output of a wearable system worn by the user) may also be recorded. For each region of recorded data, an observer may identify whether the data region corresponds to a pause, providing intermittent sounds, or completing an utterance. This information is apparent to an observer by observing the context in which the user is speaking—typically, it is easy for humans (unlike machines) to intuitively determine whether a user has completed an utterance based on observation of the user. This process may be repeated for multiple users until a sufficiently large and diverse speech / sensor data set is generated.

[0063] With respect to the above-described systems and methods, the elements of the systems and methods may be appropriately implemented by one or more computer processors (e.g., a CPU or DSP). The present disclosure is not limited to any particular configuration of computer hardware (including computer processors) for implementing these elements. In some cases, multiple computer systems may be employed to implement the above-described systems and methods. For example, a first computer processor (e.g., a processor of a wearable device coupled to a microphone) may be used to receive input microphone signals and perform initial processing of these signals (e.g., signal conditioning and / or segmentation, such as described above). A second (perhaps more powerful) processor may then be utilized to perform more computationally intensive processing, such as determining probability values ​​associated with speech segments of these signals. Another computer device, such as a cloud server, may host a speech recognition engine, ultimately providing input signals to the speech recognition engine. Other suitable configurations will be apparent and within the scope of the present disclosure.

[0064] Although the disclosed examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. For example, elements of one or more embodiments may be combined, deleted, modified or supplemented to form further embodiments. Such changes and modifications should be understood to be included within the scope of the disclosed examples defined by the appended claims.

Claims

1. A method comprising: receiving an audio signal via a microphone of the head mounted device, wherein the audio signal includes voice activity; receiving, via one or more sensors of the head mounted device, non-verbal sensor data corresponding to a user of the head mounted device; classifying audio data corresponding to the audio signal via a first classifier; determining whether the audio signal includes a pause in the speech activity; In response to determining that the audio signal includes a pause in the voice activity, determining whether the pause in the voice activity corresponds to an end point of the voice activity based on the classification of the audio data; and In response to determining that the pause in the voice activity corresponds to an ending point of the voice activity, presenting a response to the user based on the voice activity, Wherein, determining whether the pause in the voice activity corresponds to an end point of the voice activity includes: determining a probability of interest based on the classification of the audio data and further based on the non-language sensor data received via the one or more sensors of the head-mounted device.

2. The method according to claim 1, further comprising: In response to determining that the pause in the voice activity does not correspond to an ending point of the voice activity, presenting a response to the user based on the voice activity is foregone.

3. The method according to claim 1, further comprising: In response to determining that the audio signal does not include a pause in the voice activity, determining whether the pause in the voice activity corresponds to an end point of the voice activity is abandoned.

4. The method according to claim 1, wherein: Determining whether the audio signal includes a pause in the speech activity includes determining whether an amplitude of the audio signal falls below a threshold for a predetermined period of time.

5. The method according to claim 4, further comprising: In response to determining that the pause in the voice activity does not correspond to an end point of the voice activity, determining whether the audio signal includes a second pause corresponding to the end point of the voice activity.

6. The method according to claim 4, further comprising: In response to determining that the pause in the voice activity does not correspond to an end point of the voice activity, the user is prompted to speak.

7. The method according to claim 1, wherein: Determining whether the audio signal includes a pause in the voice activity includes determining whether the audio signal includes one or more language cues corresponding to an ending point of the voice activity.

8. The method according to claim 7, wherein: The one or more language cues include characteristics of the user's prosody.

9. The method according to claim 7, wherein: The one or more verbal cues include a closing phrase.

10. The method according to claim 1, wherein: The non-verbal sensor data is indicative of a gaze of the user.

11. The method according to claim 1, wherein: The non-verbal sensor data is indicative of a facial expression of the user.

12. The method according to claim 1, wherein: The non-verbal sensor data is indicative of a heart rate of the user.

13. The method according to claim 1, wherein: Determining whether the pause in the voice activity corresponds to an end point of the voice activity includes identifying one or more interstitial sounds.

14. The method according to claim 1, wherein: Determining whether the pause in the speech activity corresponds to an end point of the speech activity includes evaluating non-verbal sensor data.

15. A system comprising: Headset microphones; one or more sensors of the head mounted device; a transmissive display of the head mounted device; as well as One or more processors configured to perform a method comprising: receiving an audio signal via the microphone of the head mounted device, wherein the audio signal includes voice activity; receiving, via the one or more sensors of the head mounted device, non-verbal sensor data corresponding to a user of the head mounted device; classifying audio data corresponding to the audio signal via a first classifier; determining whether the audio signal includes a pause in the speech activity; In response to determining that the audio signal includes a pause in the voice activity, determining whether the pause in the voice activity corresponds to an end point of the voice activity based on the classification of the audio data; and In response to determining that the pause in the voice activity corresponds to an ending point of the voice activity, presenting a response to the user based on the voice activity, Wherein, determining whether the pause in the voice activity corresponds to an end point of the voice activity comprises: determining a probability of interest based on the classification of the audio data and further based on the non-language sensor data received via the one or more sensors of the head mounted device; and Presenting the response to the user includes presenting the response via the transmissive display while presenting a view of an external environment to the user via the transmissive display.

16. The system of claim 15, further comprising: In response to determining that the pause in the voice activity does not correspond to an ending point of the voice activity, presenting a response to the user based on the voice activity is foregone.

17. The system of claim 15, further comprising: In response to determining that the audio signal does not include a pause in the voice activity, determining whether the pause in the voice activity corresponds to an end point of the voice activity is abandoned.

18. The system of claim 15, wherein: Determining whether the audio signal includes a pause in the speech activity includes determining whether an amplitude of the audio signal falls below a threshold for a predetermined period of time.

19. The system of claim 18, further comprising: In response to determining that the pause in the voice activity does not correspond to an end point of the voice activity, determining whether the audio signal includes a second pause corresponding to the end point of the voice activity.

20. The system of claim 18, further comprising: In response to determining that the pause in the voice activity does not correspond to an end point of the voice activity, the user is prompted to speak.

21. The system of claim 15, wherein: Determining whether the audio signal includes a pause in the voice activity includes determining whether the audio signal includes one or more language cues corresponding to an ending point of the voice activity.

22. The system of claim 21, wherein: The one or more language cues include characteristics of the user's prosody.

23. The system of claim 21, wherein: The one or more verbal cues include a closing phrase.

24. The system of claim 15, wherein: Determining whether the audio signal includes a pause in the speech activity includes evaluating non-speech sensor data.

25. The system of claim 15, wherein: The non-verbal sensor data is indicative of a facial expression of the user.

26. The system of claim 15, wherein: The non-verbal sensor data is indicative of a heart rate of the user.

27. The system of claim 15, wherein: Determining whether the pause in the voice activity corresponds to an end point of the voice activity includes identifying one or more interstitial sounds.

28. The system of claim 15, wherein: Determining whether the pause in the speech activity corresponds to an end point of the speech activity includes evaluating non-verbal sensor data.

29. A non-transitory computer-readable medium storing one or more instructions that, when executed by one or more processors of an electronic device, cause the device to perform a method comprising: receiving an audio signal via a microphone of the head mounted device, wherein the audio signal includes voice activity; receiving, via one or more sensors of the head mounted device, non-verbal sensor data corresponding to a user of the head mounted device; classifying audio data corresponding to the audio signal via a first classifier; determining whether the audio signal includes a pause in the speech activity; In response to determining that the audio signal includes a pause in the voice activity, determining whether the pause in the voice activity corresponds to an end point of the voice activity based on the classification of the audio data; and In response to determining that the pause in the voice activity corresponds to an ending point of the voice activity, presenting a response to the user based on the voice activity, Wherein, determining whether the pause in the voice activity corresponds to an end point of the voice activity includes: determining a probability of interest based on the classification of the audio data and further based on the non-language sensor data received via the one or more sensors of the head-mounted device.

30. The non-transitory computer readable medium of claim 29, further comprising: In response to determining that the pause in the voice activity does not correspond to an ending point of the voice activity, presenting a response to the user based on the voice activity is foregone.

31. The non-transitory computer readable medium of claim 29, the method further comprising: In response to determining that the audio signal does not include a pause in the voice activity, determining whether the pause in the voice activity corresponds to an end point of the voice activity is abandoned.

32. The non-transitory computer readable medium of claim 29, wherein: Determining whether the audio signal includes a pause in the speech activity includes determining whether an amplitude of the audio signal falls below a threshold for a predetermined period of time.

33. The non-transitory computer readable medium of claim 32, the method further comprising: In response to determining that the pause in the voice activity does not correspond to an end point of the voice activity, determining whether the audio signal includes a second pause corresponding to the end point of the voice activity.

34. The non-transitory computer readable medium of claim 32, the method further comprising: In response to determining that the pause in the voice activity does not correspond to an end point of the voice activity, the user is prompted to speak.

35. The non-transitory computer readable medium of claim 29, wherein: Determining whether the audio signal includes a pause in the voice activity includes determining whether the audio signal includes one or more language cues corresponding to an ending point of the voice activity.

36. The non-transitory computer readable medium of claim 35, wherein: The one or more language cues include characteristics of the user's prosody.

37. The non-transitory computer readable medium of claim 35, wherein: The one or more verbal cues include a closing phrase.

38. The non-transitory computer readable medium of claim 29, wherein: The non-verbal sensor data is indicative of a gaze of the user.

39. The non-transitory computer readable medium of claim 29, wherein: The non-verbal sensor data is indicative of a facial expression of the user.

40. The non-transitory computer readable medium of claim 29, wherein: The non-verbal sensor data is indicative of a heart rate of the user.

41. The non-transitory computer readable medium of claim 29, wherein: Determining whether the pause in the voice activity corresponds to an end point of the voice activity includes identifying one or more interstitial sounds.

42. The non-transitory computer readable medium of claim 29, wherein: Determining whether the pause in the speech activity corresponds to an end point of the speech activity includes evaluating non-verbal sensor data.

Images