Identification of input for speech recognition engine

Abstract

To provide a system and a method for processing an utterance signal for presenting to a speech recognition engine.SOLUTION: A method in a dynamic speech recognition (ASR) system detects 710 audio input from a microphone 602, determines whether a pause is performed or not while a user presents input utterance, indicates that present utterance of a user is ongoing when the pause is not detected 720 NO, continues detection of audio input by returning to the step 710, determines likelihood that the pause indicates completion of present utterance when the pause is detected 720 YES, evaluates 730 whether the detected pause indicates completion of present utterance, concludes 740 the utterance when the pause accompanies sufficient reliability to indicate completion of present utterance 732 YES, indicates 750 the utterance to an ASR engine 760, receives 770 a response from an NLU engine 765, and indicates 780 the response to the user.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 62 / 836,593, filed April 19, 2019, under 35 U.S.C. § 119(e), the contents of which are incorporated by reference in their entirety for all purposes.

[0002] FIELD OF THE DISCLOSURE This disclosure relates generally to systems and methods for processing speech signals, and more particularly, to systems and methods for processing speech signals for presentation to a speech recognition engine. [Background technology]

[0003] A system for speech recognition is typically tasked with receiving audio input representing human speech via one or more microphones, processing the audio input, and determining words, logical structures, or other output that correspond to the audio input. For example, an automatic speech recognition (ASR) system may generate text output based on the human speech that corresponds to the audio input signal, and a natural language processing (NLP) tool may generate logical structures or computer data that correspond to the meaning of the human speech. While some ASR systems can operate on large collections of pre-recorded speech (as an example, a system tasked with creating a written transcript of speech recorded by a microphone the previous day), some ASR systems must respond to speech input provided in real time. Real-time speech processing presents a unique set of challenges to ASR systems. For example, ASR systems typically process speech as a series of individual words or phrases ("utterances") that convey meaning, rather than as a monolithic block of input.

[0004] Identifying when an utterance begins and ends can be essential for an ASR system to accurately process a user's input utterance and provide a desired result. As an example, consider a real-time "voice assistant" ASR system that communicates with a weather service. The ASR system may receive speech input from a user asking about the weather (e.g., "What's the current weather?"), convert the speech input into a structured query (e.g., a query about data showing past, current, or predicted future weather at a specific date and time and at a specific location), submit the structured query to the weather service, receive query results from the service, and present the query results to the user. The user expects the ASR system to process its complete question (not individual fragments of the question) and provide an immediate and accurate response. The user further expects the ASR system to process naturally spoken commands that do not have to conform to a specific fixed format. In this exemplary system, it is the ASR's duty to identify the user's complete question in real time, process the question, and produce an accurate response in a timely manner, ideally as soon as the user has finished asking the question.

[0005] In this exemplary system, the accuracy of the response may depend on when the ASR system determines that the user's question (utterance) is complete. For example, a user may ask, "What's the weather tomorrow?" If the ASR system prematurely determines that the utterance is complete after "What's the weather," its corresponding query to the weather service will omit the modifier "tomorrow," and the resulting response will therefore be inaccurate (will not reflect the user's desired date / time). Conversely, if the ASR system takes a more conservative approach and waits for several seconds before processing the utterance to ensure that the entire utterance is complete, the user may consider the ASR system to be insufficiently responsive to its command. (Additionally, in some cases, such long wait periods may create inaccuracies by including non-relevant follow-up speech within the utterance.)

[0006] ASR systems suffer from this problem of determining when a speaker's utterance is complete, instantly and in real time. In some systems, a fixed timeout period is employed to determine the end of an utterance, and if no speech is received following a speech input for the duration of the timeout period (e.g., 750 ms), the speech input may be considered to be the end of the utterance. However, a fixed timeout period solution is imperfect, and in situations where, for example, the user pauses to articulate a question, the user is temporarily interrupted, or the user's speech is otherwise inarticulate (e.g., due to anxiety, speech impediment, environmental distraction, cognitive load, etc.), the timeout period may expire before the user's utterance is complete. Conversely, once the user's utterance is complete, the response is delayed for at least the duration of the timeout period (during which the ASR system ensures that no further input is received), and the user cannot provide additional speech input (e.g., belonging to a new utterance) for that duration. Such interactions can limit the usefulness of an ASR system and forcefully highlight that the user is communicating with a machine rather than another human being.

[0007] It would be desirable for an ASR system to take a more intuitive approach to determining when a user has finished providing an utterance. In normal face-to-face interactions, and to a lesser extent, telephone interactions, people use a variety of context cues to understand when another person has finished speaking. For example, when a speaker pauses, people evaluate the speaker's prosody, facial expression, eye gaze, mannerisms, gestures, and posture for indications of whether the speaker has finished speaking or simply paused in the middle of a single thought. The ASR system may use similar cues to identify where a user's utterance begins and ends. As described below, in some embodiments, the ASR system can identify such context cues from a microphone input, and in some embodiments, an ASR system that communicates with one or more sensors (e.g., as part of a wearable system) can gather additional speech cues about the speaker from the output of those sensors and use such cues to identify utterance boundaries without the problems associated with conventional solutions, as described above. Summary of the Invention [Means for solving the problem]

[0008] A method of presenting a signal to a speech recognition engine is disclosed. According to an embodiment of the method, an audio signal is received from a user. A portion of the audio signal is identified, the portion having a first time and a second time. A pause within the portion of the audio signal, i.e., the pause constituting the second time, is identified. It is determined whether the pause indicates a completion of an utterance of the audio signal. In accordance with a determination that the pause indicates a completion of an utterance, the portion of the audio signal is presented as an input to the speech recognition engine. In accordance with a determination that the pause does not indicate a completion of an utterance, the portion of the audio signal is not presented as an input to the speech recognition engine. The present invention provides, for example, the following: (Item 1) 1. A method comprising: receiving an audio signal via a microphone of a head-worn device, the audio signal comprising voice activity; determining whether the audio signal comprises a pause in the voice activity; in response to determining that the audio signal comprises a pause in the speech activity, determining whether the pause in the speech activity corresponds to an end of the speech activity; in response to determining that a pause in the voice activity corresponds to an end of the voice activity, presenting a response to a user based on the voice activity; A method comprising: (Item 2) 13. The method of claim 1, further comprising: in response to determining that a pause in the voice activity does not correspond to an end of the voice activity, not presenting a response to a user based on the voice activity. (Item 3) 2. The method of claim 1, further comprising: in response to determining that the audio signal does not include a pause in the voice activity, not determining whether a pause in the voice activity corresponds to an end of the voice activity. (Item 4) 2. The method of claim 1, wherein determining whether the audio signal comprises a pause in the voice activity comprises determining whether an amplitude of the audio signal is below a threshold for a predetermined period of time. (Item 5) 5. The method of claim 4, further comprising: in response to determining that a pause in the voice activity does not correspond to an end of the voice activity, determining whether the audio signal comprises a second pause that corresponds to an end of the voice activity. (Item 6) 5. The method of claim 4, further comprising prompting the user to speak in response to determining that a pause in the voice activity does not correspond to an end of the voice activity. (Item 7) 2. The method of claim 1, wherein determining whether the audio signal comprises a pause in the voice activity comprises determining whether the audio signal comprises one or more verbal cues corresponding to an end of the voice activity. (Item 8) 8. The method of claim 7, wherein the one or more verbal cues comprise prosodic characteristics of the user. (Item 9) 8. The method of claim 7, wherein the one or more verbal cues comprise a termination phrase. (Item 10) 2. The method of claim 1, wherein determining whether the audio signal comprises a pause in the voice activity includes evaluating non-verbal sensor data. (Item 11) Item 11. The method of item 10, wherein the non-verbal sensor data is indicative of a line of sight of the user. (Item 12) Item 11. The method of item 10, wherein the non-verbal sensor data is indicative of a facial expression of the user. (Item 13) Item 11. The method of item 10, wherein the non-oral sensor data is indicative of a heart rate of the user. (Item 14) 2. The method of claim 1, wherein determining whether a pause in the speech activity corresponds to an end of the speech activity includes identifying one or more intervening sounds. (Item 15) 2. The method of claim 1, wherein determining whether a pause in the speech activity corresponds to an end of the speech activity includes evaluating non-verbal sensor data. (Item 16) 1. A system comprising: A microphone in a head-mounted device; one or more processors, the one or more processors comprising: receiving an audio signal via a microphone of the head-worn device, the audio signal comprising voice activity; determining whether the audio signal comprises a pause in the voice activity; in response to determining that the audio signal comprises a pause in the speech activity, determining whether the pause in the speech activity corresponds to an end of the speech activity; in response to determining that a pause in the voice activity corresponds to an end of the voice activity, presenting a response to a user based on the voice activity; one or more processors configured to execute a method comprising: A system comprising: (Item 17) 17. The system of claim 16, further comprising: in response to determining that a pause in the voice activity does not correspond to an end of the voice activity, not presenting a response to a user based on the voice activity. (Item 18) 17. The system of claim 16, further comprising: in response to determining that the audio signal does not include a pause in the voice activity, not determining whether a pause in the voice activity corresponds to an end of the voice activity. (Item 19) 17. The system of claim 16, wherein determining whether the audio signal has a pause in the voice activity comprises determining whether an amplitude of the audio signal is below a threshold for a predetermined period of time. (Item 20) 20. The system of claim 19, further comprising: in response to determining that a pause in the voice activity does not correspond to an end of the voice activity, determining whether the audio signal comprises a second pause that corresponds to an end of the voice activity. (Item 21) 20. The system of claim 19, further comprising prompting the user to speak in response to determining that a pause in the voice activity does not correspond to an end of the voice activity. (Item 22) 17. The system of claim 16, wherein determining whether the audio signal comprises a pause in the voice activity comprises determining whether the audio signal comprises one or more verbal cues corresponding to an end of the voice activity. (Item 23) 23. The system of claim 22, wherein the one or more verbal cues comprise prosodic characteristics of the user. (Item 24) 23. The system of claim 22, wherein the one or more verbal cues comprise a termination phrase. (Item 25) 20. The system of claim 16, wherein determining whether the audio signal comprises a pause in the voice activity includes evaluating non-verbal sensor data. (Item 26) 26. The system of claim 25, wherein the non-verbal sensor data is indicative of the user's gaze. (Item 27) 26. The system of claim 25, wherein the non-verbal sensor data is indicative of a facial expression of the user. (Item 28) 26. The system of claim 25, wherein the non-oral sensor data is indicative of the user's heart rate. (Item 29) Item 17. The system of item 16, wherein determining whether a pause in the voice activity corresponds to an end of the voice activity includes identifying one or more intervening sounds. (Item 30) 20. The system of claim 16, wherein determining whether a pause in the voice activity corresponds to an end of the voice activity includes evaluating non-verbal sensor data. (Item 31) A non-transitory computer-readable medium having stored thereon one or more instructions that, when executed by one or more processors of an electronic device, cause the device to: receiving an audio signal via a microphone of a head-worn device, the audio signal comprising voice activity; determining whether the audio signal comprises a pause in the voice activity; in response to determining that the audio signal comprises a pause in the speech activity, determining whether the pause in the speech activity corresponds to an end of the speech activity; in response to determining that a pause in the voice activity corresponds to an end of the voice activity, presenting a response to a user based on the voice activity; A non-transitory computer readable medium for carrying out a method comprising: (Item 32) 32. The non-transitory computer-readable medium of claim 31, further comprising, in response to determining that a pause in the voice activity does not correspond to an end of the voice activity, not presenting a response to a user based on the voice activity. (Item 33) 32. The non-transitory computer-readable medium of claim 31, further comprising: in response to determining that the audio signal does not include a pause in the voice activity, not determining whether a pause in the voice activity corresponds to an end of the voice activity. (Item 34) 32. The non-transitory computer-readable medium of claim 31, wherein determining whether the audio signal comprises a pause in the voice activity comprises determining whether an amplitude of the audio signal is below a threshold for a predetermined period of time. (Item 35) 35. The non-transitory computer-readable medium of claim 34, further comprising: in response to determining that a pause in the voice activity does not correspond to an end of the voice activity, determining whether the audio signal comprises a second pause that corresponds to an end of the voice activity. (Item 36) 35. The non-transitory computer-readable medium of claim 34, further comprising prompting the user to speak in response to determining that a pause in the voice activity does not correspond to an end of the voice activity. (Item 37) Item 32. The non-transitory computer-readable medium of item 31, wherein determining whether the audio signal comprises a pause in the voice activity comprises determining whether the audio signal comprises one or more verbal cues corresponding to an end of the voice activity. (Item 38) Item 38. The non-transitory computer-readable medium of item 37, wherein the one or more verbal cues comprise prosodic characteristics of the user. (Item 39) Item 38. The non-transitory computer-readable medium of item 37, wherein the one or more verbal cues comprise a termination phrase. (Item 40) Item 32. The non-transitory computer-readable medium of item 31, wherein determining whether the audio signal comprises a pause in the voice activity includes evaluating non-oral sensor data. (Item 41) Item 41. The non-transitory computer-readable medium of item 40, wherein the non-verbal sensor data is indicative of a line of sight of the user. (Item 42) Item 41. The non-transitory computer-readable medium of item 40, wherein the non-verbal sensor data is indicative of a facial expression of the user. (Item 43) Item 41. The non-transitory computer-readable medium of item 40, wherein the non-oral sensor data is indicative of a heart rate of the user. (Item 44) Item 32. The non-transitory computer-readable medium of item 31, wherein determining whether a pause in the voice activity corresponds to an end of the voice activity includes identifying one or more intervening sounds. (Item 45) Item 32. The non-transitory computer-readable medium of item 31, wherein determining whether a pause in the voice activity corresponds to an end of the voice activity includes evaluating non-oral sensor data. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an exemplary wearable system according to an embodiment of the present disclosure.

[0010] [Diagram 2] FIG. 2 illustrates an example handheld controller that may be used with an example wearable system, according to an embodiment of the present disclosure.

[0011] [Diagram 3] FIG. 3 illustrates an example auxiliary unit that may be used with an example wearable system, according to an embodiment of the present disclosure.

[0012] [Figure 4] FIG. 4 illustrates an example functional block diagram for an example wearable system in accordance with one or more embodiments of the present disclosure.

[0013] [Diagram 5] FIG. 5 illustrates an example audio waveform for input to an example ASR system in accordance with one or more embodiments of the present disclosure.

[0014] [Figure 6] FIG. 6 illustrates a flow chart of an example of processing an acoustic speech signal using an ASR system in accordance with one or more embodiments of the present disclosure.

[0015] [Figure 7] FIG. 7 illustrates a flow chart of an example of processing an acoustic speech signal using an ASR system in accordance with one or more embodiments of the present disclosure.

[0016] [Figure 8A] 8A-8B illustrate a flow chart of an example embodiment for detecting pauses in an input speech signal, in accordance with one or more embodiments of the present disclosure. [Figure 8B] 8A-8B illustrate a flow chart of an example embodiment for detecting pauses in an input speech signal, in accordance with one or more embodiments of the present disclosure.

[0017] [Figure 9A] 9A-9B illustrate a flow chart of an example of determining whether an input utterance is complete, according to one or more embodiments of the present disclosure. [Figure 9B] 9A-9B illustrate a flow chart of an example of determining whether an input utterance is complete, according to one or more embodiments of the present disclosure.

[0018] [Figure 10] FIG. 10 illustrates a flowchart of an example of classifying input data and determining probabilities associated with the input data, in accordance with one or more embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] In the following description of the embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown, by way of illustration, specific embodiments which may be practiced. It is to be understood that other embodiments may be used and structural changes may be made without departing from the scope of the disclosed embodiments.

[0020] Exemplary Wearable System

[0021] 1 illustrates an exemplary wearable head device 100 configured to be worn on a user's head. The wearable head device 100 may be part of a broader wearable system that includes one or more components, such as a head device (e.g., wearable head device 100), a handheld controller (e.g., handheld controller 200 described below), and / or an auxiliary unit (e.g., auxiliary unit 300 described below). In some examples, the wearable head device 100 can be used for virtual reality, augmented reality, or mixed reality systems or applications. The wearable head device 100 includes one or more displays, such as displays 110A and 110B (which may comprise left and right transmissive displays and associated components for coupling light from the displays to the user's eyes, such as orthogonal pupil expansion (OPE) grating set 112A / 112B and exit pupil expansion (EPE) grating set 114A / 114B), left and right acoustic structures, such as speakers 120A and 120B (which may be mounted on temple arms 122A and 122B, respectively, and positioned adjacent the user's left and right ears), and an infrared sensor. The wearable head device 100 may include one or more sensors, such as a GPS unit, an accelerometer, a GPS unit, an inertial measurement unit (IMU, e.g., IMU 126), an acoustic sensor (e.g., microphone 150), a quadrature coil electromagnetic receiver (e.g., receiver 127 shown mounted on left temple arm 122A), left and right cameras oriented away from the user (e.g., depth (time of flight) cameras 130A and 130B), and left and right eye cameras oriented toward the user (e.g., for detecting the user's eye movements) (e.g., eye cameras 128A and 128B). However, the wearable head device 100 may incorporate any suitable display technology and any suitable number, type, or combination of sensors or other components without departing from the scope of the invention.In some embodiments, the wearable head device 100 may incorporate one or more microphones 150 configured to detect audio signals generated by the user's voice, and such microphones may be positioned within the wearable head device adjacent to the user's mouth. In some embodiments, the wearable head device 100 may incorporate networking features (e.g., Wi-Fi capabilities) for communicating with other devices and systems, including other wearable systems. The wearable head device 100 may further include components such as a battery, a processor, memory, a storage unit, or various input devices (e.g., buttons, touchpads), or may be coupled to a handheld controller (e.g., handheld controller 200) or auxiliary unit (e.g., auxiliary unit 300) that comprises one or more such components. In some embodiments, the sensors may be configured to output a set of coordinates of the head-mounted unit relative to the user's environment and may provide input to a processor to implement a simultaneous localization and mapping (SLAM) procedure and / or a visual odometry algorithm. In some embodiments, the wearable head device 100 may be coupled to a handheld controller 200 and / or an auxiliary unit 300, as described further below.

[0022] FIG. 2 illustrates an exemplary mobile handheld controller component 200 of an exemplary wearable system. In some examples, the handheld controller 200 may communicate wired or wirelessly with the wearable head device 100 and / or the auxiliary unit 300 described below. In some examples, the handheld controller 200 includes a handle portion 220 to be held by a user and one or more buttons 240 disposed along a top surface 210. In some examples, the handheld controller 200 may be configured for use as an optical tracking target, e.g., a sensor (e.g., a camera or other optical sensor) of the wearable head device 100 may be configured to detect the position and / or orientation of the handheld controller 200, which in turn may indicate the position and / or orientation of a user's hand holding the handheld controller 200. In some examples, the handheld controller 200 may include a processor, a memory, a storage unit, a display, or one or more input devices such as those described above. In some examples, the handheld controller 200 includes one or more sensors (e.g., any of the sensors or tracking components described above with respect to the wearable head device 100). In some examples, the sensors can detect a position or orientation of the handheld controller 200 relative to the wearable head device 100 or relative to another component of the wearable system. In some examples, the sensors may be located in the handle portion 220 of the handheld controller 200 and / or may be mechanically coupled to the handheld controller. The handheld controller 200 can be configured to provide one or more output signals corresponding, for example, to a press state of the button 240, or to a position, orientation, and / or movement of the handheld controller 200 (e.g., via an IMU). Such output signals may be used as inputs to a processor of the wearable head device 100, to the auxiliary unit 300, or to another component of the wearable system.In some embodiments, the handheld controller 200 may include one or more microphones to detect sounds (e.g., a user's speech, environmental sounds) and, in some cases, provide signals corresponding to the detected sounds to a processor (e.g., a processor of the wearable head device 100).

[0023] FIG. 3 illustrates an exemplary auxiliary unit 300 of an exemplary wearable system. In some examples, the auxiliary unit 300 may communicate wired or wirelessly with the wearable head device 100 and / or the handheld controller 200. The auxiliary unit 300 may include a battery to provide energy to operate one or more components of the wearable system, such as the wearable head device 100 and / or the handheld controller 200 (including a display, a sensor, an acoustic structure, a processor, a microphone, and / or other components of the wearable head device 100 or the handheld controller 200). In some examples, the auxiliary unit 300 may include a processor, a memory, a storage unit, a display, one or more input devices, and / or one or more sensors, such as those described above. In some examples, the auxiliary unit 300 includes a clip 310 for attaching the auxiliary unit to a user (e.g., a belt worn by the user). An advantage of using the auxiliary unit 300 to store one or more components of a wearable system is that doing so may allow a large or heavy component to be carried on the user's waist, chest, or back, which are relatively better suited for supporting large and heavy objects, rather than being mounted on the user's head (e.g., when stored in the wearable head device 100) or carried by the user's hands (e.g., when stored in the handheld controller 200). This may be particularly advantageous with respect to relatively heavy or bulky components, such as batteries.

[0024] FIG. 4 illustrates an example functional block diagram that may correspond to an example wearable system 400, such as may include the example wearable head device 100, handheld controller 200, and auxiliary unit 300 described above. In some examples, the wearable system 400 may be used for virtual reality, augmented reality, or mixed reality applications. As shown in FIG. 4, the wearable system 400 may include an example handheld controller 400B, referred to herein as a "totem" (and that may correspond to the handheld controller 200 described above), which may include a totem / headgear six degree of freedom (6DOF) totem subsystem 404A. The wearable system 400 may also include an example wearable head device 400A (which may correspond to the wearable headgear device 100 described above), which may include a totem / headgear 6DOF headgear subsystem 404B. In an embodiment, the 6DOF totem subsystem 404A and the 6DOF headgear subsystem 404B cooperate to determine six coordinates (e.g., offsets in three translation directions and rotations along three axes) of the handheld controller 400B relative to the wearable head device 400A. The six degrees of freedom may be expressed relative to the coordinate system of the wearable head device 400A. The three translation offsets may be expressed as X, Y, and Z offsets in such coordinate system, a translation matrix, or some other representation. The rotational degrees of freedom may be expressed as a sequence of yaw, pitch, and roll rotations, a vector, a rotation matrix, a quaternion, or some other representation. In some embodiments, one or more depth cameras 444 (and / or one or more non-depth cameras) and / or one or more optical targets (e.g., buttons 240 of the handheld controller 200 as described above or dedicated optical targets included in the handheld controller) included in the wearable head device 400A can be used for 6DOF tracking.In some embodiments, the handheld controller 400B can include a camera as described above, and the headgear 400A can include an optical target for optical tracking in conjunction with the camera. In some embodiments, the wearable head device 400A and the handheld controller 400B each include a set of three orthogonally oriented solenoids that are used to wirelessly transmit and receive three distinguishable signals. By measuring the relative magnitudes of the three distinguishable signals received in each of the coils used to receive, the 6DOF of the handheld controller 400B relative to the wearable head device 400A can be determined. In some embodiments, the 6DOF totem subsystem 404A can include an inertial measurement unit (IMU), which is useful for providing improved accuracy and / or more timely information regarding high speed movements of the handheld controller 400B.

[0025] In some implementations involving augmented or mixed reality applications, it may be desirable to transform coordinates from a local coordinate space (e.g., a coordinate space fixed relative to the wearable head device 400A) to an inertial coordinate space or to an environmental coordinate space. For example, such a transformation may be necessary for the display of the wearable head device 400A to present virtual objects in an expected position and orientation relative to the real environment (e.g., a virtual person sitting in a real chair facing forward, regardless of the position and orientation of the wearable head device 400A), rather than in a fixed position and orientation on the display (e.g., in the same position on the display of the wearable head device 400A). This can maintain the illusion that the virtual objects exist in the real environment (and do not appear unnaturally positioned in the real environment, e.g., as the wearable head device 400A shifts and rotates). In some examples, a compensation transformation between coordinate spaces can be determined by processing images from the depth camera 444 (e.g., using simultaneous localization and mapping (SLAM) and / or visual odometry procedures) to determine a transformation of the wearable head device 400A relative to an inertial or environmental coordinate system. In the example shown in FIG. 4, the depth camera 444 can be coupled to the SLAM / visual odometry block 406 and can provide images to the block 406. The SLAM / visual odometry block 406 implementation can include a processor configured to process the images and then determine a position and orientation of the user's head, which can be used to identify a transformation between the head coordinate space and the real coordinate space. Similarly, in some examples, an additional source of information regarding the user's head pose and location is obtained from the IMU 409 of the wearable head device 400A. Information from the IMU 409 can be integrated with information from the SLAM / visual odometry block 406 to provide improved accuracy and / or more timely information regarding fast adjustments of the user's head pose and position.

[0026] In some examples, the depth camera 444 can provide 3D images to a hand gesture tracker 411, which can be implemented within a processor of the wearable head device 400A. The hand gesture tracker 411 can identify the user's hand gestures, for example, by matching the 3D images received from the depth camera 444 to stored patterns representing hand gestures. Other suitable techniques for identifying the user's hand gestures will also be apparent.

[0027] In some embodiments, one or more processors 416 may be configured to receive data from the headgear subsystem 404B, the IMU 409, the SLAM / visual odometry block 406, the depth camera 444, the microphone 450, and / or the hand gesture tracker 411. The processor 416 may also send and receive control signals to the 6DOF totem system 404A. The processor 416 may be wirelessly coupled to the 6DOF totem system 404A, such as in embodiments where the handheld controller 400B is not tethered. The processor 416 may further communicate with additional components, such as an audiovisual content memory 418, a graphical processing unit (GPU) 420, and / or a digital signal processor (DSP) audio spatializer 422. The DSP audio spatializer 422 may be coupled to a head-related transfer function (HRTF) memory 425. The GPU 420 may include a left channel output coupled to a left source of imagewise modulated light 424 and a right channel output coupled to a right source of imagewise modulated light 426. The GPU 420 may output stereoscopic image data to the sources of imagewise modulated light 424, 426. The DSP audio spatializer 422 may output audio to the left speaker 412 and / or the right speaker 414. The DSP audio spatializer 422 may receive an input from the processor 419 indicating a direction vector from the user to a virtual sound source (which may be moved by the user, e.g., via the handheld controller 400B). Based on the direction vector, the DSP audio spatializer 422 may determine a corresponding HRTF (e.g., by accessing an HRTF or by interpolating multiple HRTFs). The DSP audio spatializer 422 may then apply the determined HRTF to an audio signal, such as an audio signal corresponding to a virtual sound generated by a virtual object.This can improve the believability and realism of virtual sounds by incorporating the user's relative position and orientation to the virtual sounds in the mixed reality environment, i.e., by presenting virtual sounds that match the user's expectations of what they would hear if the virtual sounds were real sounds in a real environment.

[0028] 4, one or more of the processor 416, GPU 420, DSP audio spatializer 422, HRTF memory 425, and audio / visual content memory 418 may be included within an auxiliary unit 400C (which may correspond to auxiliary unit 300 described above). The auxiliary unit 400C may include a battery 427 to power its components and / or provide power to the wearable head device 400A and / or the handheld controller 400B. Including such components within an auxiliary unit, which may be mounted on the user's waist, can limit the size and weight of the wearable head device 400A, which in turn can reduce fatigue in the user's head and neck.

[0029] 4 presents elements corresponding to various components of the exemplary wearable system 400, various other suitable arrangements of these components will be apparent to those skilled in the art. For example, elements presented in FIG. 4 as being associated with the auxiliary unit 400C may instead be associated with the wearable head device 400A or the handheld controller 400B. Furthermore, some wearable systems may dispense with the handheld controller 400B or the auxiliary unit 400C entirely. Such variations and modifications are to be understood as falling within the scope of the disclosed embodiments.

[0030] Speech Recognition System

[0031] A speech recognition system generally comprises a speech recognition engine that may receive an input audio signal corresponding to human speech (a source signal), process and analyze the input audio signal, and produce an output corresponding to the human speech as a result of the analysis. In the case of an automatic speech recognition (ASR) system, for example, the output of the speech recognition engine may be a text transcript of the human speech. In the case of a natural language processing system, the output may be one or more commands or instructions indicated by the human speech, or a semantically meaningful representation (e.g., a logical representation or data structure) of the human speech. Other types of speech recognition systems, including those that do not necessarily "recognize" speech (e.g., automatic translation systems), are also contemplated and are within the scope of the present disclosure. Additionally, as used herein, a speech recognition engine may include one or more of an automatic speech recognition engine, a natural language understanding engine, and other suitable components.

[0032] ASR systems are found in a wide variety of products and applications, namely, traditional telephone systems, automated voice messaging systems, voice assistants (including stand-alone and smartphone-based voice assistants), vehicles and aircraft, desktop and document processing software, data entry, home appliances, medical devices, language translation software, closed captioning systems, and others. An advantage of ASR systems is that they can enable users to provide input to computer systems using naturally spoken language as presented to a microphone instead of traditional computer input devices such as keyboards or touch panels. Thus, speech recognition systems can be particularly useful in environments where traditional input devices (e.g., keyboards) may be unavailable or impractical. Furthermore, by allowing users to provide intuitive voice-based input, speech recognition engines can enhance the sense of immersion. Thus, ASR may be a natural fit for wearable systems, particularly for virtual reality, augmented reality, and / or mixed reality applications of wearable systems where user immersion is a primary goal and it may be desirable to limit the use of traditional computer input devices whose presence may detract from the sense of immersion.

[0033] Identifying input utterance boundaries

[0034] The effectiveness of an ASR system may be limited by its ability to quickly present accurate input data to a speech recognition engine. Presenting accurate input data may require correctly identifying when individual sequences of input begin and end. Some ASR systems suffer from quickly and in real time determining when a speaker's utterance is complete. The present disclosure is directed to systems and methods for improving the accuracy of a speech processing system by accurately identifying the end points of an utterance presented to the speech processing system as input. Rapidly and accurately determining where an utterance ends enables the speech processing system to quickly deliver correct results in real time, i.e., in response to a live stream of input audio where the entire input audio signal cannot be known in advance.

[0035] 5 illustrates an example audio waveform 500 as may be detected by one or more microphones and presented as input to an ASR system. Waveform 500 represents an example utterance "What's the weather tomorrow in Moscow?" spoken by a user, with the intent that an ASR system receiving the utterance as input would query a weather service and respond with tomorrow's weather forecast for Moscow. The speed and accuracy of the ASR system's response will depend on where the ASR system determines to be the end point of the user's utterance. Example waveform 500 begins at an initial time t 0 The ASR system detects a time t 1 If the input utterance (i.e., time t 0 and time t 1, and the ASR system's response to this utterance would not match the user's expectations because this input utterance would lack the modifiers added by the user (i.e., "tomorrow" and "in Moscow"). For example, the system may return the current weather (not tomorrow's weather) in the user's current location (not Moscow). Similarly, if the ASR system determines that the utterance falls at time t, where the utterance falls after the phrase "What's the weather tomorrow" and before the phrase "in Moscow", the ASR system may determine that the utterance falls after the phrase "What's the weather tomorrow" and before the phrase "in Moscow". 2 If it is decided to end at the input utterance (i.e., at time t 0 and time t 2 The input utterance that falls between time t and t is determined to be "What's the weather tomorrow" and the resulting response would again not match the user's expectation (tomorrow's weather in Moscow). An ideal ASR system would correctly identify the input utterance as "What's the weather in Moscow" at time t, where the end of the utterance falls immediately after the completion of the entire input query. 3 It may be determined that the

[0036] An ideal ASR system would also not include subsequent portions of the input signal that do not belong to the input utterance. For example, if an ASR system were to detect an utterance at time t 4 If it is decided to end with t, the input utterance will contain all of the correct input utterance (i.e., "What's the weather in Moscow"), but will also contain irrelevant information (t 3 and 4 This extraneous information may introduce errors into the input utterance and may also delay the response of the ASR system (i.e., at least t 3 and 4 and the interval of signal processing time between them), would result in a lack of responsiveness perceived by the user.

[0037] Some ASR systems may incorrectly identify the end points of an input utterance. For example, when presented with the example waveform 500 as an input, some ASR systems may 3 Instead, 1 , t 2 , or t 4 may incorrectly identify the end of an utterance as

[0038] FIG. 6 illustrates an example process 600 that may be performed by an ASR system. The example process 600 uses a timeout interval to determine when an input utterance is complete, i.e., when no input utterance has been received for an amount of time that exceeds the timeout interval, the utterance is considered complete and presented to the ASR engine for processing. As shown in the figure, the input utterance is detected in real time from one or more microphones 602 in stage 610. In stage 620, the process 600 determines whether the input utterance is still being received. If so, the input utterance is considered ongoing and the process 600 returns to stage 610 to continue detecting the input utterance. If not, the process 600 queries whether the timeout interval 632 has elapsed. If not, the process 600 returns to stage 610 and, if so, the utterance is considered complete (stage 640). At stage 650, process 600 then presents the utterance (or some representation of the utterance) to ASR engine 660 for processing. ASR engine 660 can generate a speech / text representation of the input utterance. Natural language understanding (NLU) engine 665 can perform additional processing based on the representation. For example, NLU engine 665 can generate a semantic representation of the speech / text representation output from ASR engine 660, determine that the input utterance represents a request for a weather forecast, query a weather forecast service using the structured query generated from the input utterance, and receive a response from the weather forecast service. In some embodiments, process 600 may stream the utterance (or some representation of the utterance) for processing (e.g., by ASR engine 660). In some embodiments, ASR engine 660 may return or output an incremented result. In some embodiments, NLU engine 665 may receive an output from ASR engine 660 for processing after the utterance is completed. At stage 670, the process 600 receives the response from the NLU engine 665, and at stage 680, the response can be presented to the user (e.g., via a text-to-speech engine coupled to an output speaker).

[0039] The above process 600 may be prone to error because by concluding the input utterance using a simple timeout interval at stages 630 and 640, the process 600 may prematurely conclude the utterance before the user has completed the utterance being spoken. With reference to the waveform 500 described above, this may be 3 Instead, time t 1 or t 2 This may occur when a user inadvertently inserts a non-speech gap between two words of a single utterance (e.g., the pause between "weather" and "tomorrow" or between "tomorrow" and "in Moscow" in example waveform 500). If these gaps exceed the length of timeout interval 632, process 600 may prematurely determine that the input utterance is complete even if the user has not yet completed the utterance. (This situation may be particularly common with complex input queries, where the user may need additional time to articulate their question, or in users with speech impediments or who may suffer anxiety when interacting with a microphone or ASR system.)

[0040] This problem may not be completely solvable by simply increasing the length of the timeout interval 632 because there is a trade-off between the duration of this interval and the perceived responsiveness of the ASR system. That is, even if the timeout interval 632 could be increased to exceed any possible inter-utterance input gap, i.e., to prevent the process 600 from prematurely interrupting the input utterance, the ASR system would wait the duration of the extended timeout interval before deciding that the utterance is complete. This delay may be frustrating to users, who may perceive the delay as unresponsive, especially compared to face-to-face human interaction, where a listener quickly and intuitively knows when a speaker has finished speaking. In some embodiments, the delay may lead to crosstalk when a user perceives the ASR system as unresponsive and starts speaking again (e.g., re-repeats the initial input), which may result in a cascade of errors.

[0041] 7 illustrates an example process 700 (which may be performed, for example, by an ASR system) that may identify an input utterance more quickly and accurately than is possible with process 600. In process 700, as described below, a pause in the input utterance may be detected and then analyzed for contextual cues that indicate whether the pause is likely to represent the completion of the current utterance (in which case the utterance may be completed and presented to the ASR engine and the NLU engine) or whether the pause indicates that the current utterance is continuing (in which case the ASR system should continue detecting the current utterance).

[0042] In process 700, audio input submitted by a user is detected from one or more microphones 602 in stage 710. (In some embodiments, the audio input can be received as streaming data or as one or more data files instead of or in addition to the microphone output.) This audio input can be stored in an input buffer or other memory for access by process 700. In stage 720, process 700 can determine whether the user has paused while submitting the input utterance (e.g., based on the input buffer and / or sensor data, as described in further detail below). If a pause is not detected, indicating that the user's current utterance is continuing, the process may return to stage 710 and continue detecting audio input. If a pause is detected in stage 720, process 700 can determine in stage 730 the likelihood that the pause indicates the completion of the current utterance (rather than the continuation of the current utterance). For example, stage 720 can determine a confidence value as a numerical value representing the likelihood that the pause indicates that the current utterance is complete. This determination may be made based on the contents of the input buffer and / or sensor data, as described in more detail below.

[0043] At stage 732, process 700 may evaluate a determination of whether the pause detected at stage 730 indicates the completion of the current utterance. If the pause is determined with sufficient confidence (e.g., with a confidence level that exceeds a threshold) that it indicates the completion of the current utterance, process 700 may proceed to conclude the utterance (stage 740), present the utterance to an ASR engine (760) (stage 750), receive a response from an NLU engine (765) (stage 770), and present the response to a user (stage 780). These steps may correspond to stages 640, 650, ASR engine 660, NLU engine 665, 670, and 680, respectively, described above with respect to process 600.

[0044] If the process 700 determines that the pause is not likely to indicate that the current utterance is completed (e.g., the determined confidence level does not meet a threshold), the process 700 may take various actions in response thereto at stage 732. In some examples, the process 700 may adjust or reset parameters used to determine whether a pause as described herein with respect to stage 720 is detected (stage 734). For example, the process 700 may increase or reset a timeout interval used in stage 720 to detect a pause in the input utterance at stage 734. This may be beneficial if the process 700 determines that more time is needed to determine whether the user intends to complete the current utterance. In some examples, the process 700 may present the user with a prompt, such as a prompt for additional input (e.g., a visual and / or audible prompt asking the user to indicate whether they have finished speaking) (stage 736). This may be beneficial in situations where it is ambiguous whether the current utterance is completed, for example, if process 700 returns a confidence value that is below, but close to, the threshold. In some examples, in response to detecting that the pause does not indicate that the current utterance is completed, process 700 may combine the current utterance with the second utterance (stage 738). For example, the utterance preceding the pause may be concatenated with the second utterance following the pause for presentation of the combined utterance to a speech recognition engine (e.g., an ASR engine and / or an NLU engine). In some examples, process 700 may return to stage 710 and continue detecting the input utterance without taking any additional actions as described with respect to stages 734, 736, or 738. This behavior may be preferred if stage 730 returns a confidence value that is well below the threshold required to conclude that the current utterance is completed.

[0045] FIG. 8A illustrates a flow chart of an exemplary process for implementing stage 720 of process 700 as described above with respect to FIG. 7. In the figure, audio input data 810 (e.g., speech input signal stored in an input buffer) can be evaluated to determine the presence of a pause in the input. In stage 820, the process may determine whether the value of one or more properties of the input signal is above or below a threshold for a period of time that exceeds a timeout interval. In some examples, it may be determined whether the amplitude of the input signal is below a threshold amplitude level for a period of time that exceeds a timeout interval. If applicable, this may indicate a pause in the user's input speech (stage 860), as described above with respect to FIG. 6. In some embodiments, the length of the timeout interval may vary depending on the property. In some examples, a spectral analysis may identify the speech signal as distinct from other ambient or incident sounds, and it may be determined whether the output of the analysis is above or below one or more thresholds for a period of time that exceeds the timeout interval.

[0046] However, if in stage 820 it is determined that the timeout interval 822 has not elapsed, the process may still examine the audio input data 810 to determine whether the speech data includes verbal cues (other than relative silence) that indicate a pause in the input utterance (stage 830). These verbal cues may include characteristics of the user's prosodic (e.g., rhythm, intonation, tone of voice, volume), the presence of a subsequent word, the presence of a closing word or phrase (e.g., "thank you" when completing a verbal request), and the like. These verbal cues may indicate that the current utterance is complete, even if the timeout interval has not yet elapsed. In stage 840, the process may evaluate whether any such verbal cues are present and, if applicable, whether they indicate that the input utterance has paused (stage 860) or has not paused (stage 850). In some cases, stage 840 may make this determination by comparing the confidence level generated in stage 830 against a threshold value. By evaluating the presence of a verbal cue to indicate that the utterance is completed, even before the expiration of the timeout interval, the process can avoid the perception of unresponsiveness, as described above, that may result from waiting for the completion of the timeout interval before presenting the utterance for processing (e.g., by an ASR engine and / or an NLU engine).

[0047] FIG. 8B illustrates a flowchart of an example process for implementing stage 720 of process 700 in which sensor input data 844 is used instead of or in conjunction with audio input data 810 as described above. In some examples, as described above, the sensor data 844 may correspond to data from sensors as described above with respect to the example wearable head device 100 in FIG. 1. As described above, such wearable systems may include one or more sensors that may provide input about a user and / or the environment of the wearable system. For example, the wearable head device 100 may include a camera (e.g., camera 444 illustrated in FIG. 4) and output a visual signal corresponding to the environment. In some examples, the camera may be a forward-facing camera on a head-mounted unit that shows what is currently in front of the user of the wearable system. In some examples, the wearable head device 100 may include a LIDAR unit, a radar unit, and / or an acoustic sensor that may output a signal corresponding to the physical geometry of the user's environment (e.g., walls, physical objects). In some examples, the wearable head device 100 can include a GPS unit, which can indicate geographic coordinates corresponding to the current location of the wearable system. In some examples, the wearable head device 100 can include an accelerometer, a gyroscope, and / or an inertial measurement unit (IMU) to indicate the orientation of the wearable head device 100. In some examples, the wearable head device 100 can include an environmental sensor, such as a temperature or pressure sensor. In some examples, the wearable head device 100 can include a biometric sensor, such as an iris camera, a fingerprint sensor, an eye tracking sensor (e.g., an electro-oculography (EOG) sensor) for measuring the user's eye movements or eye gaze, or a sensor for measuring the user's vital signs.In embodiments where the wearable head device 100 includes a head-mounted unit, such orientation may correspond to the orientation of the user's head (and, for that matter, the direction of the user's mouth and the user's speech). Other suitable sensors may also be included and may provide sensor input data 844. Additionally, in some embodiments, sensors other than those of the wearable system may be utilized, as desired. For example, sensors associated with the microphones of the speech recognition system (e.g., GPS, IMU) may be used in conjunction with the sensors of the wearable system to determine the relative distance and orientation between the user and the speech recognition system.

[0048] At stage 842, the process 800 can examine the sensor input data 844 to determine whether the sensor data includes non-verbal cues indicative of a pause in the input utterance. These non-verbal cues can include, for example, characteristics of the user's eye gaze, head pose, gestures, vital signs (e.g., breathing pattern, heart rate), and facial expressions. These non-verbal cues can indicate that the current utterance is complete even if the timeout interval has not yet elapsed and in the absence of verbal cues as described above with respect to FIG. 8A. For example, a pause in the user's utterance can correspond to a change in the user's eye gaze target, a change in the user's head pose, a gesture performed by the user, a change in the user's vital signs (e.g., breathing pattern, heart rate), a change in the user's facial expression, a change in movement or rotation away from the microphone, a change in the user's posture or other physical characteristics indicated by the sensor input data 844, a change in orientation, and / or a rate of change of any one or more of the aforementioned characteristics. In stage 846, the process may evaluate whether any such non-verbal cues are present and, if applicable, whether they indicate that the input utterance has paused (stage 860) or has not paused (stage 850). In some cases, stage 846 may make this determination by comparing the confidence level generated in stage 842 against a threshold. As described above, by evaluating the presence of non-verbal cues to indicate that the utterance is complete, even before the expiration of the timeout interval, the process may avoid the perception of unresponsiveness, as described above, that may result from waiting for the completion of the timeout interval before presenting the utterance to the speech recognition engine. Although FIG. 8B shows the verbal and non-verbal cues analyzed in separate stages (i.e., stages 840 and 846), some embodiments may analyze the verbal and non-verbal cues in combination in a single stage.

[0049] 9A illustrates a flowchart of an example process for implementing stage 730 of process 700 as described above with respect to FIG. 7. At stage 730, process 700 determines, for a pause identified at stage 720 as described above, whether the pause is likely to correspond to the completion of the current utterance. In FIG. 9A, audio input data 910 (e.g., a speech signal stored in an input buffer, which may correspond to 810 described above) may be evaluated at stage 920 to determine the presence of an intervening sound in the audio input data. An intervening sound may be a word, phrase, syllable, or other voicing present in the input audio that may indicate that the current utterance is not yet complete (such as when a user is in the middle of a thought). For example, intervening sounds may include hesitations (e.g., "um," "uh"), extended syllables (e.g., the extended "to(ooo)" at the end of the phrase "I'm going to"), repetitions (e.g., "and, and, and..."), following fillers (e.g., "like," "I mean"), and / or other indications that the user is likely to provide additional input audio that belongs to the current utterance. Such intervening sounds may be specific to an individual user, a particular language, or a type of verbal input (e.g., a question, a declarative statement). As described below, various classifiers can be employed to identify intervening sounds.

[0050] At stage 930, the process may determine whether any such intervening sounds were detected at stage 920. If not, the process may conclude that the current utterance is completed (stage 970). If intervening sounds are present, the process may evaluate at stage 940 whether the intervening sounds indicate that the current utterance is continuing. For example, the presence of a hesitation sound may indicate that the user is in the process of articulating a complete utterance (e.g., as in "What's the weather...uh...tomorrow"). Similarly, extended syllables, repetitions, filler words, and other intervening sounds may indicate that the current utterance is not yet complete. In some implementations, stage 940 may generate a confidence value indicating the likelihood that an intervening sound is present and that the current utterance is completed.

[0051] In stage 950, if it is determined in stage 940 that the current utterance is ongoing, the process may conclude that the current utterance has not been completed (stage 960). As described above with respect to FIG. 7, this may result in the process performing various actions in response thereto. For example, process 700 may extend a timeout interval during which it detects a pause and prompts the user for additional input indicating whether the current utterance has been completed, and / or combines the current utterance with a second utterance, or performs no action at all (e.g., 822). In some embodiments, the action performed (if any) may depend on the confidence value generated in stage 940. For example, in response to a high confidence value that the utterance has not yet been completed, process 700 may simply return to stage 710 and continue detecting audio input without performing any further action, and in response to a medium confidence value (e.g., indicating uncertainty as to whether the current utterance has been completed), process 700 may explicitly prompt the user for additional input (stage 736). Similarly, if in stage 940 it is determined that the current utterance is completed, the process may indicate so (stage 970) and the process may proceed to submit the utterance to a speech recognition system as described above.

[0052] 9B illustrates a flowchart of an example process for implementing stage 730 of example process 700 in which sensor input data 942 is used instead of or in conjunction with audio input data 910 as described above. The sensor input data 942 may correspond to the sensor input data 844 described above. For example, the sensor input data 942 can be output by sensors as described above with respect to the example wearable head device 100 in FIG. 1. As described above, such sensors can include one or more cameras (e.g., RGB camera, depth camera), LIDAR units, radar units, acoustic sensors, GPS units, accelerometers, gyroscopes, IMUs, environmental sensors, biometric sensors (e.g., iris cameras, fingerprint sensors, eye tracking sensors, and / or sensors for measuring the user's vital signs). Other suitable sensors can also be included and can provide the sensor input data 942. Additionally, in some examples, sensors other than those of the wearable system can be utilized as desired. For example, as described above, sensors associated with the microphone of the speech recognition system (e.g., GPS, IMU) may be used in conjunction with sensors of the wearable system to determine the relative distance and orientation between the user and the speech recognition system.

[0053] With reference to FIG. 9B, the sensor input data 942 may be evaluated in stage 944 to determine whether the sensor data indicates that the current utterance is continuing or whether the current utterance has been completed. For example, completion (or non-completion) of an utterance may correspond to a change in the user's eye gaze target, a change in the user's head pose, a gesture performed by the user, a change in the user's vital signs (e.g., breathing pattern, heart rate), a change in the user's facial expression, a change in movement or rotation away from the microphone, a change in the user's posture or other physical characteristics indicated by the sensor input data 944, a change in orientation, and / or a rate of change of any one or more of the aforementioned characteristics. In some embodiments, stage 944 may generate a confidence level with which to indicate the likelihood that the current utterance has been completed. Based on the determination made in stage 944 (e.g., by comparing the confidence level to a threshold), the process in stage 950 may indicate that the utterance has either been completed (stage 970) or not been completed (stage 960).

[0054] In the process 700 described above, the input data (e.g., audio data, sensor data) may be evaluated in one or more stages for its significance to the way the data is presented to a speech recognition engine (e.g., an ASR engine and / or an NLU engine). For example, in stage 830 of process 720, as described above, the audio input data may be evaluated to determine whether the data includes verbal cues that the current utterance has completed. In stage 842, as described above, the sensor data may be evaluated for non-verbal cues (e.g., changes in facial expression) that the current utterance has completed. In stage 920, as described above, the audio input data may be evaluated to identify the presence of intervening sounds, and in stage 940, whether those intervening sounds indicate that the current utterance is ongoing may be evaluated. Also, in stage 944, as described above, the sensor input data may be evaluated to determine whether the sensor input data indicates that the current utterance is ongoing.

[0055] In some examples, the audio input data and / or sensor input data used as described above can be classified according to one or more parameters resulting in one or more classifiers that represent the data. These classifiers can be used (e.g., by example process 700) to evaluate the significance of the data (e.g., a probability associated with the data). FIG. 10 illustrates an example process 1000 for classifying input data 1010 and determining a probabilistic attention associated with the input data. As used herein, a probabilistic attention may correspond to a probability described above with respect to example process 700, a probability that the audio input data and / or sensor input data indicates a pause in the input utterance, a probability that the pause indicates the completion of the utterance, and / or a probability that the presence of intervening sounds indicates that the utterance is ongoing, or another suitable probability. With respect to FIG. 10, this determination can be performed using the audio input data 1016 alone or in combination with the sensor input data 1020. The step of determining a probability value for the input data 1010 may be referred to as a "classification" of the input data 1010, and a module or process for performing this determination (eg, 1074) may be referred to as a "classifier."

[0056] In the exemplary process shown in FIG. 10, the input data 1010 (e.g., audio input data 1016 and / or sensor input data 1020) may be used in conjunction with speech / sensor data 1029 (e.g., from a database) to determine one or more probabilities of interest for the input data 1010. In some embodiments, the audio input data 1016 and / or sensor input data 1020 may be parameterized / characterized according to one or more parameters to facilitate classification of speech segments based on the speech / sensor data 1029 at stage 1075. A Fourier transform of the input data 1010 may be performed to provide a spectral representation of the input data 1010 (e.g., a function of frequency indicative of the relative distribution of various frequency parameters within the audio input data 1016 and / or sensor input data 1020). For example, the process may identify the levels (or changes) of amplitude or component frequencies of the user's speech, position, eye gaze, and / or body movement. These values ​​may indicate a pause in the user's speech, the presence of an intervening sound, or the completion of the user's utterance as described above. In some examples, characteristics of the user, such as the user's age, gender, and / or native language, may be used as parameters to characterize the input data 1010. Other ways in which the input data 1010 may be parameterized with such parameters used to determine the probability of attention of the input data will be apparent to those skilled in the art.

[0057] In an embodiment, at step 1076, a probability value 1078 is determined with respect to the probability of interest of the input data 1010. In some embodiments, the probability value 1078 can be determined using the speech / sensor data 1029, such as when a database including the speech / sensor data 1029 identifies, for elements of the speech and / or sensor data in the database, whether those elements correspond to the input speech. In some embodiments, the audio input data 1016 can include a set of audio waveforms corresponding to speech segments, and for each waveform, can indicate whether the corresponding speech segment exhibits a pause or an intervening sound. In some embodiments, instead of or in addition to the audio waveforms, the audio input data 1016 can include audio parameters corresponding to the speech segments. The audio input data 1016 may be compared to speech segments of the speech / sensor data 1029, for example, by comparing audio waveforms of the audio input data 1016 with similar waveforms of the speech / sensor data 1029, or by comparing parameters of the audio input data 1016 (as may be characterized in stage 1075) with similar parameters of the speech / sensor data 1029. Based on such comparisons, a probability value 1078 may be determined with respect to the audio input data 1016.

[0058] Similar techniques can also be applied to the sensor input data 1020. For example, the sensor input data 1020 can include a sequence of raw sensor data and, with respect to the raw sensor data, can indicate whether the data indicates a pause or the completion or continuation of an utterance. Similarly, the sensor input data 1020 can include sensor input parameters that correspond to the sensor data. The sensor input data 1020 can be compared to elements of the speech / sensor data 1029 as described above with respect to the audio input data 1016.

[0059] Techniques for determining the probability 1078 based on the input data 1010 will be familiar to those skilled in the art. For example, in some embodiments, nearest neighbor interpolation may be used in stage 1076 to compare an element of the input data 1010 to similar data elements in an N-dimensional space (wherein the N-dimension may comprise, for example, audio parameters, audio waveform data, sensor parameters, or raw sensor data as described above) and determine a probability value 1078 based on the relative distance between the element of the input data 1010 and its neighbors in the N-dimensional space. As another example, a support vector machine may be used in stage 1076 to determine a basis for classifying an element of the input data 1010 as either indicative of a completed utterance or indicative of an incomplete utterance based on the speech / sensor database 1029 and classify the input data 1010 according to the basis (e.g., determine a probability value 1078 that the input data 1010 is indicative of the presence of a completed utterance, a pause, or an intervening sound). Other suitable techniques will become apparent for analyzing the input data 1010 and / or the speech / sensor data 1029, comparing the input data 1010 and the speech / sensor data 1029, and / or classifying the input data 1010 based on the speech / sensor data 1029 to determine a probability 1078. This disclosure is not limited to any particular technique or combination of techniques.

[0060] In some examples, machine learning techniques may be used, alone or in combination with other techniques described herein, to determine the probability values ​​1078. For example, a neural network may be trained on the speech / sensor data 1029 and applied to the input data 1010 to determine a probability value 1078 for the input data. As another example, a genetic algorithm may be used to determine a function for determining a probability value 1078 corresponding to the input data 1010 based on the speech / sensor data 1029. Other suitable machine learning techniques, as would be familiar to one of ordinary skill in the art, will also be apparent. This disclosure is not limited to any particular technique or combination of techniques.

[0061] In some examples, speech / sensor data 1029 can be generated by recording sets of speech data and / or sensor data for various users and identifying, for elements of that data, whether the user completed an utterance, paused their utterance, or provided an intervening sound. For example, a user interacting with a group of people can be observed using a speech recognition system present in the same room as the user's speech is recorded, and sensor data for the user (e.g., output by a wearable system worn by the user) can also be recorded. For each region of the recorded data, the observer can identify whether that region of the data corresponds to a pause, the provision of an intervening sound, or the completion of an utterance. This information can be obvious to the observer by observing the context in which the user is speaking, and it is generally easy and intuitive for a human (unlike a machine) to determine whether a user has completed an utterance based on the user's observation. This process can be repeated for multiple users until a sufficiently large and diverse set of speech / sensor data is generated.

[0062] With respect to the systems and methods described above, the elements of the systems and methods can be implemented by one or more computer processors (e.g., CPU or DSP) as appropriate. The present disclosure is not limited to any particular configuration of computer hardware, including computer processors, used to implement these elements. In some cases, multiple computer systems can be employed to implement the systems and methods described above. For example, a first computer processor (e.g., a processor of a wearable device coupled to a microphone) can be utilized to receive input microphone signals and perform initial processing of those signals (e.g., signal conditioning and / or segmentation as described above). A second (possibly more computationally powerful) processor can then be utilized to perform more computationally intensive processing, such as determining probability values ​​associated with speech segments of those signals. Another computing device, such as a cloud server, can host a speech recognition engine to which the input signals are ultimately provided. Other suitable configurations will be apparent and are within the scope of the present disclosure.

[0063] Although the disclosed embodiments have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. For example, elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. Such changes and modifications are to be understood as being included within the scope of the disclosed embodiments as defined by the appended claims.

Claims

1. A method, the method comprising: receiving an audio signal, the audio signal including voice activity; receiving non-verbal sensor data corresponding to a user of a head-worn device via one or more sensors of the head-worn device, the non-verbal sensor data indicative of one or more of a gaze of the user, a facial expression of the user, and a heart rate of the user; classifying said audio data into classes indicative of the presence of completed utterances or pauses or intervening sounds; determining an end point of the speech activity based on the class; presenting a response to the user based on the voice activity in response to determining the end point of the voice activity; and Including, Determining the end point of the voice activity includes: determining a probability of attention based on the class and further based on the non-verbal sensor data; determining whether the probability of interest exceeds a threshold; determining that a pause in the speech activity corresponds to the end of the speech activity in response to a determination that the probability of attention exceeds the threshold; determining that the pause in the speech activity does not correspond to the end of the speech activity in response to a determination that the probability of attention does not exceed the threshold; A method comprising:

2. The audio signal is received further via a first processor, the audio signal comprises a plurality of audio signal segments; The method comprises: for each of the plurality of audio signal segments, receiving via a second processor a respective audio signal segment of the plurality of audio signal segments; determining, via the second processor, whether the respective audio signal segment includes a pause in the speech activity; The method of claim 1 further comprising:

3. The method of claim 1, further comprising, in response to determining that the pause in the voice activity does not correspond to the end point of the voice activity, not presenting the response to the user based on the voice activity.

4. The method of claim 1, further comprising determining whether the audio signal includes the pause within the voice activity.

5. The method of claim 4, wherein determining whether the audio signal includes the pause in the voice activity includes determining whether the amplitude of the audio signal is below a second threshold for a predetermined period of time.

6. The method of claim 4, wherein whether the audio signal includes the pause in the voice activity is determined based on frequency components of the audio signal.

7. The method of claim 4, wherein determining whether the audio signal includes the pause in the voice activity includes determining whether the audio signal includes one or more verbal cues corresponding to the end of the voice activity.

8. The method of claim 7, wherein the one or more verbal cues include prosodic characteristics of the user.

9. The method of claim 7, wherein the one or more verbal cues include a termination phrase.

10. The method of claim 4, wherein the end point of the voice activity is determined according to a determination that the audio signal includes the pause within the voice activity.

11. The method of claim 1, wherein determining the end point of the voice activity includes identifying the intervening sounds.

12. The method of claim 1, further comprising, in response to determining that the pause in the voice activity does not correspond to the end of the voice activity, determining whether the audio signal includes a second pause that corresponds to an end of the voice activity, the second pause having a different duration compared to the pause in the voice activity.

13. The method of claim 1, further comprising prompting the user to speak in response to determining that the pause in the voice activity does not correspond to the end of the voice activity.

14. The method of claim 1, wherein the non-verbal sensor data further indicates the user's position and orientation.

15. A system comprising: one or more sensors in a head-worn device; One or more processors Equipped with The one or more processors: receiving an audio signal, the audio signal including voice activity; receiving non-verbal sensor data corresponding to a user of the head-worn device via the one or more sensors, the non-verbal sensor data indicative of one or more of a gaze of the user, a facial expression of the user, and a heart rate of the user; classifying said audio data into classes indicative of the presence of completed utterances or pauses or intervening sounds; determining an end point of the speech activity based on the class; presenting a response to the user based on the voice activity in response to determining the end point of the voice activity; and The method is configured to perform a method comprising: Determining the end point of the voice activity includes: determining a probability of attention based on the class and further based on the non-verbal sensor data; determining whether the probability of interest exceeds a threshold; determining that a pause in the speech activity corresponds to the end of the speech activity in response to a determination that the probability of attention exceeds the threshold; determining that the pause in the speech activity does not correspond to the end of the speech activity in response to a determination that the probability of attention does not exceed the threshold; Including, the system.

16. A non-transitory computer-readable medium storing one or more instructions, the one or more instructions, when executed by one or more processors, receiving an audio signal, the audio signal including voice activity; receiving non-verbal sensor data corresponding to a user of a head-worn device via one or more sensors of the head-worn device, the non-verbal sensor data indicative of one or more of a gaze of the user, a facial expression of the user, and a heart rate of the user; classifying said audio data into classes indicative of the presence of completed utterances or pauses or intervening sounds; determining an end point of the speech activity based on the class; presenting a response to the user based on the voice activity in response to determining the end point of the voice activity; and causing the one or more processors to execute a method including: Determining the end point of the voice activity includes: determining a probability of attention based on the class and further based on the non-verbal sensor data; determining whether the probability of interest exceeds a threshold; determining that a pause in the speech activity corresponds to the end of the speech activity in response to a determination that the probability of attention exceeds the threshold; determining that the pause in the speech activity does not correspond to the end of the speech activity in response to a determination that the probability of attention does not exceed the threshold; A non-transitory computer readable medium comprising:

Images