Device and method for context-aware control

Abstract

The voice interaction device includes multiple input and output components configured to facilitate interaction between the voice interaction device and a target user. The multiple input and output components may include: a microphone configured to sense sound and generate an audio input signal; a speaker configured to output an audio signal to the target user; and an input component configured to sense at least one inaudible interaction from the target user. The context controller monitors the multiple input and output components and determines the current usage context. The virtual assistant module facilitates voice communication between the voice interaction device and the target user, and configures one or more of the input and output components in response to the current usage context. The current usage context may include whisper detection, target user proximity, gaze direction tracking, and other usage contexts.

Description

[0001] Jochen Huber, Mohamed Sheik-Nainar and Anna Ostberg

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 62 / 653,469, filed on April 5, 2018, entitled “CONTEXT-AWARE CONTROL FOR SMARTDEVICES,” which is hereby incorporated by reference in its entirety. Technical Field

[0004] According to one or more embodiments, the present disclosure generally relates to voice interaction devices, and more particularly, for example, to an intelligent voice assistant system including multiple input and / or output modalities. Background Art

[0005] In recent years, voice interaction devices such as intelligent voice assistants have gained popularity. These devices usually receive audio through one or more microphones, and then process the received audio input to detect human voice, which may include one or more trigger words and voice commands for controlling the device. Feedback is provided to the user through the computer-generated voice output via an audio speaker. The device may also include other input / output (I / O) modalities, such as one or more physical buttons, lights, and touch screen displays. Voice interaction devices are usually optimized for voice command processing, which may not be optimal for specific environments or user interactions. Therefore, there is a continuous need for improved systems and methods for operating voice interaction devices in various user environments. Summary of the invention

[0006] In various embodiments of the present disclosure, an intelligent voice assistant device includes at least one speaker, at least one microphone and an audio input circuit, which is operable to receive an audio input signal and process the audio input signal to generate a target audio sample. The intelligent voice assistant device also includes a voice processor, which is operable to detect voice and execute associated voice commands. The input / output controller facilitates user input and output through additional I / O components (which may include lights, displays, touch screen displays, user input components (such as buttons, sliders or dials), image sensors and other input sensing components or output devices). The context controller monitors the input components and adaptively configures the input / output modality in response thereto. In various embodiments, the voice processor and the context controller enable an artificial intelligence-based virtual assistant that adapts the input and output modalities to the current usage context.

[0007] In one embodiment, the intelligent voice assistant device includes at least one image sensor for acquiring an image of a field of view, and a context controller that analyzes the acquired image to detect and track the target user and the usage context. The context controller can use object or person recognition image processing technology (such as facial recognition and motion detection and tracking as known in the art) to identify the target user in the image. In one embodiment, the context controller evaluates whether the target user is looking at the display direction of the smart device, and turns the display on (if the user's gaze is directed to the display) or turns the display off (if the user's gaze is not sufficiently directed to the display). If the user's gaze is directed to the display, the device can provide attention-aware output rendering, wherein the output fidelity is rendered based on the distance of the target user from the system to facilitate readability from various distances. In one embodiment, the distance between the intelligent voice assistant device and the target user can be estimated by analyzing an image input from a multi-sensor camera (e.g., a 3D camera).

[0008] In various embodiments, the context controller adaptively facilitates spatial relational input modality selection based on analysis of (one or more) received images. When the user is away from the smart device (e.g., the smart voice assistant device is out of the user's reach), the system can use voice as the primary input and output modality. When the user approaches the smart voice assistant device, the visual output through the display may become more desirable to the user. When the target user is close to the smart voice assistant device (e.g., within arm's reach), the interface can adaptively provide a touch-enabled interface, such as by rendering buttons on a touch screen display.

[0009] In various embodiments of the present disclosure, the intelligent voice assistant detects user voice interaction, determines at least one voice characteristic, and modulates the output volume of the intelligent voice assistant according to the determined voice characteristic. In one embodiment, the context controller detects the volume of the user's voice based on the characteristics of the voice, which may include the amplitude of the input audio signal, the vocal characteristics, the distance between the user and the intelligent voice assistant, ambient noise, and other characteristics. The context controller analyzes the characteristics of the input voice signal and modulates the output volume of the intelligent voice assistant to match the detected usage scenario. For example, if the user is whispering in a quiet room, the context controller can lower the output volume to respond with a corresponding quiet voice output. In another example, if the user is located across the room from the device, the context controller can increase the volume to project the voice output to the user across the room to hear clearly.

[0010] A more complete understanding of embodiments of the present disclosure, as well as a realization of additional advantages thereof, will be afforded to those skilled in the art by consideration of the following detailed description of one or more embodiments. Reference will be made to the appended sheets of drawings which will first be briefly described. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Various aspects of the present disclosure and its advantages may be better understood with reference to the following drawings and the following detailed description. It should be understood that the same reference numerals are used to identify the same elements illustrated in one or more of the drawings, wherein the displays in the drawings are for the purpose of illustrating embodiments of the present disclosure and not for the purpose of limiting the embodiments of the present disclosure. The components in the drawings are not necessarily to scale, but emphasis is placed on clearly illustrating the principles of the present disclosure.

[0012] Figure 1 An example voice interaction device in a noisy environment is illustrated in accordance with one or more embodiments.

[0013] Figure 2 Illustrated is an example attention-aware output rendering in accordance with one or more embodiments.

[0014] Figure 3 An example context-aware voice interaction device is illustrated in accordance with one or more embodiments.

[0015] Figure 4 is a flow diagram illustrating example operation of a context aware controller in accordance with one or more embodiments.

[0016] Figure 5 is a flow diagram illustrating example operations of a context-aware voice interaction device in accordance with one or more embodiments.

[0017] Figure 6 is a flow diagram illustrating example operations of a context-aware voice interaction device using speech characteristics to modulate an audio response in accordance with one or more embodiments. DETAILED DESCRIPTION

[0018] The system and method of the present disclosure provide improved techniques for operating, configuring and optimizing the performance of a voice interaction device. In various embodiments, the voice interaction device includes multiple input and output modalities, which may include audio input, audio output, image sensor, display and / or touch screen display. The virtual assistant module facilitates voice communication between the voice interaction device and the target user, and adapts the input and output modalities to the current usage scenario. The scenario module monitors the input data received from the input component and updates the scenario parameters defining the current usage scenario. The voice interaction device can be implemented as a voice interaction intelligent voice assistant with a display and supporting multiple input modes including sensors for receiving voice, touch and image input.

[0019] In one embodiment, the voice interaction device supports multiple input modalities, such as voice commands received by at least one audio sensor (e.g., a microphone), and touch input such as a sensor in a touch screen display. Certain input modalities may be more suitable than other input modalities for performing specific tasks in specific situations. For example, when the voice interaction device is out of reach of the user, voice input may be more effective and desirable than touch input. The present disclosure provides systems and methods that are operable to adapt the fidelity of available input modes and outputs to the detected usage context and user gaze.

[0020] In one method of operation, when a voice interaction device is outside the reach of a user, the preferred input modality is voice, and the visualization on the display screen is rendered large enough for the user to see from the detected distance. The context controller tracks the user using images captured from a camera, audio input, and / or other sensor inputs during operation. When the user moves closer to the voice interaction device, the visual output can be adapted to the distance between the user and the display accordingly. For example, when the user moves closer to the voice interaction device, more information can be displayed at a smaller size. When the user is within the reach of the device, a touch-enabled interface can be provided. When the user moves away from the voice interaction device, the displayed information can be adapted to readability, wherein less information is displayed at a larger size, and the touch-enabled interface can be disabled (e.g., to save power and / or to optimize the user experience). In one embodiment, when the user is close to the device, the available touch-enabled input controls can be replaced with voice prompts played through the speaker of the voice interaction device.

[0021] During operation, the voice interaction device typically enters a low power mode (e.g., sleep mode) while waiting for a keyword to be spoken to trigger further voice command processing. In one embodiment, the trigger and wake-up functionality can be further enhanced by using one or more built-in cameras, which can detect the presence of the user in the field of view (e.g., by comparing the image frame with the background image, facial recognition, motion detection, etc.). The camera can also be used to analyze whether the user's gaze is directed to the device (e.g., near the device and / or looking at the device), and adapt both the input and output modalities to specific situations. The captured image can be analyzed to detect the portrait, which can be analyzed to determine the distance from the user to the voice interaction device (e.g., by estimating the size of the object in the image, by determining the spatial position in the 3-dimensional image captured by the 3D camera) to intelligently adapt the input method and output fidelity to the corresponding situation. This detected user context information is provided to an artificial intelligence-based virtual assistant to provide a better conversation experience for the user, and to adapt the input method and render the visual output depending on the user's context and gaze.

[0022] In one embodiment, attention-aware output rendering is provided. Attention-aware devices can adapt the visual interface to the user's attention. As a first stage, the system can evaluate whether the user's gaze is directed at the display, and if not, turn the display to blank. If the user's gaze is directed at the device (so that the user appears to be paying attention to the device), the system can render the output fidelity based on the distance to the system, making it more comfortable to read from a distance and adding more details as the user moves closer to the device (such as through semantic zoom). Figure 2 Three examples of potential outputs are illustrated as the user moves closer to the display.

[0023] In another embodiment, the spatial relational input modality selection is provided by a context controller that is configured to monitor one or more input components to track the target user. The device can adaptively configure the input modality based on the user's proximity to the voice interaction device. When the user is close to the device, such as within arm's reach, the interface can provide touch-enabled interface elements (see Figure 2 , where an exit button is displayed). As the user moves further away from the device, these interface elements are removed for the benefit of clarity from a distance, and user interaction can be limited to voice as the primary input modality. In one embodiment, the device tracks multiple users and adapts the input modality in the context of a multi-user environment. For example, if any user moves close to the device, the touch interface can be enabled even if other input / output modalities are configured for one or more other users.

[0024] In various embodiments, a system and method for adaptively adjusting voice output is also provided. In one embodiment, a voice interaction device receives an audio input signal including a target audio signal, enhances the target audio signal, determines the audio characteristics associated with the target audio signal, and modulates the audio output according to the determined characteristics and other available context information. The context controller can, for example, determine the appropriate output volume level for the use context by analyzing multiple characteristics of the user's voice and the environment, the characteristics can include the voice volume in the target audio signal as measured at the device input, environmental noise (e.g., background noise level), the distance between the target user and the voice interaction device (e.g., via an estimated voice arrival time from the user, object tracking from an image sensor), voicing (e.g., whether the voice is a whisper, a neutral voice, a shout) and other voice characteristics. For example, if the target audio signal includes a whispered voice command spoken by a user in a quiet room near the voice interaction device, the context controller can reduce the device output volume to deliver a voice response output that approximates the input context. In another example, if the target audio signal includes a shouted voice command received from a target user in a noisy environment who is a distance away from the voice interaction device (e.g., across the room), the context controller can adjust the device output volume to deliver the voice response output at a higher level to project the voice response above the ambient noise and across the room to the target user.

[0025] The method disclosed herein can also be applied to dynamic content, such as video output. The device can track whether the user's gaze is directed to the display. For example, one or more image sensors can be used to track the user's position, including the distance from the voice interaction device and the user's gaze. The user's position and distance can be determined by analyzing the captured images received from the device camera (e.g., a 3D camera), and the user's gaze can be tracked, for example, by analyzing the eye direction and movement. When the user pays attention from a distance, the frame rate of the video stream and the resolution of the video stream can be reduced. When the user approaches the device, both rates can be increased. This operation is beneficial for devices that run on batteries (such as portable voice interaction devices, phones, and tablets).

[0026] Proximity, gaze tracking, modulated voice output, and other functionality disclosed herein may be implemented through system hardware such that a proximity module, gaze tracking module, voice processing module, and audio processing module may interact to detect user presence. The user input / output space may be controlled by hardware that provides I / O, context, and tracking functionality for voice, gaze, display, and touch. Facial recognition may also be used to look up the person's interaction history to customize the user experience (e.g., by using accents, specific dictionaries, etc. to make the conversation simpler).

[0027] Figure 1 An exemplary operating environment 100 for a voice interaction device according to various embodiments of the present disclosure is illustrated. The operating environment 100 includes a voice interaction device 105, a target audio source 110 (eg, a user), and one or more noise sources 135, 140, and 145. Figure 1 In the example illustrated in , the operating environment 100 is illustrated as the interior of a room 150, but it is contemplated that the operating environment may include other environments. The voice interaction device 105 may include one or more audio sensing components (e.g., microphones) 115a-15d and one or more audio output components (e.g., speakers) 120a-120b. In the illustrated embodiment, the voice interaction device 105 includes four microphones and two speakers, but other configurations may be implemented. The voice interaction device 105 also includes at least one user input / output component 130, such as a touch screen display and an image sensor 132, to provide (one or more) additional modes of user interaction with the voice interaction device 105.

[0028] The voice interaction device 105 is operable to sense sounds from the environment via audio sensing components 115a-115d and generate an audio input signal, which may include two or more audio input channels. The voice interaction device 105 may process the audio input signal using audio processing techniques to detect and enhance the audio signal received from the target audio source 110. For example, a spatial filter (e.g., a beamformer) may receive the audio input signal, identify the direction of the target signal, and use constructive interference and noise cancellation techniques to output an enhanced audio signal (also referred to as an enhanced target signal) that enhances the sound (e.g., speech) produced by the target audio source 110. The enhanced audio signal may then be transmitted to other components within the voice interaction device 105, such as a voice recognition engine or a voice command processor, or transmitted to an Internet phone (VoIP) application as an input signal during a VoIP call. In various embodiments, the voice interaction device 105 may operate as a communication device that facilitates VoIP communications or video conferencing across a network, and the voice interaction device may also receive audio and video signals from a remote device and output audio signals via audio output components 120a-120b.

[0029] The target audio source 110 can be any source that produces the target audio 125 that can be detected by the voice interaction device 105. In the illustrated example, the target audio 125 is defined as human voice, and the target audio source 110 is a human. In addition to the target audio source 110, the operating environment 100 may include one or more noise sources 135-145. In various embodiments, sounds that are not target audio are processed as noise. In the illustrated example, the noise sources 135-145 may include, for example, a speaker 135 playing music, a television 140 playing a television program, a movie, or a sporting event, and a background conversation between non-target speakers 145. It will be understood that other noise sources may exist in various operating environments.

[0030] The target audio and noise can reach the microphone 115a-115d of the voice interaction device 105 from different directions, and the position of each audio source can change over time. For example, the noise source 135-145 can generate noise at different positions in the room 150, and the target audio source (person) 110 can speak while walking around the room 150. In addition, the target audio and / or noise may be reflected from the fixture (e.g., wall) in the room 150. The voice interaction device 105 can estimate the position of the target audio source 110 based on the audio input signal received by the microphone 115a-115d, and process the audio input signal based on the estimated position to enhance the target audio and suppress noise. The information related to the direction of the target audio source 110 and the distance from the voice interaction device 105 can be provided to the context controller 134, which monitors the user position and activity and adaptively configures the operation of the voice interaction device 105 according to the current usage context.

[0031] In one embodiment, the voice interaction device 105 also includes one or more image sensors 132, such as an image sensor that can be implemented in a 3D camera. In operation, the image sensor 132 and the touch screen display 130 can provide video call functionality, video recording, and image capture functions. The image sensor 132 can also be used to identify the location of the target audio source 110, whether the gaze of the target source is directed to the voice interaction device 105, and other information about the environment 100 and the target audio source 110. In one embodiment, the context controller 134 is operable to monitor audio and visual context information and adaptively configure the voice interaction device 105 in response thereto.

[0032] In one embodiment, the target audio source 110 (also referred to herein as the target user) may initially be located away from the voice interaction device 105. The context controller 134 may use the audio and visual information received from the microphone and image sensor 132 to estimate the location of the target audio source 110. For example, when the user is away from the voice interaction device and out of the reach of the voice interaction device, the touch controls are ineffective, and detailed information on the touch screen display 130 may be unreadable. If the user is too far away from the voice interaction device to read the display, or if the user is not paying attention to the touch screen display 130 as determined by tracking the user's gaze, the context controller 134 may turn off the touch screen display 130. In this way, the voice interaction device 105 can save power, and the context controller 134 can adjust the output modality to enhance the user experience (e.g., by alternatively preparing and directing the output to the audio output components 120a-b).

[0033] In various embodiments, the context controller 134 may monitor the location and gaze of the target audio source 110 and modify the modality based on the current context. For example, if the user is located across the room, the touch screen display 130 may be used to display a reduced amount of information at a larger size that is visible across the room (such as Figure 2 As the user moves closer to the voice interaction device 105 (such as to position 110b), the user may be able to see the display better, thereby allowing more visual output to be displayed (such as Figure 2 As the user moves closer to and within reach of the voice interaction device 105 (such as to position 110c), the user may be able to read additional details displayed on the screen and get close enough to activate touch controls (such as Figure 2 as shown on the screen (c) of the

[0034] In another embodiment, the voice interaction device 105 is operable to monitor the noise level of the environment and adjust the output volume of the voice prompt in view of the environmental noise conditions. When the noise level is above a certain threshold, the context controller 134 can switch the output prompt to a visual representation on the touch screen display.

[0035] Figure 3 An exemplary device 300 according to various embodiments of the present disclosure is illustrated. In some embodiments, the device 300 may be implemented as Figure 1 The voice interaction device 105 of the present invention. The device 300 includes an audio sensor array 305, at least one speaker 310a-b, an audio signal processor 320 and a device module 350.

[0036] The audio sensor array 305 includes two or more sensors, each of which can be implemented as a transducer that converts an audio input in the form of a sound wave into an audio signal. In the illustrated environment, the audio sensor array 305 includes a plurality of microphones 305a-305n, each of which generates an audio input signal that is provided to an audio input circuit 322 of an audio signal processor 320. In one embodiment, the audio sensor array 305 generates a multi-channel audio signal, where each channel corresponds to an audio input signal from one of the microphones 305a-n.

[0037] The audio signal processor 320 includes an audio input circuit 322, a digital signal processor 324, and an audio output circuit 326. In various embodiments, the audio signal processor 320 can be implemented as an integrated circuit including an analog circuit, a digital circuit, and a digital signal processor 324, which is operable to execute program instructions stored in firmware. The audio input circuit 322 can, for example, include an interface to the audio sensor array 305, an anti-aliasing filter, an analog-to-digital converter circuit, an echo cancellation circuit, and other audio processing circuits and components as disclosed herein. The digital signal processor 324 is operable to process multi-channel digital audio signals to generate enhanced target audio signals, which are output to one or more of the device modules 350. In various embodiments, the digital signal processor 324 can be operable to perform echo cancellation, noise cancellation, target signal enhancement, post-filtering, and other audio signal processing functions. In some embodiments, the device module 350 is configured to enter a low power mode (e.g., sleep mode) during an inactive period, and the audio signal processor 320 is operable to wait for a trigger word and wake up one or more of the device modules 350 when a trigger word is detected.

[0038] The audio output circuit 326 processes the audio signals received from the digital signal processor 324 for output to at least one speaker, such as speakers 310a and 310b. In various embodiments, the audio output circuit 326 may include a digital-to-analog converter that converts one or more digital audio signals into analog audio signals and one or more amplifiers for driving the speakers 310a-310b.

[0039] The device 300 can be implemented as any voice interaction system or device, such as, for example, an intelligent voice assistant, a mobile phone, a tablet computer, a laptop computer, a desktop computer, a voice interaction application, or a car. The device module 350 may include various hardware and software components for facilitating the operation of the device 300. In the illustrated embodiment, the device module 350 includes a processor 352, a user interface component 354, a communication interface 356 for communicating with external devices and networks, such as a network 382 (e.g., the Internet, a cloud, a local area network, or a cellular network) and an external device 384 (e.g., a mobile device), and a memory 358. The device 300 may further include one or more applications 364, a voice processor 370, a context controller 376, and a virtual assistant module 380.

[0040] The processor 352 and the digital signal processor 324 may include one or more of a processor, a microprocessor, a single-core processor, a multi-core processor, a microcontroller, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)), a digital signal processing (DSP) device, or other logic device, which may be configured to perform various operations discussed herein for embodiments of the present disclosure by hard wiring, executing software instructions, or a combination of both. The device module 350 is configured to interface and communicate with the audio signal processor 320 and other device modules 350, such as through a bus or other electronic communication interface.

[0041] It should be understood that although the audio signal processor 320 and device module 350 are illustrated as comprising a combination of hardware components, circuits, and software, in some embodiments, at least some or all of the functionality of the hardware components and circuits operable to perform may be implemented as software modules executed by the processor 352 and / or the digital signal processor 324 in response to software instructions and / or configuration data stored in the memory 358 or firmware of the digital signal processor 324.

[0042] The memory 358 and other memory components disclosed herein may be implemented as one or more memory devices operable to store data and information, including audio data and program instructions. The memory 358 may include one or more memory devices of various types, including volatile and non-volatile memory devices, such as RAM (random access memory), ROM (read-only memory), EEPROM (electrically erasable read-only memory), flash memory, hard disk drives, and / or other types of memory.

[0043] The processor 352 may be operable to execute software instructions stored in the memory 358 and other memory components. In various embodiments, the speech processor 370 may be implemented in hardware, software instructions executed by the processor 352, or a combination of hardware and software. The speech processor 370 includes a speech recognition engine 372 that is operable to process target audio signals received from the audio signal processor 320 to recognize speech and detect trigger words and / or voice commands.

[0044] The user interface components 354 may include a display 390, a user input component 392 (such as a touch panel display, a keypad, one or more buttons, dials, or knobs), and / or other input / output components operable to enable a user to directly interact with the device 300. The user interface components 354 may also include one or more sensors, such as one or more image sensors (e.g., cameras) for capturing images and video. In some embodiments, the user interface components 354 include at least one input component for receiving inaudible target user interactions, and / or at least one output component for generating inaudible responses to the target user.

[0045] The device module 350 further includes a context controller 376, which can be implemented in hardware, software instructions executed by the processor 352, or a combination of hardware and software. The context controller 376 includes a context monitor 378 for receiving, processing, and monitoring the current environment and usage context of the device 300, which may include monitoring one or more inaudible target user interactions. For example, the context controller 376 can be configured to detect the location and movement of the target user and determine whether the target user is paying attention to the device 300 by tracking the user's gaze (e.g., by determining the gaze direction from a captured image of the target user).

[0046] In one embodiment, context controller 376 is operable to process video images from a 3D image sensor (sensor 394), detect the presence of a target human within the field of view, track the movement of the target human, and determine how far the target human is from device 300 (e.g., by analyzing the relative size or position of the human detected in the image). In one embodiment, context monitor 378 is operable to analyze images received from image sensor 394 to determine whether the target human is paying attention to device 300, such as by tracking eye movement. Context information can be stored in memory as context parameters for access by other modules.

[0047] The virtual assistant module 380 is operable to provide a conversational experience to the target user, adapt the input / output modality, and facilitate the execution of user commands (e.g., voice commands received from the target user). In various embodiments, the virtual assistant module includes an artificial intelligence-based system (e.g., an artificial intelligence module) that includes a trained neural network that receives voice commands from the voice processor 370 and receives context parameters from the context controller 376, and adapts the user experience to the current usage context as disclosed herein. The application 364 includes instructions and associated data that can be executed by the processor 352, and can include devices and user applications that can be launched by the user and / or the virtual assistant module 380. In one embodiment, the application 364 includes a VoIP application that facilitates voice communication with one or more external devices (such as an external device 384 or a server 386). In various embodiments, the virtual assistant module 380 is operable to adaptively update the input and output modalities based on context information from the context controller 376.

[0048] The communication interface 356 facilitates communication between the device 300 and external devices. For example, the communication interface 356 may enable a Wi-Fi (e.g., 802.11) or Bluetooth connection between the device 300 and one or more local devices, such as the external device 384, or a wireless router such as one that provides network access to the server 386 via the network 382. In various embodiments, the communication interface 356 may include other wired and wireless communication components to facilitate direct or indirect communication between the device 300 and one or more other devices.

[0049] Reference Figure 4, the operation of the context-aware voice interaction device according to an embodiment of the present disclosure will now be described. The voice interaction device initially operates according to a default input / output configuration (such as configured by the user) (step 400). During operation, the context controller detects and monitors audio and sensor data related to target user activities (step 402). The audio data may indicate the presence and position of the target user and the noise level in the environment. Other sensor data may include image sensors and image processing functionality for detecting and tracking objects in the field of view of the image sensor. For example, the distance and position of the target user relative to the device can be used to set the output configuration of the display screen. For example, the position of the target user can be determined by image analysis of the size and position of the detected target user in the field of view of the image sensor. The position can also be estimated, for example, by analyzing the direction of the target audio received by the device microphone array. The estimated distance and position of the target user can be stored as context parameters defining the current context. In various embodiments, contextual parameters may include numerical data (e.g., the number of target users detected, the distance from the device in meters, the direction of the target users relative to the display in degrees)), state information (e.g., whether the target users are within reach of the device, whether the display can be read from the target users' position, whether the target users' gaze is directed toward the device), and other contextual information as required by the system.

[0050] Analyze the situational parameters in view of the user preferences to determine the best input / output mode (step 404), and update the device input / output configuration (step 406) according to the current use situation. For example, the target user may be in a first position that is too far away or in a certain viewing angle, so that the information displayed on the display device cannot be viewed. The display device can be turned off (for example, to save power), and other output modes can be used for user output (for example, audio information and prompts are transmitted through the device speaker). The target user may move to a second position outside the reach of the device, but close enough to view the basic information provided on the display device. When the target user approaches the device, the situational parameters will be updated to display the new target user position, and the output configuration can be adaptively configured to provide the target user with more detailed information that can be read from a new closer distance.

[0051] In various embodiments, when the device is out of reach of the target user, the device input and output can be configured for voice command processing with audio prompts. The target user can move to a third location within the reach of the device (e.g., by comparing the current distance to a reach threshold), thereby creating conditions for touch controls that may be superior to voice processing. The activated application adaptively processes the input and output according to the changed device input / output configuration (step 408), and the device continues to operate according to the new contextual configuration (e.g., to guide input and output interactions through the desired modality).

[0052] Reference Figure 5 , example operations of a context-aware voice interaction device according to one or more embodiments will now be described. In step 500, a context controller receives a captured image stream from one or more image sensors (such as a 3D camera of a voice interaction device). The context controller analyzes the images to identify and track humans within the captured images (step 502). Next, the physical position of each tracked human relative to the device is estimated (step 504). Based on the physical position and the estimated distance from the device to the human, context areas can be assigned (step 506), such as areas that identify whether the tracked person is within the reach of the voice interaction device, areas where the device is outside the reach and the display is readable, far areas where the device is outside the reach and the readability of the display is limited, and areas indicating that the display cannot be viewed by the tracked person (e.g., the person is too far away from the display or at a location where there is no line of sight to the display). In step 508, the image is further analyzed to determine whether the tracked person is paying attention to the display (e.g., by determining gaze points and eye movements). Parameters are stored in a context parameter memory 510, which can be used by other system components (such as Figure 3 Virtual assistant module 380) access.

[0053] Reference Figure 6 , example operations of a context-aware voice interaction device that uses voice characteristics to modulate voice response output according to one or more embodiments will now be described. In various embodiments, a voice interaction device receives an audio input signal including a target audio signal and background noise (step 600). In step 602, voice feature characteristics of the target audio signal are determined. The voice feature characteristics may include amplitude and voicing determination of speech in the received target audio signal. In one embodiment, the voicing determination includes determining from a voice processor whether the target speaker is whispering, using a soft voice, using a neutral voice, using a loud voice, or shouting.

[0054] In step 604, the audio input signal is evaluated to determine an estimate of the distance between the user and the voice interaction device. In one embodiment, the volume of the received target voice, the time difference of the target voice arriving at two or more microphones of the voice interaction device, and the vocalization information (e.g., whispering may indicate that the user is very close to the device, while shouting may indicate that the speaker is farther away) are used to estimate the distance. In various embodiments, other audio and environmental information may be used, including room characteristics and reverberation signal characteristics. In some embodiments, the voice interaction device may include additional sensors that provide inputs that can be used in distance determination (step 608). For example, as previously discussed herein, an image sensor may be used to capture an image of the field of view and track a person in the captured image (e.g., using object detection / motion detection), and map the position of the person relative to the voice interaction device. In step 610, background noise characteristics are determined from (one or more) background noise signals, which may include determining an average noise level.

[0055] In step 612, the output mode of the voice interaction device is adaptively modified by the context controller using voice feature characteristics (from step 602), distance information (from step 604) and background noise characteristics (from step 610). For example, in one embodiment, the output audio level is adjusted so that the volume of the voice output received by the target user is approximately the same level as the volume of the target user speaking. In other embodiments, the output audio level can be adjusted to provide a clear voice response to the user, and the voice response can be independent of the speaker's voice level (for example, based on ambient noise and the distance from the device). In another embodiment, the context controller uses additional voice characteristics, such as the speed, pitch, pronunciation, tension of the target user speaking, to further determine the emotional state of the speaker and adjust the output mode accordingly. In step 614, the context controller reconfigures the audio output settings according to the voice feature characteristics, the target user distance and the background noise. In step 616, the voice interaction device processes the voice output according to the reconfigured output mode.

[0056] In operation, if the received target audio signal includes a whispered voice command received in a quiet room (e.g., a user's bedroom with another person sleeping nearby at night), the context controller can adjust the device output volume to deliver a voice response (a whispered response) at a lower level that matches the input context. In this way, the present disclosure provides advantages over conventional methods. If the voice interaction device is to respond in standard voice to confirm a command or provide feedback, the volume may be surprising and disruptive to the usage context. Although the user can physically adjust the volume by touching the device (e.g., by pressing one or more buttons), this will require additional effort on the part of the user, but the user may not be able to reach the device and / or the user may not realize that the current volume setting is too high until the voice response is to be delivered.

[0057] In another scenario, the user may be in a noisy environment, in which it is difficult for the user to hear the response of the device. If the target audio signal includes a voice command shouted by the device in a noisy environment from a target user (the target user is at a distance from the voice interaction device (e.g., across the room)), the context controller can adjust the device output volume to deliver a voice response at a higher than normal level (e.g., higher than the configured conversation volume level) to project the voice output response onto the ambient noise and across the room to the target user. If a user in a noisy environment is detected approaching the device and speaking in a normal voice or a quiet voice, the context controller can adaptively adjust the output to match the user's tone, thereby outputting a voice response in a softer voice, which will be discernible to the user near the device. If the environment changes (e.g., after a noisy event has ended), the voice interaction device of the present disclosure will automatically adjust the output volume to a setting suitable for the new use scenario, which can be determined using the target user's voice volume, the ambient noise level, the distance between the target user and the device, and other context information.

[0058] In one or more embodiments, the voice interaction device is configured to distinguish between whispered speech and non-whispered speech received from the target user, and adaptively adjust the voice output to a lower volume setting when the whispered speech is detected. For example, the voice interaction device can classify the target audio signal as including "whispered" speech by detecting the low amplitude of the target audio signal, "whispering" pronunciation characteristics (e.g., speech duration, energy distribution, spectral slope, harmonic components, and other speech characteristics different from normal speech), and context information (such as low ambient background noise and the user's proximity to the voice interaction device). In various embodiments, the voice interaction device can identify a quiet mode, which includes a classification of both whispered speech (voice without vibration of the vocal cords) and soft speech with a lower volume than normal speech (some vibration of the vocal cords). The context controller can analyze the detected usage context to determine whether the quiet mode or the whispering mode should be used for audio output.

[0059] Where applicable, hardware, software, or a combination of hardware and software may be used to implement the various embodiments provided by the present disclosure. Moreover, where applicable, without departing from the scope of the present disclosure, the various hardware components and / or software components set forth herein may be combined into composite components comprising software, hardware, and / or both. Where applicable, without departing from the scope of the present disclosure, the various hardware components and / or software components set forth herein may be divided into subcomponents comprising software, hardware, or both. In addition, where applicable, it is contemplated that software components may be implemented as hardware components, and vice versa.

[0060] Software (such as program code and / or data) according to the present disclosure may be stored on one or more computer-readable media. It is also contemplated that the software identified herein may be implemented using one or more general or special-purpose computers and / or computer systems networked and / or otherwise. Where applicable, the order of the various steps described herein may be changed, combined into composite steps, and / or divided into sub-steps to provide the features described herein.

[0061] The foregoing disclosure is not intended to limit the disclosure to the precise form disclosed or to the specific field of use. Thus, it is contemplated that various alternative embodiments and / or modifications to the disclosure are possible in light of the disclosure, whether or not explicitly described or implied herein. Having thus described embodiments of the disclosure, it will be appreciated by those of ordinary skill in the art that changes may be made in form and detail without departing from the scope of the disclosure.

Claims

1. A voice interaction device, comprising: A plurality of input and output components configured to facilitate interaction between the voice interaction device and a target user, the plurality of input and output components comprising: a microphone configured to sense sound and generate an audio input signal; a speaker configured to output an audio signal to the target user; and an input component configured to sense at least one inaudible interaction from the target user; a display configured to present a visual display element to the target user; A context controller, which is configured to: monitoring the plurality of input and output components and determining a current usage context, detecting and / or tracking the position of the target user relative to the voice interaction device, and estimating a distance between the target user and the voice interaction device based at least in part on the location; and A virtual assistant module configured to facilitate voice communication between the voice interaction device and the target user, and to configure one or more of the input and output components in response to the current usage context, wherein configuring one or more of the input and output components in response to the current usage context comprises: If the distance between the target user and the voice interaction device is greater than a distance threshold, the frame rate of the video stream displayed on the display is reduced.

2. The voice interaction device according to claim 1, further comprising: an audio input circuit configured to receive the audio input signal and generate an enhanced target signal including audio generated by the target user; as well as a speech processor configured to detect a speech command in the enhanced target signal; The virtual assistant module is configured to execute the detected voice command according to the current usage context.

3. The voice interaction device of claim 1 , wherein the input component comprises an image sensor configured to capture a digital image of a field of view; and The context controller is further configured to analyze the digital image to detect and / or track the position of the target user relative to the voice interaction device, and determine a current usage context based at least in part on the position of the target user.

4. The voice interaction device according to claim 3, wherein the context controller is further configured to analyze the digital image to determine the gaze direction of the target user; and The virtual assistant module is further configured to direct the interaction to the target user via the visual display element on the display in response to the gaze direction being directed toward the display, and to direct the interaction via the audio output signal and the speaker in response to the gaze direction being directed away from the display.

5. A voice interaction device according to claim 4, wherein when the target user moves relative to the voice interaction device, the virtual assistant module adjusts the size of the visual display element, wherein the size of the visual display element is adjusted to facilitate readability at a distance between the target user and the voice interaction device.

6. The voice interaction device according to claim 3, wherein the context controller is configured as: Attention-aware output rendering is provided, wherein output fidelity is determined based at least in part on the distance of the target user from the voice interaction device to facilitate interaction from various distances.

7. The voice interaction device according to claim 3, wherein the input component comprises a touch control; wherein the context controller is configured to select a proxemics input modality based at least in part on an analysis of the digital image; and The virtual assistant module is configured to: activate the input component if it is determined that the target user is within the reach of the input component; and activate voice communication when the target user is out of the reach of the voice interaction device.

8. The voice interaction device of claim 1, wherein the virtual assistant module detects speech from the target user, determines at least one speech characteristic, and modulates an output volume according to the determined speech characteristic.

9. A voice interaction device according to claim 8, wherein the context controller is configured to detect the voice volume of the target user based at least in part on the amplitude of the voice, the at least one voice characteristic, the distance between the target user and the voice interaction device and / or ambient noise.

10. A voice interaction device according to claim 9, wherein the context controller analyzes the voice characteristics and modulates the output volume of the voice interaction device to match the detected usage context; wherein if the voice is determined to be a whisper, the context controller may instruct a lower volume to respond with a corresponding voice output level; and wherein if the target user is far away, the context controller may instruct volume adjustment to project the voice output to the target user at a corresponding volume level.

11. A method comprising: facilitating communication between a voice interaction device and a target user using a plurality of input and output components, including sensing sound to generate an audio input signal, outputting the audio signal to the target user, and sensing, by the input component, at least one inaudible interaction from the target user; Outputting information to the target user via a display; monitoring, by a context controller, current usage contexts of the plurality of input and output components; Detecting and / or tracking a relative position of the target user relative to the voice interaction device; estimating a distance between the target user and the voice interaction device based at least in part on the location; as well as Adapting one or more of the input and output components to the current usage context includes reducing a frame rate of a video stream displayed on the display if the distance between the target user and the voice interaction device is greater than a distance threshold.

12. The method according to claim 11, further comprising: generating an enhanced target signal comprising audio generated by the target user from the audio input signal; detecting speech in the enhanced target signal; as well as A voice command is extracted from the speech and executed according to the current usage context.

13. The method according to claim 11, further comprising: Acquire a digital image of the field of view; Analyzing the acquired image to detect and / or track the relative position of the target user with respect to the voice interaction device; as well as A current usage context is determined based at least in part on the relative position of the target user.

14. The method according to claim 13, wherein the method further comprises: Analyzing the gaze direction of the target user; In response to the gaze direction being directed toward the display, turning on the display and providing a visual output to the target user; as well as In response to the gaze direction being directed away from the display, turning off the display and providing speech output to the target user through a speaker.

15. The method according to claim 13, further comprising: Attention-aware output rendering is provided, wherein output fidelity is rendered based at least in part on a distance of the target user from the voice interaction device to facilitate readability from various distances.

16. The method according to claim 15 also includes selecting a spatial relational input modality based on an analysis of the acquired image; and wherein if it is determined that the target user is within the reach of the input component, the voice interaction device activates the input component; and wherein when the target user is out of the reach of the voice interaction device, the voice interaction device utilizes voice input and output.

17. The method according to claim 15, further comprising: When the target user moves relative to the voice interaction device, adjusting the size of the displayed element, wherein the size of the displayed element is adjusted for readability at a distance between the target user and the voice interaction device; as well as Wherein when it is determined that the target user is within arm's reach of the touch-enabled interface, the voice interaction device activates touch screen input.

18. The method of claim 11, further comprising detecting speech from the target user, determining at least one speech characteristic, and modulating an output volume according to the determined speech characteristic.

19. The method according to claim 18 also includes detecting the voice volume of the target user based at least in part on the amplitude of the voice, the at least one voice characteristic, the distance between the target user and the voice interaction device and / or ambient noise.

20. The method according to claim 19 also includes analyzing the voice characteristics and modulating the output volume according to the detected usage context, including determining whether the voice is a whisper, and the context controller can reduce the volume to respond with a corresponding voice output level; and if the target user is far away, the context controller can adjust the volume to project the voice output to the target user at a corresponding volume level.

Images