Voice user interface with radio frequency sensing assistance

Abstract

Systems and techniques are provided for speech recognition assisted by radio frequency (RF) sensing. For example, a process for voice recognition assisted by radio frequency (RF) sensing may include obtaining, at a voice user interface (UI) device, audio data including a voice command from a speaking entity; obtaining RF sensing data corresponding to the audio data; processing the audio data to determine an audio voice command output; processing the RF sensing data to determine an RF sensing voice command output; determining the voice command based on the audio voice command output and the RF sensed voice command output; and performing, at the voice UI device, an operation based on the voice command.

Description

Technical Field

[0001] The present disclosure generally relates to enhancing speech recognition using radio frequency (RF) sensing by a voice user interface (UI) device. In some examples, aspects of the present disclosure relate to systems and techniques for obtaining RF data from an environment to enhance disambiguation of voice commands issued by a speaking entity within the environment. Background Art

[0002] There are devices that are capable of receiving audio input from a user, converting the audio input into one or more commands, and performing one or more actions based on the commands. However, in some scenarios, the environment in which such a device is present may experience an increased amount of noise, which may obscure the commands, thereby rendering the device ineffective in performing the requested one or more operations. In other scenarios, a user may wish to issue commands to such a device without having to speak at a specific volume level to produce a command that can be obtained by the audio input component of such a device.

[0003] To implement various functions, electronic devices may include hardware and software components configured to send and receive radio frequency (RF) signals. For example, wireless devices may be configured to communicate via Wi-Fi, 5G / New Radio (NR), Bluetooth TM (Bluetooth TM ) and / or ultra-wideband (UWB), millimeter wave (mmWave), etc. for communication. Summary of the invention

[0004] In some examples, systems and techniques for speech recognition assisted by radio frequency (RF) sensing are described. According to at least one illustrative example, a method for speech recognition assisted by radio frequency (RF) sensing is provided. The method includes: obtaining audio data including a voice command from a speaking entity at a voice user interface (UI) device; obtaining RF sensing data corresponding to the audio data; processing the audio data to determine an audio voice command output; processing the RF sensing data to determine an RF sensing voice command output; determining the voice command based on the audio voice command output and the RF sensing voice command output; and performing an operation based on the voice command at the voice UI device.

[0005] In another illustrative example, an apparatus for speech recognition assisted by radio frequency (RF) sensing is provided, comprising a memory device and a processor coupled to the memory device. The processor is configured to: obtain audio data including a voice command from a speaking entity at a voice user interface (UI) device; obtain RF sensing data corresponding to the audio data; process the audio data to determine an audio voice command output; process the RF sensing data to determine an RF sensing voice command output; determine the voice command based on the audio voice command output and the RF sensing voice command output; and perform an operation based on the voice command at the voice UI device.

[0006] In another illustrative example, a non-transitory computer-readable medium is provided having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: obtain audio data including a voice command from a speaking entity at a voice user interface (UI) device; obtain RF sensing data corresponding to the audio data; process the audio data to determine an audio voice command output; process the RF sensing data to determine an RF sensing voice command output; determine the voice command based on the audio voice command output and the RF sensing voice command output; and perform operations based on the voice command at the voice UI device.

[0007] In another illustrative example, an apparatus for speech recognition assisted by radio frequency (RF) sensing is provided, comprising: a component for obtaining audio data including a voice command from a speaking entity at a voice user interface (UI) device; a component for obtaining RF sensing data corresponding to the audio data; a component for processing the audio data to determine an audio voice command output; a component for processing the RF sensing data to determine an RF sensing voice command output; a component for determining the voice command based on the audio voice command output and the RF sensing voice command output; and a component for performing an operation based on the voice command at the voice UI device.

[0008] In some aspects, one or more of the apparatuses described herein are mobile or wireless communication devices (e.g., mobile phones or other mobile devices), extended reality (XR) devices or systems (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), wearable devices (e.g., network-connected watches or other wearable devices), vehicles or computing devices or components of vehicles, cameras, personal computers, laptop computers, server computers or server devices (e.g., edge or cloud-based servers, personal computers acting as server devices, mobile devices such as mobile phones acting as server devices, XR devices acting as server devices, vehicles acting as server devices, network routers, or other devices acting as server devices), any combination thereof, and / or other types of devices. In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus may include one or more sensors (e.g., one or more RF sensors), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors.

[0009] This summary is not intended to identify the key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. This subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and the appropriate parts of each claim.

[0010] The foregoing, together with other features and examples, will become more apparent by reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Illustrative examples of the present application are described in detail below with reference to the following drawings:

[0012] Figure 1 is a block diagram illustrating a voice user interface (UI) device according to some examples;

[0013] Figure 2 is a block diagram illustrating a wireless device according to some examples;

[0014] Figure 3 is a block diagram illustrating an environment including a voice UI device and a user according to some examples;

[0015] Figure 4 is a block diagram illustrating an environment including a voice UI device, an RF device, and a user according to some examples;

[0016] Figure 5is a block diagram illustrating an environment including a voice UI device, an RF device, and a user according to some examples;

[0017] Figure 6 is a block diagram illustrating an environment including a voice UI device, an occluding object, and a user according to some examples;

[0018] Figure 7 is a flow chart illustrating an example process for speech recognition assisted by radio frequency (RF) sensing according to some examples;

[0019] Figure 8 is a flow chart illustrating an example process for speech recognition assisted by radio frequency (RF) sensing according to some examples;

[0020] Fig. 9 is a diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0021] Certain aspects and examples of the present disclosure are provided below. Some of these aspects and examples can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the examples of the present application. However, it is obvious that various examples can be practiced without these specific details. The accompanying drawings and descriptions are not intended to be restrictive. In addition, certain details known to those of ordinary skill in the art can be omitted to avoid vague descriptions.

[0022] In the following description of the drawings, any component described with respect to the drawings in the various examples described herein may be equivalent to one or more similarly named components described with respect to any other drawings. For the sake of brevity, the description of these components may not be completely repeated with respect to each figure. Therefore, each example of the components of each drawing is incorporated by reference and is assumed to be optionally present in each other drawing with one or more similarly named components. In addition, according to the various examples described herein, any description of the components of the drawings should be interpreted as an optional example, which can be implemented in addition to, in combination with, or in place of the examples described with respect to the corresponding similarly named components in any other drawings.

[0023] The following description provides only illustrative examples and is not intended to limit the scope, applicability or configuration of the present disclosure. Instead, the following description of the illustrative examples will provide an enabling description for implementing the exemplary examples for those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0024] As used herein, the phrase operably connected or operatively connected means that there is a direct or indirect connection between elements / components / devices that allows the elements to interact with each other in some way. For example, the phrase "operably connected" can refer to any direct (e.g., directly wired between two devices or components) or indirect (e.g., a wired and / or wireless connection between any number of devices or components that connect an operatively connected device) connection. Therefore, any path through which information can travel can be considered an operative connection. In addition, operatively connected devices and / or components can exchange things other than information, such as, for example, electric current, radio frequency signals, etc.

[0025] Many electronic devices such as smart phones, smart speakers, smart TVs, tablets, laptops, smart refrigerators, and / or various other Internet of Things (IoT) devices can be used to access different types of services, applications, and / or media content. For example, a smart speaker may provide a virtual assistant function that can be used to process user queries, respond to commands, present media content, provide communication functions, and / or control other smart devices, among other uses and / or applications. Such devices may be referred to herein as voice user interface (UI) devices.

[0026] In order to use the smart UI device, the voice commands spoken by the user in the environment where such a device is present (e.g., living room, bedroom, etc.) should be clearly obtained by one or more audio input components (e.g., microphone, microphone array, etc.), so that the voice UI device can determine the one or more commands issued by the user (e.g., speaking entity). Once the voice commands are obtained in this way, the voice UI device can process the received audio data to determine one or more operations to be performed in response to the one or more voice commands.

[0027] However, there are certain scenarios in which the audio data obtained by the voice UI device is insufficient to determine one or more voice commands spoken by the user and / or can be improved to improve the recognition efficiency of the voice UI device. As an example, when the environment is noisy (e.g., oversaturated from an audio perspective), all or any part of one or more voice commands may not be perceived and / or incorrectly perceived by the device voice UI, such as when one or more words of the voice command are incomprehensible due to other noises in the environment. As another example, the recognition of the voice UI device can be improved by changing the sensing characteristics of the voice UI device, for example, by performing beamforming on the microphone array to focus on the direction of receiving the voice command, by adjusting the gain level of the audio component of the voice UI device, etc. As another example, there may be such a situation (e.g., in a room with a sleeping child) where the user may want to whisper and lip-sync commands, which may not be understood by the audio sensing component of the voice UI device. Therefore, in order to address the improvement of the voice UI device, additional capabilities should be implemented to enhance the command recognition of such devices. Therefore, systems and technologies are needed to determine the location, direction and / or commands issued to these devices.

[0028] Systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively, "systems and techniques") are described herein for enhancing the capabilities of Voice UI devices to improve the ability of these devices to receive commands and perform operations based on these commands. The systems and techniques provide a device with RF sensing capabilities to collect RF sensing data from an environment in which Voice UI devices are present, and use such data to improve the ability of Voice UI devices to perform speech recognition related operations and capabilities.

[0029] In some examples, RF sensing data can be collected by utilizing a wireless interface that is capable of performing both transmit and receive functions (e.g., a single-station configuration). As an example, a voice UI device may include an audio component for receiving voice commands, and an RF sensing component for performing single-station RF sensing. In other examples, RF sensing data can be collected by utilizing a dual-station configuration, where the transmit and receive functions are performed by different devices (e.g., a first wireless device transmits an RF waveform, and a second wireless device receives the RF waveform and any corresponding reflections). Some examples will be described herein using Wi-Fi as an illustrative example of RF sensing technology. However, these systems and techniques are not limited to Wi-Fi. Any suitable technology for RF sensing using RF spectrum signals may be used without departing from the scope of the examples described herein. For example, in some cases, these systems and techniques can be implemented using 5G / New Radio (NR) (e.g., using millimeter wave (mmWave) technology). In some cases, the systems and techniques can be implemented using other wireless technologies, such as Bluetooth. TM, ultra-wideband (UWB), etc.

[0030] In some examples, a device may include an RF interface configured to implement algorithms with different levels of RF sensing resolution based on the bandwidth of the transmitted RF signal, the number of spatial streams, the number of antennas configured to transmit RF signals, the number of antennas configured to receive RF signals, the number of spatial links (e.g., the number of spatial streams multiplied by the number of antennas configured to receive RF signals), the sampling rate, or any combination thereof. For example, the RF interface of the device may be configured to implement a low-resolution RF sensing algorithm that consumes a small amount of power and can operate in the background when the device is in a "locked" state and / or in a "sleep" mode. In some cases, the device may use the low-resolution RF sensing algorithm as a coarse detection mechanism that is capable of determining the position, direction, and / or distance of a user in an environment relative to the voice UI device. Such information may be used, for example, to perform actions such as beamforming and / or gain control of an audio component of the voice UI device in order to improve the ability of the voice UI device to obtain relevant audio data. As another example, the device's RF interface may be configured to perform higher resolution RF sensing (e.g., a medium resolution RF sensing algorithm, a high resolution RF sensing algorithm, or other higher resolution RF sensing algorithms as discussed herein) to obtain more information about the environment and / or users therein, who may issue voice commands to the voice UI device.

[0031] In some examples, the RF interface of the device can be configured to implement a medium-resolution RF sensing algorithm. The transmitted RF signal for the medium-resolution RF sensing algorithm can be different from the low-resolution RF sensing algorithm by having a higher bandwidth, a greater number of spatial streams, a greater number of spatial links (e.g., a greater number of antennas and / or a greater number of spatial streams configured to receive RF signals), a higher sampling rate (corresponding to a smaller sampling interval), or any combination thereof. In some cases, the medium-resolution RF sensing algorithm can be used to detect the presence of a user (e.g., detecting a head or other body parts such as lips, tongue, etc.) and other information such as speech rate, speaker identity (e.g., based on speech characteristics), etc.

[0032] In another example, the RF interface of the device can be configured to implement a high-resolution RF sensing algorithm. The RF signal transmitted for the high-resolution RF sensing algorithm can be different from the medium-resolution RF sensing algorithm and the low-resolution RF sensing algorithm by having a higher bandwidth, a greater number of spatial streams, a greater number of spatial links (e.g., a greater number of antennas configured to receive RF signals and / or a greater number of spatial streams), a higher sampling rate, or any combination thereof. In some cases, the high-resolution RF sensing algorithm can be used to detect sufficient information about the environment (e.g., a depth map) to identify a speaking entity in the environment, determine the location of the entity's mouth area, determine movement within the mouth area (e.g., lip movement, tongue movement, etc.), etc. Such information can be used, for example, to determine that the speaking entity has issued certain commands or portions of commands, which can be combined with audio data obtained by the voice UI device to enhance the ability of the voice UI device to distinguish one or more commands issued by the user. As an example, audio data obtained by the audio component of a voice UI device may obtain audio data in which a portion of a command is discernible but another portion is not (e.g., “Alexa, turn on <audio data missing> lights”), and high-resolution RF sensing data may be used to provide the missing audio data (e.g., “garage”).

[0033] In some examples, the systems and techniques may perform RF sensing associated with any of the foregoing algorithms by implementing an RF interface of the device having at least two antennas that can be used to simultaneously send and receive RF signals (e.g., a single-station configuration). In some cases, the antenna may be omnidirectional so that RF signals can be received and sent in all directions. For example, a device may utilize the transmitter of its RF interface to send an RF signal and simultaneously enable the RF receiver of the RF interface so that the device can receive any reflected signals (e.g., from reflectors such as objects or people). The RF receiver may also be configured to detect leakage signals that are transmitted from the antenna of the RF transmitter to the antenna of the RF receiver without being reflected from any object. In doing so, the device may collect RF sensing data in the form of channel state information (CSI) data related to the direct path of the transmitted signal (leakage signal), and data related to the reflected path of the signal received corresponding to the transmitted signal.

[0034] In some aspects, the systems and techniques can perform RF sensing associated with each of the foregoing algorithms using a dual-station configuration in which the transmit and receive functions are performed by different devices. For example, a first device can utilize a transmitter of its RF interface to transmit an RF signal, and a second device can enable an RF receiver of the RF interface to receive any RF signals corresponding to the transmission. The received signals can include signals that travel directly from the transmitter to the receiver (e.g., line of sight (LOS) signals) as well as reflected signals (e.g., from reflectors such as objects or people).

[0035] In some aspects, CSI data can be used to calculate the distance and angle of arrival of the reflected signal. The distance and angle of the reflected signal can be used to detect the user's position in the environment, the direction between the user and the voice UI device, generate a depth map of the environment, identify relevant features within the depth map (e.g., the position of the mouth area of ​​the speaking entity that issues a voice command), etc. In some examples, signal processing, machine learning algorithms, any other suitable technology, or any combination thereof can be used to determine the distance and angle of arrival of the reflected signal. In one example, the distance of the reflected signal can be calculated by measuring the time difference from receiving the leakage signal to receiving the reflected signal. In another example, the angle of arrival can be calculated by receiving the reflected signal using an antenna array and measuring the difference in the received phase at each element of the antenna array. In some cases, the distance of the reflected signal together with the angle of arrival of the reflected signal can be used to identify the presence and orientation characteristics of the user or any part of the user.

[0036] In some examples, audio data is obtained by a voice UI device, and RF sensing data corresponding to the audio data is obtained by an RF component, which may be part of a voice UI device or may be part of a separate device. In some examples, audio data is processed to determine an audio voice command output. The audio voice command output may be, for example, the following facts: attempting a voice command, all or any part of one or more voice commands, whether the audio data is sufficient to determine the voice command, the missing part of the voice command, whether the audio data quality has the desired quality (e.g., above a threshold quality level to allow effective voice recognition), etc. In some examples, RF sensing data is processed to determine the RF sensing voice command output. Examples of RF sensing voice command outputs include, but are not limited to, the direction between the user and the voice UI device, the distance between the user and the voice UI device, all or any part of the voice command (e.g., using a machine learning model to associate lip and / or tongue movement with all or any part of the voice command), etc. In some examples, the audio voice command output and the RF sensing voice command output are at least partially combined to allow the voice UI device to better perform the voice recognition function, and perform one or more operations based thereon.

[0037] The examples described herein address the need to enhance the speech recognition capabilities of a voice UI device by using RF sensing data to obtain additional information about a user in an environment in which one or more voice commands are issued to a voice UI device to enhance the audio data obtained by the voice UI device. Such enhancements may include, but are not limited to, allowing the voice UI device to perform beamforming for the audio capture component therein, adjusting various characteristics of the audio component (e.g., gain level), helping to filter audio data, detecting movement of various features of a speaking entity (e.g., in the mouth area) to determine all or any portion of a voice command, etc.

[0038] Various aspects of the systems and techniques described herein are discussed below with respect to the accompanying figures. Figure 1 An example of a computing system 170 of a voice UI device 107 is shown. The voice UI device 107 is an example of a device that may include hardware and software for connecting and exchanging data with other devices and systems using a computer network (e.g., the Internet). The voice UI device 107 may be any device capable of obtaining audio data from an environment, processing audio data to determine one or more voice commands, and performing one or more operations (e.g., turning on a light, turning off a TV, playing a song, putting on a movie, performing a search, locking a door, activating or stopping an alarm, making a phone call, sending a text message, checking social media feed updates, etc.). For example, the voice UI device 107 may be or include a virtual assistant device, a smart speaker, a smart TV, a smart home appliance, a mobile phone, a router, a tablet computer, a laptop computer, a tracking device, a wearable device (e.g., a smart watch, glasses, an XR device, etc.), a vehicle (or a computing device of a vehicle), and / or another device used by a user to communicate via a wireless communication network. In some cases, a device may be referred to as a station (STA), such as when referring to a device configured to communicate using a Wi-Fi standard. In some cases, a device may be referred to as a user equipment (UE), such as when referring to a device configured to communicate using 5G / New Radio (NR), Long Term Evolution (LTE), or other telecommunication standards. Any suitable wireless communication technology may be used without departing from the scope of the examples described herein.

[0039] The computing system 170 may include software and hardware components that may be electrically or communicatively coupled (e.g., operably connected) via a bus 189 (or may communicate in other ways as appropriate). For example, the computing system 170 includes one or more processors 184. The one or more processors 184 may include one or more CPUs, ASICs, FPGAs, APs, GPUs, VPUs, NSPs, microcontrollers, dedicated hardware, any combination thereof, and / or other processing devices and / or systems. The bus 189 may be used by the one or more processors 184 to communicate between cores and / or to communicate with one or more memory devices 186 and / or other components or devices.

[0040] The computing system 170 may also include one or more memory devices 186, one or more digital signal processors (DSPs) 2, one or more subscriber identity modules (SIMs) 174, one or more modems 176, one or more wireless transceivers 178, one or more antennas 187, one or more input devices 172 (e.g., a camera, a mouse, a keyboard, a touch-sensitive screen, a touchpad, a keypad, a microphone or a microphone array, etc.), and one or more output devices 180 (e.g., a display, a speaker, a printer, etc.). In some examples, all or any portion of the input device 172 and / or the output device (180) may be referred to as an audio component of the voice UI device 107. For example, a microphone or a microphone array and a speaker may be considered an audio component of the voice UI device 107.

[0041] One or more wireless transceivers 178 (which may be referred to herein as all or any portion of an RF sensing component) may receive wireless signals (e.g., signal 188) via antenna 187 from one or more other devices, such as other user devices, network devices (e.g., base stations such as eNBs and / or gNBs, WiFi access points (APs) such as routers, range extenders, etc.), cloud networks, etc. In some examples, computing system 170 may include multiple antennas or antenna arrays that can facilitate simultaneous transmission and reception capabilities. Antenna 187 may be an omnidirectional antenna so that RF signals can be received and transmitted in all directions. Wireless signal 188 may be sent via a wireless network. The wireless network may be any wireless network, such as a cellular or telecommunications network (e.g., 3G, 4G, 5G, etc.), a wireless local area network (e.g., a WiFi network), a Bluetooth TMnetwork and / or any wireless other network. In some examples, one or more wireless transceivers 178 may include an RF front end that includes one or more components such as an amplifier, a mixer for down-converting a signal (also known as a signal multiplier), a frequency synthesizer (also known as an oscillator) that provides a signal to the mixer, a baseband filter, an analog-to-digital converter (ADC), one or more power amplifiers, and other components. The RF front end may generally handle the selection and conversion of the wireless signal 188 to a baseband or intermediate frequency, and may convert the RF signal to the digital domain.

[0042] In some examples, computing system 170 may include an encoding-decoding device (or codec) (not shown) configured to encode and / or decode data transmitted and / or received using one or more wireless transceivers 178. In some examples, computing system 170 may include an encryption-decryption device or component (not shown) configured to encrypt and / or decrypt data transmitted and / or received by one or more wireless transceivers 178 (e.g., in accordance with the Advanced Encryption Standard (AES) and / or Data Encryption Standard (DES) standards).

[0043] One or more SIMs 174 can each securely store an International Mobile Subscriber Identity (IMSI) number and associated keys assigned to a user of the Voice UI device 107. The IMSI and keys can be used to identify and authenticate the subscriber when accessing a network provided by a network service provider or operator associated with the one or more SIMs 174.

[0044] One or more modems 176 (which in some examples may be considered part of the RF sensing component) may modulate one or more signals to encode information for transmission using one or more wireless transceivers 178. One or more modems 176 may also demodulate signals received by one or more wireless transceivers 178 in order to decode the transmitted information. In some examples, one or more modems 176 may include a WiFi modem, a 4G (or LTE) modem, a 5G (or NR) modem, and / or any other type of modem, or any combination of such modems.

[0045] The computing system 170 may also include (and / or communicate with) one or more non-transitory machine-readable storage media or storage devices (e.g., one or more memory devices 186), which may include (but are not limited to) local and / or network accessible storage, disk drives, drive arrays, optical storage devices, solid-state storage devices such as RAM and / or ROM, which may be programmable, flash-updatable, and / or the like. Such storage devices may be configured to implement any suitable data storage, including but not limited to various file systems, database structures, and / or the like.

[0046] In various examples, the functionality may be stored as one or more computer program products (e.g., instructions or code) in the memory device 186 and executed by the one or more processors 184 and / or the one or more DSPs 182. The computing system 170 may also include software elements (e.g., located within the one or more memory devices 186), including, for example, an operating system, device drivers, executable libraries, and / or other code, such as one or more applications, which may include a computer program that implements the functionality provided by the various examples, and / or may be designed to implement methods and / or configure systems, as described herein.

[0047] although Figure 1 A specific number of components in a specific configuration are shown, but one of ordinary skill in the art will appreciate that the voice UI device 107 may include more components or fewer components, and / or components arranged in any number of alternative configurations without departing from the scope of the examples described herein. Therefore, the examples disclosed herein should not be limited to Figure 1 Configuration of the components shown in .

[0048] Figure 2 is a diagram showing an example of a wireless device 200 that utilizes radio frequency (RF) sensing technology to perform one or more functions (such as detecting the presence of a user 202, detecting an orientation characteristic of a user, performing facial recognition, determining movement of a portion of a user (e.g., lips, tongue, etc.), any combination thereof), and / or performing other functions. In some examples, the wireless device 200 may be a voice UI device 107 or any portion thereof, such as a voice command assistant device, a smart speaker, a smart home appliance, a mobile phone, a tablet computer, a wearable device, or any other device that includes at least one RF interface. In some examples, the wireless device 200 may be a device that provides connectivity for a user device (e.g., for the voice UI device 107), such as a wireless access point (AP), a base station (e.g., a gNB, an eNB, etc.), or any other device that includes at least one RF interface. In some examples, the wireless device 200 is a voice UI device (e.g., Figure 1 The wireless device 200 may be a wireless device that is separate from the voice UI device and is in the same environment (e.g., the same room, home, etc.) and that is configured to sense the voice of the user.

[0049] In some examples, the wireless device 200 may include one or more components for transmitting RF signals. The wireless device 200 may include a digital-to-analog converter (DAC) 204 that is capable of receiving a digital signal or waveform (e.g., from a microprocessor, not shown) and converting the signal or waveform into an analog waveform. The analog signal as an output of the DAC 204 may be provided to an RF transmitter 206. The RF transmitter 206 may be a Wi-Fi transmitter, a 5G / NR transmitter, a Bluetooth transmitter, or a wireless transmitter. TM transmitter, or any other transmitter capable of sending RF signals.

[0050] RF transmitter 206 may be coupled to one or more transmit antennas, such as TX antenna 212. In some examples, TX antenna 212 may be an omnidirectional antenna capable of transmitting RF signals in all directions. For example, TX antenna 212 may be an omnidirectional Wi-Fi antenna that may radiate Wi-Fi signals (e.g., 2.4 GHz, 5 GHz, 6 GHz, etc.) in a 360-degree radiation pattern. In another example, TX antenna 212 may be a directional antenna that transmits RF signals in a specific direction. Although Figure 2 TX antenna 212 and RX antenna 214 are shown as separate components, but one of ordinary skill in the relevant art will appreciate that the TX and RX antennas may be the same antenna.

[0051] In some examples, the wireless device 200 may also include one or more components for receiving RF signals. For example, a receiver lineup in the wireless device 200 may include one or more receive antennas, such as an RX antenna 214. In some examples, the RX antenna 214 may be an omnidirectional antenna capable of receiving RF signals from multiple directions. In other examples, the RX antenna 214 may be a directional antenna configured to receive signals from a specific direction. In another example, both the TX antenna 212 and the RX antenna 214 may include multiple antennas (e.g., elements) configured as an antenna array.

[0052] The wireless device 200 may also include an RF receiver 210 coupled to an RX antenna 214. The RF receiver 210 may include a signal processor for receiving RF waveforms (such as Wi-Fi signals, Bluetooth signals, etc.). TM The output of RF receiver 210 may be coupled to an analog-to-digital converter (ADC) 208. ADC 208 may be configured to convert a received analog RF waveform into a digital waveform, which may be provided to a processor such as a digital signal processor (not shown).

[0053] In some examples, the wireless device 200 implements the RF sensing technology by causing a TX waveform 216 to be transmitted from the TX antenna 212. Although the TX waveform 216 is shown as a single line, in some cases, the TX waveform 216 may be transmitted in all directions by the omnidirectional TX antenna 212. In some examples, the TX waveform 216 may be a Wi-Fi waveform transmitted by a Wi-Fi transmitter in the wireless device 200. In some examples, the TX waveform 216 may correspond to a Wi-Fi waveform transmitted simultaneously or nearly simultaneously with a Wi-Fi data communication signal or a Wi-Fi control function signal (e.g., a beacon transmission). In some examples, the TX waveform 216 may be transmitted using the same or similar frequency resources as the Wi-Fi data communication signal or the Wi-Fi control function signal (e.g., a beacon transmission). In some examples, the TX waveform 216 may correspond to a Wi-Fi waveform transmitted separately from the Wi-Fi data communication signal and / or the Wi-Fi control signal (e.g., the TX waveform 216 may be transmitted at a different time and / or using different frequency resources).

[0054] In some examples, TX waveform 216 may correspond to a 5G NR waveform that is transmitted simultaneously or nearly simultaneously with a 5G NR data communication signal or a 5G NR control function signal. In some examples, TX waveform 216 may be transmitted using the same or similar frequency resources as the 5G NR data communication signal or the 5G NR control function signal. In some examples, TX waveform 216 may correspond to a 5G NR waveform that is transmitted separately from the 5G NR data communication signal and / or the 5G NR control signal (e.g., TX waveform 216 may be transmitted at a different time and / or using different frequency resources).

[0055] In some examples, one or more parameters associated with the TX waveform 216 may be modified, which may be used to increase or decrease RF sensing resolution. These parameters may include frequency, bandwidth, number of spatial streams, number of antennas configured to transmit the TX waveform 216, number of antennas configured to receive reflected RF signals corresponding to the TX waveform 216, number of spatial links (e.g., number of spatial streams multiplied by number of antennas configured to receive RF signals), sampling rate, or any combination thereof.

[0056] In some examples, the TX waveform 216 can be implemented to have a sequence that has perfect or nearly perfect autocorrelation properties. For example, the TX waveform 216 can include a single-carrier Zadoff sequence, or can include symbols similar to orthogonal frequency division multiplexing (OFDM) long training field (LTF) symbols. In some examples, the TX waveform 216 can include a linear frequency modulation (chirp) signal, such as used in a frequency modulated continuous wave (FM-CW) radar system. In some configurations, the linear frequency modulation signal can include a signal whose frequency increases and / or decreases periodically in a linear and / or exponential manner.

[0057] In some examples, the wireless device 200 may also implement RF sensing technology by performing concurrent transmit and receive functions. For example, the wireless device 200 may enable its RF receiver 210 to receive at or near the same time as it enables the RF transmitter 206 to transmit the TX waveform 216. In some examples, the transmission of a sequence or pattern included in the TX waveform 216 may be repeated continuously so that the sequence is transmitted a specific number of times or for a specific duration. In some examples, if the RF receiver 210 is enabled after the RF transmitter 206, repeating the pattern in the transmission of the TX waveform 216 may be used to avoid missing any reception of reflected signals. In some examples, the TX waveform 216 may include a sequence having a sequence length L that is transmitted two or more times, which may allow the RF receiver 210 to be enabled at a time less than or equal to L so as to receive reflections corresponding to the entire sequence without missing any information.

[0058] By implementing simultaneous transmit and receive functionality, the wireless device 200 can receive any signal corresponding to the TX waveform 216. For example, the wireless device 200 can receive a signal reflected from an object or person (e.g., a speaking entity) within the range of the TX waveform 216, such as the RX waveform 218 reflected from the user 202. The wireless device 200 can also receive a leakage signal (e.g., TX leakage signal 220) that is directly coupled from the TX antenna 212 to the RX antenna 214 without reflecting from any object. For example, the leakage signal can include a signal transmitted from a transmitter antenna (e.g., TX antenna 212) on the wireless device to a receiving antenna (e.g., RX antenna 214) on the wireless device without reflecting from any object. In some examples, the RX waveform 218 can include multiple sequences corresponding to multiple copies of the sequence included in the TX waveform 216. In some examples, the wireless device 200 can combine multiple sequences received by the RF receiver 210 to improve the signal-to-noise ratio (SNR).

[0059] The wireless device 200 may also implement RF sensing techniques by obtaining RF sensing data associated with each received signal corresponding to the TX waveform 216. In some examples, the RF sensing data may include channel state information (CSI) data related to a direct path of the TX waveform 216 (e.g., the leakage signal 220), and data related to a reflected path corresponding to the TX waveform 216 (e.g., the RX waveform 218).

[0060] In some examples, the RF sensing data (e.g., CSI data) may include information that can be used to determine the manner in which an RF signal (e.g., TX waveform 216) propagates from the RF transmitter 206 to the RF receiver 210. The RF sensing data may include data corresponding to effects on the transmitted RF signal due to scattering, fading, and / or power attenuation with distance, or any combination thereof. In some examples, the RF sensing data may include imaginary data and real data (e.g., I / Q components) corresponding to each tone in the frequency domain over a particular bandwidth.

[0061] In some examples, the RF sensing data can be used to calculate the distance and angle of arrival corresponding to the reflected waveform (such as RX waveform 218). In other examples, the RF sensing data can also be used to detect physical characteristics, detect motion, determine position, determine the direction between the user and the voice UI device, detect changes in position or motion patterns (e.g., movement of one or more features in the mouth area of ​​the speaking entity), obtain channel estimates, or any combination thereof. In some cases, the distance and angle of arrival of the reflected signal can be used to identify the size, position, movement, or orientation of a user (e.g., user 202) in the surrounding environment in order to determine the user's position, determine the direction between the user and the voice UI device, identify specific areas of the user (e.g., mouth area), identify various features within a given area (e.g., lips, tongue, etc. in the user's mouth area), determine the movement of these features, generate a depth map of the environment or any part thereof, etc.

[0062] The wireless device 200 can calculate the distance and angle of arrival corresponding to the reflected waveform (e.g., the distance and angle of arrival corresponding to the RX waveform 218) by utilizing signal processing, machine learning algorithms, using any other suitable techniques, or any combination thereof. In other examples, the wireless device 200 can send or transmit the RF sensing data to another computing device, such as a server, which can perform calculations to obtain the distance and angle of arrival corresponding to the RX waveform 218 or other reflected waveform.

[0063] In some examples, the distance of the RX waveform 218 may be calculated by measuring the time difference from receiving the leakage signal to receiving the reflected signal. For example, the wireless device 200 may determine a baseline distance of zero based on the difference (e.g., propagation delay) from the time the wireless device 200 transmits the TX waveform 216 to the time it receives the leakage signal 220. The wireless device 200 may then determine the distance associated with the RX waveform 218 based on the difference (e.g., time of flight) from the time the wireless device 200 transmits the TX waveform 216 to the time it receives the RX waveform 218, which may then be adjusted based on the propagation delay associated with the leakage signal 220. In doing so, the wireless device 200 may determine the distance traveled by the RX waveform 218, which may be used to generate a depth map of the environment, which may include different distances to various elements of the environment. As an example, the depth map may include the distance difference and relative positioning of the user's lips over time, which may be used as an input to a machine learning model that is trained to identify certain keywords (e.g., voice commands or portions of voice commands) that correspond to specific locations of the lips.

[0064] In some examples, the angle of arrival of the RX waveform 218 may be calculated by measuring the time difference of arrival of the RX waveform 218 between various elements of a receive antenna array (e.g., antenna 214). In some examples, the time difference of arrival may be calculated by measuring the difference in the receive phase at each element in the receive antenna array.

[0065] In some examples, the distance and angle of arrival of RX waveform 218 may be used to determine the distance between wireless device 200 and user 202 (or any one or more portions of a user), as well as the location of user 202 relative to wireless device 200 and / or relative to any other devices (not shown) within the environment. The distance and angle of arrival of RX waveform 218 may also be used to determine the presence, movement, proximity, attention, identity, or any combination thereof of user 202.

[0066] As described above, the wireless device 200 may include or be part of various devices, such as a voice UI device, a mobile device (e.g., an IoT device, a smart phone, a laptop computer, a tablet computer, etc.), a smart appliance, and / or any other type of device configured to send and / or receive RF signals to perform RF sensing, as discussed herein. In some examples, the wireless device 200 may be configured to obtain device location data and device orientation data as well as RF sensing data. In some examples, the device location data and device orientation data may be used to determine or adjust the distance and angle of arrival of a reflected signal (such as RX waveform 218). For example, when the user 202 walks toward the table during the RF sensing process, the wireless device 200 may be set on a table facing the ceiling. In this example, the wireless device 200 may use its location data and orientation data as well as the RF sensing data to determine the direction in which the user 202 is walking.

[0067] In some examples, device location data may be collected by the wireless device 200 using techniques including round trip time (RTT) measurements, passive positioning, angle of arrival, received signal strength indicator (RSSI), CSI data, using any other suitable technique, or any combination thereof. In further examples, device orientation data may be obtained from electronic sensors on the wireless device 200, such as a gyroscope, accelerometer, compass, magnetometer, barometer, global positioning system (GPS) receiver, any other suitable sensor, or any combination thereof.

[0068] Although Figure 2 A specific number of components in a specific configuration is shown, but one of ordinary skill in the art will appreciate that the wireless device 200 may include more components or fewer components, and / or components arranged in any number of alternative configurations without departing from the scope of the examples described herein. Figure 2 Configuration of the components shown in .

[0069] Figure 3 An example environment 300 is shown according to one or more examples described herein. Figure 3 As shown, environment 300 includes user 308 (which may also be referred to as a speaking entity) and voice UI device 302. Figure 3 The voice UI device shown includes an audio capture component and an RF sensing component. Each of these components is described below.

[0070] In some examples, the voice UI device 302 is any device that can capture voice commands and perform actions based on voice commands. Examples of the voice UI device 302 may include, but are not limited to, voice assistant devices, smart speakers, smart phones, smart home appliances, smart watches, extended reality (XR) devices (e.g., augmented reality, virtual reality, etc.), tablets, computing devices (e.g., mobile computing devices, server computing devices, desktop computing devices, etc.), smart TVs, vehicle computing devices, navigation devices, etc. The voice UI device 302 may be Figure 1 The voice UI device 107 shown in and described above, Figure 2 The wireless device 200 shown in and described above, Fig. 9 The computing device 900 shown in and described below may include all or any portion of, or include all or any portion of, the computing device 900 , and / or any other computing device described herein.

[0071] In some examples, environment 300 includes user 308, which can be referred to as a speaking entity. In some examples, a speaking entity is any entity, such as a person, that is capable of issuing voice commands to voice UI device 302. Figure 3 User 308 is depicted as a person, but user 308 may be any other entity capable of issuing voice commands (eg, a speaker device, a robotic device, etc.).

[0072] In some examples, voice commands are any number of spoken words, phrases, etc. that the voice UI device 302 is configured to understand. Voice commands may include any number of keywords. Such keywords may include, but are not limited to, wake-up words or phrases (e.g., "Alexa", "Siri", "OK Google", etc.), command words or phrases (e.g., "turn off the lights", "turn on the alarm", "play [any song]", "set the timer to five minutes", "record [any TV show]", "lower the temperature", "search [any topic]", "tell me a joke", etc.), question words or phrases (e.g., "what time", "what is the weather near me", "when does the movie play", "when does Astros play", etc.), words that modify phrases (e.g., specifying a location such as a room), etc. Voice commands may be spoken at any volume level (e.g., loudly, at a standard conversation level, in a low voice, etc.). As used herein, the term voice command also includes commands issued when not at an audible level. As an example, in certain scenarios (e.g., in a room with a sleeping child, when a sporting game is on television, etc.), user 308 may wish to issue voice commands silently (e.g., without producing or intending to produce sound), and therefore, may mouth the voice commands rather than speaking them at an audible level.

[0073] In some examples, the voice UI device responds to the voice command by performing the operation indicated by the voice command. Examples of such operations include, but are not limited to, turning on or off items (e.g., lights, alarms, appliances, televisions, music playback devices, fans, computing devices, monitors, sound machines, etc.), raising or lowering volume levels, performing searches, answering questions, etc. Operations may include multiple actions. As an example, a voice command asking about the current weather may cause the UI device to perform a search to determine the current weather, and then use an audio output device to tell the user 308 the current weather where the user 308 lives.

[0074] In some examples, the voice UI device 302 includes an audio capture component 304. In some examples, the audio capture component 304 is any part of the element of the voice UI device 302, which is configured to capture the audio data in the environment 300, including but not limited to voice commands (for example, voice commands issued by user 308). As an example, the audio capture component may include a microphone and / or a microphone array. In some examples, the audio capture component 304 includes and / or is operably connected to a storage device (not shown) for storing the captured audio data. In some examples, the audio capture component 304 includes and / or is operably connected to any number of processing elements (not shown). As an example, such a processing element may be configured to process the audio data captured from the environment to determine when (for example, by user 308) to use voice commands. As an example, the audio capture component 304 may use a processing element to filter the audio data including other sounds from the environment 300 to obtain filtered audio data including one or more voice commands. In such an example scenario, the filtered audio data may be provided as input to a trained machine learning model that processes the filtered audio data to determine what the voice command is and / or causes the voice UI device 302 to perform one or more operations in response to one or more voice commands. The audio capture component 304 may include Figure 1 All or any portion of any element of the voice UI device 302 shown in and described above (e.g., input device(s) 172, processor(s) 184, memory device(s) 186, DSP(s) 182, etc.).

[0075] In some examples, the voice UI device 302 includes an RF sensing component 306. In some examples, the RF sensing component 306 is any portion of an element of the voice UI device 302 that is configured to perform RF sensing in the environment 300. As described above, RF sensing includes sending and receiving RF signals within the environment, and processing the results of the sending and receiving to obtain additional information. In some examples, the RF sensing component 306 sends and receives RF signals. Therefore, the RF sensing component 306 can be considered a single-station configuration.

[0076] RF sensing may be performed using any suitable wireless technology using RF signals of any suitable frequency. Examples of such wireless technologies include, but are not limited to, Wi-Fi, mmWave, UWB, Bluetooth, etc. RF sensing may include using, for example, Figure 1 and Figure 2 RF sensing may include determining the distance and angle between the RF sensing component 306 and an object (e.g., user 308) in the environment 300. As discussed above, RF sensing may be performed at a relatively low or high resolution level (e.g., based on the purpose of the RF sensing being performed).

[0077] RF sensing component 306 may include one or more wireless transceivers (e.g., Figure 1 and described above), one or more antennas (e.g., Figure 1 ) and / or a wireless device (e.g., Figure 2 In some examples, the RF sensing component 306 includes and / or is operably connected to a storage device (e.g., Figure 1 In some examples, the RF sensing component 306 includes and / or is operably connected to one or more processing elements (e.g., Figure 1 302 and described above). As an example, such a processing element may process the RF sensing data to obtain additional information about one or more objects in the environment (e.g., user 308). Such additional information may include, but is not limited to, the distance between user 308 and voice UI device 302, the angle between user 308 and voice UI device 302, a depth map of environment 300, identification of one or more regions in the depth map (e.g., the mouth region of user 308), identification of one or more features in such regions (e.g., lips, tongue, etc. of the mouth region of user 308), movement of such features over time, etc.

[0078] The RF sensing component 306 may include and / or be operably connected to an environment configured to execute any number of ML models. As an example, the RF sensing component may be configured to generate a depth map of the environment 300. The depth map may be flattened into a two-dimensional representation of the environment. Alternatively, the voice UI device 302 may include a camera (not shown) from which a two-dimensional representation of the environment is obtained. In either case, the two-dimensional representation may be processed by a trained machine learning model to identify relevant elements in the environment 300, such as the mouth area of ​​the user 308. In this example scenario, the depth map may then be filtered to focus on the mouth area of ​​the user 308. The RF sensing data obtained over time (e.g., when a voice command is spoken) for the filtered area of ​​interest may then be further processed by an ML model, which is trained to associate movement (e.g., tongue movement, lip movement, airflow from the mouth area, any combination thereof, etc.) with the keywords of the voice command. There may be any number of ML models, each configured for different situations. As an example, there may be different ML models for different genders, age ranges, languages, etc. of users 308. Thus, processing the RF sensing data may include determining which one or more ML models are appropriate for the situation.

[0079] In some examples, the RF sensing data can be used to enhance, supplement, improve, etc. the audio data captured by the audio capture component 304. The following are various examples of using RF sensing data to enhance the voice recognition capabilities of the voice UI device 302. The following embodiments are only for explanation purposes and are not intended to limit the scope of the embodiments described herein. In addition, although the examples illustrate certain aspects of the examples described herein, all possible aspects of these examples may not be shown in these specific examples.

[0080] Consider an example scenario where the audio capture component 304 has difficulty determining the voice command of the user 308 from the environment 300. Such difficulties may occur, for example, because the environment 300 is noisy, because the user 308 speaks at a low volume, etc. In this case, the RF sensing component can process the RF sensing data from the environment to identify the position of the user 308 relative to the voice UI device in the environment (e.g., the distance and angle between the user 308 and the voice UI device 302). The position information can be provided to the audio capture component 304, which can then use the information to make one or more configuration changes to better capture the voice command from the user 308. As an example, the audio capture device can perform beamforming on the microphone array to point the array to the user. As another example, the audio capture component 304 can use the position information to adjust the gain level of one or more microphones to improve audio capture. As another example, the RF sensing data can be used to determine how to better filter the audio data captured by the audio capture component (e.g., remove background noise, embed and adjust, etc.).

[0081] Consider another example scenario in which a user 308 issues a voice command to the Voice UI device 302. In this case, the voice command is "devices, turn on the bedroom lights." However, at the moment the word "bedroom" is spoken, a door slams shut near the Voice UI device 302. Thus, the noise from the slamming door masks the word bedroom in the voice command. As a result, the audio capture component 304 is only able to determine "devices, turn on the [missing audio data] lights." Therefore, the Voice UI device 302 cannot perform the operation of turning off the bedroom lights because it cannot identify the location of the lights to be turned off.

[0082] In this scenario, the audio data can be enhanced by RF sensing data to provide missing words. The RF sensing component 306 can first use the RF sensing data captured from the environment to generate a depth map of the environment. The depth map can be flattened into a two-dimensional representation of the environment. The two-dimensional representation is provided to an ML model trained to identify aspects of the two-dimensional representation. The output of the ML model is the position of the mouth area of ​​the user 308 within the environment. The position of the mouth area is used to filter the depth map to focus on the mouth area during the user's voice command, which can include filtering out the data outside the depth range and / or outside the mouth area. The filtered depth map includes a representation of the lips and tongue of the user 308 (e.g., features in the mouth area), and when they move, the lips and tongue are at different positions and distances relative to the RF sensing component 306. Based on the movement and optional other RF sensing data (e.g., the size and / or shape of the user 308), the RF sensing component selects a trained ML model suitable for the age, gender and language of the user 308. The movement information from the lips and tongue during the time when the missing part of the voice command is spoken is provided as input to the selected ML model. The selected ML model generates as output the keyword (bedroom) spoken by the user 308 by correlating movement with the keyword. The keyword bedroom is then provided to the audio capture component 304. Now having the missing parts of the voice command, the audio capture component 304 can cause the voice UI device 302 to perform the correct operation of turning on the bedroom lights.

[0083] Consider another example scenario, where user 308 desires to issue a voice command to voice UI device 302. However, user 308 may wish to issue the voice command silently or softly so as not to wake up a sleeping person who is also in environment 300. In such a scenario, user 308 may lip-sync or whisper the voice command instead of speaking the voice command. In this scenario, the RF sensing component may process data (e.g., as described in the previous example scenario) to determine the voice command based on the relative position and movement of one or more features in the mouth area of ​​user 308. Thus, voice UI device 302 is able to perform the operation requested by the voice command without the audio capture component capturing the voice command (e.g., the audio capture component may only capture audio data representing the background noise of environment 300).

[0084] There may be other example scenarios in which RF sensing data may be used to enhance the ability of the voice UI device 302 to perform operations in response to voice commands. As an example, the voice UI device 302 may have a camera (not shown) that is sometimes used at least in part to assist the voice recognition capabilities of the audio capture component 304. In such a scenario, the conditions in the environment 300 may prevent the camera from providing such assistance (e.g., the room is dark, the user covers the user's mouth, etc.). Therefore, the RF sensing data may be used to determine all or any part of the voice command, whether or not it is audibly issued, to allow the voice UI device 302 to perform the requested operation. As another example, the RF sensing data may be used to determine the gesture made by the user 308 in conjunction with the voice command. In such a scenario, the user may say "device, turn off the lights" while pointing to a specific light in the environment 300. The RF sensing data may be processed to determine the position of the user's arm and fingers, and identify the user 308 pointing to, etc. Therefore, the audio data combined with the RF sensing data allows the voice UI device 302 to perform the operation of turning off a specific light.

[0085] Although Figure 3 A specific number of components in a specific configuration are shown, but one of ordinary skill in the art will appreciate that environment 300 may include more components or fewer components, and / or components arranged in any number of alternative configurations without departing from the scope of the examples described herein. Figure 3 Configuration of the components shown in .

[0086] Figure 4 An example environment 400 is shown according to one or more examples described herein. Figure 4 The environment 400 shown includes a user 410, a voice UI device 402 including an audio capture component 404, and an RF device 406 including an RF sensing component 408. Each of these components is described below.

[0087] In some examples, user 410 is substantially similar to Figure 3 In some examples, the audio capture component 404 is substantially similar to the user 308 shown in FIG. Figure 3 In some examples, the voice UI device 402 is substantially similar to the audio capture component 304 shown in FIG. Figure 3 , except that the voice UI device 402 does not include an RF sensing component. The RF sensing component 408 is substantially similar to Figure 3 , and described above, except that the RF sensing component 408 is not included in the voice UI device 402.

[0088] On the contrary, Figure 4As shown, the RF sensing component 408 is included in the RF device 406. In some examples, the RF device 406 is any device that is separate from the voice UI device 402 and includes the RF sensing component 408. Figure 4 It is intended to illustrate an example in which the RF sensing component 408 is included in a device (e.g., RF device 406) that is separate from the voice UI device 402 and operably connected to the voice UI device 402. In this scenario, the RF sensing component 408 can perform the above-mentioned Figure 3 , but modified with awareness of the position of the voice UI device 402 relative to the RF device 406. Thus, the information obtained using the RF sensing data can be modified to take into account the relative positions of the two devices. Such information can be transmitted from the RF device 406 and thus can be used by the voice UI device 402, as described above in Figure 3 as discussed in the description of .

[0089] Although Figure 4 A specific number of components in a specific configuration are shown, but one of ordinary skill in the art will appreciate that environment 400 may include more components or fewer components, and / or components arranged in any number of alternative configurations without departing from the scope of the examples described herein. Figure 4 Configuration of the components shown in .

[0090] Figure 5 An example environment 500 is shown according to one or more examples described herein. Figure 5 The environment 500 shown includes a user 512, a voice UI device 502 including an audio capture component 504 and an RF sensing receiver 506, and an RF device 508 including an RF sensing transmitter 510. Each of these components is described below.

[0091] In some examples, user 512 is substantially similar to Figure 3 In some examples, the audio capture component 504 is substantially similar to the user 308 shown in FIG. Figure 3 The audio capture component 304 shown and described above. In some examples, the voice UI device 402 is substantially similar to Figure 3 , except that voice UI device 402 does not include an RF sensing component. Instead, in some examples, voice UI device 502 includes an RF sensing receiver 506. In some examples, RF device 508 is substantially similar to Figure 4 4 and described above, except that RF device 508 includes an RF sensing transmitter 510 rather than an RF sensing component.

[0092] Figure 5 It is intended to illustrate an example where the RF sensing assembly is configured in a bistatic configuration (as described above), where RF signals are transmitted from one device and received by a second device. Figure 5 In the example shown, the RF sensing transmitter 510 transmits an RF signal that is received by the RF sensing receiver 506 of the voice UI device 502. The RF signal can be received after being reflected by an object in the environment 500 and / or directly received without reflection. The RF sensing data obtained by the RF sensing receiver 506 can then be used, for example, to perform the above Figure 3 500. Thus, the RF sensing transmitter 510 and the RF sensing receiver 506 can be collectively considered to be RF sensing components in the environment 500. In some examples, the RF sensing data is processed by the voice UI device 502. In other examples, the RF sensing data is transmitted to the RF device 508, where the RF sensing data is processed and the results are returned to the voice UI device 502. In some examples, the RF sensing receiver 506 and / or the RF sensing transmitter are configured with the relative positions of the voice UI device 502 and the RF device 508 so that the difference in position can be taken into account when processing the RF sensing data.

[0093] Although Figure 5 A certain number of components in a particular configuration are shown, but one of ordinary skill in the art will appreciate that environment 500 may include more components or fewer components, and / or components arranged in any number of alternative configurations without departing from the scope of the examples described herein. Figure 5 Configuration of the components shown in .

[0094] Figure 6 An example environment 600 is shown according to one or more examples described herein. Figure 6 The illustrated environment 600 includes a user 610, a voice UI device 602, and an occluding object 608. The voice UI device 602 includes an audio capturing component 604 and an RF sensing component 606. Each of these components is described below.

[0095] In some examples, user 610 is substantially similar to Figure 3 In some examples, the voice UI device 602 is substantially similar to the user 308 shown in FIG. Figure 3 In some examples, the audio capture component 404 is substantially similar to Figure 3 In some examples, the RF sensing component 606 is generally similar to the audio capture component 304 shown in FIG. Figure 3The RF sensing component 306 shown in and described above.

[0096] Figure 6 It is intended to illustrate an example in which the environment 600 includes an occluding object 608 between the voice UI device 602 and the user 610. In some examples, the occluding object is any object (e.g., a wall, a column, furniture, a staircase, a feature of a room, a door, etc.) that is located between the voice UI device 602 and the user 610 and that obscures some aspect of the user 610 to the voice UI device 602. As an example, the occluding object 608 may mute a voice command issued from the user 610 to the voice UI device 602. As another example, the occluding object 608 may prevent a camera (not shown) of the voice UI device 602 from seeing the user 610, thereby preventing any camera-related functions of the voice UI device 602 from being performed. In an embodiment such as Figure 6 In the scenario shown in environment 600, RF sensing component 606 can be configured to send and receive RF signals of one or more frequencies that can pass through blocking object 608. Therefore, the voice recognition capability of voice UI device 602 can still be as described above. Figure 3 608 is enhanced, augmented, improved, etc. as described in the description of , even when an occluding object 608 obscures one or more aspects of the user 610 to the voice UI device 602.

[0097] Although Figure 6 A certain number of components in a particular configuration are shown, but one of ordinary skill in the art will appreciate that environment 400 may include more components or fewer components, and / or components arranged in any number of alternative configurations without departing from the scope of the examples described herein. Figure 6 Configuration of the components shown in .

[0098] Figure 7 is a flow chart illustrating an example of a process 700 for speech recognition assisted by RF sensing according to examples described herein. The process 700 may be performed at least in part by, for example Figure 1 The voice UI device 107 shown in and described above, Figure 2 The wireless device 200 shown in and described above, Figure 3 The voice UI device 302 shown in and described above, Figure 4 The voice UI device 402 and RF device 406 shown in and described above, Figure 5 The voice UI device 502 and RF device 508 shown in and described above, Figure 6 and / or the voice UI device 602 shown in and described above. Fig. 9 The present invention is performed by the computing device 900 shown in and described below.

[0099] At box 702, process 700 includes obtaining audio data including voice commands from a speaking entity at a voice user interface (UI) device. In some examples, the speaking entity is any entity (e.g., a person) that can speak commands to a voice UI device. In some examples, the audio data includes any sound released from the speaking entity. In some examples, a voice command is any set of one or more sounds intended to cause a voice UI device to perform any one or more actions (e.g., increase the volume, turn off a light, set an alarm, start a timer, etc.). In some examples, obtaining audio data includes receiving audio data at an audio receiver (e.g., one or more microphones) of a voice UI device.

[0100] At block 704, process 700 includes obtaining RF sensing data corresponding to the audio data. In some examples, the RF sensing data is acquired using any one or more RF sensing components of the voice UI device (e.g., Figure 3 As an example, the RF sensing data may include transmitting an RF waveform and receiving reflections of the RF waveform during a time period when a voice command is issued from a speaking entity, which may be used to generate a depth map of the environment during that time period.

[0101] At box 706, process 700 includes processing audio data to determine an audio voice command output. In some examples, the audio voice command output includes at least a portion of a voice command issued to a voice UI device. As an example, a voice UI device may record a voice command issued by a speaking entity, and process the voice command to determine various characteristics of the audio recording. Such characteristics may be used as an input to a voice command processing algorithm that is trained to interpret voice command audio data in an attempt to determine the voice command issued by the speaking entity. In some examples, voice commands may be determined using only audio data, and the voice UI device may perform one or more operations based thereon. However, in some examples, the audio data may not include enough information to allow the voice UI device to recognize the voice command.

[0102] At box 708, process 700 includes processing the RF sensing data to determine an RF sensing command output. In some examples, the RF sensing command output includes any data corresponding to the RF sensing data obtained when the voice command is issued by the speaking entity. As an example, the RF sensing command output may include a depth map of the environment obtained when the voice command is issued. Such a depth map can be processed to flatten the depth map into a two-dimensional representation of the environment. Image processing techniques can then be used to determine feature information (e.g., the location of the mouth area of ​​the speaking entity). Based on the feature information, further processing may include determining information about one or more portions of the feature information from the depth map. As an example, the depth map can be processed to determine the movement of the tongue and / or lips of the speaking entity when the voice command is issued.

[0103] At box 710, process 700 includes determining a voice command based on an audio voice command output and an RF sensing voice command output. In some examples, determining a voice command includes combining an audio voice command output and an RF sensing voice command output. As an example, the RF sensing voice command output can be used to determine the direction between a voice UI device and a speaking entity, and combining the RF sensing voice command output and the audio voice command output can include performing beamforming on one or more microphones of the voice UI device to point the microphone to the speaking entity. As another example, the RF sensing voice command output can be used to determine the distance between the voice UI device and the speaking entity, and the distance information can be used to adjust the gain level of the audio sensing component of the voice UI device. As another example, the RF sensing voice command output can be processed to determine various voice characteristics of the speaking entity, which can be used to enhance the ability of the voice UI device to correctly interpret voice commands. As another example, the RF sensing voice command output can be processed (e.g., using a trained ML model) to determine one or more words or parts of words spoken when issuing a voice command based on the movement of one or more features (e.g., tongue, lips, etc.) of the speaking entity, and such information can be used to fill in the gaps in the audio voice command output to complete the expected voice command. As another example, RF sensing voice command output may be processed to determine one or more gestures made by the speaking entity during the voice command that indicate additional information related to the issued voice command (e.g., gesturing at a particular light).

[0104] At block 712, process 700 includes performing an operation based on the voice command at the voice UI device. In some examples, performing an operation includes performing any action based on the voice command. Examples include, but are not limited to, turning a light on or off, raising or disarming an alarm, adjusting volume, performing a search, answering a query, etc. For example, such an operation may be performed based on processing of the voice command determined at block 710 by the voice UI device.

[0105] Figure 8 8 is a flow chart illustrating an example of a process 800 for speech recognition assisted by RF sensing according to examples described herein. The process 800 may be performed at least in part by, for example, Figure 1 The voice UI device 107 shown in and described above, Figure 2 The wireless device 200 shown in and described above, Figure 3 The voice UI device 302 shown in and described above, Figure 4 The voice UI device 402 and RF device 406 shown in and described above, Figure 5 The voice UI device 502 and RF device 508 shown in and described above, Figure 6 and / or the Voice UI device 602 shown in and described above Fig. 9 The present invention is performed by the computing device 900 shown in and described below.

[0106] At block 802, process 800 includes obtaining RF sensing data of a device at a voice user interface (UI), the RF sensing data including commands from a user. In some examples, there may be certain scenarios where a speaking entity may desire to issue voice commands that are inaudible to a voice UI device (e.g., silently, whispering, etc.). As an example, an environment in which a speaking entity is present may include a sleeping child, a companion watching a sports game, etc. In such a scenario, a speaking entity may desire to issue a command to a voice UI device without speaking the voice command (e.g., speaking the command by mouthing).

[0107] At block 804, process 800 includes processing the RF sensing data to determine a command output. In some examples, although no audible voice commands are issued by the speaking entity, the RF sensing data may be processed to determine an area (e.g., a mouth area) within the environment of the relevant portion of the speaking entity where features (e.g., tongue, lips, etc.) are present. The RF sensing data corresponding to such an area may be further processed to determine movement therein, which may then be processed to determine one or more commands issued by the speaking entity without actual speech.

[0108] At block 806, process 800 includes performing an operation at the voice UI device based on the command determined at block 804. In some examples, performing the operation includes performing any action based on the voice command. Examples include, but are not limited to, turning a light on or off, raising or disarming an alarm, adjusting volume, performing a search, answering a query, etc.

[0109] In some examples, process 700, process 800, or any other process described herein can be performed by a computing device or apparatus, and / or one or more components therein and / or to which the computing device is operably connected. As an example, process 700 and / or process 800 can be performed in whole or in part by Figure 1 The voice UI device 107 shown and described above, Figure 2 The wireless device 200 shown and described above, Figure 3 The voice UI device 302 shown and described above, Figure 4 The voice UI device 402 and RF device 406 shown and described above, Figure 5 The voice UI device 502 and RF device 508 shown and described above, Figure 6 The voice UI device 602 shown and described above, and / or Fig. 9 The present invention is performed by the computing device 900 shown and described below.

[0110] The voice UI device and / or RF device may include any suitable device, such as a vehicle or a computing device of the vehicle (e.g., a driver monitoring system (DMS) of the vehicle), a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, an AR glasses, a network-connected watch or smart watch, or other wearable device), a server computer, a robotic device, a television, a smart speaker, a voice assistant device, and / or any other computing device having resource capabilities to perform the processes described herein (including process 700) and / or other processes described herein. In some cases, a computing device or apparatus (e.g., a voice UI device) may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the operations of the processes described herein. In some examples, the computing device may include a display, a network interface configured to send and / or receive data, an RF sensing component, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive data based on an Internet Protocol (IP) or other types of data.

[0111] The components of the voice UI device and / or RF device may be implemented at least in part in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or may be implemented at least in part using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0112] Figure 7 The process 700 shown in Figure 8 The process 800 shown in is shown as a logical flow diagram, whose operations represent a series of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media, which, when executed by one or more processors, performs the operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0113] In addition, process 700, process 800, and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, by hardware, or a combination thereof. As described above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0114] Fig. 9 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. In particular, Fig. 9 An example of a computing system 900 is shown, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 905. Connection 905 may be a physical connection using a bus, or a direct connection into processor 910, such as in a chipset architecture. Connection 905 may also be a virtual connection, a networked connection, or a logical connection.

[0115] In some examples, computing system 900 is a distributed system, where the functionality described in the present disclosure can be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some examples, one or more of the described system components represent a number of such components, each of which performs some or all of the functionality for which the component is described. In some examples, a component can be a physical or virtual device.

[0116] The example system 900 includes at least one processing unit (CPU or processor) 910 and connections 805 coupling various system components including system memory 915, such as read-only memory (ROM) 920 and random access memory (RAM) 925, to the processor 910. The computing system 900 may include a cache 912 of high-speed memory directly connected to the processor 910, in close proximity to the processor 910, or integrated as part of the processor 910.

[0117] Processor 910 may include any general purpose processor and hardware services or software services, such as services 932, 934, and 936 stored in storage device 930, which are configured to control processor 910 as well as special purpose processors where the software instructions are incorporated into the actual processor design. Processor 910 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0118] To enable user interaction, the computing system 900 includes an input device 945, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, and the like. The computing system 900 can also include an output device 935, which can be one or more of a plurality of output mechanisms. In some instances, a multimodal system can enable a user to provide multiple types of input / output to communicate with the computing system 900. The computing system 900 can include a communication interface 940, which can generally govern and manage user input and system output. The communication interface can use a wired and / or wireless transceiver to perform or facilitate receiving and / or sending wired or wireless communications, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Proprietary wired ports / plugs, Wireless signal transmission, Low energy (BLE) wireless signal transmission, Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), global microwave access interoperability (WiMAX), infrared (IR) communication wireless signal transmission. Public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, special network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 940 may also include one or more global navigation satellite system (GNSS) receivers or transceivers, which are used to determine the location of the computing system 900 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There is no restriction to operate on any particular hardware arrangement, and thus the basic features herein may be easily replaced with improved hardware or firmware arrangements as they are developed.

[0119] The storage device 930 may be a non-volatile and / or non-transitory and / or computer-readable memory device of the device and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory ... card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM) card, a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0120] Storage device 930 may include software services, servers, services, etc., which cause the system to perform functions when the code defining such software is executed by processor 910. In some examples, hardware services that perform a particular function may include software components stored in a computer-readable medium that are combined with the necessary hardware components (such as processor 910, connection 905, output device 935, etc.) to perform the function.

[0121] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data can be stored and do not include carrier waves and / or temporary electronic signals that are propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, disks or tapes, optical storage media (such as compact disks (CDs) or digital versatile disks (DVDs)), flash memories, memories, or memory devices. Computer-readable media may store thereon codes and / or machine-executable instructions, which may represent any combination of processes, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Code segments may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or sent using any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0122] In some examples, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0123] Specific details are provided in the above description to provide a thorough understanding of these examples and the examples provided herein. However, it will be appreciated by those skilled in the art that examples can be put into practice without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, which include functional blocks including devices, device components, operations, steps or routines in methods implemented with software, hardware or a combination of hardware and software. In addition to those components shown in the accompanying drawings and / or described herein, additional components may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form, so as not to confuse examples with unnecessary details. In other instances, well-known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details, so as to avoid confusing examples.

[0124] Various examples may be described above as processes or methods depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. A process terminates when its operations are completed, but may have additional operations not included in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or a main function.

[0125] The processes and methods according to the above examples can be implemented using computer executable instructions stored in or otherwise available from computer readable media. Such instructions may include, for example, instructions and data that enable or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a specific function or group of functions. Parts of the computer resources used may be accessed over a network. Computer executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, etc.

[0126] The device implementing the processes and methods disclosed herein may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may adopt any of various form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. (Multiple) processors may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be embodied in peripheral devices or add-on cards. As another example, such functions may also be implemented between different chips or different processes executed in a single device on a circuit board.

[0127] The instructions, the media for carrying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functions described in this disclosure.

[0128] In the foregoing description, various aspects of the present application are described with reference to their specific examples, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative examples of the present application have been described in detail herein, it should be understood that the inventive concept may be implemented and adopted differently in other ways, and the appended claims are intended to be interpreted as including such variations, except for being limited by the prior art. The various features and aspects of the above-mentioned application may be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, the examples described herein may be used in any number of environments and applications other than the environments and applications described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For the purpose of illustration, the method is described in a specific order. It should be understood that in an alternative example, the method may be performed in an order different from the described order.

[0129] One of ordinary skill will understand that the less than ("<") and greater than (">") symbols ≤ or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0130] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform those operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform those operations, or any combination thereof.

[0131] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is in direct or indirect communication with another component (e.g., connected to another component via a wired or wireless connection, and / or other suitable communication interface).

[0132] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of a set or multiple members of a set (in any combination) satisfy the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0133] Various illustrative logic blocks, modules, circuits, and algorithmic operations described in conjunction with the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and operations have been generally described above with respect to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art may implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0134] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handset, or an integrated circuit device with multiple uses (including applications in wireless communication device handsets and other devices). Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions for executing one or more of the methods described above when executed. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (e.g., a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the techniques may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0135] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. This processor may be configured to perform any of the techniques described in the present invention. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for the implementation of the techniques described herein.

[0136] Illustrative aspects of the disclosure include:

[0137] Aspect 1: A method for speech recognition assisted by radio frequency (RF) sensing, the method comprising: obtaining audio data including a voice command from a speaking entity at a voice user interface (UI) device; obtaining RF sensing data corresponding to the audio data; processing the audio data to determine an audio voice command output; processing the RF sensing data to determine an RF sensing voice command output; determining the voice command based on the audio voice command output and the RF sensing voice command output; and performing an operation based on the voice command at the voice UI device.

[0138] Aspect 2: A method according to Aspect 1, wherein: the RF sensing voice command output includes a direction from the voice UI device to the speaking entity; and determining the voice command includes performing beamforming on the audio capture component of the voice UI device based on the direction.

[0139] Aspect 3: A method according to Aspect 1 or 2, wherein: the RF sensing voice command output includes the distance between the voice UI device and the speaking entity; and determining the voice command output includes adjusting the gain level of the audio capture component of the voice UI device based on the distance.

[0140] Aspect 4: A method according to any one of Aspects 1-3, wherein: the RF sensing voice command output includes speech characteristics of a speaking entity; and determining the voice command includes using the speech characteristics to enhance the speech recognition operation of the voice UI device.

[0141] Aspect 5: The method according to any one of aspects 1 to 4, wherein the RF sensing data includes depth map information for an environment including the speaking entity.

[0142] Aspect 6: A method according to any one of Aspects 1-5, wherein: the RF sensing data includes mouth region data corresponding to the mouth region of the speaking entity; and processing the RF sensing data includes processing depth map information to obtain feature information corresponding to the position of features in the mouth region.

[0143] Aspect 7: A method according to any one of Aspects 1-6, wherein the feature information corresponds at least in part to a tongue of a speaking entity.

[0144] Aspect 8: The method according to any one of Aspects 1 to 7, wherein the feature information corresponds at least partially to the lips of the speaking entity.

[0145] Aspect 9: The method according to any one of Aspects 1-8 further includes filtering the RF sensing data to obtain filtered RF sensing data before processing the RF sensing data, wherein the filtered RF sensing data includes mouth area data without other RF sensing environmental data from the environment.

[0146] Aspect 10: The method according to any one of aspects 1-9, wherein determining the voice command includes providing missing portions of the voice command to determine one or more operations to be performed.

[0147] Aspect 11: A method according to any one of Aspects 1 to 10, wherein: the RF sensing voice command output includes gesture data corresponding to a gesture made by the speaking entity; and determining the voice command includes using the gesture data and the audio voice command output to determine an operation to be performed.

[0148] Aspect 12: A method according to any one of aspects 1-11, wherein processing the RF sensing data includes providing the RF sensing data to a trained machine learning (ML) model to determine the RF sensing voice command output.

[0149] Aspect 13: The method according to any one of Aspects 1-12 further includes: before processing the RF sensing data, selecting the trained ML model from a plurality of trained ML models corresponding to different speech patterns.

[0150] Aspect 14: The method according to any one of Aspects 1-13, wherein the trained ML model is trained using a voice command dataset comprising a plurality of voice command keywords.

[0151] Aspect 15: The method according to any one of Aspects 1 to 14 further includes: before obtaining the RF sensing data, sending an RF signal toward an environment including a speaking entity, wherein the RF signal is sent by an RF sensing component, and wherein the RF sensing data is based on one or more reflections of the transmitted RF signal from the speaking entity.

[0152] Aspect 16: The method according to any one of aspects 1 to 15, wherein the speaking entity is obscured from the perspective of the RF sensing component.

[0153] Aspect 17: A method according to any one of aspects 1-16, wherein the voice UI device includes an RF sensing component.

[0154] Aspect 18: The method according to any one of Aspects 1 to 17, further comprising: obtaining additional RF sensing data, wherein the additional RF sensing data is obtained when the speaking entity does not emit sound audible to the voice UI device; processing the RF sensing data to obtain depth map information of an environment including the speaking entity, wherein the depth map information includes mouth region data corresponding to the mouth region of the speaking entity; processing the mouth region data to obtain feature information corresponding to the position of features in the mouth region; and performing a second operation by the voice UI device based on the feature information.

[0155] Aspect 19: A method according to any one of Aspects 1 to 18, wherein the RF sensing data includes depth map information, and wherein processing the RF sensing data includes: using two-dimensional data to determine the location of features in mouth area data corresponding to the mouth area of ​​the speaking entity; and identifying the location of the features in the depth map information.

[0156] Aspect 20: The method according to any one of aspects 1-19, wherein the two-dimensional data is obtained by flattening the depth map information.

[0157] Aspect 21: The method according to any one of aspects 1-20, wherein the two-dimensional data is obtained from a camera.

[0158] Aspect 22: A method according to any one of Aspects 1-21, wherein processing RF sensing data includes: performing initial processing to determine a depth range of interest; filtering the RF sensing data to exclude data outside the depth range of interest and obtain filtered RF sensing data; and providing the filtered RF sensing data to a trained machine learning (ML) model to obtain an RF sensing voice command output.

[0159] Aspect 23: An apparatus for speech recognition assisted by radio frequency (RF) sensing, the apparatus comprising: a memory device; and a processor coupled to the memory device and configured to: obtain audio data including a voice command from a speaking entity at a voice user interface (UI) device; obtain RF sensing data corresponding to the audio data; process the audio data to determine an audio voice command output; process the RF sensing data to determine an RF sensing voice command output; determine the voice command based on the audio voice command output and the RF sensing voice command output; and perform operations based on the voice command at the voice UI device.

[0160] Aspect 24: An apparatus according to Aspect 23, wherein: the RF sensing voice command output includes a direction from the voice UI device to the speaking entity, and the processor is further configured to: determine that the voice command includes performing beamforming on the audio capture component of the voice UI device based on the direction.

[0161] Aspect 25: An apparatus according to Aspect 23 or 24, wherein: the RF sensing voice command output includes the distance between the voice UI device and the speaking entity, and the processor is further configured to: determine that the voice command output includes adjusting the gain level of the audio capture component of the voice UI device based on the distance.

[0162] Aspect 26: An apparatus according to any of Aspects 23-25, wherein: the RF sensing voice command output includes speech characteristics of the speaking entity, and the processor is further configured to: determine that the voice command includes using the speech characteristics to enhance the speech recognition operation of the voice UI device.

[0163] Aspect 27: An apparatus according to any one of Aspects 23 to 26, wherein the RF sensing data includes depth map information of an environment including a speaking entity.

[0164] Aspect 28: An apparatus according to any one of Aspects 23 to 27, wherein: the RF sensing data includes mouth region data corresponding to the mouth region of the speaking entity, and the processor is further configured to: process the RF sensing data by processing the depth map information to obtain feature information corresponding to the position of features in the mouth region.

[0165] Aspect 29: An apparatus according to any one of Aspects 23 to 28, wherein the feature information corresponds at least in part to a tongue of the speaking entity.

[0166] Aspect 30: An apparatus according to any one of Aspects 23 to 29, wherein the feature information corresponds at least in part to the lips of the speaking entity.

[0167] Aspect 31: An apparatus according to any one of Aspects 23-30, wherein the processor is further configured to filter the RF sensing data before processing the RF sensing data to obtain filtered RF sensing data, wherein the filtered RF sensing data includes mouth area data without other RF sensing environmental data from the environment.

[0168] Aspect 32: An apparatus according to any one of Aspects 23-31, wherein the processor is further configured to determine the voice command by providing missing portions of the voice command in order to determine one or more operations to be performed.

[0169] Aspect 33: An apparatus according to any of Aspects 23-32, wherein: the RF sensing voice command output includes gesture data corresponding to a gesture made by the speaking entity, and the processor is further configured to determine the voice command by using the gesture data and the audio voice command output to determine an operation to be performed.

[0170] Aspect 34: An apparatus according to any one of Aspects 23-33, wherein, in order to process the RF sensing data, the processor is further configured to provide the RF sensing data to a trained machine learning (ML) model to determine the RF sensing voice command output.

[0171] Aspect 35: An apparatus according to any one of Aspects 23-34, wherein the processor is further configured to select a trained ML model from a plurality of trained ML models corresponding to different speech patterns before processing the RF sensing data.

[0172] Aspect 36: An apparatus according to any one of Aspects 23-35, wherein the trained ML model is trained using a voice command dataset comprising a plurality of voice command keywords.

[0173] Aspect 37: An apparatus according to any one of Aspects 23 to 36, wherein the processor is further configured to send an RF signal to an environment including a speaking entity before obtaining the RF sensing data, wherein the RF signal is sent by the RF sensing component, and wherein the RF sensing data is based on one or more reflections of the transmitted RF signal from the speaking entity.

[0174] Aspect 38: An apparatus according to any one of aspects 23 to 37, wherein the speaking entity is obscured from the perspective of the RF sensing component.

[0175] Aspect 39: An apparatus according to any one of Aspects 23-38, wherein the voice UI device includes an RF sensing component.

[0176] Aspect 40: An apparatus according to any one of Aspects 23-39, wherein the processor is further configured to: obtain additional RF sensing data, wherein the additional RF sensing data is obtained when a speaking entity does not emit sound audible to a voice UI device; process the RF sensing data to obtain depth map information of an environment including the speaking entity, wherein the depth map information includes mouth region data corresponding to a mouth region of the speaking entity; process the mouth region data to obtain feature information corresponding to a position of a feature in the mouth region; and perform a second operation based on the feature information.

[0177] Aspect 41: An apparatus according to any one of Aspects 23 to 40, wherein the RF sensing data includes depth map information, and wherein, in order to process the RF sensing data, the processor is further configured to: use the two-dimensional data to determine the location of features in the mouth area data corresponding to the mouth area of ​​the speaking entity; and identify the location of the features in the depth map information.

[0178] Aspect 42: The apparatus according to any one of Aspects 23 to 41, wherein the two-dimensional data is obtained by flattening the depth map information.

[0179] Clause 43: An apparatus according to any one of Clauses 23-42, wherein the two-dimensional data is obtained from a camera.

[0180] Aspect 44: An apparatus according to any one of Aspects 23-43, wherein, in order to process RF sensing data, the processor is further configured to: perform initial processing to determine a depth range of interest; filter the RF sensing data to exclude data outside the depth range of interest and obtain filtered RF sensing data; and provide the filtered RF sensing data to a trained machine learning (ML) model to obtain an RF sensing voice command output.

[0181] Aspect 45: A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform the operations according to any one of aspects 1 to 22.

[0182] Aspect 46: An apparatus for speech recognition assisted by radio frequency (RF) sensing, comprising one or more components for performing the operations according to any one of aspects 1 to 22.

Claims

1. A method for speech recognition assisted by radio frequency (RF) sensing, comprising: obtaining, at a voice user interface (UI) device, audio data including a voice command from a speaking entity; obtaining RF sensing data corresponding to the audio data; processing the audio data to determine an audio voice command output; processing the RF sensing data to determine an RF sensing voice command output; determining the voice command based on the audio voice command output and the RF sensing voice command output; as well as An operation is performed at the voice UI device based on the voice command.

2. The method according to claim 1, wherein: The RF sensing voice command output includes a direction from the voice UI device to the speaking entity; and Determining the voice command includes performing beamforming on an audio capture component of the voice UI device based on the direction.

3. The method according to claim 1, wherein: The RF sensing voice command output includes a distance between the voice UI device and the speaking entity; and Determining the voice command includes adjusting a gain level of an audio capture component of the voice UI device based on the distance.

4. The method according to claim 1, wherein: The RF sensed voice command output includes speech characteristics of the speaking entity; and Determining the voice command includes using the speech characteristics to enhance a speech recognition operation of the voice UI device.

5. The method according to claim 1, wherein: The RF sensing data includes depth map information for an environment including the speaking entity.

6. The method according to claim 5, wherein: The RF sensing data includes mouth region data corresponding to a mouth region of the speaking entity; and Processing the RF sensing data includes processing the depth map information to obtain feature information corresponding to locations of features in the mouth region.

7. The method according to claim 6, wherein: The characteristic information corresponds at least in part to a tongue of the speaking entity.

8. The method according to claim 6, wherein: The feature information corresponds at least in part to lips of the speaking entity.

9. The method according to claim 6, further comprising: Prior to processing the RF sensing data, the RF sensing data is filtered to obtain filtered RF sensing data, wherein the filtered RF sensing data includes the mouth region data without other RF sensing environmental data from the environment.

10. The method according to claim 1, wherein: Determining the voice command includes providing missing portions of the voice command to determine one or more operations to be performed.

11. The method according to claim 1, wherein: the RF sensing voice command output includes gesture data corresponding to a gesture made by the speaking entity; as well as Determining the voice command includes using the gesture data and the audio voice command output to determine an operation to be performed.

12. The method according to claim 1, wherein: Processing the RF sensing data includes providing the RF sensing data to a trained machine learning (ML) model to determine the RF sensing voice command output.

13. The method according to claim 12, further comprising: Prior to processing the RF sensing data, the trained ML model is selected from a plurality of trained ML models corresponding to different speech patterns.

14. The method according to claim 12, wherein: The trained ML model is trained using a voice command dataset including a plurality of voice command keywords.

15. The method according to claim 1, further comprising: Prior to obtaining the RF sensing data, an RF signal is transmitted toward an environment including the speaking entity, wherein the RF signal is transmitted by an RF sensing component, and wherein the RF sensing data is based on one or more reflections of the transmitted RF signal from the speaking entity.

16. The method according to claim 15, wherein: From the perspective of the RF sensing component, the speaking entity is obscured.

17. The method according to claim 15, wherein: The voice UI device includes the RF sensing component.

18. The method of claim 1, further comprising: obtaining additional RF sensing data, wherein the additional RF sensing data is obtained when the speaking entity does not emit sound audible to the voice UI device; processing the RF sensing data to obtain depth map information of an environment including the speaking entity, wherein the depth map information includes mouth region data corresponding to a mouth region of the speaking entity; processing the mouth region data to obtain feature information corresponding to locations of features in the mouth region; and The voice UI device performs a second operation based on the feature information.

19. The method of claim 1, wherein the RF sensing data includes depth map information, and wherein processing the RF sensing data comprises: using the two-dimensional data to determine a location of a feature in the mouth region data corresponding to a mouth region of the speaking entity; as well as The location of the feature in the depth map information is identified.

20. The method according to claim 19, wherein: The two-dimensional data is obtained by flattening the depth map information.

21. The method according to claim 19, wherein: The two-dimensional data is obtained from a camera.

22. The method of claim 1, wherein processing the RF sensing data comprises: Perform initial processing to determine the depth range of interest; filtering the RF sensing data to exclude data outside the depth range of interest and obtain filtered RF sensing data; as well as The filtered RF sensing data is provided to a trained machine learning (ML) model to obtain the RF sensing voice command output.

23. An apparatus for speech recognition aided by radio frequency (RF) sensing, the apparatus comprising: at least one memory; as well as at least one processor coupled to the at least one memory and configured to: obtaining, via a voice user interface (UI) device, audio data including a voice command from a speaking entity; obtaining RF sensing data corresponding to the audio data; processing the audio data to determine an audio voice command output; processing the RF sensing data to determine an RF sensing voice command output; determining the voice command based on the audio voice command output and the RF sensing voice command output; as well as An operation is performed at the voice UI device based on the voice command.

24. The apparatus of claim 23, wherein: The RF sensing voice command output includes a direction from the voice UI device to the speaking entity; and The at least one processor is further configured to determine the voice command, including performing beamforming on an audio capture component of the voice UI device based on the direction.

25. The apparatus of claim 23, wherein: The RF sensing voice command output includes a distance between the voice UI device and the speaking entity; and The at least one processor is further configured to determine the voice command, including adjusting a gain level of an audio capture component of the voice UI device based on the distance.

26. The apparatus of claim 23, wherein: The RF sensed voice command output includes speech characteristics of the speaking entity; and The at least one processor is further configured to determine the voice command, including using the speech characteristics to enhance a speech recognition operation of the voice UI device.

27. The device according to claim 23, wherein: The RF sensing data includes depth map information for an environment including the speaking entity.

28. The apparatus of claim 27, wherein: The RF sensing data includes mouth region data corresponding to a mouth region of the speaking entity; and The at least one processor is further configured to process the RF sensing data by processing the depth map information to obtain feature information corresponding to locations of features in the mouth region.

29. The device according to claim 28, wherein The characteristic information corresponds at least in part to a tongue of the speaking entity.

30. The device according to claim 28, wherein The feature information corresponds at least in part to lips of the speaking entity.

31. The apparatus of claim 28, wherein the at least one processor is further configured to, before processing the RF sensing data, filter the RF sensing data to obtain filtered RF sensing data, wherein the filtered RF sensing data includes the mouth region data without other RF sensing environmental data from the environment.

32. The device according to claim 23, wherein: The at least one processor is further configured to determine the voice command by providing missing portions of the voice command to determine one or more operations to be performed.

33. The apparatus of claim 23, wherein: The RF sensing voice command output includes gesture data corresponding to a gesture made by the speaking entity; and The at least one processor is further configured to determine a voice command by determining an operation to be performed using the gesture data and the audio voice command output.

34. The apparatus of claim 23, wherein: To process the RF sensing data, the at least one processor is further configured to provide the RF sensing data to a trained machine learning (ML) model to determine the RF sensing voice command output.

35. The device according to claim 34, wherein The at least one processor is further configured to select a trained ML model from a plurality of trained ML models corresponding to different speech patterns before processing the RF sensing data.

36. The device according to claim 34, wherein The trained ML model is trained using a voice command dataset including a plurality of voice command keywords.

37. The apparatus of claim 23, wherein: The at least one processor is also configured to, before obtaining the RF sensing data, send an RF signal toward an environment including the speaking entity, wherein the RF signal is sent by an RF sensing component, and wherein the RF sensing data is based on one or more reflections of the sent RF signal from the speaking entity.

38. The device according to claim 37, wherein From the perspective of the RF sensing component, the speaking entity is obscured.

39. The device according to claim 37, wherein: The voice UI device includes the RF sensing component.

40. The apparatus of claim 23, wherein: The at least one processor is further configured to: obtaining additional RF sensing data, wherein the additional RF sensing data is obtained when the speaking entity does not emit sound audible to the voice UI device; processing the RF sensing data to obtain depth map information of an environment including the speaking entity, wherein the depth map information includes mouth region data corresponding to a mouth region of the speaking entity; processing the mouth region data to obtain feature information corresponding to locations of features in the mouth region; and A second operation is performed based on the characteristic information.

41. The apparatus of claim 23, wherein: The RF sensing data includes depth map information; and In order to process the RF sensing data, the processor is further configured to: Using the two-dimensional data, determining a location of a feature in the mouth region data corresponding to a mouth region of the speaking entity; and The location of the feature in the depth map information is identified.

42. The device according to claim 41, wherein The two-dimensional data is obtained by flattening the depth map information.

43. The device according to claim 41, wherein The two-dimensional data is obtained from a camera.

44. The apparatus of claim 23, wherein: In order to process the RF sensing data, the processor is further configured to: Perform initial processing to determine the depth range of interest; filtering the RF sensing data to exclude data outside the depth range of interest and obtain filtered RF sensing data; as well as The filtered RF sensing data is provided to a trained machine learning (ML) model to obtain the RF sensing voice command output.

Images