Smart device-based radar system using spatiotemporal neural network for gesture recognition

Abstract

Techniques and apparatus are described to implement a smart device-based radar system capable of performing gesture recognition using a spatiotemporal neural network (222). The spatiotemporal neural network (222) employs machine learning to recognize a user's gesture based on composite radar data. The spatiotemporal neural network (222) is implemented using a multi-level machine learning architecture, which enables the radar system (102) to save power and recognize the user's gesture in real time (e.g., as the gesture is performed). The spatiotemporal neural network (222) is also adaptable and can be extended to recognize multiple types of gestures, such as sweeping gestures and reaching gestures, without significantly increasing size, computational requirements, or latency.

Description

Background Art

[0001] Radar is a useful device that can detect objects. Relative to other types of sensors, such as cameras, radar can provide improved performance in the presence of different environmental conditions, such as low lighting and fogging, or with moving or overlapping objects. Radar can also detect objects through one or more obstructions, such as a wallet or pocket. Although radar has many advantages, there are many challenges associated with integrating radar in electronic devices.

[0002] One challenge involves power constraints within small or mobile electronic devices. The operation of some radars significantly drains the battery of the electronic device and causes the user to frequently recharge the electronic device. Therefore, the advantages of utilizing radar may not be realized in situations where the effective operation of the radar is curtailed or disabled due to limitations on available power.

[0003] Another challenge involves the constraints that small consumer devices may impose on the design or operation of the radar. To meet size or layout constraints, for example, fewer antenna elements and various antenna element spacings may be used. Other constraints may limit the bandwidth, transmission power, or update rate of the radar signal. As a result, the design of the radar may result in reduced signal-to-noise performance, which may make it challenging to achieve sufficient accuracy for some applications. As a result, the performance of the radar integrated within an electronic device may be significantly reduced. This may limit the types of applications that the radar can support or the types of electronic devices that can be incorporated with the radar. Summary of the invention

[0004] Techniques and apparatus are described for implementing a smart device-based radar system capable of performing gesture recognition using a spatiotemporal neural network. The spatiotemporal neural network employs machine learning to analyze composite radar data (e.g., both amplitude and phase information) for gesture recognition. By analyzing both amplitude and phase information, the spatiotemporal neural network can achieve improved accuracy in detecting distributed objects and recognizing gestures relative to other signal processing techniques, heuristic techniques, or machine learning techniques that process amplitude information.

[0005] The spatiotemporal neural network is implemented using a multi-level machine learning architecture. In the first level, a spatial recurrent network analyzes the composite radar data in the spatial domain to generate feature data. The feature data identifies one or more features (e.g., characteristics) of a gesture performed by a user. The feature data is stored in a memory element of a circular buffer for at least a portion of the time. Over time, additional feature data is stored in other memory elements of the circular buffer. In the second level, a temporal recurrent network analyzes the feature data in the temporal domain to recognize the gesture.

[0006] This multi-stage design enables the radar system to save power and recognize gestures in real time (e.g., as the gestures are performed). For example, the use of a circular buffer enables the radar system to save power and memory by alleviating the need to regenerate feature data or store composite radar data. Storing feature data instead of composite radar data also reduces the computation time used to recognize gestures. The spatiotemporal neural network is also adaptable and can be extended to recognize multiple types of gestures, such as swipe gestures and hand gestures, without significantly increasing size, computational requirements, or latency.

[0007] Aspects described below include a method for gesture recognition performed by a radar system. The method includes transmitting a radar transmit signal using an antenna array of the radar system. The method also includes receiving a radar receive signal using the antenna array. The radar receive signal includes a version of the radar transmit signal reflected by at least one user. The method also includes generating composite radar data based on the radar receive signal and providing the composite radar data to a spatiotemporal neural network of the radar system. The spatiotemporal neural network includes a multi-level machine learning architecture. The method further includes analyzing the composite radar data using the spatiotemporal neural network to recognize a gesture performed by at least one user.

[0008] The aspects described below also include an apparatus including a radar system. The radar system includes an antenna array and a transceiver. The radar system also includes a processor and a computer-readable storage medium configured to perform any of the described methods.

[0009] Aspects described below include a computer-readable storage medium including computer-executable instructions that, in response to being executed by a processor, implement a spatiotemporal neural network configured to receive composite radar data associated with a radar receive signal reflected by at least one object. The spatiotemporal neural network is further configured to analyze the composite radar data in a spatial domain using machine learning to generate feature data. Additionally, the spatiotemporal neural network is configured to analyze the feature data in a temporal domain using machine learning to generate radar application data. The spatiotemporal neural network is configured to deliver the radar application data to a radar-based application.

[0010] Aspects described below also include a system having a spatiotemporal neural network device for performing gesture recognition based on composite radar data. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Apparatus and techniques for implementing a smart device-based radar system capable of performing gesture recognition using a spatiotemporal neural network are described with reference to the following figures. The same reference numerals are used throughout the figures to refer to the same features and components:

[0012] Figure 1An example environment is illustrated in which a smart device-based radar system capable of performing gesture recognition using a spatiotemporal neural network may be implemented.

[0013] Figure 2 An example implementation of a radar system as part of a smart device is illustrated.

[0014] Figure 3-1 The operation of an example radar system is illustrated.

[0015] Figure 3-2 An example radar frame structure for gesture recognition is illustrated.

[0016] Figure 4 An example antenna array for a radar system and an example transceiver for a radar system are illustrated.

[0017] Figure 5 An example scheme implemented by a radar system for performing gesture recognition using a spatiotemporal neural network is illustrated.

[0018] Figure 6-1 An example hardware abstraction module for gesture recognition is illustrated.

[0019] Figure 6-2 Illustrated is example hardware-independent composite radar data generated by a hardware abstraction module for gesture recognition.

[0020] Figure 7-1 An example spatiotemporal neural network for performing gesture recognition is illustrated.

[0021] Figure 7-2 An example chirp-level analysis module of a spatiotemporal neural network is illustrated.

[0022] Figure 7-3 Illustration of an example feature-level analysis module of a spatiotemporal neural network.

[0023] Figure 7-4 An example application-level analysis module of a spatiotemporal neural network is illustrated.

[0024] Figure 8 An example method for performing operation of a smart device-based radar system capable of performing gesture recognition using a spatiotemporal neural network is illustrated.

[0025] Fig. 9 An example computing system is illustrated that embodies or can implement techniques that can use a radar system that can perform gesture recognition using a spatiotemporal neural network. DETAILED DESCRIPTION

[0026] Overview

[0027] Integrating radar systems within electronic devices can be challenging. For example, electronic devices may have limited available power. As a result, the radar system may not be able to increase transmit power or utilize higher update rates. In some cases, size or layout constraints of the electronic devices may limit the number of antenna elements or result in suboptimal antenna element spacing, which may limit angular resolution. Other hardware limitations or frequency restrictions may limit the bandwidth of the radar signal, which may limit the range resolution of radars using pulse compression techniques. Due to these limitations, achieving target accuracy for some radars may be challenging.

[0028] Target accuracy can also be challenging for detecting distributed objects with non-deterministic radar signatures. Example types of distributed objects include human body parts (e.g., fingers, hands, faces, or appendages) or human bodies. From the radar's perspective, distributed objects have multiple phase centers (e.g., scattering points), which causes the radar signature of the distributed objects to vary at different observation angles. When observing a distributed object, the observation angle may change due to the movement of the distributed object or the movement of the radar. As the observation angle changes, the amplitude or phase of the radar signature may change, which increases the error or uncertainty of the position data generated using radar processing techniques suitable for non-distributed objects (e.g., objects with deterministic radar signatures).

[0029] To address these issues, this document describes techniques and devices for implementing a smart device-based radar system capable of performing gesture recognition using a spatiotemporal neural network. The spatiotemporal neural network employs machine learning to analyze composite radar data (e.g., amplitude and phase information) for gesture recognition. By analyzing both amplitude and phase information, the spatiotemporal neural network can achieve improved accuracy for detecting distributed objects and recognizing gestures relative to other signal processing techniques, heuristic techniques, or machine learning techniques that process amplitude information. Unless otherwise specified in the specific context of this document, increased accuracy refers to increased refinement, increased consistency with reality, or both increased refinement and increased consistency with reality.

[0030] The spatiotemporal neural network is implemented using a multi-level machine learning architecture. In the first level, a spatial recurrent network analyzes the composite radar data in the spatial domain to generate feature data. The feature data identifies one or more features (e.g., characteristics) of a gesture performed by a user. The feature data is stored in a memory element of a circular buffer for at least a portion of the time. Over time, additional feature data is stored in other memory elements of the circular buffer. In the second level, a temporal recurrent network analyzes the feature data in the temporal domain to recognize the gesture.

[0031] This multi-stage design enables the radar system to save power and recognize gestures in real time (e.g., as the gestures are performed). For example, the use of a circular buffer enables the radar system to save power and memory by alleviating the need to regenerate feature data or store composite radar data. Storing feature data instead of composite radar data also reduces the computation time used to recognize gestures. The spatiotemporal neural network is also adaptable and can be extended to recognize multiple types of gestures, such as sweeping gestures and reaching gestures, without significantly increasing size, computational requirements, or latency.

[0032] Example Environment

[0033] Figure 1 1 is an illustration of example environments 100-1 to 100-6 in which techniques for using a smart device-based radar system and apparatus including a smart device-based radar system capable of performing gesture recognition using a spatiotemporal neural network may be embodied. In the depicted environments 100-1 to 100-6, smart device 104 includes a smart device capable of using ( Figure 2 ) spatiotemporal neural network to detect one or more objects (e.g., users) radar system 102. Smart devices 104 are shown as smart phones in environments 100-1 to 100-5 and smart vehicles in environment 100-6.

[0034] In environments 100-1 to 100-4, users perform different types of gestures that are detected by radar system 102. In some cases, users perform gestures using appendages or body parts. Alternatively, users may also perform gestures using a stylus, a handheld object, a ring, or any type of material that can reflect radar signals.

[0035] In environment 100-1, a user makes a rolling gesture by moving a hand over the smart device 104 along a horizontal dimension (e.g., from the left side of the smart device 104 to the right side of the smart device 104). In environment 100-2, a user makes a reaching gesture, which reduces the distance between the smart device 104 and the user's hand. A user in environment 100-3 makes a gesture to play a game on the smart device 104. In one case, a user makes a pushing gesture by moving a hand over the smart device 104 along a vertical dimension (e.g., from the bottom of the smart device 104 to the top of the smart device 104). In environment 100-4, the smart device 104 is stored in a wallet, and the radar system 102 provides occlusion gesture recognition by detecting gestures that are occluded by the wallet.

[0036] Radar system 102 can also identify Figure 1Other types of gestures or motions not shown in the figure. Example types of gestures include handle turning gestures, in which the user curls their fingers to grip an imaginary door handle and rotates their fingers and hand in a clockwise or counterclockwise manner to simulate the action of turning an imaginary door handle. Another example type of gesture includes an axis twisting gesture, which the user performs by rubbing the thumb and at least another finger together. The gesture can be two-dimensional, such as gestures used with a touch-sensitive display (e.g., two-finger pinch, two-finger spread, or tap). The gesture can also be three-dimensional, such as many sign language gestures, for example, gestures of American Sign Language (ASL) and other sign languages ​​around the world. Once each of these gestures is detected, the smart device 104 can perform actions such as displaying new content, moving a cursor, activating one or more sensors, opening an application, etc. In this way, the radar system 102 provides contactless control of the smart device 104.

[0037] In environment 100-5, radar system 102 generates a three-dimensional map of the surrounding environment for contextual awareness. Radar system 102 also detects and tracks multiple users so that both users can interact with smart device 104. Radar system 102 can also perform vital sign detection. In environment 100-6, radar system 102 monitors the vital signs of the user driving the vehicle. Example vital signs include heart rate and breathing rate. For example, if radar system 102 determines that the driver is falling asleep, radar system 102 can cause smart device 104 to warn the user. Alternatively, if radar system 102 detects a life-threatening emergency, such as a heart attack, radar system 102 can cause smart device 104 to alert a medical professional or emergency service.

[0038] Some implementations of the radar system 102 are particularly advantageous when applied in the context of a smart device 104, whereby there is a convergence of issues. This may include the need for spacing and layout of the radar system 102 and low power constraints. An exemplary overall lateral dimension of the smart device 104 may be, for example, approximately eight centimeters by approximately fifteen centimeters. An exemplary footprint of the radar system 102 may be even more limited, such as approximately four millimeters by six millimeters including the antenna. An exemplary power consumption of the radar system 102 may be on the order of a few milliwatts to tens of milliwatts (e.g., between approximately two milliwatts and twenty milliwatts). This limited footprint and power consumption requirement of the radar system 102 enables the smart device 104 to include other desired parts (e.g., camera sensors, fingerprint sensors, displays, etc.) in a space-constrained package. Reference Figure 2 Smart device 104 and radar system 102 are further described.

[0039] Figure 2The radar system 102 is illustrated as part of a smart device 104. The smart device 104 is illustrated with various non-limiting example devices including a desktop computer 104-1, a tablet computer 104-2, a laptop computer 104-3, a television 104-4, a computing watch 104-5, computing glasses 104-6, a gaming system 104-7, a microwave oven 104-8, and a vehicle 104-9. Other devices may also be used, such as home service devices, smart speakers, smart thermostats, security cameras, baby monitors, Wi-Fi TM routers, drones, trackpads, drawing tablets, netbooks, e-readers, home automation and control systems, wall displays, and other home appliances. Note that the smart device 104 can be wearable, non-wearable but mobile, or relatively immobile (e.g., desktops and appliances). The radar system 102 can be used as a standalone radar system, or used with or embedded in many different smart devices 104 or peripherals, such as embedded in control panels that control home appliances and systems, embedded in cars to control internal functions (e.g., volume, cruise control, or even car driving), or used as an accessory to a laptop to control computing applications on the laptop.

[0040] The smart device 104 includes one or more computer processors 202 and one or more computer-readable media 204, including memory media and storage media. Applications and / or operating systems (not shown) embodied as computer-readable instructions on the computer-readable media 204 can be executed by the computer processor 202 to provide some of the functionality described herein. The computer-readable media 204 also includes radar-based applications 206 that use radar data generated by the radar system 102 to perform functions such as presence detection, gesture-based contactless control, collision avoidance for autonomous driving, human vital sign notification, etc.

[0041] The smart device 104 may also include a network interface 208 for communicating data via a wired, wireless, or optical network. For example, the network interface 208 may communicate data via a local area network (LAN), a wireless local area network (WLAN), a personal area network (PAN), a wired local area network (WAN), an intranet, the Internet, a peer-to-peer network, a point-to-point network, a mesh network, etc. The smart device 104 may also include a display (not shown).

[0042] Radar system 102 includes a communication interface 210 for transmitting radar data to a remote device, although this interface need not be used when radar system 102 is integrated within smart device 104. Typically, the radar data provided by communication interface 210 is in a format usable by radar-based application 206.

[0043] The radar system 102 also includes at least one antenna array 212 and at least one transceiver 214 to transmit and receive radar signals. The antenna array 212 includes at least one transmit antenna element and at least two receive antenna elements. In some cases, the antenna array 212 includes multiple transmit antenna elements to implement a multiple input multiple output (MIMO) radar capable of transmitting multiple different waveforms at a given time (e.g., each transmit antenna element transmits a different waveform). The antenna elements can be circularly polarized, horizontally polarized, vertically polarized, or a combination thereof.

[0044] The receive antenna elements of antenna array 212 may be positioned in a one-dimensional shape (e.g., a line) or a two-dimensional shape (e.g., a rectangular arrangement, a triangular arrangement, or an "L" shaped arrangement) for implementations including three or more receive antenna elements. A one-dimensional shape enables radar system 102 to measure one angular dimension (e.g., azimuth or elevation), while a two-dimensional shape enables radar system 102 to measure two angular dimensions (e.g., to determine the azimuth and elevation of an object). The element spacing associated with the receive antenna elements may be less than, greater than, or equal to half the center wavelength of the radar signal.

[0045] Using antenna array 212, radar system 102 can form a beam that is directed or undirected, wide or narrow, or shaped (e.g., hemispherical, cubic, fan-shaped, conical, cylindrical). Steering and shaping can be achieved through analog beamforming or digital beamforming. One or more transmitting antenna elements can have, for example, an undirected omnidirectional radiation pattern, or can produce a wide steerable beam to illuminate a large volume of space. In order to achieve target angular accuracy and angular resolution, receiving antenna elements can be used to generate hundreds or thousands of narrow steerable beams using digital beamforming. In this way, radar system 102 can effectively monitor the external environment and detect one or more users.

[0046] The transceiver 214 includes circuits and logic for transmitting and receiving radar signals via the antenna array 212. The components of the transceiver 214 may include amplifiers, mixers, switches, analog-to-digital converters, or filters for adjusting radar signals. The transceiver 214 also includes logic for performing in-phase / quadrature (I / Q) operations such as modulation or demodulation. Various modulations may be used, including linear frequency modulation, triangular frequency modulation, step frequency modulation, or phase modulation. Alternatively, the transceiver 214 may generate a radar signal with a relatively constant frequency or a single tone. The transceiver 214 may be configured to support continuous wave or pulse radar operations.

[0047] The spectrum (e.g., frequency range) used by the transceiver 214 to generate the radar signal may include frequencies between 1 and 400 gigahertz (GHz), between 4 and 100 GHz, between 1 and 24 GHz, between 2 and 4 GHz, between 57 and 64 GHz, or at approximately 2.4 GHz. In some cases, the spectrum may be divided into multiple sub-spectra with similar or different bandwidths. The bandwidth may be on the order of 500 megahertz (MHz), 1 GHz, 2 GHz, etc. Different frequency sub-spectra may include, for example, frequencies between approximately 57 and 59 GHz, 59 and 61 GHz, or 61 and 63 GHz. Although the above example frequency sub-spectra are continuous, other frequency sub-spectra may not be continuous. In order to achieve coherence, the transceiver 214 may use multiple frequency sub-spectra (continuous or discontinuous) with the same bandwidth to generate multiple radar signals, which are transmitted simultaneously or separated in time. In some cases, multiple continuous frequency sub-spectra may be used to transmit a single radar signal so that the radar signal has a wide bandwidth.

[0048] The radar system 102 also includes one or more system processors 216 and one or more system media 218 (e.g., one or more computer-readable storage media). The system media 218 optionally includes a hardware abstraction module 220. The system media 218 also includes at least one spatiotemporal neural network 222 and at least one circular buffer 224. The hardware abstraction module 220, the spatiotemporal neural network 222, and the circular buffer 224 can be implemented using hardware, software, firmware, or a combination thereof. In this example, the system processor 216 implements the hardware abstraction module 220 and the spatiotemporal neural network 222. The system processor 216 or a memory controller can implement and manage the circular buffer 224. The hardware abstraction module 220, the spatiotemporal neural network 222, and the circular buffer 224 together enable the system processor 216 to process responses from the receive antenna elements in the antenna array 212 to detect users, determine the location of objects, and / or recognize gestures performed by the user 302.

[0049] In an alternative implementation (not shown), the hardware abstraction module 220, the spatiotemporal neural network 222, or the circular buffer 224 are included within the computer-readable medium 204 and implemented by the computer processor 202. This enables the radar system 102 to provide raw data to the smart device 104 via the communication interface 210 so that the computer processor 202 can process the raw data for the radar-based application 206.

[0050] Hardware abstraction module 220 transforms the raw data provided by transceiver 214 into hardware-independent composite radar data that can be processed by spatiotemporal neural network 222. In particular, hardware abstraction module 220 conforms composite radar data from a variety of different types of radar signals to the expected inputs of spatiotemporal neural network 222. This enables spatiotemporal neural network 222 to process different types of radar signals received by radar system 102, including those utilizing different modulation schemes for frequency modulated continuous wave radar, phase modulated spread spectrum radar, or pulse radar. Hardware abstraction module 220 can also normalize composite radar data from radar signals having different center frequencies, bandwidths, transmit power levels, or pulse widths.

[0051] Additionally, hardware abstraction module 220 accommodates composite radar data generated using different hardware architectures. Different hardware architectures may include different antenna arrays 212 or different sets of antenna elements within antenna array 212 located on different surfaces of smart device 104. By using hardware abstraction module 220, spatiotemporal neural network 222 may process composite radar data generated by different sets of antenna elements with different gains, different sets of antenna elements with various numbers, or different sets of antenna elements with different antenna element spacings.

[0052] By using the hardware abstraction module 220, the spatiotemporal neural network 222 can operate in radar systems 102 having different limitations that affect the available radar modulation schemes, transmission parameters, or types of hardware architectures. Figure 6-1 to Figure 6-2 The hardware abstraction module 220 is further described.

[0053] Spatiotemporal neural network 222 uses machine learning to analyze the hardware-independent composite radar data and generate radar application data for radar-based application 206. Example types of radar application data include the location of a user, the movement of a user, the type of gestures performed by a user, measured vital signs of a user, or characteristics of an object. Spatiotemporal neural network 222 analyzes both amplitude and phase information of the composite radar data to improve the accuracy of detecting distributed objects relative to other signal processing techniques, heuristic techniques, or machine learning techniques that process amplitude information.

[0054] For gesture recognition, the spatiotemporal neural network 222 can be designed to recognize a specific gesture or multiple gestures, such as those described above with reference to Figure 1Those gestures described. In some cases, the spatiotemporal neural network 222 includes a set of machine learning architectures that can be individually selected based on the type of smart device 104 or radar-based application 206. The design of the machine learning architecture can be customized to support smart devices 104 with different amounts of available memory, different amounts of available power, or different computing capabilities. For example, a low-power, non-computationally intensive spatiotemporal neural network 222 can process information from a smaller number of receive channels (e.g., two to four receive channels), have fewer hidden layers, or operate on compressed input data. Conversely, a high-power, computationally intensive spatiotemporal neural network 222 can process information from a larger number of receive channels (e.g., more than four receive channels), execute, have a larger number of hidden layers, or operate on uncompressed input data.

[0055] The circular buffer 224 is a fixed-size memory buffer implemented using an allocated portion of memory within the system medium 218. The circular buffer 224 includes a plurality of memory elements and provides a first-in, first-out queue in which data is sequentially stored and accessed using the memory elements. Once all memory elements store data, the oldest data stored within the circular buffer 224 is overwritten. The circular buffer 224 is implemented between two stages of the spatiotemporal neural network 222. In this way, the circular buffer 224 stores data generated by the first stage of the spatiotemporal neural network 222 and enables the second stage of the spatiotemporal neural network 222 to access the stored data. Figure 7-3 Circular buffer 224 is further described.

[0056] A multi-level machine learning architecture is used to implement the spatiotemporal neural network 222. At the first level, a spatial recurrent network analyzes the composite radar data in the spatial domain to generate feature data. The feature data identifies one or more features (e.g., characteristics) of a gesture performed by a user. In other words, the feature data is a compressed representation of the composite radar data. By using smaller components of the feature data to represent the composite radar data, the complexity of the subsequent levels within the spatiotemporal neural network 222 can be significantly reduced. This can enable the spatiotemporal neural network 222 to be implemented within memory-constrained or computationally-constrained devices.

[0057] The feature data is stored in a memory element of the circular buffer 224 for at least a portion of the time. As time passes, additional feature data is stored in other memory elements of the circular buffer 224. In the second stage, the temporal recursive network analyzes the feature data across multiple memory elements within the circular buffer 224 to recognize the gesture. The circular buffer 224 and the second stage together extract global or temporal information from the local information contained in the feature data stored in the circular buffer 224. Figure 7-1 to Figure 7-4 The spatiotemporal neural network 222 is further described.

[0058] Figure 3-1 1 illustrates an example operation of the radar system 102. In the depicted configuration, the radar system 102 is implemented as a frequency modulated continuous wave radar. However, other types of radar architectures may be implemented, as described above with reference to Figure 2 In environment 300, user 302 is located at a specific tilt range 304 from radar system 102. User 302 represents a distributed object having multiple phase centers 305 (e.g., multiple scatter points). As an example, the hand of user 302 has five phase centers 305 at a specific observation angle. The observation angle is the angle of incidence relative to the line of sight along which radar system 102 "sees" an object (e.g., the hand of user 302). If the observation angle is identified by considering the orientation of the velocity vector of the object, the observation angle can also be considered as a viewing angle.

[0059] The number and corresponding positions of phase centers 305 may vary at different observation angles. When the observation angle changes, such as when user 302 performs a gesture, the amplitude or phase of the radar signature may vary. This variation (also known as speckle) may make it challenging to identify gestures using other signal processing techniques, heuristic techniques, or machine learning techniques that process amplitude information.

[0060] To detect user 302, radar system 102 transmits radar transmit signal 306. At least a portion of radar transmit signal 306 is reflected by user 302. The reflected portion represents radar receive signal 308. Radar system 102 receives radar receive signal 308 and processes radar receive signal 308 to extract data for radar-based application 206. As depicted, the amplitude of radar receive signal 308 is less than the amplitude of radar transmit signal 306 due to losses incurred during propagation and reflections.

[0061] Radar transmit signal 306 includes a sequence of chirps 310-1 to 310-N, where N represents a positive integer greater than one. Radar system 102 may transmit chirps 310-1 to 310-N in a continuous burst, or as time-separated pulses, as shown in FIG. Figure 3-2 As further described, for example, the duration of each chirp 310 - 1 to 310 -N may be on the order of tens or thousands of microseconds (eg, between approximately 30 microseconds (μs) to 5 milliseconds (ms)).

[0062] The individual frequencies of the chirps 310-1 to 310-N may increase or decrease over time. In the depicted example, the radar system 102 employs a dual slope period (e.g., triangular frequency modulation) to linearly increase and linearly decrease the frequencies of the chirps 310-1 to 310-N over time. The dual slope period enables the radar system 102 to measure the Doppler shift caused by the motion of the user 302. In general, the transmit characteristics (e.g., bandwidth, center frequency, duration, and transmit power) of the chirps 310-1 to 310-N may be customized to achieve a specific detection range, range resolution, or Doppler sensitivity to detect one or more features of the user 302 or one or more actions performed by the user 302.

[0063] At the radar system 102, the radar receive signal 308 represents a delayed version of the radar transmit signal 306. The amount of delay is proportional to the tilt range 304 (e.g., distance) from the antenna array 212 of the radar system 102 to the user 302. Specifically, the delay represents the sum of the time it takes for the radar transmit signal 306 to propagate from the radar system 102 to the user 302 and the time it takes for the radar receive signal 308 to propagate from the user 302 to the radar system 102. If the user 302 is moving, the radar receive signal 308 is shifted in frequency relative to the radar transmit signal 306 due to the Doppler effect. Similar to the radar transmit signal 306, the radar receive signal 308 is composed of one or more of the chirps 310-1 to 310N.

[0064] The plurality of chirps 310-1 to 310-N allow the radar system 102 to make a plurality of observations of the user 302 within a predetermined period of time. The radar frame structure determines the timing of the chirps 310-1 to 310-N, as shown in FIG. Figure 3-2 Further described.

[0065] Figure 3-2 An example radar framing structure 312 for gesture recognition is illustrated. In the depicted configuration, the radar framing structure 312 includes three different types of frames. At the highest level, the radar framing structure 312 includes a sequence of main frames 314, which can be in an active state or an inactive state. Generally speaking, an active state consumes a greater amount of power than an inactive state. At an intermediate level, the radar framing structure 312 includes a sequence of feature frames 316, which can similarly be in an active state or an inactive state. The different types of feature frames 316 include a pulse mode feature frame 318 (in the Figure 3-2 ) and the burst mode feature frame 320 (shown in the lower left of Figure 3-2 At a low level, radar framing structure 312 includes a sequence of radar frames (RF) 322, which may also be in an active state or an inactive state.

[0066] Radar system 102 transmits and receives radar signals during active radar frames 322. In some cases, radar frames 322 are analyzed individually for basic radar operations such as search and tracking, clutter map generation, user position determination, etc. Radar data collected during each active radar frame 322 may be saved to a buffer after radar frame 322 is completed or provided directly to the radar system 102. Figure 2 system processor 216.

[0067] Radar system 102 analyzes radar data across multiple radar frames 322 (e.g., across a set of radar frames 322 associated with active feature frames 316) to identify specific features associated with one or more radar functions (e.g., gesture recognition, presence detection, collision avoidance, or human vital sign detection). Example feature types include specific types of motion, motion associated with specific appendages or gesture devices (e.g., a hand, individual fingers, or a stylus), features associated with different parts of a gesture, and the location of at least a portion of a user or another object. For example, to identify a gesture performed by user 302 during active primary frame 314, radar system 102 analyzes radar data associated with one or more active feature frames 316.

[0068] Depending on the type of radar function, the duration of the main frame 314 can be on the order of milliseconds or seconds (e.g., between approximately 10 ms and 10 seconds). After the active main frames 314-1 and 314-2 occur, the radar system 102 is inactive, as shown by the inactive main frames 314-3 and 314-4. The duration of the inactive main frames 314-3 and 314-4 is characterized by a deep sleep time 324, which can be on the order of tens of milliseconds or longer (e.g., greater than 50 ms). In an example implementation, the radar system 102 turns off all active components within the transceiver 214 (e.g., amplifiers, active filters, voltage-controlled oscillators (VCOs), voltage-controlled buffers, multiplexers, analog-to-digital converters, phase-locked loops (PLLs), or crystal oscillators) to save power during the deep sleep time 324.

[0069] In the depicted radar framing structure 312, each main frame 314 includes K feature frames 316, where K is a positive integer. If a main frame 314 is inactive, all feature frames 316 associated with the main frame 314 are also inactive. In contrast, an active main frame 314 includes J active feature frames 316 and KJ inactive feature frames 316, where J is a positive integer less than or equal to K. The number of feature frames 316 can be based on the complexity of the gesture, and can include several to one hundred feature frames 316 (e.g., K can be equal to 2, 10, 30, 60, or 100). The duration of each feature frame 316 can be on the order of milliseconds (e.g., between approximately 1 ms and 50 ms).

[0070] To save power, active feature frames 316-1 to 316-J appear before inactive feature frames 316-(J+1) to 316-K. The duration of inactive feature frames 316-(J+1) to 316-K is characterized by sleep time 326. In this way, the inactive feature frames 316-(J+1) to 316-K are executed consecutively, so that the radar system 102 can be in a powered-off state for a longer duration relative to other techniques that may interleave the inactive feature frames 316-(J+1) to 316-K with the active feature frames 316-1 to 316-J. In general, increasing the duration of sleep time 326 enables the radar system 102 to shut down components within the transceiver 214 that require longer startup time.

[0071] Each feature frame 316 includes L radar frames 322, where L is a positive integer that may or may not be equal to J or K. In some implementations, a certain number of radar frames 322 may vary across different feature frames 316 and may include a few frames or hundreds of frames (e.g., L may be equal to 5, 15, 30, 100, or 500). The duration of a radar frame 322 may be on the order of tens or thousands of microseconds (e.g., between approximately 30 μs and 5 ms). A radar frame 322 within a particular feature frame 316 may be customized for a predetermined detection range, range resolution, or Doppler sensitivity, which helps detect a particular feature or gesture. For example, a radar frame 322 may utilize a particular type of modulation, bandwidth, frequency, transmit power, or timing. If a feature frame 316 is inactive, all radar frames 322 associated with the feature frame 316 are also inactive.

[0072] Pulse mode feature frame 318 and burst mode feature frame 320 include sequences of different radar frames 322. Generally speaking, radar frames 322 within active burst mode feature frame 318 transmit pulses that are spaced apart in time by a predetermined amount. This is observed over time dispersion, which may make it easier for radar system 102 to identify gestures due to the larger variations in chirps 310-1 to 310-N observed within pulse mode feature frame 318 relative to burst mode feature frame 320. In contrast, radar frames 322 within active burst mode feature frame 320 transmit pulses continuously across a portion of burst mode feature frame 320 (e.g., pulses are not spaced apart by a predetermined amount of time). This will cause active burst mode feature frame 320 to consume less power than pulse mode feature frame 318 by shutting down a larger number of components, including components with longer startup times, as further described below.

[0073] Within each active pulse mode feature frame 318, a sequence of radar frames 322 alternates between an active state and an inactive state. Each active radar frame 322 transmits a chirp 310 (e.g., a pulse), which is illustrated by a triangle. The duration of the chirp 310 is characterized by an active time 328. During the active time 328, components within the transceiver 214 are powered on. During a short idle time 330, which includes the remaining time within the active radar frame 322 and the duration of the subsequent inactive radar frame 322, the radar system 102 saves power by shutting down one or more active components within the transceiver 214 that have an activation time within the duration of the short idle time 330.

[0074] The active burst mode characteristic frame 320 includes P active radar frames 322 and LP inactive radar frames 322, where P is a positive integer less than or equal to L. In order to save power, the active radar frames 322-1 to 322-P appear before the inactive radar frames 322-(P+1) to 322-L. The duration of the inactive radar frames 322-(P+1) to 322-L is characterized by a long idle time 332. By grouping the inactive radar frames 322-(P+1) to 322-L together, the radar system 102 can be in a powered-off state for a longer duration relative to the short idle time 330 that occurs during the pulse mode characteristic frame 318. In addition, the radar system 102 can shut down additional components within the transceiver 214 that have a startup time that is longer than the short idle time 330 and shorter than the long idle time 332.

[0075] Each active radar frame 322 within active burst mode feature frame 320 transmits a portion of chirp 310. In this example, active radar frames 322-1 through 322-P alternate between transmitting a portion of chirp 310 that increases in frequency and a portion of chirp 310 that decreases in frequency.

[0076] Radar framing structure 312 enables power savings through adjustable duty cycles within each frame type. A first duty cycle 334 is based on the number of active feature frames 316 (J) relative to the total number of feature frames 316 (K). A second duty cycle 336 is based on the number of active radar frames 322 relative to the total number of radar frames 322 (L) (e.g., L / 2 or P). A third duty cycle 338 is based on the duration of chirp 310 relative to the duration of radar frame 322.

[0077] Consider an example radar framing structure 312 for a power state, which consumes approximately 2 milliwatts (mW) of power and has a main frame update rate between approximately 1 and 4 hertz (Hz). In this example, the radar framing structure 312 includes a main frame 314, whose duration is between approximately 250ms and 1 second. The main frame 314 includes thirty-one pulse mode feature frames 318 (e.g., K equals 31). One of the thirty-one pulse mode feature frames 318 is in an active state. This results in a duty cycle 334 that is approximately equal to 3.2%. The duration of each pulse mode feature frame 318 is between approximately 8 and 32ms. Each pulse mode feature frame 318 is composed of eight radar frames 322 (e.g., L equals 8). In the active pulse mode feature frame 318, all eight radar frames 322 are in an active state. This results in a duty cycle 336 that is equal to 100%. The duration of each radar frame 322 is between approximately 1 and 4ms. The active time 328 within each active radar frame 322 is between approximately 32 and 128 μs. Thus, the resulting duty cycle 338 is approximately 3.2%. It has been found that this example radar framing structure 312 produces good performance results. These good performance results in terms of good gesture recognition and presence detection, while also producing good power efficiency results in the context of a handheld smartphone in a low power state. Reference Figure 4 Further description ( Figure 3-1 The generation of radar transmission signal 306 and ( Figure 3-1 ) processing of radar received signal 308.

[0078] Figure 4 An example antenna array 212 and an example transceiver 214 of the radar system 102 are illustrated. In the depicted configuration, the transceiver 214 includes a transmitter 402 and a receiver 404. The transmitter 402 includes at least one voltage controlled oscillator 406 (VOC 406) and at least one power amplifier 408 (PA 408). The receiver 404 includes at least two receive channels 410-1 to 410-M, where M is a positive integer greater than 1. Each receive channel 410-1 to 410-M includes at least one low noise amplifier 412 (LNA 412), at least one mixer 414, at least one filter 416, and at least one analog-to-digital converter 418 (ADC 418).

[0079] The antenna array 212 includes at least one transmit antenna element 420 and at least two receive antenna elements 422-1 through 422-M. The transmit antenna element 420 is coupled to the transmitter 402. The receive antenna elements 422-1 through 422-M are coupled to the receive channels 410-1 through 410-M, respectively.

[0080] During transmission, voltage controlled oscillator 406 generates frequency modulated radar signal 424 at a radio frequency. Power amplifier 408 amplifies frequency modulated radar signal 424 for transmission via transmit antenna element 420. Transmitted frequency modulated radar signal 424 is represented by radar transmission signal 306, which may include frequency modulated radar signal 424 based on the frequency modulated radar signal 424. Figure 3-2 The radar achieves a plurality of chirps 310-1 to 310-N of a frame structure 312. As an example, according to Figure 3-2 The radar transmit signal 306 is generated using a burst mode feature frame 320 of , and the radar transmit signal 306 includes 16 chirps 310 (eg, N is equal to 16).

[0081] During reception, each receive antenna element 422-1 to 422-M receives a version of the radar receive signal 308-1 to 308-M. Typically, the relative phase difference between these versions of the radar receive signal 308-1 to 308-M is due to the positional differences of the receive antenna elements 422-1 to 422-M. Within each receive channel 410-1 to 410-M, the low noise amplifier 412 amplifies the radar receive signal 308, and the mixer 414 mixes the amplified radar receive signal 308 with the frequency modulated radar signal 424. In particular, the mixer performs a beat operation, which down-converts and demodulates the radar receive signal 308 to generate a beat signal 426.

[0082] The frequency of the beat signal 426 represents the frequency difference between the frequency modulated radar signal 424 and the radar received signal 308, which is related to Figure 3-1 304. Although not shown, beat signal 426 may include multiple frequencies representing reflections from different parts of user 302 (e.g., different fingers, different parts of a hand, or different body parts) or from different objects. In some cases, these different parts move at different speeds, move in different directions, or are positioned at different tilt ranges relative to radar system 102.

[0083] The filter 416 filters the beat signal 426, and the analog-to-digital converter 418 digitizes the filtered beat signal 426. The receive channels 410-1 to 410-M generate digital beat signals 428-1 to 428-M, respectively, which are provided to the system processor 216 for processing. The receive channels 410-1 to 410-M of the transceiver 214 are coupled to the system processor 216, as shown in FIG. Figure 5 as shown in .

[0084] Figure 5An example scheme implemented by radar system 102 for performing gesture recognition using spatiotemporal neural network 222 is illustrated. In the depicted configuration, system processor 216 implements hardware abstraction module 220 and spatiotemporal neural network 222. System processor 216 is connected to receive channels 410-1 to 410-M. System processor 216 may also communicate with computer processor 202. Although not shown, hardware abstraction module 220 and / or spatiotemporal neural network 222 may be implemented by computer processor 202.

[0085] In this example, hardware abstraction module 220 accepts digital beat signals 428-1 to 428-M from receive channels 410-1 to 410-M. Digital beat signals 428-1 to 428-M represent raw or unprocessed composite radar data. Hardware abstraction module 220 performs one or more operations to generate hardware-independent composite radar data 502-1 to 502-M based on digital beat signals 428-1 to 428-M. Hardware abstraction module 220 transforms the composite radar data provided by digital beat signals 428-1 to 428-M into a form expected by spatiotemporal neural network 222. In some cases, hardware abstraction module 220 normalizes the amplitudes associated with different transmit power levels or transforms the composite radar data into a frequency domain representation.

[0086] The hardware-independent composite radar data 502-1 to 502-M include both amplitude and phase information (e.g., in-phase and quadrature components). In some implementations, the hardware-independent composite radar data 502-1 to 502-M include a range-Doppler map for each receive channel 410-1 to 410-M and for a particular active feature frame 316. In other implementations, the hardware-independent composite radar data 502-1 to 502-M include composite interferometry data that is an orthogonal representation of the range-Doppler map. As another example, the hardware-independent composite radar data 502-1 to 502-M include a frequency domain representation of the digital beat signals 428-1 to 428-M of the active feature frame 316. Although not shown, other implementations of the radar system 102 may provide the digital beat signals 428-1 to 428-M directly to the spatiotemporal neural network 222.

[0087] The spatiotemporal neural network 222 uses a trained regression or classification model to analyze the hardware-independent composite radar data 502-1 to 502-M and generate radar application data 504. For gesture-based applications, the radar application data 504 provides information about the gesture performed by the user 302. For example, the radar application data 504 indicates whether the gesture performed by the user 302 is a left sweep gesture or a right sweep gesture. In other implementations, the radar application data 504 indicates different probabilities associated with different types of gestures, such as the probability of a sweep gesture and the probability of a hand reaching gesture. Although described with reference to gesture recognition, the training process performed by the spatiotemporal neural network 222 and the machine learning architecture of the spatiotemporal neural network 222 can be adapted to support other types of applications, including presence detection, collision avoidance, and human vital sign detection.

[0088] In some implementations, spatiotemporal neural network 222 relies on supervised learning and records measured (e.g., real) data for machine learning training purposes. Training enables spatiotemporal neural network 222 to learn a nonlinear mapping function for generating radar application data 504 based on hardware-independent composite radar data 502-1 to 502-M. In other implementations, spatiotemporal neural network 222 relies on unsupervised learning to determine the nonlinear mapping function.

[0089] The example training process prompts the user 302 to select and perform a particular gesture, such as a reaching gesture or a sweeping gesture. While performing the gesture, the spatiotemporal neural network 222 records the composite radar data provided as input (e.g., hardware-independent composite radar data 502-1 to 502-M or digital beat signals 428-1 to 428-M). The spatiotemporal neural network 222 adjusts the machine learning parameters (e.g., coefficients, weights, or biases) to recognize the gesture based on the recorded input data. The determined machine learning parameters are stored by the system media 218 or pre-programmed into the spatiotemporal neural network 222 to enable recognition of future gestures. In some cases, the process may be repeated multiple times to enable the spatiotemporal neural network 222 to account for slight changes in how the user 302 performs the gesture.

[0090] The example offline training process uses a motion capture system to generate real data for training the spatiotemporal neural network 222. The motion capture system may include multiple optical sensors, such as infrared sensors or cameras, to measure the positions of multiple markers placed on different parts of the human body (such as arms, hands, torso, or head). When the person moves to different positions or performs different gestures relative to the radar system 102, the input data is recorded together with the position data from the motion capture system. The position data recorded from the motion capture system is converted into a position measurement relative to the radar system 102 and represents real data. Additionally or alternatively, the real data includes the type of gesture performed. The spatiotemporal neural network 222 analyzes the training data and the real data together and adjusts the machine learning parameters to minimize errors. In some cases, the offline training process can provide a relatively noise-free environment and high-resolution real data for training the spatiotemporal neural network 222.

[0091] Additionally or alternatively, a real-time training process may use available sensors within the smart device 104 to generate real data for training the spatiotemporal neural network 222. In this case, the training process may be initiated by a user 302 of the smart device 104. As the user 302 moves around the smart device 104 or performs gestures, data from optical sensors (e.g., cameras or infrared sensors) of the smart device 104 and the radar system 102 is collected and provided as real data to the spatiotemporal neural network 222. The real-time training process enables the spatiotemporal neural network 222 to be customized to the user 302, take into account current environmental conditions, and take into account the current position or orientation of the smart device 104.

[0092] The spatiotemporal neural network 222 includes two or more artificial neural networks (e.g., neural networks). A neural network includes a set of connected nodes (e.g., neurons or perceptrons) organized into one or more layers. As an example, the spatiotemporal neural network 222 includes a deep neural network that includes an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. The nodes of the deep neural network can be partially connected or fully connected between layers.

[0093] In some cases, the deep neural network is a recursive deep neural network (e.g., a long short-term memory (LSTM) recursive deep neural network) in which the connections between nodes form a loop to retain information from a previous portion of the input data sequence for a subsequent portion of the input data sequence. In other cases, the deep neural network is a feedforward deep neural network in which the connections between nodes do not form a loop. Additionally or alternatively, the spatiotemporal neural network 222 may include another type of neural network, such as a convolutional neural network. Reference Figure 7-1 An example implementation of the spatiotemporal neural network 222 is further described.

[0094] Figure 6-1 An example hardware abstraction module 220 for gesture recognition is illustrated. In the depicted configuration, the hardware abstraction module 220 includes a pre-processing stage 602 and a signal transformation stage 604. The pre-processing stage 602 operates on each chirp 310-1 to 310-N within the digital beat signals 428-1 to 428-M. In other words, the pre-processing stage 602 performs operations on each active radar frame 322. In this example, the pre-processing stage 602 includes one-dimensional (1D) fast Fourier transform (FFT) modules 606-1 to 606-M, which process the digital beat signals 428-1 to 428-M, respectively. Other types of modules that perform similar operations are also possible, such as Fourier transform modules.

[0095] The signal conversion stage 604 operates on the sequence of chirps 310-1 to 310-M within each of the digital beat signals 428-1 to 428-M. In other words, the signal conversion stage 604 operates on each active feature frame 316. In this example, the signal conversion stage 604 includes buffers 608-1 to 608-M and two-dimensional (2D) FFT modules 610-1 to 610-M.

[0096] During reception, the one-dimensional FFT modules 606-1 to 606-M perform individual FFT operations on the chirps 310-1 to 310-M within the digital beat signals 428-1 to 428-M. Assuming that the radar receive signals 308-1 to 308-M include 16 chirps 310-1 to 310-N (e.g., N is equal to 16), each of the one-dimensional FFT modules 606-1 to 606-M performs 16 FFT operations to generate pre-processed composite radar data for each chirp 612-1 to 612-M. When each operation is performed, the buffers 608-1 to 608-M store the results. Once all chirps 310-1 through 310-M associated with active feature frame 316 have been processed by pre-processing stage 602, information stored by buffers 608-1 through 608-M represents pre-processed composite radar data for each feature frame 614-1 through 614-M for respective receive channels 410-1 through 410-M.

[0097] The two-dimensional FFT modules 610-1 to 610-M process the pre-processed composite radar data of each feature frame 614-1 to 614-M, respectively, to generate hardware-independent composite radar data 502-1 to 502-M. In this case, the hardware-independent composite radar data 502-1 to 502-M include composite range-Doppler maps, as shown in FIG. Figure 6-2 Further described.

[0098] Figure 6-2Example hardware-independent composite radar data 502-1 for gesture recognition generated by the hardware abstraction module 220 is illustrated. The hardware abstraction module 220 is shown as processing a digital beat signal 428-1 associated with the receive channel 410-1. The digital beat signal 428-1 includes chirps 310-1 to 310-M as time domain signals. The chirps 310-1 to 310-M are passed to the one-dimensional FFT module 606-1 in the order in which they are received and processed by the transceiver 214.

[0099] As described above, one-dimensional FFT module 606-1 performs an FFT operation on first chirp 310-1 of digital beat signal 428-1 at a first time. Buffer 608-1 stores a first portion of pre-processed composite radar data 612-1 associated with first chirp 310-1. One-dimensional FFT module 606-1 continues to process subsequent chirps 310-2 to 310-N, and buffer 608-1 continues to store corresponding portions of pre-processed composite radar data 612-1. This process continues until buffer 608-1 stores the last portion of pre-processed composite radar data 612-M associated with chirp 310-M.

[0100] At this point, buffer 608-1 stores pre-processed composite radar data associated with a particular feature frame 614-1. The pre-processed composite radar data of each feature frame 614-1 represents amplitude information (as shown) and phase information (not shown) across different chirps 310-1 through 310-N and across different range bins 616-1 through 616-A, where A represents a positive integer.

[0101] Two-dimensional FFT 610-1 accepts the pre-processed composite radar data of each feature frame 614-1 and performs a two-dimensional FFT operation to form hardware-independent composite radar data 502-1 representing a range-Doppler map. The range-Doppler map includes composite radar data for range intervals 616-1 to 616-A and Doppler intervals 618-1 to 618-B, where B represents a positive integer. In other words, each range interval 616-1 to 616-A and Doppler interval 618-1 to 618-B includes a complex number having a real part and an imaginary part, which together represent amplitude and phase information. The number of range intervals 616-1 to 616-A can be on the order of tens or hundreds, such as 64 or 128 (e.g., A is equal to 64 or 128). The number of Doppler intervals can be on the order of tens or hundreds, such as 32, 64, or 124 (e.g., B is equal to 32, 64, or 124). Hardware-independent composite radar data 502-1 and ( Figure 6-1 The hardware-independent composite radar data 502-1 to 502-M are provided to the spatiotemporal neural network 222, such as Figure 7-1 as shown in .

[0102] Figure 7-1 An example spatiotemporal neural network 222 for detecting gesture detection is illustrated. In the depicted configuration, the spatiotemporal neural network 222 includes two levels implemented by a spatial recurrent network 702 and a temporal recurrent network 704, respectively. The spatial recurrent network 702 includes a chirp-level analysis module 706 and a feature-level analysis module 708. In general, the spatial recurrent network 702 analyzes the hardware-independent composite radar data 502-1 to 502-M in the spatial domain for each active feature frame 316. The resulting data is stored by the circular buffer 224. The temporal recurrent network 704 includes an application-level analysis module 710 that analyzes the data stored in the circular buffer 224 for two or more active feature frames 316. In this manner, the temporal recurrent network 704 analyzes data in the temporal domain for at least a portion of the active primary frame 314. In some implementations, the size of the circular buffer 224 is tailored to the duration of the active primary frame 314 so as to enable storage of all active feature frames 316 of the active primary frame 314. For gesture recognition, the duration of active major frame 314 may be associated with the duration of one or more gestures. Generally, application-level analysis module 710 is designed to support one or more radar-based applications, including gesture recognition, presence detection, collision avoidance, and human vital sign detection.

[0103] During reception, the chirp-level analysis module 706 processes the composite radar data across each range interval 616-1 to 616-A to generate channel Doppler data 712. The feature-level analysis module 708 analyzes the channel Doppler data 712 to generate feature data 714, which characterizes one or more features associated with the application-level analysis module 710. For example, the feature data 714 can characterize features of a gesture performed by the user 302 for gesture recognition. The circular buffer 224 stores the feature data 714.

[0104] Over time, the circular buffer 224 stores feature data 714 associated with different active feature frames 316. Feature data 714 associated with two or more active feature frames 316 is referred to as a temporal set 716 of feature data. The temporal set 716 of feature data is provided to or accessed by the application-level analysis module 710. The application-level analysis module 710 analyzes the temporal set 716 of feature data to generate radar application data 504. As an example, the radar application data 504 includes a prediction about the type of gesture performed by the user 302. When the circular buffer 224 stores feature data 714 associated with a larger number of active feature frames 316, the accuracy of the prediction increases. In some cases, the application-level analysis module 710 continuously generates or updates the radar application data 504 as subsequent feature frames 316 associated with the main frame 314 are processed by the spatial recurrent network 702. Alternatively, the application-level analysis module 710 delays the generation of the radar application data 504 until all (or at least a portion) of the feature frames 316 associated with the main frame 314 have been processed by the spatial recurrent network 702. Figure 7-2 to Figure 7-4 The implementation of the chirp-level analysis module 706 , the feature-level analysis module 708 , and the application-level analysis module 710 is further described.

[0105] Figure 7-2 An example chirp level analysis module 706 of the spatiotemporal neural network 222 is illustrated. In the depicted configuration, the chirp level analysis module 706 includes channel Doppler processing modules 718-1 to 718-A. Each channel Doppler processing module 718-1 to 718-A includes a neural network having one or more layers 720-1 to 720-Q, where Q is a positive integer. The value of Q may vary depending on the implementation. As an example, Q may be equal to 2, 4, or 10. The layers 720-1 to 720-Q may be fully connected, partially connected, connected in a feedforward manner, and / or connected in a recursive manner. For example, the nodes within the layers 720-1 to 720-Q may perform a nonlinear rectifier activation function. The channel Doppler processing modules 718-1 to 718-A may also perform addition and multiplication.

[0106] Channel Doppler processing modules 718-1 to 718-A receive corresponding portions of hardware-independent composite radar data 502-1 to 502-M according to range intervals 616-1 to 616-A. In particular, channel Doppler processing module 718-1 receives composite radar data associated with first range interval 616-1 across all receive channels 410-1 to 410-M and across all Doppler intervals 618-1 to 618-B. Each complex number is provided as an input to each node of layer 720-1. Layers 720-1 to 720-Q analyze the data using a nonlinear rectifier activation function to generate channel Doppler data 712-1. Similar operations are also performed by channel Doppler processing modules 718-2 to 718-A. Combined channel Doppler data 712-1 to 712-A represent vectors. For example, assuming there are three receive channels 410-1 to 410-M (e.g., M equals 3), 32 Doppler intervals 618-1 to 618-B (e.g., B equals 32), and 16 range intervals 616-1 to 616-A (e.g., A equals 16), the channel Doppler data 712-1 to 712-A form a 1x16 vector of values ​​that represents the relationship across the receive channels in the Doppler domain so that the feature level analysis module 708 can identify one or more features of the gesture.

[0107] Figure 7-3 An example feature-level analysis module 708 of the spatiotemporal neural network 222 is illustrated. In the depicted configuration, the feature-level analysis module 708 is implemented using one or more recursive layers 722-1 to 722-V, where V represents a positive integer. Within the recursive layers 722-1 to 722-V, the connections between nodes form loops that retain information from previous active feature frames 316 for subsequent active feature frames 316. Using the recursive layers 722-1 to 722-V, the feature-level analysis module 708 can implement a long short-term memory (LSTM) network or some other temporal convolutional network.

[0108] As described above, the feature level analysis module 708 processes the channel Doppler data 712-1 to 712-A across the range intervals 616-1 to 616-A to generate feature data 714. Although not shown, some implementations of the spatial recurrent network 702 may include an additional fully connected layer 720 connected to the output of the recurrent layer 722-V. Similar to the layers 720 of the chirp level analysis module 706, these layers 720 may also perform nonlinear transformations.

[0109] Over time, feature data 714-1 to 714-J associated with active feature frames 316-1 to 316-J are sequentially stored in different memory elements by the circular buffer 224. Feature data 714-1 to 714-J represent a temporal collection 716 of feature data processed by the application-level analysis module 708, as described with reference to FIG. Figure 7-4 Further described.

[0110] Figure 7-4 An example application-level analysis module 710 of the spatiotemporal neural network 222 is illustrated. In the depicted configuration, the application-level analysis module 708 is implemented using one or more recursive layers 724-1 to 724-W, where W represents a positive integer that may or may not be equal to V. Using the recursive layers 724-W to 724-W, the application-level analysis module 710 can implement a long short-term memory (LSTM) network or a temporal convolutional network. The application-level analysis module 710 can optionally include a classifier module 726, which represents one or more layers of the application-level analysis module 710. For gesture recognition, the classifier module 726 can distinguish different types of gestures, such as a left sweep gesture, a right sweep gesture, or a hand-reaching gesture. In some implementations, the classifier module 726 has multiple heads (e.g., multiple submodules) that can support different radar-based applications. For example, a first head can detect the presence of a user 302 to support presence detection, while a second head detects gestures performed by the user 302 to support gesture recognition.

[0111] The application-level analysis module 710 processes two or more feature data 714-1 to 714-J stored in the circular buffer 224. For gesture recognition, the application-level analysis module 710 forms a prediction about the type of gesture performed based on two or more of the feature data 714-1 to 714-J. In some cases, the application-level analysis module 710 may wait to form a prediction until a specified number of feature data 714-1 to 714-J are available, such as 15. If the active main frame 314 includes more than 15 active feature frames 316 (e.g., J is greater than 15), the application-level analysis module 710 may continue to update its prediction based on the last 15 active feature frames 316. Typically, the accuracy of the prediction increases over time or when a larger amount of feature data 714-1 to 714-J is analyzed.

[0112] Optionally, the classifier module 726 can further refine the predictions made using the recursive layers 724-1 to 724-W. For example, the classifier module 726 can distinguish between different directional sweeps as they relate to the orientation of the smart device 104. Specifically, the classifier module 726 can determine whether the sweep gesture is an upward sweep gesture in the portrait mode or a left sweep gesture in the landscape mode. To implement this logic, the classifier module 726 can include at least one fully connected layer having class nodes activated according to the gesture type detected by the classifier module 726.

[0113] Although described with reference to radar systems, the spatiotemporal neural network 222 can be modified to operate with data provided by other types of sensors or electromagnetic systems. Typically, these sensors or electromagnetic systems have the ability to encode one or more characteristics of a signal propagating through space, such as the amplitude, phase, or frequency of the signal. Examples include Bluetooth TM System, WiGig TM Systems, ultrasonic sensors, lidar sensors, etc.

[0114] Example Method

[0115] Figure 8 An example method 800 is depicted for performing operations of a smart device-based radar system capable of performing gesture recognition using a spatiotemporal neural network 222. The method 800 is shown as a collection of operations (or actions) performed, but is not necessarily limited to the order or combination of operations shown herein. In addition, any one or more of the operations may be repeated, combined, reorganized, or linked to provide a variety of additional and / or alternative methods. In the following discussion, reference may be made to Figure 1 Environments 100-1 to 100-6, and Figure 4 5, references thereto are by way of example only. The technology is not limited to being performed by one entity or multiple entities operating on one device.

[0116] At 802, a radar transmission signal is transmitted using an antenna array of a radar system. Figure 4 As shown in FIG. 3 , radar system 102 transmits radar transmit signal 306 using at least one transmit antenna element 420. In some implementations, radar transmit signal 306 includes a plurality of frequency modulated chirps 310-1 through 310-N, as shown in FIG.

[0117] At 804, a radar receive signal is received using an antenna array. The radar receive signal includes a version of the radar transmit signal reflected by at least one user. For example, radar system 102 uses at least one receive antenna element 422 to receive a version of radar receive signal 308 reflected by user 302, such as Figure 4 as shown in .

[0118] At 806, composite radar data is generated based on the radar receive signal. For example, receive channel 410 of radar system 102 generates digital beat signal 428 based on radar receive signal 308. The digital beat signal represents the composite radar data and includes both amplitude and phase information (e.g., in-phase and quadrature information).

[0119] At 808, the composite radar data is provided to a spatiotemporal neural network of the radar system. The spatiotemporal neural network includes a multi-level machine learning architecture. For example, the receiving channels 410-1 to 410-M provide (e.g., directly or indirectly through the hardware abstraction module 220) the digital beat signals 428-1 to 428-M to the spatiotemporal neural network 222, such as Figure 5 As shown. The spatiotemporal neural network 222 includes a multi-level machine learning architecture, which includes Figure 7-1 2 and 3. The spatial recurrent network 702 and the temporal recurrent network 704 are shown in FIG. 2. The spatial recurrent network 702 represents the first level of the spatiotemporal neural network 222 and analyzes the composite radar data in the spatial domain. The temporal recurrent network 704 represents the second level of the spatiotemporal neural network 222 and analyzes the data provided by the spatial recurrent network 702 in the temporal domain.

[0120] At 810, the composite radar data is analyzed using a spatiotemporal neural network to identify a gesture performed by at least one user. For example, spatiotemporal neural network 222 analyzes digital beat signals 428-1 through 428-M or hardware-independent composite radar data 502-1 through 502-M to generate radar application data 504 that identifies a gesture performed by user 302. Although described with respect to gesture recognition, similar operations may be performed with respect to other applications, including presence detection, collision avoidance, vital sign detection, etc.

[0121] Example computing system

[0122] Fig. 9 Illustrate various components of an example computing system 900, which may be implemented as described with reference to the previous Figure 2 Any type of client, server, and / or computing device described may be used to implement gesture recognition using the spatiotemporal neural network 222 .

[0123] The computing system 900 includes a communication device 902 that can implement wired and / or wireless communication of device data 904 (e.g., received data, data being received, data scheduled for broadcast, or data packets of data). Although not shown, the communication device 902 or computing system 900 may include one or more radar systems 102. The device data 904 or other device content may include configuration settings of the device, media content stored on the device, and / or information associated with the user 302 of the device. The media content stored on the computing system 900 may include any type of audio, video, and image data. The computing system 900 includes one or more data inputs 906 that can receive any type of data, media content, and / or input, such as human speech, radar-based applications 206, user-selectable input (explicit or implicit), messages, music, television media content, recorded video content, and any other type of audio, video, and / or image data received from any content and / or data source.

[0124] The computing system 900 also includes a communication interface 908, which can be implemented as any one or more of a serial and / or parallel interface, a wireless interface, any type of network interface, a modem, and as any other type of communication interface. The communication interface 908 provides a connection and / or communication link between the computing system 900 and a communication network through which other electronic, computing, and communication devices communicate data with the computing system 900.

[0125] The computing system 900 includes one or more processors 910 (e.g., any of a microprocessor, controller, etc.) that process various computer-executable instructions to control the operation of the computing system 900 and enable techniques for gesture recognition in the presence of saturation, or in which they may be embodied. Alternatively or additionally, the computing system 900 may be implemented using any one or combination of hardware, firmware, or fixed logic circuits implemented together with processing and control circuits generally identified at 912. Although not shown, the computing system 900 may include a system bus or data transmission system that couples various components within the device. The system bus may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any one of a variety of bus architectures.

[0126] The computing system 900 also includes computer-readable media 914, such as one or more memory devices capable of persistent and / or non-transitory data storage (i.e., as opposed to mere signal transmission), examples of which include random access memory (RAM), non-volatile memory (e.g., any one or more of read-only memory (ROM), flash memory, EPROM, EEPROM, etc.), and disk storage devices. The disk storage device can be implemented as any type of magnetic or optical storage device, such as a hard drive, a recordable and / or rewritable compact disk (CD), any type of digital versatile disk (DVD), etc. The computing system 900 may also include a mass storage media device (storage media) 916.

[0127] The computer-readable medium 914 provides a data storage mechanism for storing device data 904 as well as various device applications 918 and any other types of information and / or data related to operational aspects of the computing system 900. For example, an operating system 920 may be maintained as a computer application and executed on the processor 910 using the computer-readable medium 914. The device applications 918 may include a device manager, such as any form of control application, software application, signal processing and control module, code native to a particular device, a hardware abstraction layer for a particular device, and the like.

[0128] Device applications 918 also include any system components, engines, or managers that use machine learning to implement angle estimation. In this example, device applications 918 include Figure 2 Radar-based applications 206 and spatiotemporal neural networks 222.

[0129] in conclusion

[0130] Although techniques for and apparatuses including smart device-based radar systems that use a smart device-based radar system that performs gesture recognition using a spatiotemporal neural network have been described in language specific to features and / or methods, it should be understood that the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as example implementations of a smart device-based radar system that performs gesture recognition using a spatiotemporal neural network.

[0131] Some examples are described below.

[0132] Example 1: A method performed by a radar system, the method comprising:

[0133] transmitting a radar transmit signal using an antenna array of the radar system;

[0134] receiving, using the antenna array, a radar receive signal, the radar receive signal comprising a version of a radar transmit signal reflected by at least one user;

[0135] generating composite radar data based on the radar received signal;

[0136] providing the composite radar data to a spatiotemporal neural network of the radar system, the spatiotemporal neural network comprising a multi-level machine learning architecture; and

[0137] The composite radar data is analyzed using the spatiotemporal neural network to recognize a gesture performed by the at least one user.

[0138] Example 2: The method of Example 1, wherein:

[0139] Analysis of the composite radar data includes analyzing both amplitude and phase information of the composite radar data using machine learning techniques to identify the gesture.

[0140] Example 3: The method according to Example 1 or 2, wherein:

[0141] The multi-level machine learning architecture includes a spatial recurrent network and a temporal recurrent network; and

[0142] The analysis of the composite radar data includes:

[0143] analyzing the composite radar data in a spatial domain using the spatial recurrent network to generate feature data associated with the gesture; and

[0144] The feature data is analyzed in the time domain using the time recursive network to recognize the posture.

[0145] Example 4: The method according to Example 3 further comprises:

[0146] storing the feature data in a circular buffer; and

[0147] The feature data stored in the circular buffer is accessed by the time recursive network.

[0148] Example 5: The method of Example 3 or 4, wherein analyzing the composite radar data in the spatial domain comprises:

[0149] separately processing portions of the composite radar data associated with different range bins using a non-linear activation function to generate channel Doppler data for each range bin; and

[0150] The channel Doppler data is analyzed across the different range intervals to generate the signature data.

[0151] Example 6: A method according to any one of Examples 3 to 5, wherein analyzing the feature data in the time domain includes: forming a prediction about the posture by analyzing the feature data associated with at least two different time frames.

[0152] Example 7: The method according to any one of Examples 3 to 6, wherein the spatial recursive network includes a chirp-level analysis module and a feature-level analysis module.

[0153] Example 8: The method of any one of Examples 3 to 7, wherein the time recursive network includes an application-level analysis module.

[0154] Example 9: A method according to any preceding example, wherein the composite radar data includes at least one of the following:

[0155] Composite range-Doppler map;

[0156] Composite interferometry data;

[0157] a plurality of digital beat signals associated with the radar receive signal; or

[0158] A frequency domain representation of the plurality of digital beat signals.

[0159] Example 10: The method of any preceding example, wherein the gesture comprises at least one of:

[0160] Sweeping gesture;

[0161] reaching hand gesture;

[0162] Handle turning gesture; or

[0163] Axis twist attitude.

[0164] Example 11: A method according to any of the preceding examples, wherein:

[0165] Receiving the radar receive signal includes receiving multiple versions of the radar receive signal using different antenna elements of the radar system; and

[0166] Generating the composite radar data includes generating a digital beat signal using corresponding receive channels of the radar system, the corresponding receive channels being connected to the different antenna elements, respectively.

[0167] Example 12: A method according to any one of Examples 4 to 11, wherein feature data processed by the spatial recurrent network is stored by a circular buffer and a temporal set of feature data is accessed by the temporal recurrent network.

[0168] Example 13: An apparatus comprising:

[0169] A radar system, the radar system comprising:

[0170] Antenna arrays;

[0171] transceiver; and

[0172] A processor and a computer-readable storage medium configured to perform the method according to any one of Examples 1 to 12.

[0173] Example 14: The apparatus of Example 13, wherein the apparatus comprises a smart device, the smart device comprising one of the following:

[0174] Smartphone;

[0175] Smart watches;

[0176] Smart speakers;

[0177] Smart thermostat;

[0178] Security cameras;

[0179] Vehicle; or

[0180] Household appliances.

[0181] Example 15: A computer-readable storage medium comprising computer-executable instructions that, in response to being executed by a processor, implement:

[0182] A spatiotemporal neural network, wherein the spatiotemporal neural network is configured to:

[0183] receiving composite radar data associated with a radar receive signal reflected by at least one object;

[0184] analyzing the composite radar data in a spatial domain to generate signature data;

[0185] Analyzing the characteristic data in the time domain to generate radar application data; and

[0186] The radar application data is communicated to a radar based application.

[0187] Example 16: The computer-readable storage medium of Example 15, wherein the spatiotemporal neural network is further configured to analyze both amplitude and phase information of the composite radar data to generate the radar application data.

[0188] Example 17: The computer-readable storage medium of Example 15 or 16, wherein:

[0189] The at least one object comprises a user;

[0190] The radar application data identifies a gesture performed by the user; and

[0191] The computer executable instructions, in response to being executed by the processor, implement the radar-based application, the radar-based application being configured to:

[0192] receiving the radar application data; and

[0193] Content is displayed to the user based on the recognized gesture.

[0194] Example 18: The computer-readable storage medium of Example 17, wherein the spatiotemporal neural network is further configured to:

[0195] prompting the user to perform the gesture;

[0196] recording the composite radar data as training data while the user performs the gesture; and

[0197] A training process is performed to recognize future gestures performed by the user.

[0198] Example 19: The computer-readable storage medium of Example 15 or 16, wherein:

[0199] The at least one object comprises a stylus; and

[0200] The radar application data identifies a gesture performed by a user using the stylus.

[0201] Example 20: The computer-readable storage medium of Example 15 or 16, wherein:

[0202] The at least one object includes an inanimate object and a user;

[0203] The radar application data identifies a potential collision between the inanimate object and the user; and

[0204] The computer executable instructions, in response to being executed by the processor, implement a radar-based application, the radar-based application being configured to:

[0205] receiving the radar application data; and

[0206] The user is warned of the potential collision.

[0207] Example 21: The computer-readable storage medium of Example 15 or 16, wherein:

[0208] The at least one object comprises a user;

[0209] The radar application data includes measured vital signs of the user; and

[0210] The computer executable instructions, in response to being executed by the processor, implement a radar-based application, the radar-based application being configured to:

[0211] receiving the radar application data; and

[0212] The user is notified of the measured vital signs.

[0213] Example 22: The computer-readable storage medium of any one of Examples 15 to 21, wherein the computer-executable instructions, in response to being executed by the processor, implement a hardware abstraction module, the hardware abstraction module being configured to:

[0214] generating hardware-independent composite radar data based on the composite radar data; and

[0215] The hardware-independent composite radar data is provided to the spatiotemporal neural network as the composite radar data.

[0216] Example 23: A computer-readable storage medium comprising computer-executable instructions that, in response to being executed by a processor, implement:

[0217] A spatiotemporal neural network, comprising a multi-level machine learning architecture, wherein the spatiotemporal neural network is configured to:

[0218] receiving composite radar data associated with a radar receive signal reflected by at least one object; and

[0219] The composite radar data is analyzed to identify a gesture performed by the at least one user.

[0220] Example 24: The computer-readable storage medium of Example 23, wherein the spatiotemporal neural network is further configured to:

[0221] The composite radar data is analyzed in a spatial domain to generate signature data; the signature data is analyzed in a time domain to generate radar application data; and the radar application data is communicated to a radar-based application.

Claims

1. A method performed by a radar system, the method comprising: transmitting a radar transmit signal using an antenna array of the radar system; receiving, using the antenna array, a radar receive signal, the radar receive signal comprising a version of a radar transmit signal reflected by at least one user; generating composite radar data based on the radar received signal; Providing the composite radar data to a spatiotemporal neural network of the radar system, the spatiotemporal neural network comprising a spatial recurrent network and a temporal recurrent network; as well as Analyzing the composite radar data using the spatiotemporal neural network to identify a gesture performed by the at least one user, analyzing the composite radar data comprises: generating channel Doppler data for each range interval associated with the composite radar data by separately processing portions of the composite radar data associated with different range intervals using the spatially recurrent network; generating feature data associated with the gesture by analyzing the channel Doppler data across the different range intervals using the spatial recursive network; as well as A prediction of the pose is formed by analyzing the feature data associated with at least two different time frames using the time recursive network.

2. The method according to claim 1, wherein: Analysis of the composite radar data includes analyzing both amplitude and phase information of the composite radar data using machine learning techniques to identify the gesture.

3. The method according to claim 1, further comprising: storing the feature data in a circular buffer; as well as The feature data stored in the circular buffer is accessed by the time recursive network.

4. The method according to claim 1, wherein: The parts that process the composite radar data individually include: Portions of the composite radar data associated with the different range intervals are processed separately using a non-linear activation function to generate the channel Doppler data for each range interval.

5. The method according to claim 1, wherein: The composite radar data includes at least one of the following: Composite range-Doppler map; Composite interferometry data; a plurality of digital beat signals associated with the radar receive signal; and A frequency domain representation of the plurality of digital beat signals.

6. The method according to claim 1, wherein: The gesture includes at least one of the following: Sweeping gesture; reaching hand gesture; Handle turning gesture; and Axis twist attitude.

7. The method according to any one of claims 1 to 6, wherein: Receiving the radar receive signal includes receiving multiple versions of the radar receive signal using different antenna elements of the radar system; as well as Generating the composite radar data includes generating a digital beat signal using corresponding receive channels of the radar system, the corresponding receive channels being connected to the different antenna elements, respectively.

8. An apparatus comprising: A radar system, the radar system comprising: Antenna arrays; A transceiver, the transceiver being configured to: transmitting a radar transmission signal using the antenna array; receiving a radar receive signal using the antenna array, the radar receive signal comprising a version of the radar transmit signal reflected by at least one user; and generating composite radar data based on the radar receive signal; and A processor and a computer-readable storage medium, the processor and the computer-readable storage medium configured to implement a spatiotemporal neural network, the spatiotemporal neural network configured to: receiving the composite radar data; generating channel Doppler data for each range interval associated with the composite radar data by separately processing portions of the composite radar data associated with different range intervals using a spatially recurrent network of the spatiotemporal neural network; generating feature data associated with a gesture performed by the at least one user by analyzing the channel Doppler data across the different range intervals using the spatial recursive network; and A prediction of the pose is formed by analyzing the feature data associated with at least two different time frames using a temporal recurrent network of the spatiotemporal neural network.

9. The device according to claim 8, wherein: The apparatus includes a smart device, and the smart device includes one of the following: Smartphone; Smart watches; Smart speakers; Smart thermostat; Security cameras; Vehicle; or Household appliances.

10. The apparatus according to claim 8, further comprising: a circular buffer configured to store the feature data, Wherein the spatiotemporal neural network is configured to form the prediction by accessing the feature data stored in the circular buffer using the temporal recursive network.

11. The device according to claim 10, wherein: The antenna array comprises a plurality of antenna elements; and The transceiver is configured to: receiving multiple versions of the radar receive signal using the multiple antenna elements; and The composite radar data is generated based on the multiple versions of the radar receive signal.

12. A non-transitory computer-readable storage medium comprising computer-executable instructions that, in response to being executed by a processor, implement: A spatiotemporal neural network, wherein the spatiotemporal neural network is configured to: receiving composite radar data associated with a radar receive signal reflected by at least one object; generating channel Doppler data for each range interval associated with the composite radar data by separately processing portions of the composite radar data associated with different range intervals using a spatially recurrent network of the spatiotemporal neural network; generating feature data associated with a gesture performed with the object by analyzing the channel Doppler data across the different range intervals using the spatially recurrent network of the spatiotemporal neural network; and A prediction of the pose is formed by analyzing the feature data associated with at least two different time frames using a temporal recurrent network of the spatiotemporal neural network.

13. The computer-readable storage medium of claim 12, wherein: The spatiotemporal neural network is further configured to recognize the gesture based on the prediction; and The gesture includes at least one of the following: Sweeping gesture; reaching hand gesture; Handle turning gesture; and Axis twist attitude.

14. The computer-readable storage medium of claim 13, wherein: The at least one object comprises a user; The spatiotemporal neural network is configured to transmit the recognized gesture to a radar-based application; as well as The computer-executable instructions, in response to being executed by the processor, implement the radar-based application, the radar-based application configured to display content to the user based on the recognized gesture.

15. The computer-readable storage medium of claim 14, wherein: The spatiotemporal neural network is further configured as follows: prompting the user to perform the gesture; recording the composite radar data as training data when the user performs the gesture; as well as A training process is performed to recognize future gestures performed by the user.

16. The computer-readable storage medium of claim 12, wherein: The at least one object comprises a stylus; and The gesture is performed by a user using the stylus.

17. The computer-readable storage medium of claim 12, wherein: The at least one object includes an inanimate object and a user; The spatiotemporal neural network is configured to identify a potential collision between the inanimate object and the user based on the prediction; as well as The computer-executable instructions, in response to being executed by the processor, implement a radar-based application configured to warn the user of the potential collision.

18. The computer-readable storage medium of claim 12, wherein: The at least one object comprises a user; The spatiotemporal neural network is configured to measure vital signs of the user; as well as The computer-executable instructions, in response to being executed by the processor, implement a radar-based application configured to notify the user of the measured vital signs.

19. The computer-readable storage medium according to any one of claims 12 to 18, wherein: The computer executable instructions, in response to being executed by the processor, implement a hardware abstraction module, the hardware abstraction module being configured to: generating hardware-independent composite radar data based on the composite radar data; and The hardware-independent composite radar data is provided to the spatiotemporal neural network as the composite radar data.

Images