RF-based micro-motion tracking for gesture tracking and recognition

Abstract

The present application relates to RF-based micro-motion tracking for gesture tracking and recognition. Techniques for radio frequency (RF)-based micro-motion tracking are described herein. These techniques enable even millimeter-scale hand movements to be tracked. To do so, radar signals from radar systems are used, which, using conventional techniques, only allow a resolution of centimeters or more.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201680021212.5, filed on April 29, 2016.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This application claims priority under 35 USC §119(e) to U.S. Provisional Application Serial No. 62 / 155,357 filed on April 30, 2015 and U.S. Provisional Application Serial No. 62 / 167,823 filed on May 28, 2015, the disclosures of which are incorporated herein by reference in their entirety. Background Art

[0005] Small screen computing devices continue to grow, such as smart phones, computing bracelets, rings and watches. Like many computing devices, these small screen devices often use virtual keyboards to interact with users. However, on some small screens, many people find it difficult to interact through virtual keyboards because they often result in slow and inaccurate input. This makes users feel cumbersome and limits the applicability of small screen computing devices.

[0006] To address this problem, optical finger and hand tracking techniques have been developed that enable tracking of gestures that are not made on the screen. However, these optical techniques are bulky, costly, or inaccurate, thereby limiting their usefulness in addressing the usability problem of small-screen computing devices. Other conventional techniques have been tried with limited success, including radar tracking systems. These radar tracking systems have difficulty determining small gesture movements without a large, complex, or expensive radar system because the resolution of the radar tracking system is constrained by the hardware of the radar system. Summary of the invention

[0007] This article describes techniques for radio frequency (RF) based micro-motion tracking. These techniques enable even millimeter-scale hand movements to be tracked. To do so, radar signals from radar systems are used, which, using conventional techniques, only allow a resolution of centimeters or more.

[0008] This Summary is provided to introduce simplified concepts related to RF-based micromotion tracking that is further described below in the Detailed Description. This Summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments of the technology and apparatus for RF-based micro-motion tracking are described with reference to the following drawings. The same numerals are used throughout the drawings to refer to the same features and components:

[0010] Figure 1 The resolution of the hardware constraints of a conventional system is illustrated.

[0011] Figure 2 An example environment in which the techniques for RF-based micro-motion tracking can be implemented is illustrated. The environment illustrates a fairly simple radar system through which the techniques for micro-motion tracking can overcome the problems of Figure 1 The hardware limitations of conventional radar systems are shown.

[0012] Figure 3 A computing device by which determination of RF-based micro-motion tracking can be implemented is illustrated.

[0013] Figure 4 Picture shows Figure 2 A fairly simple radar system in the avionics package along with hands that move within the provided radar field.

[0014] Figure 5 The velocity distribution of the hand points, the relative velocity diagram, and the relative displacement diagram are shown.

[0015] Figure 6 An example gesture determined by RF-based micro-motion tracking is illustrated, with micro-motions of the thumb relative to the fingers, similar to rotating the sawtooth gears of a traditional mechanical watch.

[0016] Figure 7 An example method for implementing gesture recognition through RF-based micro-motion tracking is illustrated.

[0017] Figure 8 An example gesture is illustrated in three sub-gesture steps, the gesture being used to press a virtual button.

[0018] Fig. 9 An example rotational micro-motion gesture that allows fine movement and control through RF-based micro-motion tracking is illustrated in four steps.

[0019] Fig.10 An example computing system that embodies or in which techniques that enable the use of RF-based micromotion tracking are illustrated. DETAILED DESCRIPTION

[0020] Overview

[0021] This article describes techniques for implementing RF-based micro-motion tracking that tracks millimeter-scale hand motions from radar signals even from conventional resolution radar systems with hardware constraints coarser than the millimeter-scale resolution being tracked.

[0022] The gesturing hand is a complex, non-rigid object with multiple dynamic components. Because of this, the range and velocity of hand subcomponents such as fingertips, palms, or thumbs are often sub-resolution-limited by conventional hardware. As a result, conventional hardware must be large, expensive, or complex to track small movements. Even for those conventional hardware that are capable of tracking small movements, the tracking algorithms are computationally limited for real-time gesture recognition applications.

[0023] consider Figure 1 1. The hardware-constrained resolution of a conventional system is shown. Here, the hardware-constrained spatial resolution 102 consists of a cross-range resolution 104 and a range resolution 106. The cross-range resolution 104 depends on the antenna beamwidth 108, while the range resolution 106 depends on the bandwidth 110, both of which are based on the hardware of the conventional radar system. The bandwidth 110 can be expressed as a pulse width or a wavelength.

[0024] In order to obtain better resolution, multiple antennas are often used in conventional radar systems, which increases complexity and cost. This is illustrated by a radar field 112 provided by a conventional radar system 114 having three separate radar transmitting elements 116 and antenna 118. Reflections are received from a hand 120 that is moving within the radar field 112 for each separate radar transmitting element 116. Therefore, each of the twelve elements 122 is constrained by the hardware of the radar system in terms of their size. Note that a micro-motion of the hand 120, such as moving the index finger relative to the thumb, will be within a specific element 122-1 in the element 122. In such a case, conventional systems and techniques cannot determine that the micro-motion was made.

[0025] Will Figure 1 and Figure 2 In contrast, the latter illustrates an environment 200 in which RF-based micro-motion tracking techniques can overcome the hardware limitations of conventional radar systems. In this illustration, a relatively simple radar system 202 is shown, which has a single radar transmit element 204 and a single antenna element 206. Figure 1 and comparing a single radar antenna 206 with multiple radar transmitting elements 116. Figure 11. The simple radar system 202 is compared to the multiple antennas 118 of the conventional radar system 114. The simple radar system 202 is simpler and may be less expensive, smaller, or less complex than the conventional radar system 114. In addition, even at the greater cost, size, or complexity of the conventional radar system 114, the conventional radar system 114 is unable to determine micro-movements of the hand 120, which requires higher resolution than the size of the element 122 allows.

[0026] As mentioned, for conventional techniques, radar systems have displacement sensing resolution limitations based on hardware parameters. These limitations are based on parameters of the system hardware, such that for a hardware-constrained spatial resolution 212 (shown with three examples), the resolution of a simple radar system 202 has a range resolution 208 and a cross-range resolution 210. However, as described below, the RF-based micro-motion tracking technique enables micro-motion tracking of smaller motions and, therefore, finer resolution than the limitations of the hardware constraints would conventionally suggest. Thus, the technique allows for relative displacement resolution that is finer than the wavelength or beamwidth of the radar system.

[0027] The present document now turns to an example computing device in which RF-based micro-motion tracking can be used, followed by example methods and gestures, and concludes with an example computing system.

[0028] Example computing device

[0029] Figure 3 A computing device by which RF-based micro-motion tracking can be determined is illustrated. Computing device 302 is illustrated using various non-limiting example devices, a desktop computer 302-1, a computing watch 302-2, a smart phone 302-3, a tablet computer 302-4, a computing ring 302-5, computing glasses 302-6, and a microwave oven 302-7, but other devices such as home automation and control systems, entertainment systems, audio systems, other home appliances, security systems, netbooks, automobiles, and e-readers may also be used. Note that computing device 302 can be wearable, non-wearable but mobile, or relatively non-mobile (e.g., desktop and appliances).

[0030] The computing device 302 includes one or more computer processors 304 and computer readable media 306, the latter including memory media and storage media. Applications and / or operating systems (not shown) embodied as computer readable instructions on the computer readable media 306 can be executed by the processor 304 to provide some of the functions described herein. The computer readable media 306 also includes a micro-motion tracking module 308 and a recognition module 310 described below.

[0031] The computing device 302 may also include one or more network interfaces 312 for transmitting data via a wired, wireless, or optical network, and a display 314. The network interface 312 may transmit data via a local area network (LAN), a wireless local area network (WLAN), a personal area network (PAN), a wide area network (WAN), an intranet, the Internet, a peer-to-peer network, a point-to-point network, a mesh network, etc. The display 314 may be integrated or associated with the computing device 302, such as integrated or associated with the desktop computer network 302-1.

[0032] The computing device 302 may also include or be associated with a radar system, such as Figure 2 The radar system 202 includes a radar transmitting element 204 and an antenna element 206. As mentioned above, the radar system 202 can be simpler, less expensive, or less complex than a conventional radar system that still cannot determine micro-motions on a millimeter scale using conventional technology.

[0033] The micro-motion tracking module 308 is configured to extract relative dynamics from the radar signal representing the superposition of reflections from two or more points of the hand within the radar field. Figure 4 Environment 400 Figure 2 4 , wherein the radar system 202 provides a radar field 402 in which a hand 404 can make movements. The hand 404 has various points of interest, some moving toward the radar antenna element 206, some moving away from it, and some not moving. This is illustrated at the thumb point 406, the index finger point 408, and the knuckle point 410. Assume that for a micro-motion gesture, the thumb point 406 moves away from the antenna element 206, the index finger point 408 moves toward the antenna element 206, and the knuckle point 410 does not move.

[0034] In more detail, for each of these points, the micro-motion tracking module 308 can determine their relative speed and energy. Thus, assume that the speed of the thumb point 406 is 1.7 meters per second away, the index finger point 408 is 2.1 meters per second approaching, and the knuckle point 410 is 0 meters per second. The micro-motion tracking module 308 uses the radar signal to determine the speed distribution of these points.

[0035] For example, consider the velocity profile 502 shown in FIG. Figure 5, the velocity profile 502 shows the velocity and energy of three points of the hand 404. The velocity profile 502 shows velocity vs. energy in arbitrary units with a higher energy measure for the thumb point 406, the index finger point 408, and the knuckle point 410. The velocity axis shows the knuckle point 410, which has not moved relative to the antenna element 206, but also shows the movement toward and away from the thumb point 406 and the index finger point 408. The absolute velocity of each point is shown for clarity, but is not required for this technique, relative velocities are sufficient, and a more lightweight process can be used than determining the absolute velocities and then comparing them to determine the relative velocities.

[0036] Using this velocity profile 502 and other previously determined or subsequently determined velocity profiles, the technique can determine the relative velocity between points of the hand 404. Here, the highest relative velocity is between the thumb point 406 and the index finger point 408. The micro-motion tracking module 308 can determine the relative velocity (and therefore the displacement) between the thumb point 406 and the knuckle point 410 or the index finger point 408, although the relative displacement between the thumb point 406 and the index finger point 408 is the largest relative displacement, which can improve gesture recognition and control refinement. However, in addition or alternatively, the resolution may be better than other points, such as in the case of noise or other signal quality issues for one or more points of the hand 404.

[0037] As mentioned, velocity profile 502 indicates the energy of each point of hand 404. The energy is a measure of the intensity of reflected energy as a function of the target range from each point to the transmitter or antenna element, e.g., radial distance from the radar transmitting element. The time delay between the transmitted signal and the reflection is observed by the Doppler frequency, and the radial velocity is determined and then integrated for the radial distance. The observation of the Doppler frequency can be made by the range-Doppler-time data cube of the radar signal, although such a format is not required. Regardless of the form of the data of the radar signal with the superposition of the reflections of the points, integrating the relative velocity can quantitatively combine the Doppler-determined relative dynamics of the radar signal and the unfolded signal phase. Alternatively or in addition, an extended Kalman filter can be used to integrate the raw phase with the Doppler centroid for the points of the hand, which allows for nonlinear phase unfolding.

[0038] In more detail, the following equations represent the manner in which the velocity profile 502 is determined. Equation 1 represents the incremental change in phase as a function of the incremental change in distance over a time period. More specifically, is the phase, so is the phase change, r i is the distance, Δr i is the displacement, and λ is the wavelength, so Δri (t,T) / λ is the variation of displacement with wavelength. Each incremental change in phase is equal to four times the displacement.

[0039] Equation 1

[0040]

[0041] Equation 2 expresses the time derivative of the phase 1 / 2π Proportional to the frequency f Doppler,i (T). Subsequently, inserting the time derivative of the displacement and wavelength 2 / λdr(t,T) / dT results in the velocity v which is also relative to the wavelength.

[0042] Equation 2

[0043]

[0044] Equations 1 and 2 show the relationship between the incremental velocity - such as the points of the hand making micro-movements - and how this is shown in the signals reflected from those points of the hand.

[0045] Equation 3 shows how to estimate the frequency of micromotion. This technique uses the Doppler centroid to calculate the Doppler spectrum f Doppler,centroid (T), which shows how much energy is at each frequency. This technique uses a central summation And each frequency corresponding to each micro-motion is taken out.

[0046] Equation 3

[0047] f Doppler,centroid (T) = Σ f f F(f)

[0048] Therefore, the technique constructs energy distributions such as Figure 2 4. An example velocity profile 502 of a finger moving at various velocities, such as thumb point 406, index finger point 408, and knuckle point 410. Based on the profile, the technique estimates specific micro-motions at specific energies in the profile as disclosed below.

[0049] Relative velocity graph 504 illustrates relative velocity 506 over time. Although shown for clarity of explanation, the absolute thumb velocity 508 of thumb point 406 and the absolute index velocity 510 of index finger 408 are not required. Relative velocity 506 can be determined without determining the absolute velocity. However, showing these graphs illustrates the relative velocity between these velocities and how it can change over time (note the deceleration of thumb point 406 from 2.1 units to 1.9 units in 6 time units).

[0050] Using the relative velocity 506 determined over the 6 time units, the relative displacement can then be determined by integrating the relative velocity. This is shown using a relative displacement graph 512, which illustrates a displacement trajectory 514. The displacement trajectory 514 is the change in displacement of the thumb point 406 relative to the index finger point 408 over the 6 time units. Therefore, the thumb point 406 and the index finger point 408 move apart by 24 arbitrary displacement units over the 6 time units.

[0051] In some cases, the micromotion tracking module 308 determines a weighted average of the relative velocities and then integrates the weighted average to find their relative displacement. The weighted average can be weighted based on velocity readings with a higher probability of having an accurate reading, lower noise, or other factors.

[0052] like Figure 5 As shown in the example of , the technology enables tracking of micro-movements, including the use of low-bandwidth RF signals. This allows tracking with standard RF devices such as Wi-Fi routers without having to modify the RF system or add complex or expensive radar systems.

[0053] return Figure 3 The recognition module 310 is configured to determine the gesture made by the hand based on the relative displacement of the points on the hand. The recognition module 310 can then pass the gesture to the application or device.

[0054] For example, suppose the gesture being determined is a micro-motion of the thumb against the fingers, similar to the rotation of a sawtooth wheel in a traditional mechanical watch. Figure 6 , which shows that the micro-gesture begins at a starting position 602, and that the micro-gesture ends at an ending position 604. Note that a movement 606 is made from the beginning to the end, but is not shown in the intermediate positions for visual simplicity. At the starting position 602, the thumb point 608 and the index finger point 610 are positioned relative to each other, with the end of the thumb at the tip of the finger. At the ending position 604, the thumb and therefore the thumb point 608 have moved a few millimeters on the finger and therefore the index finger point 610, which are each displaced relative to each other by those few millimeters. The technology is configured to track the gesture with a resolution finer than a few millimeters, but this shows the beginning and end without making intermediate measurements.

[0055] Using the displacement made by the micro-motion tracking module 308 between the thumb point 608 and the index finger point 610, the recognition module 310 determines the gesture and passes the gesture (usually as multiple sub-gestures when making a full gesture with sub-gesture parts) to the application - here to the application of the smart watch, which then changes the user interface 612 to scroll the text being displayed upward (the scrolling shown by the scroll arrow 614, and the results shown in the start text 616 and the end text 618). The tracked gestures can be large or small - it does not require millimeter scale, and does not use a single hand or even a human hand, because it is possible to track devices such as robotic arms that are tracked to determine control of a robot. Therefore, the micro-motion tracking module 308 can track micro-gestures with millimeter or finer resolution and overall relative displacements of up to five centimeters, or track a user's arm, hand or finger relative to another hand, arm or object, or track larger gestures, such as multi-hand gestures with relative displacements of up to one meter in size.

[0056] Example Method

[0057] Figure 7 A method 700 for identifying gestures using RF-based micro-motion tracking is depicted. The method 700 receives a radar signal from a radar system in which a hand is making a gesture, determines a displacement based on parameters of the radar system at a finer resolution than conventional techniques allow, and then determines a gesture, even a micro-motion gesture on a millimeter scale, based on the displacement. The method is shown as a collection of boxes specifying operations performed, but is not necessarily limited to the order or combination shown for performing the operations performed by the corresponding boxes. In various portions of the following discussion, reference may be made to Figure 2-6 , Figure 8 as well as Fig. 9 , references to which are made are by way of example only. The technology is not limited to being performed by one entity or multiple entities operating on one device, or those entities depicted in these figures.

[0058] At 702, providing Figure 2 The radar field shown. This radar field can be provided by a simple radar system including existing WiFi radars, and does not require the use of complex, multi-transmitter or multi-antenna or narrow beam scanning radars. Instead, a wide beam, fully continuous radar field can be used, such as 57-64 or 59-61 GHz, but other frequency bands and even acoustic waves can also be used.

[0059] At 704, a radar signal representing a superposition of reflections of the hand's points within the radar field is received. As mentioned, this can be received from as few as a single antenna. Each point of the hand has movement relative to the transmitter or antenna, and therefore has movement relative to each other point. As mentioned below, as few as two points can be represented and analyzed.

[0060] The radar signal may be filtered, such as with a moving target indicator (MTI) filter, at 706. Filtering the radar signal is not required, but can remove noise and help locate elements of the signal, such as those points that represent greater movement than other points.

[0061] At 708, a velocity profile is determined based on the radar signal. Examples of this determination are described above such as Figure 5 is provided.

[0062] At 710, relative velocities are extracted from the velocity profile. In order to determine multiple relative velocities over time, one or more predetermined or subsequently determined velocities are also determined. Figure 7 As shown by the repeating arrows in FIG. 7 , the technique can repeat operations 704 and 708 .

[0063] At 712, a displacement trajectory is determined by integrating a plurality of relative velocities. Relative velocities extracted from a plurality of velocity profiles at a plurality of times are integrated. An example of this is shown in Figure 5 This is shown in the relative displacement diagram 512 .

[0064] At 714, a gesture is determined based on the displacement trajectory between the plurality of points of the hand. As mentioned above, the gesture can be delicate and small, such as a micro-gesture performed by one hand, or multiple hands or objects, or be larger in size.

[0065] At 716, the gesture is communicated to the application or device. The gesture, when received by the application or device, can control the application or device, such as controlling or changing the display, function, or capability of the application or device. The device can be remote, peripheral, or a system on which method 700 is executed.

[0066] The displacement trajectory is determined in Figure 6 The example of shows the displacement between a point at a knuckle and another point at a fingertip, both of which are moving. This RF-based micro-motion tracking technique is not required to track all points of the hand, or even many points, or even two points in three-dimensional space. Instead, determining the relative displacement from one point to another can be sufficient to determine gestures, even those that are a millimeter or finer.

[0067] Through the operation of method 700, relative dynamics are extracted from the radar signal representing the superposition of reflections of multiple points of the hand within the radar field. These relative dynamics indicate the displacement of the points of the hand relative to each other, from which micro-motion gestures can be determined. As mentioned above, in some cases, the micro-Doppler centroid of the points is determined from the superposition. These micro-Doppler centroids enable the determination of computationally lightweight super-resolution velocity estimates. Therefore, the required computing system is relatively low compared to conventional radar technology, which further enables the use of these RF-based micro-motion technologies in small or resource-constrained devices such as some wearable devices and applications. These technologies can not only be used on resource-constrained devices, but also the computationally lightweight determination can allow faster response to gestures, such as real-time response when making small fine gestures (e.g., micro-gestures) to perform small fine control of the device.

[0068] Additionally, by using the micro-Doppler centroid, the RF-based micro-motion technique allows for greater robustness against noise and clutter than using the Doppler profile peak.To improve resolution, the micro-motion tracking module 308 can use phase changes of the radar signal to extract millimeter and sub-millimeter displacements of high frequency movements of points.

[0069] Example gestures

[0070] Figure 2-7 The RF-based micro-motion techniques described in

[14] enable gestures even at the millimeter or sub-millimeter scale. For example, consider Figure 8 and 9 , which illustrates two such gestures.

[0071] Figure 8 A gesture with a resolution of 10 mm or less is illustrated in three sub-gesture steps. Each gesture can be tracked by the micro-motion module 308 and the sub-gestures determined by the recognition module 310. Each of these sub-gestures can effect a control, or may require the last sub-gesture to be completed before a control is effected. This can be driven by the application or recognition module 310 receiving the gesture, as the recognition module 310 may have parameters from the application or device intended to receive the gesture, such as data indicating that only thumb and finger contact completions should be passed to the application or device.

[0072] for Figure 8This is the case for the hand gesture 808, which shows the starting position 802 of two points 804 and 806 of the hand 808, where the user interface shows an unpressed virtual button 810. A first sub-gesture 812 is shown, where the two points 804 and 806 (fingertips and thumb tip) move closer to each other. A second sub-gesture 814 is also shown, where the two points 804 and 806 move closer. A third sub-gesture 816 completes the gesture, where the two points 804 and 806 touch or are close to touching (depending on whether the points 804 and 806 are completely at the fingertips of the finger and thumb or are offset). It is assumed here that the micro-motion tracking module 308 determines the displacement between the points 804 and 806 in each of the three sub-gestures 812, 814 and 816, and passes these to the recognition module 310. The recognition module 310 waits to pass the completion gesture when the points touch or are close to touching, at which time the application receives the gesture and indicates in the user interface that the button has been pressed, which is shown as a pressed virtual button 818.

[0073] As another example, consider Fig. 9 , which illustrates a rotation micro-motion gesture 900. The rotation micro-motion gesture 900 involves movement of a thumb 902 relative to an index finger 904, wherein the thumb 902 and the index finger 904 move in generally opposite directions, a thumb direction 906 and an index finger direction 908.

[0074] The rotation micro-motion gesture 900 is shown at a starting position 910, and has four sub-gesture positions 912, 914, 916, and 918, but these are shown for visual simplicity, as many additional movements can be identified in the full gesture even at sub-millimeter resolution. To better visualize the effect of the rotation micro-motion gesture 900, consider the labeled wheel 920. The labeled wheel 920 is not held by the thumb 902 and index finger 904, but is shown to assist the reader in seeing the manner in which the gesture can be identified and used for fine resolution control when it is performed, similar to the manner in which the marker 922 moves as the labeled wheel 920 is rotated from the starting position at marker 922-1 to marker 922-2, to marker 922-3, to marker 922-4, and ends at marker 922-5.

[0075] When performing the rotation micro-motion gesture 900, the micro-motion tracking module 308 determines the displacement trajectory between one or more points of each of the thumb 902 and the index finger 904, and transmits these to the gesture module 310, which in turn determines the gesture or part thereof being performed. The gesture is transmitted to the device or application, which is thus controlled by the micro-motion gesture. For this type of micro-motion, the application can advance the media being played (or vice versa when the gesture is performed backward), scroll through the text or content in the display, turn up the volume of the music, the temperature of the thermostat, or another parameter. In addition, because the RF-based micro-motion technology has high resolution and lightweight computing requirements, it is possible to recognize fine movements in real time, which allows the user to easily set the exact desired control by moving her thumb and fingers back and forth, such as an exact amount 34 on a 100 scale or accurately finding a frame in the playing video.

[0076] Example computing system

[0077] Fig.10 The various components of an example computing system 1000 are illustrated, which can be implemented as described above with reference to FIG. Figure 2-8 Any type of client, server, and / or computing device described may be used to implement RF-based micro-motion tracking.

[0078] The computing system 1000 includes a communication device 1002 that supports wired and / or wireless communication of device data 1004 (e.g., received data, data being received, data scheduled for broadcast, data packets of data, etc.). The device data 1004 or other device content may include configuration settings of the device, media content stored on the device, and / or information associated with a user of the device (e.g., the identity of an operator performing a gesture). The media content stored on the computing system 1000 may include any type of audio, video, and / or image data. The computing system 1000 includes one or more data inputs 1006, through which any type of data, media content, and / or input can be received, such as user speech, interaction with a radar field, user selectable input (explicit or implicit), messages, music, television media content, recorded video content, and any other type of audio, video, and / or image data received from any content and / or data source.

[0079] The computing system 1000 also includes a communication interface 1008, which can be implemented as any one or more of a serial and / or parallel interface, a wireless interface, any type of network interface, a modem, and any other type of communication interface. The communication interface 1008 provides a connection and / or communication link between the device and a communication network through which other electronic, computing, and communication devices can transmit data with the computing system 1000.

[0080] The computing system 1000 includes one or more processors 1010 (e.g., any microprocessor, controller, etc.) that process various computer executable instructions to control the operation of the computing system 1000 and implement techniques for or in which RF-based micromotion tracking can be implemented. Alternatively or in addition, the computing system 1000 can be implemented using any one or combination of hardware, firmware, or fixed logic circuits implemented in conjunction with processing and control circuits generally identified at 1012. Although not shown, the computing system 1000 can include a system bus or data transmission system that couples various components within the device. The system bus can include any one or combination of different bus structures, such as a memory bus or memory controller, an external bus, a universal serial bus, and / or a processor or local bus utilizing various bus architectures.

[0081] The computing system 1000 also includes computer-readable storage media 1014, such as one or more memory devices that implement persistent and / or non-persistent data storage (i.e., as opposed to mere signal transmission), examples of which include random access memory (RAM), non-volatile memory (e.g., any one or more of read-only memory (ROM), flash memory, EPROM, EEPROM, etc.), and disk storage devices. The disk storage device may be implemented as any type of magnetic or optical storage device, such as a hard drive, a recordable and / or writable compact disk (CD), any type of digital versatile disk (DVD), etc. The computing system 1000 may also include a mass storage media device (storage media) 1016.

[0082] The computer-readable storage medium 1014 provides a data storage mechanism to store device data 1004, as well as various device applications 1018 and any other type of information and / or data related to the operational aspects of the computing system 1000. For example, the operating system 1020 can be saved as a computer application using the computer-readable storage medium 1014 and executed on the processor 1010. The device applications 1018 can include device managers, such as any form of control applications, software applications, signal processing and control modules, code local to a specific device, abstract modules or gesture modules, etc. The device applications 1018 also include any system components, engines or managers used to implement RF-based micro-motion tracking, such as the micro-motion tracking module 308 and the recognition module 310.

[0083] Computing system 1000 also includes or has access to one or more of the radar systems, such as radar system 202 having radar transmit element 204 and antenna element 206. Although not shown, one or more elements of micromotion tracking module 308 and identification module 310 may be operated in whole or in part through hardware or firmware.

[0084] in conclusion

[0085] Although techniques for using RF-based micro-motion tracking and apparatuses including RF-based micro-motion tracking have been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features and acts described above. Rather, the specific features and methods are disclosed as example implementations of ways to determine RF-based micro-motion tracking.

Claims

1. One or more non-transitory computer-readable storage media having instructions stored thereon, the instructions implementing a micro-motion tracking module and a recognition module in response to being executed by one or more computer processors: The micro-motion tracking module is configured to: receiving a first radar signal representing reflections of a radar field from first and second points of a hand at a first time, the first and second points of the hand moving relative to each other within the radar field; determining velocities of the first point and the second point of the hand at the first time based on energy and Doppler frequency observations of the first radar signal; receiving a second radar signal representing reflections of the radar field from the first point and the second point of the hand at a second time; determining velocities of the first point and the second point of the hand at the second time based on energy and Doppler frequency observations of the second radar signal; calculating a relative speed between the first point and the second point of the hand based on the speeds of the first point and the second point of the hand at the first time and the second time; as well as determining a millimeter-scale change in displacement between the first point and the second point of the hand based on the relative velocity between the first point and the second point of the hand and an elapsed time between the first time and the second time; And the identification module is configured to: Based on the displacement change, determining the gestures of the first point and the second point of the hand; as well as The gestures are communicated for controlling or changing the display, functions, or capabilities of the device.

2. The one or more non-transitory computer-readable storage media of claim 1, wherein: The velocities of the first point and the second point at the first time and the second time are determined based on the micro-Doppler centroids of the first radar signal and the second radar signal.

3. The one or more non-transitory computer-readable storage media of claim 1, wherein: The energy of the first radar signal and the second radar signal is a measure of the intensity of reflected energy as a function of target range to the first point and the second point of the hand.

4. The one or more non-transitory computer-readable storage media of claim 1, wherein: The first radar signal and the second radar signal are received via a single antenna element.

5. The one or more non-transitory computer-readable storage media of claim 1, wherein: An extended Kalman filter is used to integrate the raw phase with the Doppler centroid for the first point and the second point of the hand to allow for nonlinear phase unwrapping of the first radar signal and the second radar signal.

6. A computer-implemented method comprising: receiving a first radar signal representing reflections of a radar field from first and second points of a hand at a first time, the first and second points of the hand moving relative to each other within the radar field; determining velocities of the first point and the second point of the hand at the first time based on energy and Doppler frequency observations of the first radar signal; receiving a second radar signal representing reflections of the radar field from the first point and the second point of the hand at a second time; determining velocities of the first point and the second point of the hand at the second time based on energy and Doppler frequency observations of the second radar signal; calculating a relative speed between the first point and the second point of the hand based on the speeds of the first point and the second point of the hand at the first time and the second time; as well as Based on the relative velocity between the first point and the second point of the hand and the elapsed time between the first time and the second time, a millimeter-scale change in displacement between the first point and the second point of the hand is determined.

7. The method according to claim 6, wherein: The radar field is emitted by a single radar element, and the first radar signal and the second radar signal are received by a single antenna.

8. The method according to claim 6, wherein: The energy of the first radar signal and the second radar signal is a measure of the intensity of reflected energy as a function of target range to the first point and the second point of the hand.

9. The method according to claim 6, wherein: The velocities of the first point and the second point at the first time and the second time are radial velocities determined by the Doppler frequencies of the first radar signal and the second radar signal.

10. The method according to claim 9, wherein: The velocities of the first point and the second point at the first time and the second time are determined based on the micro-Doppler centroids of the first radar signal and the second radar signal.

11. The method according to claim 6, wherein: The first radar signal and the second radar signal include a superposition of energy corresponding to the first point and the second point of the hand.

12. The method according to claim 6, wherein: The change in displacement is determined by integrating the relative velocity.

13. A device for gesture tracking and recognition, comprising: one or more computer processors; Radar system, including: one or more radar transmit elements configured to provide a radar field; and one or more antenna elements configured to receive radar signals representing reflections of the radar field from two or more points of a hand that move relative to each other within the radar field; and One or more computer-readable storage media having instructions stored thereon, wherein the instructions, in response to being executed by the one or more computer processors, implement a micro-motion tracking module and a recognition module: The micro-motion tracking module is configured to: causing the radar transmitting element to provide the radar field; receiving a first radar signal from the antenna element, the first radar signal representing reflections of the radar field from a first point and a second point on a hand at a first time; determining velocities of the first point and the second point of the hand at the first time based on energy and Doppler frequency observations of the first radar signal; receiving a second radar signal from the antenna element, the second radar signal representing reflections of the radar field from the first point and the second point of the hand at a second time; determining velocities of the first point and the second point of the hand at the second time based on energy and Doppler frequency observations of the second radar signal; calculating a relative speed between the first point and the second point of the hand based on the speeds of the first point and the second point of the hand at the first time and the second time; and determining a millimeter-scale displacement change between the first point and the second point of the hand based on the relative speed between the first point and the second point of the hand and the elapsed time between the first time and the second time; and the recognition module is configured to: Determining gestures of the first point and the second point of the hand based on the displacement change; and The gestures are communicated for controlling or changing the display, functions, or capabilities of the device.

14. The device according to claim 13, wherein: The velocities of the first point and the second point at the first time and the second time are determined based on the micro-Doppler centroids of the first radar signal and the second radar signal.

15. The device according to claim 13, wherein: The energy of the first and second radar signals is a measure of the intensity of reflected energy as a function of target range to the first and second points of the hand.

16. The device according to claim 13, wherein: The antenna element is a single antenna element.

17. The device according to claim 13, wherein: An extended Kalman filter is used to integrate the raw phase with the Doppler centroid for the first point and the second point of the hand to allow for nonlinear phase unwrapping of the first radar signal and the second radar signal.

18. The device according to claim 13, wherein: The device comprises a mobile phone or a tablet computer.

19. A computer-implemented method comprising: receiving a radar signal representing a superposition of reflections from two or more points of a hand within a radar field provided by a radar system, a first point of the two or more points having a first movement and a second point of the two or more points having a second movement; determining a velocity profile of the radar signal, the velocity profile indicating energies at a plurality of velocities, a first amount of the energies representing reflections of the first point having the first movement and a second amount of the energies representing reflections of the second point having the second movement; extracting a relative velocity between the first point and the second point of the hand from the velocity profile and one or more predetermined or subsequently determined velocity profiles over time; determining a displacement trajectory by integrating the relative velocity, the displacement trajectory representing the relative displacement between the first point and the second point of the hand; determining a gesture based on a relative displacement between the first point and the second point during the first movement and the second movement; as well as The gesture is communicated to an application or device to effectively control or change capabilities associated with the application or the device.

20. The method according to claim 19, wherein: A velocity profile of the radar signal representing the superposition is determined to determine a first micro-Doppler centroid and a second micro-Doppler centroid for the first point and the second point, respectively.

21. The method of claim 20, further comprising determining the first point and the second point from the two or more points based on energies associated with the first micro-Doppler centroid and the second micro-Doppler centroid being higher than other energies associated with other points of the two or more points.

22. The method of claim 19, further comprising increasing the resolution of the displacement trajectory by extracting millimeter or sub-millimeter displacements using phase changes of the radar signal.

23. The method of claim 19, further comprising filtering the radar signal prior to determining the velocity profile.

24. The method according to claim 23, wherein: Filtering the radar signal includes isolating the first point from the second point using a moving target indicator (MTI) filter to determine a point of the two or more points having a greater movement.

25. The method of claim 19, wherein: The first point is a first finger and the second point is a second finger, and a relative displacement between the first point and the second point during the first movement and the second movement indicates a millimeter scale movement of the first finger relative to the second finger.

26. The method according to claim 25, wherein: The resolution of the relative displacement of the millimeter-scale movement is one millimeter or finer.

27. One or more computer-readable storage media having stored thereon instructions that, in response to being executed by one or more computer processors, implement a micro-motion tracking module and a recognition module: The micro-motion tracking module is configured to: receiving a radar signal representing a superposition of reflections from two or more points of a hand within a radar field provided by a radar system, a first point of the two or more points having a first movement and a second point of the two or more points having a second movement; determining a velocity profile of the radar signal, the velocity profile indicating energies at a plurality of velocities, a first amount of the energies representing reflections of the first point having the first movement and a second amount of the energies representing reflections of the second point having the second movement; extracting a relative velocity between the first point and the second point of the hand from the velocity profile and one or more predetermined or subsequently determined velocity profiles over time; as well as A displacement trajectory is determined by integrating the relative velocity, the displacement trajectory representing the relative displacement between the first point and the second point of the hand, and the recognition module is configured to: determining gestures of the first point and the second point of the hand based on the displacement trajectory; as well as The ability to transmit said gestures to effectively control or change the device.

28. The computer-readable storage medium of claim 27, wherein: The micro-motion tracking module is further configured to determine a weighted average of the relative velocity and integrate the relative velocity to integrate the weighted average of the relative velocity.

29. The computer-readable storage medium of claim 27, wherein: Determining the velocity profile of the radar signal uses a range-Doppler-time data cube of the radar signal.

30. The computer-readable storage medium of claim 29, wherein: Integrating the relative velocity quantitatively combines the relative velocity from the range-Doppler-time data cube of the radar signal and the unwrapped signal phase of the radar signal.

31. The computer-readable storage medium of claim 27, wherein: The displacement trajectory represents sub-gesture steps of a complete gesture, and wherein the micro-motion tracking module is further configured to determine multiple subsequent displacement trajectories, and the recognition module is further configured to determine sub-gestures of the complete gesture and transmit the sub-gestures to effectively control or change the display, function or capability of the device in real time.

32. A device for gesture tracking and recognition, comprising: one or more computer processors; Radar system, including: one or more radar transmit elements configured to provide a radar field; and one or more antenna elements configured to receive a radar signal representing a superposition of reflections from two or more points of a hand within the radar field; and one or more computer-readable storage media having stored thereon instructions that in response to being executed by the one or more computer processors implement a micro-motion tracking module and a recognition module: The micro-motion tracking module is configured to: receiving a radar signal representing said superposition of reflections of said two or more points of a hand within a radar field provided by said radar system, a first point of said two or more points having a first movement and a second point of said two or more points having a second movement; determining a velocity profile of the radar signal, the velocity profile indicating energies at a plurality of velocities, a first amount of the energies representing reflections of the first point having the first movement and a second amount of the energies representing reflections of the second point having the second movement; extracting a relative velocity between the first point and the second point of the hand from the velocity profile and one or more predetermined or subsequently determined velocity profiles over time; and A displacement trajectory is determined by integrating the relative velocity, the displacement trajectory representing the relative displacement between the first point and the second point of the hand, and the recognition module is configured to: determining gestures of the first point and the second point of the hand based on the displacement trajectory; and The gesture is communicated to an application of the apparatus or a remote or peripheral device to effectively control or change the capabilities of the apparatus or the remote or peripheral device.

33. The device according to claim 32, wherein: The radar system has a displacement sensing resolution limit based on hardware parameters, and the resolution of the relative displacement is finer than the displacement sensing resolution limit based on hardware parameters.

34. The device according to claim 33, wherein The hardware parameter based displacement sensing resolution limit is based on a wavelength of the radar field provided by the radar system, and wherein a resolution of the relative displacement is finer than the wavelength.

35. The device according to claim 32, wherein: The one or more radar transmit elements is a radar transmit element and the one or more antenna elements is a radar antenna element, and the radar field provided by the radar system is a wide beam, fully continuous radar field.

Images