Wireless Measurement of Human-Product Interaction

Abstract

The present invention provides for inferring product activity, which includes: providing a first product with a first harmonic tag attached thereto; directing a first transmission signal of a first frequency at a first area where the first product is located; and receiving a first return signal of a first return frequency from the first harmonic tag, wherein the first harmonic tag radiates the first return signal after receiving the first transmission signal such that the first return frequency is a harmonic of the first frequency. A computer can then infer a first activity of using the first product based on the first return signal.

Description

Background Art

[0001] The present disclosure relates to tracking information about consumer products and, more particularly, to tracking motion-related information associated with the use of consumer products.

[0002] There are several methods for human activity recognition (HAR) based on spectrogram data (obtained via the Doppler effect, radar, sonar, etc.). Traditionally, manually selected features (e.g., low-level statistical parameters such as mean, variance, frequency, and amplitude) are used as inputs to train matching learning classifiers for HAR. Commonly used classifiers include support vector machines (SVMs), decision trees, and dynamic time warping (DTW). Such feature-based classifiers rely on domain knowledge and experience and typically have drawbacks such as poor robustness and generality. Recently, a method for classifying HAR includes feeding the raw amplitude spectrogram into a deep neural network (DNN) such that the feature extraction step can be bypassed. Popular choices for DNN architectures include convolutional neural networks (CNNs), recurrent neural networks (RNNs), and autoencoders (AEs). It has been shown that hybrid models that combine various DNN structures (such as CNNs or AEs) as automatic feature extractors plus using an RNN as a classifier provide excellent performance. Summary of the Invention

[0003] One aspect of the present invention relates to a method for inferring product activity, the method comprising: providing a first product having a first harmonic tag attached thereto; directing a first transmission signal of a first frequency at a first region where the first product is located; and receiving a first return signal of a first return frequency from the first harmonic tag, wherein the first harmonic tag radiates the first return signal after receiving the first transmission signal such that the first return frequency is a harmonic of the first frequency. A computer then infers a first activity of using the first product based on the first return signal.

[0004] Another aspect of the present invention relates to a method for inferring cumulative usage of a product having a harmonic tag attached thereto, the method comprising: directing a transmission signal of a first frequency at a first region; and receiving a return signal of a second frequency from the harmonic tag, wherein the harmonic tag radiates the return signal after receiving the transmission signal such that the second frequency is a harmonic of the first frequency. A computer can then determine one or more movement events of the harmonic tag over a period of time based on the return signal; and infer the cumulative usage of the product over the period of time based on the one or more movement events of the harmonic tag.

[0005] Another aspect of the present invention relates to a method for determining the expiration of the defined useful life of a product with an attached harmonic tag. The method includes: storing, by a computer, a value indicating the useful life of the product; directing a transmitted signal of a first frequency at a first region; and receiving a returned signal of a second frequency from the harmonic tag, wherein the harmonic tag radiates the returned signal after receiving the transmitted signal such that the returned signal is a harmonic of the transmitted signal. The computer can then determine, based on the returned signal, one or more movement events of the harmonic tag during a time period starting from the earliest determined movement event in the current life cycle of the product; and accumulate a count of the one or more movement events that occurred during the time period. The computer can also determine whether the defined life cycle of the product has expired based on the count of the one or more movement events that occurred during the time period.

[0006] Another aspect of the present invention relates to a method for inferring movement, the method including: providing a first product with an attached first harmonic tag for use by a person; directing a first transmitted signal of a first transmission frequency at a first region where the first product is located; receiving a first returned signal of a first returned frequency, wherein the first returned frequency and the first transmission frequency are substantially the same; and receiving a second returned signal of a second returned frequency from the first harmonic tag, wherein the first harmonic tag radiates the second returned signal after receiving the first transmitted signal such that the second returned frequency is a harmonic of the first transmission frequency. The computer can then determine the movement of the person based on the first returned signal and the movement of the first harmonic tag based on the second returned signal.

[0007] One aspect of the present invention relates to a system for inferring product activity, the system including: a first product having an attached first harmonic tag; a radar configured to direct a first transmitted signal of a first frequency at a first region where the first product is located, the radar being configured to receive a first returned signal of a first returned frequency from the first harmonic tag, wherein the first harmonic tag radiates the first returned signal after receiving the first transmitted signal such that the first returned frequency is a harmonic of the first frequency. The system further includes: a memory storing executable instructions; and a processor in communication with the memory. Specifically, execution of the executable instructions by the processor causes the processor to infer a first activity of using the first product based on the first returned signal.

[0008] Another aspect of the present invention relates to a system for inferring the cumulative usage of a product with an attached harmonic tag. The system includes a radar configured to direct a transmitted signal of a first frequency at a first region and configured to receive a returned signal of a second frequency from the harmonic tag, where the harmonic tag radiates the returned signal after receiving the transmitted signal such that the second frequency is a harmonic of the first frequency. The system further includes: a memory storing executable instructions; and a processor in communication with the memory. Specifically, execution of the executable instructions by the processor causes the processor to determine one or more movement events of the harmonic tag over a period of time based on the returned signal; and infer the cumulative usage of the product over the period of time based on the one or more movement events of the harmonic tag.

[0009] Another aspect of the present invention relates to a system for determining the expiration of a defined useful life cycle of a product with an attached harmonic tag. The system includes a radar configured to direct a transmitted signal of a first frequency at a first region and configured to receive a returned signal of a second frequency from the harmonic tag, where the harmonic tag radiates the returned signal after receiving the transmitted signal such that the returned signal is a harmonic of the transmitted signal. The system further includes: a memory storing executable instructions; and a processor in communication with the memory. Specifically, execution of the executable instructions by the processor causes the processor to store a value indicating the defined useful life cycle of the product; determine one or more movement events of the harmonic tag over a period of time starting from the earliest determined movement event in the current life cycle of the product based on the returned signal; accumulate a count of the one or more movement events that occur during the period; and determine whether the defined life cycle of the product has expired based on the count of the one or more movement events that occur during the period.

[0010] Another aspect of the present invention relates to a system for inferring movement. The system includes: a first product having an attached first harmonic tag for human use; a radar configured to direct a first transmitted signal of a first transmitted frequency at a first region where the first product is located, the radar being configured to receive a first returned signal of a first returned frequency, where the first returned frequency and the first transmitted frequency are substantially the same, and the radar being configured to receive a second returned signal of a second returned frequency from the first harmonic tag, where the first harmonic tag radiates the second returned signal after receiving the first transmitted signal such that the second returned frequency is a harmonic of the first transmitted frequency. The system further includes a memory storing executable instructions and a processor in communication with the memory. Specifically, execution of the executable instructions by the processor causes the processor to determine human movement based on the first returned signal; and determine movement of the first harmonic tag based on the second returned signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1AShows an example environment of a product with a harmonic tag in accordance with the principles of the present disclosure.

[0012] Figure 1B Is an illustration that provides more details about Figure 1A the example environment.

[0013] Figure 1C Shows a radar and a tagged object in accordance with the principles of the present disclosure.

[0014] Figure 1D Shows an example antenna of a harmonic tag in accordance with the principles of the present disclosure.

[0015] Figures 2A to 2C Is a flowchart of an example method for inferring information about a harmonic tag in accordance with the principles of the present disclosure.

[0016] Figures 3 to 4B Shows an example power spectrum diagram in accordance with the principles of the present disclosure.

[0017] Figures 5 to 7 Shows a process for utilizing different activity recognition models in accordance with the principles of the present disclosure.

[0018] Figure 8 Is a flowchart of an example process for determining the movement of a person and a harmonic tag in accordance with the principles of the present disclosure. Detailed Description

[0019] In the following detailed description of the illustrated embodiments, reference is made to the accompanying drawings, which form a part hereof, and in which are shown, by way of illustration and not limitation, specific embodiments in which the invention may be practiced. It is to be understood that other embodiments may be utilized and changes may be made without departing from the spirit and scope of the various embodiments of the invention.

[0020] Embodiments in accordance with the principles of the present disclosure relate to monitoring the use of relatively low-cost consumer products in at least some cases where it is not possible to embed radios or RFID tags in the products due to cost and RF exposure issues. As explained below, useful information (e.g., usage data for restocking) can be inferred if activities involving these types of products can be identified by non-video means. Examples of such products include toothbrushes, hairbrushes, and containers such as those that hold laundry detergent, shampoo, toothpaste, and the like.

[0021] Generally speaking, indoor radars have been used in research and commercial applications to analyze people and objects. For people, the phenomenon known as the micro-Doppler effect (a very small frequency shift caused by the movement of an object reflecting a radar signal) can be used to infer activities such as walking, running, falling, heartbeat, breathing, etc. However, it may be difficult to accurately measure and interpret signals from these activities, especially for activities that may look the same to the radar. The term "micromotion" is generally used to refer to the movement of limbs (e.g., legs, arms, hands) that move relative to a larger object (e.g., a person's torso). Micromotion characteristics can be used to infer the activities that a monitored person may be performing.

[0022] Harmonic tags can be made very small and typically consist of only a bent piece of wire and a non-linear electrical component such as a diode. In some cases, harmonic tags have been used to detect presence. A harmonic tag is a tag that receives electromagnetic energy at one frequency (e.g., the fundamental frequency) but then re-transmits the electromagnetic energy at a second frequency. Typically, the second frequency of the re-transmitted or re-radiated energy is a harmonic of the fundamental frequency.

[0023] Since harmonic tags re-transmit at different frequencies, their presence can be clearly distinguished from radar echoes at the fundamental frequency, which can consist of reflections, constructive interference, and destructive interference, etc. As described below, harmonic tags can be associated with or attached to a specific object. Thus, the presence or movement of an object in the returned radar data from the tag can be used to refine the estimate of the activity involving that object.

[0024] In addition, harmonic tags can be made to resonate at different frequencies. By transmitting at different frequencies and monitoring when different tags "appear" or "disappear", unique objects can be identified.

[0025] Now referring to the drawings and specifically to Figure 1A , a general environment is shown in which harmonic tags and Doppler radars can be implemented in accordance with the principles of the present disclosure.

[0026] Consumer 104 uses product 102 in a first area 100 such as a kitchen, laundry room, bathroom, etc. In the following description, using a toothbrush is given as an example of an activity that a user may perform. The toothbrush is provided by way of example only, and within the scope of the present invention, many different products are contemplated.

[0027] Product 102 (e.g., a toothbrush) may include a harmonic tag 103. Generally, the harmonic tag 103 is attached to the product 102 in an unobtrusive manner. The term "unobtrusive" is intended to mean that the tag 103 does not interfere with or affect the normal use of the product 103. The harmonic tag 103 may be attached by the manufacturer before the product 102 is sold to a consumer, or the harmonic tag 103 may be a separate item attached to the product 102 after the product 102 has been obtained by the consumer 104.

[0028] The first region 100 may include other objects and furniture 106 that do not have an attached or associated harmonic tag. As described below, the radar 108 is used to radiate energy as a continuous wave or a pulsed wave at one frequency and may detect the resulting return signal. The return signal may include a signal having the frequency of the transmitted signal, and due to the harmonic tag, the return signal may include a signal having a certain frequency that is a harmonic of the transmitted signal. Additionally, the radar 108 may radiate signals at multiple frequencies, for example, by sweeping through discrete frequencies, thereby generating multiple return signals each at a different frequency.

[0029] The radar 108 may include a processor or computer 110 that processes and analyzes the return signal. It is also contemplated that the processor or computer 110 may be separate from but coupled to the radar 108 to receive signals from the radar 108. As explained below, the analysis of the return signal may be used to infer an activity or movement event involving the product 102 having the associated harmonic tag 103.

[0030] Figure 1B More details of the radar 108 for the illustrated embodiment are shown. Specifically, the radar 108 has a transmitter section 122 that radiates or transmits electromagnetic energy at one or more different base frequencies. As Figure 1B shown, four different base frequencies 130A to 130D are shown by way of example. Thus, for example purposes, four different harmonic tags 128A to 128D are also shown. In the radar 108, a separate transmitter may be included for each base frequency, or alternatively, a single multi-frequency transmitter section that sweeps through different base frequencies may be used. Sweeping through frequencies means transmitting each base frequency individually for a predetermined period of time (e.g., 0.5 seconds or 2.0 seconds, etc.) and then repeating the transmission of the base frequency signal.

[0031] Radar 108 has a receiving section for receiving return signals generated from the fundamental frequency signals. A receiver section 124 may be tuned to receive return signals 132A to 132D that are at the fundamental frequency and caused by reflections of the transmitted fundamental frequency signals 130A to 130D. Different receiver sections 126 will be tuned to receive return signals 134A to 134D that are harmonics of the transmitted fundamental frequency signals 130A to 130D. The presence of harmonic tags within the first region 100 will generate return harmonic signals that can be detected by the radar 108. If there are no harmonic tags in the first region 100, then the return signals at the harmonic frequencies will not be reradiated to be detected by the radar 108. Each of the tags 128A to 128D may be associated with one of the fundamental frequencies 130A to 130D such that the tags 128A to 128D are capable of generating corresponding return signals 134A to 134D that are harmonics of the transmitted fundamental frequency signals 130A to 130D. Based on these return signals 134A to 134D, the radar 108 can detect the presence of a plurality of different products. Additionally, examples of using objects with attached harmonic tags to infer human activities are provided below. Using different harmonic tags for different objects allows for inferring the corresponding activities associated with the use of each different object. The different activities may occur simultaneously or may occur separately at different times from each other.

[0032] Figure 1C Details regarding an example radar 108A are shown in accordance with the principles of the present disclosure. Different from the example of Figure 1B the radar 108A of Figure 1C is shown as transmitting a single fundamental frequency. The transmitter 150 transmits a first signal 160 having the fundamental frequency such that the antenna 122A directs the signal to the first region where the product 102 is located. The harmonic tag 103 is associated with the object 102 and the object 102 is in motion as indicated by the "V" vector in Figure 1C . Accordingly, the radar 108A detects two return signals. One return signal 162 is the reflected fundamental frequency signal and is detected by the antenna 124A. In the example circuit shown, the return signal 162 may be filtered by the filter 156 to remove unwanted frequency components, such as frequencies above or below the fundamental frequency. In other words, the return signal may pass through a band - pass filter centered at the fundamental frequency. Then, the filtered signal may be mixed with the transmitted first fundamental frequency signal 160 in the mixer 154. Combining these two signals in this manner removes the fundamental frequency component, resulting in a base - band time - domain signal centered at 0 Hz.

[0033] As is well known, the movement of person 104 and object 102 causes the fundamental return signal 162 to include components indicating a slight Doppler shift due to this movement. Although the movements of both object 102 and person 104 can contribute to the fundamental return signal 162, the movement of person 104 contributes more to the fundamental return signal 162. Therefore, the analysis of the fundamental return signal 162 allows determination of how person 104 moves. The Doppler-shifted fundamental return signal 162 includes a component corresponding to the fundamental frequency and a component corresponding to the Doppler shift caused by the movement of person 104. The Doppler-shifted fundamental return signal has a frequency f RS = f(1 + 2v / c), where f is the fundamental frequency in Hz, c is the speed of light in m / s, and v is the speed of the person in m / s. As described above, the Doppler-shifted fundamental return signal 162 is filtered by filter 156 and then mixed with the transmitted first fundamental frequency signal 160 in mixer 154. Also as described above, combining the two signals in this way removes the fundamental frequency component, resulting in a baseband time-domain signal centered at 0 Hz.

[0034] The baseband time-domain signal can be processed using a well-known short-time Fourier transform (STFT) 166. In this way, a series of individual time segments (e.g., two seconds) of the fundamental return signal 162 can be processed and converted into a frequency-domain signal. As a result, a fundamental power spectrogram 178 is obtained, which includes the frequency components and their corresponding amplitudes in the fundamental return signal 162 due to the movement of person 104 using object 102. The movement of person 104 towards antenna 158 produces a Doppler shift in one direction (e.g., a positive sign), and the movement of person 104 away from antenna 158 produces a Doppler shift in a second direction (e.g., a negative sign). In the fundamental power spectrogram 178, the frequency values provide information about the movement speed of person 104, and the frequency amplitude values provide information about the "certainty" of the frequency values. For example, the power spectrogram 178 may indicate the presence of a frequency component of approximately 20 Hz with a very high amplitude and a frequency component of approximately 10 Hz with a relatively low amplitude. Automated analysis of this amplitude information can determine that due to the higher amplitude, the 20 Hz frequency component is not caused by noise, interference, or some other artifact of the radar's signal detection circuit. However, there may be some uncertainty about the 10 Hz frequency component because its amplitude is low, such that the automated analysis process can determine that the 10 Hz frequency component may not actually be present in the fundamental power spectrogram 177. As described above, the presence of frequency components in the fundamental power spectrogram indicates the movement of the person, and more specifically, the speed of that movement. The duration of the movement (i.e., the integral of the speed over time) can provide a rough estimate of the amount or magnitude of the person's movement (e.g., 6 cm).

[0035] Figure 1CThe radar also includes an antenna 126A that receives a harmonic return signal 176 from the harmonic tag 103. The return signal 176 includes a component indicating a slight Doppler shift caused by the movement of the object 102. Although it is a tag that provides a harmonic return signal, the tag is attached to the object 102. Therefore, the harmonic return signal 176 indicates the movement of the object 102. The Doppler-shifted harmonic return signal 176 includes a component corresponding to the harmonic frequency and a component corresponding to the Doppler shift caused by the movement of the object 102. The harmonic return signal 176 has a frequency f HRS = nf(1 + 2v / c), where n is the number of harmonics (e.g., 2), f is the fundamental frequency in Hz, c is the speed of light in m / s, and v is the speed of the person in m / s. The harmonic return signal 176 can also be filtered 170, such as using a band-pass filter, and mixed with a signal corresponding to the harmonic as the fundamental frequency. If the tag 103 re-radiates energy at the second harmonic, a frequency multiplier 164 is used to generate the second harmonic and feed it into the mixer 168. Similar to the above operations, the harmonic return signal 176 is converted into a baseband signal fluctuating around approximately 0 Hz, and this baseband signal can be transformed by the well-known short-time Fourier transform STFT 174 to generate a harmonic power spectrum 180, which includes the frequency components and their corresponding amplitudes in the harmonic return signal 176 due to the movement of the object 102. The movement of the object 102 towards the antenna 172 generates a Doppler shift in one direction (e.g., a positive sign), and the movement of the object 102 away from the antenna 172 generates a Doppler shift in a second direction (e.g., a negative sign).

[0036] Figure 1D An example geometry of the harmonic tag 103 according to the principles of the present disclosure is shown. Specifically, Figure 1D a dual-band slot dipole antenna that can be constructed on a laminated substrate is shown. The laminated substrate is a high-frequency circuit material with a thickness of 1.52 mm that can be commercially obtained from Rogers Corporation (Rogers Company) under the trade name RO3003. The antenna can be constructed from a 17-μm-thick copper laminate. The overall dimensions of the substrate can be 9.7 cm × 2.8 cm. As Figure 1D shown, the top horizontal leg can have a height of 2 mm and a length of 19.5 mm. The vertical connecting section can have a length of 11 mm and a height of 3.8 mm. The bottom horizontal leg can have a height of 1 mm and a length of 31 mm. The bottom vertical element can have a height of 11.2 mm and a length of 1 mm.

[0037] Accordingly, the exemplary harmonic tag 103 includes an antenna 105 and a substrate 184 coupled to the antenna 105. The antenna 105 includes a top portion 180 and a bottom portion 182, respectively. The bottom portion 182 is designed to receive a transmit signal (e.g., 2.5 GHz), and the top portion 180 is designed to transmit or re-radiate a signal having a harmonic frequency of the transmit signal (e.g., 5 GHz). In this example, a diode 184 is connected between two legs 182A and 182B that define the bottom portion 182 of the antenna 105. The antenna 105 can be constructed of a copper laminate as described above or of a conductive material that permits the reception and transmission of electromagnetic energy, such as copper, nickel, tin, silver, aluminum, zinc, and / or alloys thereof. The substrate 184 enables the tag 103 to be conveniently and unobtrusively attached to a variety of objects. As described above, the substrate 184 can be constructed of RO3003 material or can include polyester, polyimide, or a similar material, and the antenna 105 can be coupled to the substrate 184 using an adhesive, such as an acrylic pressure-sensitive adhesive.

[0038] As described above, with reference to Figures 1A to 1C One use of the radar and harmonic tags described is to determine mobile events and / or the activities of a consumer using a product having an associated or attached harmonic tag. For example, with respect to Figure 1CIn an embodiment, the movement of person 104 is detected by determining the Doppler shift information in the fundamental return signal 162 received by radar antenna 124A. The movement of object 102 is detected by determining the Doppler shift information in the harmonic return signal 176 received by radar antenna 126A. The return signal (either one) can be divided into time segments (e.g., about 2 seconds), and each time segment will include Doppler-related information. The well-known STFT can be used to process each time segment to produce the corresponding part of the power spectrogram. The STFT is a series of (possibly overlapping) Fourier transforms along with a window function (e.g., Hamming window function) to reduce the start / end effects of pulling out a finite segment of the signal from the series. The STFT provides a timeline of the activity. The STFT will have to be suitable for the time segments of the activity. The term "suitable" depends on the detectable activity. For example, when a person is brushing their teeth, a pair of toothbrush strokes (i.e., one stroke in each opposite direction) can occur every 1 second. A suitable time segment will be about 0.5 seconds to 1 second. A series of time segments each having this duration will allow capturing multiple pairs of toothbrush strokes as separate movement events. Compared to the STFT, a more conventional Fourier transform will capture all frequency information but may hide individual "events". The individual STFT time segments can be arranged in sequence to show how an object or person moves over a longer period of time (e.g., 5 to 45 seconds). As described above, the fundamental power spectrogram relates to the movement of person 104, while the harmonic power spectrogram relates to the movement of object 102. This Doppler information includes information about the amount of movement experienced by the person or object. For example, the Doppler shift frequency present in the return signal can correspond to the speed at which the object (or person) is moving, the amplitudes of the various frequencies present in the return signal can indicate the confidence or certainty that a particular frequency actually exists, and the periodicity (if any) of the return signal can indicate the time interval between successive occurrences of a particular frequency component. For example, when a consumer uses a toothbrush with an attached tag, a series of time segments of the harmonic return signal can indicate that the harmonic tag moves back and forth at 2 Hz, with a maximum speed of about 1 m / s. A single time segment can provide information about a single brush stroke or a pair of brush strokes, but may not reflect the periodicity of successive brush strokes or the peak speed in multiple brush strokes. However, multiple STFT time segments can be analyzed to determine the peak speed in multiple brush strokes and whether there is a periodicity associated with any frequency component within the power spectrogram formed by the series of STFT time segments. The sequence of time segments can be arranged to indicate a back-and-forth movement that repeats every 2 seconds. The information that can be extracted from the return signal can be considered as defining the "characteristics" of brushing teeth. Whenever a return signal with characteristics similar to the brushing teeth characteristics is captured at a later time, the computer can infer that a human-related brushing activity is occurring.

[0039] By inferring human-related activities involving a product (i.e., product activities), the use of the product can be determined. Brushing teeth, combing hair, styling hair, shaving, lifting a container, using a tissue, etc. are examples of human-related activities involving products to which harmonic tags can be attached. This information can be used to determine when a product may need to be replenished or replaced.

[0040] Figures 2A to 2C Each is an example flowchart of a high-level view of a consumer product using a harmonic tag in accordance with the principles of the present disclosure. The following figures provide further details of the general steps listed in the Figures 2A to 2C flowcharts.

[0041] Starting from Figure 2A step 202, a first product with an attached harmonic tag is provided such that the product is available for human use. In step 204, a transmission signal is directed at a first area where the product is located. Since it is desired to detect the movement of the object being used by the consumer, the direction of the transmission signal can be at a direction and height commensurate with how the product might be used. The transmission signal has a first frequency that can be conveniently labeled as the fundamental frequency. The transmission signal can be a continuous wave (CW) signal that is always transmitting or a pulsed wave signal that is transmitted periodically for a defined period of time (e.g., transmitting for 0.2 seconds every 1.0 second or transmitting for 0.02 seconds every 0.1 second). Alternatively, the transmission signal can be provided by a system that also detects the presence of a person in the first area (e.g., passive infrared detection) before being powered on to transmit the transmission signal.

[0042] Such as Figure 1D the harmonic tag shown is designed to receive the fundamental frequency signal and radiate a return signal having a second frequency that is a harmonic of the fundamental frequency. Thus, in step 206, a return signal is received from the harmonic tag that radiates the return signal after receiving the transmission signal. Steps 204 and 206 can be performed by conventional radar circuitry.

[0043] A processor, computer, or other type of processing device (such as a microprocessor, such as one from the Sitara TM series available from Texas Instruments (TI), or an application processor, such as one from the OMAP TM series also available from Texas Instruments, or a digital signal processor, such as one from the C6000 series also available from Texas Instruments, or a microcontroller, such as the STM32 TMOne in the series) can be incorporated into the radar or can be a separate processor, computer, or other processing device such that the radar provides the return signal to the separate processor, computer, or other processing device. As discussed above, as the harmonic tag moves, the radiated harmonic return signal can undergo Doppler frequency shifts (both positive and negative), rather than simply being a pure harmonic of the fundamental frequency. By removing the harmonic frequencies from the harmonic return signal, a baseband signal is generated that will vary over time as the tag is moving. The presence of variations in the signal indicates that the tag and the object are moving. The return signal from a stationary tag will not include variations due to Doppler shifting of the radiated signal. As described above, the STFT can be used to construct a harmonic spectrogram from the harmonic return signal. An automated process using a processor, computer, or other processing device can determine whether the harmonic tag (or the object to which it is attached) is moving by analyzing the pixel values present in the harmonic spectrogram. If there is no movement of the object, the amplitude of any frequency component above or below 0 Hz in the harmonic spectrogram will be essentially zero. However, if there is movement of the object, one or more frequency components of the harmonic spectrogram will have a non-zero amplitude. One of ordinary skill in the art will recognize that a predetermined threshold can be applied such that frequency components with amplitude values (i.e., pixel values in the harmonic spectrogram) below the predetermined threshold are still considered to be absent, even if the amplitude value is not precisely 0. Noise, interference, and other unintended artifacts in the receiving and processing circuitry may inadvertently cause frequency components of the harmonic spectrogram to have non-zero but very small amplitude values, even if the frequency component is not actually generated by object movement. A similar analysis can be performed with respect to the pixel data of the fundamental spectrogram to determine or detect the presence of movement associated with a person.

[0044] As discussed above, the movement of the tag and the object can be characterized, for example, by a harmonic spectrogram that includes the different frequencies in the baseband form of the harmonic return signal and the timeline of their amplitudes. Multiple samples of an activity (such as brushing teeth) can be captured along with their corresponding spectrograms. The different samples can involve multiple people of different ages and body sizes. One or more of these spectrograms corresponding to the sample activities can be compared to the most recently captured and generated harmonic spectrogram to see if the most recently captured harmonic spectrogram is similar to one of the one or more spectrograms corresponding to the sample activities. Different spectrograms can be generated for a variety of different sample activities (such as shaving, brushing teeth, etc.), and thus the most recently captured and generated spectrogram can be compared to the different spectrograms corresponding to the sample activities to determine the spectrogram that is similar to the most recently captured spectrogram. Thus, in step 208, the computer or processor can infer the activity of the product to which the harmonic tag is attached based on the harmonic return signal and the resulting harmonic spectrogram that involves information regarding the movement of the harmonic tag.

[0045] For comparing one spectrogram with another or comparing a portion of a most recently captured spectrogram with one or more other spectrograms corresponding to sample activities, conventional image analysis techniques such as, for example, cross-correlation can be utilized. For image processing applications where the brightness of the image and the template can vary due to lighting and exposure conditions, the image can be normalized first. This is typically done in each step by subtracting the mean and dividing by the standard deviation. That is, the cross-correlation of a template t(x,y) and a sub-image f(x,y) is

[0046]

[0047] where n is the number of pixels in t(x,y) and f(x,y), μ t is the mean of t(x,y), μ f is the mean of f(x,y), σ t is the standard deviation of t(x,y), and σ f is the standard deviation of f(x,y). One of ordinary skill in the art will readily recognize that the cross-correlation can also be calculated without explicitly subtracting the means μ t and μ f in the above formula. Normalized correlation is one of the methods for template matching and is a process for finding the incidence of patterns or objects within an image. The template is moved to different positions of a spectrogram of a known activity, and the cross-correlation value is calculated at each different position. According to the principles of the present disclosure, a current spectrogram or a portion of a current spectrogram (fundamental or harmonic) can be considered as a template for comparison with a spectrogram of a known activity. A cross-correlation score above a predetermined threshold indicates that the image features of the current spectrogram or a portion of the current spectrogram are similar to the features in the spectrogram of the known activity. Thus, the activity of a person using an object with an attached harmonic tag can be inferred from the return signal for generating the harmonic tags of the current spectrogram.

[0048] Figure 2B The flowchart of depicts a method for inferring information about the usage amount of a product with an attached harmonic tag. The meaning of "usage amount" will vary depending on the type of product. The usage amount of a razor can involve how many times it is used, where each "use" corresponds to the razor moving from an original position and returning to the original position, or can involve how far the razor has traveled, i.e., the distance. The usage amount of a toothpaste container can involve the frequency with which it moves from an original position and returns to the original position. The usage amount of a toothbrush can involve how many different brushing strokes are detected. The usage amount can also involve a specific time period. For an air freshener, the relevant time period (and usage amount) can be a fixed time period (e.g., 3 months) after replacing the cartridge. The relevant time period for a toothbrush can be open-ended, but will start being measured when the product is first used.

[0049] Similar to Figure 2A the flowchart ofFigure 2B The flowchart of Figure 2B begins at step 230, where the transmitted signal at the first frequency or fundamental frequency is directed to a first area where the product with the attached harmonic tag is located. As previously mentioned, the harmonic tag is designed to receive the fundamental frequency signal and radiate a harmonic return signal having a second frequency that is a harmonic of the fundamental frequency. Thus, at step 232, the harmonic return signal is received from the harmonic tag that radiates the harmonic return signal after receiving the transmitted signal. Steps 230 and 232 may be performed by a conventional radar circuit.

[0050] At step 234, the computer may determine one or more movement events of the harmonic tag based on the harmonic return signal and the Doppler information contained therein. Similar to the discussion above regarding "usage amount", the term "movement event" may vary depending on the product to which the harmonic tag is attached. Screwing a cap onto or off a container (i.e., the cap moving through a predetermined angle) may define a movement event. Determining that a container (e.g., for laundry detergent) has moved from and / or returned to its original position may include one or more movement events for that type of product, i.e., the container moving from its original position to another position such as on or near a washing machine may include one movement event, and the container moving back from on or near the washing machine to its original position may include another movement event. It is also contemplated that one or more movement events may include activities, e.g., two movement events of a laundry detergent container may include a single activity of a consumer's one use of the laundry detergent container. A movement event for a razor may be each of one or more different strokes in one or more directions. Movement events may be determined over a specific time period. Generally, this time period will begin when the consumer first uses the product and will continue until the product is replaced or replenished. The first detection of the presence of the harmonic tag may be one way for the computer or processor to automatically determine that the time period should begin. Alternatively, a system is envisioned where the consumer may use a user interface to indicate that the time period for detecting movement events of the product should begin. For example, a graphical user interface screen may list the products detected in the first area and allow the consumer to select the appropriate product and select to reset or start the time period for which movement events are to be detected.

[0051] This document provides examples of how activity can be inferred from a harmonic return signal or a combination of a harmonic return signal and a fundamental return signal. Relative to a "movement event", and as described above, an activity can include one or more movement events. As discussed above, in one example, a user's activity can be moving a detergent container between two locations. In this case, a first movement event can also be defined as moving the detergent container from an original location to another location, and a second movement event can be defined by moving the detergent container from the other location back to the original location. Thus, the techniques and methods related to inferring activity described herein are equally applicable to inferring or determining movement events. In other cases, an activity can be defined as "brushing teeth". This activity can be composed of more than one movement event. For example, a single movement event can be defined as a brushing stroke in one direction, a single movement event can be defined as a pair of successive brushing strokes with one stroke in each direction, or two movement events can be defined as a pair of successive brushing strokes with one stroke in each direction. As described above, the power spectral diagrams of known activities can be used to infer activity by comparing them with the most recently captured and generated power spectral diagrams. In a similar manner, known power spectral diagrams can provide a finer granularity such that power spectral diagrams associated with corresponding known or defined movement events can be used. Thus, the techniques and methods related to inferring activity described herein are equally applicable to inferring or determining movement events based on return signals (e.g., harmonic return signals).

[0052] Based on the movement events determined during a period corresponding to the use of a product having a harmonic tag, in step 236, the computer can infer the cumulative usage of the product associated with the harmonic tag during that period. Such information can be displayed to the consumer (using the GUI described above), the information can be collected and sent to a wireless device, or the information can be collected and transmitted to a store, distributor, manufacturer, or other data collector for possible automated purchase of replenishment products when appropriate.

[0053] Figure 2C is a flowchart of a method that builds on many concepts of Figure 2B First, in step 250, a value is stored in the computer that indicates or represents what is considered to be the defined useful life cycle of a product attached with a harmonic tag.

[0054] Similar to Figure 2A the flowchart of Figure 2CThe flowchart includes step 252, where the transmitted signal at the first frequency or fundamental frequency is directed to the first area where the product with the attached harmonic tag is located. As previously mentioned, the harmonic tag is designed to receive the fundamental frequency signal and radiate a return signal with a second frequency that is a harmonic of the fundamental frequency. Thus, in step 254, the harmonic return signal is received from the harmonic tag that radiates the harmonic return signal after receiving the transmitted signal. Steps 252 and 254 can be performed by a conventional radar circuit.

[0055] In step 256, a processor or computer can determine one or more movement events of the harmonic tag based on the harmonic return signal and the Doppler information contained therein, similar to the discussion above regarding Figure 2B . However, in step 256, the time period for determining the movement events is defined as the time period starting from the earliest determined movement event in the current life cycle of the product. In step 258, the number of movement events that occurred during that time period is represented by accumulating the counts of the different movement events that occurred during that time period. Finally, in step 260, the computer that accumulates the count and stores the value indicating the defined useful life cycle can use these two data elements to determine whether the defined useful life cycle of the product has expired.

[0056] Figure 3 depicts a simplified schematic representation 302 of an amplitude spectrogram (such as Figure 1C the harmonic spectrogram 180 of Figure 1C or the fundamental wave spectrogram 178 of

[0057] The vertical axis 304 represents the frequency in Hz, and the horizontal axis 306 represents time. As described above, the return signal (fundamental or harmonic) can be filtered and down-converted to produce a time-domain baseband signal. A short portion of the time-domain baseband signal can be operated on by a short-time Fourier transform to produce a series of frequency-domain samples arranged in the spectrogram representation 302 of Figure 3 . The spectrogram signal 300 represents the micro-Doppler frequency shift information in the return signal. In the bottom portion 310, the Doppler frequency shift is negative, which corresponds to movement away from the transmitter. In the top portion 308, the Doppler frequency shift is positive, which corresponds to movement towards the transmitter.

[0058] Figure 4A and Figure 4B depict actual movement-related spectrograms according to the principles of the present invention. Although shown in grayscale herein, spectrograms are typically colored such that the color represents the amplitude associated with a particular frequency. Figure 4A represents the fundamental wave spectrogram depicting information related to human movement.Figure 4B A harmonic spectrogram representing information related to the movement of harmonic tags.

[0059] Further in accordance with the present disclosure, radar echo data can be used to classify activities (especially human-related activities) into a limited number of categories through a process of spectrogram preprocessing and machine learning classification. The preprocessing of the spectrogram can include computing the short-time Fourier transform of the time-series radar signal. This preprocessing can also include a contrast enhancement step, where a filter is applied to amplify frequency features considered relevant to the classification and attenuate those frequency features considered attributable to noise. Contrast enhancement can be beneficial when the pixel values of an image are clustered close to each other. As is known in the field of image processing, an automated process can analyze a range of pixel values present in the image and increase the contrast by spreading the actual pixel values across the entire range of potential pixel values.

[0060] One type of machine learning classifier is a support vector machine (SVM). Using this type of classifier, the preprocessing of the spectrogram will include one or more feature extraction or identification steps, where the presence or periodicity of energy in the spectrogram is identified. Rules provided by the model architecture can be used to extract one or more features. For example, the frequency fluctuations are divided into intervals (e.g., components), and the magnitude of the energy in a particular interval is viewed relative to other intervals. Or an interval or frequency component is selected and observed over time to estimate periodicity. Another technique for automatically determining periodicity is to apply a two-dimensional Fourier transform to some or all of the most recently collected spectrograms. The resulting transformed image will indicate the periodicity of one or more frequency components in the power spectrogram. In other words, applying the STFT to create an initial power spectrogram can indicate the presence of a 20 Hz frequency component in the return signal as indicated by the pixel values in the power spectrogram, while the two-dimensional Fourier transform of the power spectrogram can indicate that the 20 Hz frequency component appears every 2 seconds.

[0061] For example, in the spectrogram of the Doppler echo of a person walking, the torso may provide a small frequency shift as the person walks away from the radar, but the swinging arms and legs will provide oscillatory traces that vary between negative and positive frequency shifts. Once all the extracted or identified features required for SVM model inference have been extracted, they can be fed into the SVM model as vectors for classification.

[0062] As those of ordinary skill in the art will recognize, the SVM model is generated by a computer or similar processing device using known test data. For example, a large number (e.g., hundreds or thousands) of spectrograms can be collected for different activities, and for each spectrogram, the relevant features are identified. In cases where the activity associated with the spectrogram is known and the relevant features of the spectrogram are identified, a machine learning algorithm can automatically construct the SVM model without additional human assistance.

[0063] Figure 5 shows an overview of the process of using SVM to identify and classify human activities. As described with reference to the previous figures, baseband time-domain signal data 502 is generated. Next, a short-time Fourier transform 504 can be applied to produce spectrogram data 506. This spectrogram 506 can be a fundamental spectrogram (such as Figure 1C 178) or a harmonic spectrogram (such as Figure 1C 180).

[0064] Before the Figure 5 process shown, subject matter experts have identified features in the spectrogram that are relevant to classifying the spectrogram as being associated with a particular activity. An example of a feature can be that a particular frequency component (e.g., 20 Hz) occurs with a period of approximately 1 second. Another feature can be that a different frequency component (e.g., -20 Hz) appears between each pair of adjacent 20 Hz components. Thus, a rule-based feature extraction step 508 can be performed. In other words, a set of analysis and calculation steps can be (automatically or manually) applied to evaluate the spectrogram 506 to determine the extent to which certain features are present in the spectrogram 506. Generally, a feature vector includes elements and corresponding values for each extracted feature. Some of the feature values can be binary values, such as a value of "1" if the feature is present, or "0" if the feature is not present. Feature values can involve percentages, such as "50% of the spectrogram pixels are green". Feature values can be the amplitudes of each frequency component in the spectrogram. An example feature and corresponding value can be the standard deviation (or some other moment) of the frequency energy at a particular time segment. Another example feature and corresponding value can be the magnitude of a particular frequency value normalized by other frequency values. One of ordinary skill in the art will recognize that many possible features and values can be determined by subject matter experts such that rule-based feature extraction 508 can be performed.

[0065] Also, a support vector machine model 514 is generated before other steps in Figure 5 . As described above, the SVM model 514 is generated by providing training data to a machine learning algorithm to automatically produce the SVM model 514, which serves as an activity recognition model 512 to evaluate the feature vector 510 obtained from the features extracted via the spectrogram 506. The result is that the activity recognition model classifies 516 the spectrogram as being associated with a particular activity. The activity recognition model 512 that defines the SVM model 514 may be implemented on a computer or processing device with sufficient resources to produce the classification 516 within the time frame required by the system designer. This time frame can vary depending on whether the system is a real-time system.

[0066] Other types of machine learning classifiers, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), can take the frequency data from the preprocessing step as input. In this case, short time periods of spectrograms are fed into a classifier model, which may include a convolutional function that emphasizes or attenuates features based on training data, and a recurrent function that infers activity based on temporal correlations in time series data, such as long short-term memory (LSTM) cells. The duration of the time series data given to the model for classification will vary based on the activity, but can range from 0.1 s to 2 s. These models are trained using large amounts of labeled data so that feature extraction and temporal correlation weights will be sufficiently generalized and not limited to how one or two individuals perform an activity.

[0067] As those of ordinary skill in the art will recognize, CNN models are generated by a computer or similar processing device using known test or training data. For example, a large number (e.g., hundreds or thousands) of training spectrograms can be collected for different activities. Then a process can begin where a machine learning algorithm applies a series of convolutional kernels to the image of each spectrogram. The series can typically be randomly selected convolutional kernels of different sizes and different weights in each cell of the kernel. Different kernels can tend to emphasize different graphical features of the spectrogram, such as edges, colors, object sizes, proximity of different objects. The result of the machine learning algorithm is to automatically identify one or more convolutional kernels that can effectively classify the activity associated with the spectrogram. The generation of the CNN model is performed automatically by a computer or similar processing device, except for human assistance in collecting training spectrograms, labeling the activity associated with each spectrogram, and providing the labeled training spectrograms to the machine learning algorithm.

[0068] As described above, an RNN can identify a time series of different spectrograms. In other words, a first spectrogram with a first set of features can be followed in time by a second spectrogram with a second set of features. Thus, individual spectrograms not only provide information that helps classify an activity, but also the sequence of spectrograms that are related to each other can provide relevant information. The training of the RNN is similar to the training of the CNN in that training data (i.e., spectrograms) are provided to a computer or similar processing device that automatically constructs the RNN model. When non-training spectrograms of a user are collected according to an embodiment of the present disclosure, the spectrogram can have the learned convolutional kernels of the CNN model applied to extract features from the spectrogram. Then the time-ordered sequence of these spectrograms can be fed into the RNN model that infers the user's activity.

[0069] Figure 6 A process overview of using CNNs and RNNs to identify and classify human activities is shown. As described with reference to the previous figures, baseband radar data 602 is generated. Next, a short-time Fourier transform 604 can be applied to produce spectrogram data 606. The spectrogram 606 can be a fundamental wave spectrogram (such asFigure 1C of 178) or a harmonic spectrogram (such as Figure 1C of 180).

[0070] As described above, a variety of training data related to the activity to be recognized is collected. The training data includes many spectrograms, each of which has been labeled as being associated with a specific activity. In addition, each of the training data "elements" may include a time-ordered sequence of spectrograms rather than just a single spectrogram. The training data can first be used in a deep learning algorithm to automatically generate a multi-layer convolutional neural network (CNN). The CNN is generated in such a way that it learns which convolutional kernels extract (or recognize) spectrogram features that seem to be effective in correctly classifying the training data. For a time-ordered sequence of spectrograms, the features from each spectrogram can be arranged in order and used as training data for a recurrent neural network (RNN).

[0071] The CNN 610 and the RNN 612 are used as an activity recognition model 608 to evaluate the spectrogram 606. The spectrogram 606 is treated as an image, and the CNN 610 can operate on this image to extract one or more features (i.e., detect their presence in the image). Figure 6 The process is not necessarily a one-time calculation, but may include a series of individual spectrograms 606 received by the activity recognition model 608, where each spectrogram is processed by the CNN 610. The resulting series of feature extraction data can then be provided to the RNN 612 for classifying the series of spectrograms 614.

[0072] More details about deep learning algorithms (such as CNNs and RNNs) are provided in "A Survey of Deep Learning-Based Human Activity Recognition in Radar" by Li et al., Remote Sensing 2019, 11, 1068, the entire disclosure of which is incorporated herein by reference. More details about feature extraction for SVM machine learning models can be found in "Human Activity Classification Based on Micro-Doppler Signatures Using a Support Vector Machine" by Kim et al., IEEE Trans. Geosci. Remote Sens. 2009, 47, 1328 to 1337, the entire disclosure of which is incorporated herein by reference.

[0073] As Figure 1CAs shown, both the fundamental return signal 162 and the harmonic return signal 176 can be received and processed by the radar 108. To improve the accuracy of the classification of sensed activity, it may be advantageous to analyze the fundamental radar echo and the harmonic radar echo simultaneously. Figure 7 A combined machine learning model is depicted where the echoes from the two receivers can be passed to separate convolutional layers where features can be extracted based on the learned data. The combined data can be passed to a recurrent layer for activity recognition.

[0074] Thus, in Figure 7 , the fundamental baseband radar data 702 is operated on by the STFT 166 to produce a spectrogram 178. This spectrogram 178 is a fundamental spectrogram and captures Doppler shift related information caused by the movement of the person 104. The CNN 716 is similar to the Figure 6 CNN 610 in that training data is collected and labeled to be provided to the machine learning algorithm that generates the convolutional layers of the CNN 716. Each convolutional layer includes a convolutional kernel that is found by the machine learning algorithm to identify or emphasize features in the training data that are effective for classifying the activity. Thus, each convolutional kernel can be applied to the fundamental spectrogram 178 to effectively extract or emphasize features of the fundamental spectrogram 178 that contribute to classifying the spectrogram 178 as corresponding to a particular activity. The CNN 716 can be used to extract or identify features in the fundamental spectrogram 178, and then these features can be provided to the RNN 720.

[0075] Additionally, the harmonic band radar data 704 is operated on by the STFT 174 to produce a spectrogram 180. This spectrogram 180 is a harmonic spectrogram and captures Doppler shift related information caused by the movement of the harmonic tag 103. The CNN 718 is similar to the Figure 6 CNN 610 in that training data is collected and labeled to be provided to the machine learning algorithm that generates the convolutional layers of the CNN 718. Each convolutional layer includes a convolutional kernel that is found by the machine learning algorithm to identify or emphasize features in the training data that are effective for classifying the activity. Thus, each convolutional kernel can be applied to the harmonic spectrogram 180 to effectively extract or emphasize features of the harmonic spectrogram 180 that contribute to classifying the spectrogram 180 as corresponding to a particular activity. The CNN 718 can be used to extract or identify features in the harmonic spectrogram 180, and then these features can be provided to the RNN 720.

[0076] The RNN 720 is similar to the RNN 612 in that training data is collected and provided to a machine learning algorithm to generate the RNN 720. However, in this case, the training data may include features extracted from a pair of spectrogram types (i.e., fundamental spectrogram and harmonic spectrogram). In some cases, the spectrogram may be associated with data of a movement that occurs approximately simultaneously. In other cases, the extracted features may be from fundamental spectrogram data and harmonic spectrogram data that occur at different times. Ultimately, the RNN 720 performs a classification 722 of the activities captured by the radar and analyzed by the activity recognition model 714.

[0077] Figure 8 is a flowchart of an example method or process that relies on both a fundamental spectrogram and a harmonic spectrogram to infer information about the movement a user may be engaged in. Steps 802 and 804 are similar to Figure 2A steps 202 and 204 of. The first region where the product with the harmonic label is located is illuminated with an emission signal having a first frequency (e.g., fundamental frequency). Next, in steps 806 and 808, a return signal is received due to the emission signal. One return signal is the fundamental return signal related to the movement of a person using the product with the harmonic label. The movement may include moving legs and arms as done when walking, or the movement may include moving arms, hands, feet, and other body parts when using the product. The frequency of the fundamental return signal is substantially the same as the frequency of the emission signal.

[0078] Another return signal is the harmonic return signal and is generated by the harmonic label that radiates the harmonic return signal when the emission signal is received. The frequency of the harmonic return signal is a harmonic of the fundamental frequency. In step 810, the computer determines the movement of the person using the product based on the fundamental return signal. As explained above, a CNN can be used to extract or identify features in the spectrogram that indicate a specific type of movement of the person and their body. The presence of the movement can be determined by detecting the micro-Doppler shift frequency in the fundamental spectrogram.

[0079] In step 812, the movement of the harmonic label can also be determined by the computer. As explained above, a CNN can be used to extract or identify features in the spectrogram that indicate a specific movement type of the harmonic label. The presence of the movement can be determined by detecting the micro-Doppler shift frequency in the harmonic spectrogram.

[0080] The dimensions and values disclosed herein should not be understood as being strictly limited to the exact numerical values recited. Instead, unless otherwise specified, each such dimension is intended to represent the recited value and a range functionally equivalent around that value. For example, a dimension disclosed as "40 mm" is intended to represent "about 40 mm".

[0081] Each document cited herein, including any cross-referenced or related patent or patent application and any patent application or patent from which this application claims priority or the benefit of, is hereby incorporated by reference in its entirety unless expressly excluded or otherwise limited. The citation of any document is not an admission that it is prior art with respect to any invention disclosed herein or claimed in the claims hereof, or that it alone or in any combination with any one or more other references teaches, suggests, or discloses any such invention. Further, when any meaning or definition of a term in this invention conflicts with any meaning or definition of the same term in a document incorporated by reference, the meaning or definition assigned to the term in this invention shall govern.

[0082] While the specific embodiments of the invention have been illustrated and described, it will be apparent to those skilled in the art that various other changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, it is intended that all such changes and modifications that fall within the scope of the invention be covered by the appended claims.

Claims

1. A method for inferring product activity, the method comprising: providing a first product with an attached first harmonic tag and a radar that radiates signals at a plurality of frequencies by sweeping through discrete frequencies, wherein the radar includes a computer that processes and analyzes the returned signals; directing a first transmitted signal at a first frequency at a first area where the first product is located; receiving a first returned signal at a first returned frequency from the first harmonic tag, wherein the first harmonic tag radiates the first returned signal after receiving the first transmitted signal such that the first returned frequency is a harmonic of the first frequency; determining movement of the first product based on the first returned signal, wherein the first returned signal includes a Doppler shift; inferring, by the computer, a first activity of using the first product based on the movement of the first product, generating, by the computer, a power spectrogram based on a series of short-time Fourier transforms applied to the first returned signal, wherein the first returned signal is converted into a baseband signal fluctuating at 0 Hz, and the power spectrogram is generated from the baseband signal by short-time Fourier transform, the power spectrogram including frequency components and their corresponding amplitudes in the first returned signal due to the movement of the first product, and wherein the computer infers the activity of using the first product based on the harmonic returned signal and the resulting harmonic spectrogram, and determines the use of the first product by inferring the activity of the first product, wherein the harmonic returned signal and the resulting harmonic spectrogram relate to information about the movement of the first harmonic tag.

2. The method according to claim 1, the method comprising: analyzing, by the computer, the power spectrogram with a convolutional neural network to determine the identified activity, wherein the identified activity includes the first activity.

3. The method according to claim 1, the method comprising: analyzing, by the computer, the power spectrogram with a support vector machine to determine the identified activity, wherein the identified activity includes the first activity.

4. The method according to claim 1, wherein inferring the first activity comprising: comparing the power spectrogram signal with one or more pre-stored activity models.

5. The method according to claim 1, the method comprising: directing a second transmitted signal at a second frequency different from the first frequency at the first area; receiving a second returned signal at a second returned frequency from a second harmonic tag, wherein the second harmonic tag is attached to a second product located in the first area, and the second harmonic tag radiates the second returned signal after receiving the second transmitted signal such that the second returned frequency is a harmonic of the second frequency; and inferring, by the computer, a second activity of using the second product based on the second returned signal.

6. The method according to claim 1, the method comprising: receiving a second returned signal at the first frequency from a user of a product in the first area that reflects the first transmitted signal.

7. A system for inferring product activity, the system comprising: A first product having an attached first harmonic tag; A radar that radiates signals at a plurality of frequencies by sweeping through discrete frequencies and is configured to direct a first transmitted signal at a first frequency to a first area where the first product is located, The radar is configured to receive a first return signal at a first return frequency from the first harmonic tag, where the first harmonic tag radiates the first return signal after receiving the first transmitted signal such that the first return frequency is a harmonic of the first frequency; A memory that stores executable instructions; and A processor that communicates with the memory, where execution of the executable instructions by the processor causes the processor to: Determine a Doppler shift included in the first return signal; Determine movement of the first product based on the Doppler shift; and Infer a first activity of using the first product from the movement of the first product, where execution of the executable instructions by the processor causes the processor to: Generate a power spectrogram based on a series of short-time Fourier transforms applied to the first return signal, where the first return signal is converted to a baseband signal that fluctuates at 0 Hz, and the power spectrogram is generated from the baseband signal by short-time Fourier transform, the power spectrogram including frequency components and their corresponding amplitudes in the first return signal due to movement of the first product, and where the processor infers an activity in which the first product is being used based on the harmonic return signal and the resulting harmonic spectrogram, and determines use of the first product by inferring the activity of the first product, where the harmonic return signal and the resulting harmonic spectrogram relate to information about movement of the first harmonic tag.

8. The system of claim 7, where execution of the executable instructions by the processor causes the processor to: Analyze the power spectrogram with a convolutional neural network to determine an identified activity, where the identified activity includes the first activity.

9. The system of claim 7, where execution of the executable instructions by the processor causes the processor to: Analyze the power spectrogram with a support vector machine to determine an identified activity, where the identified activity includes the first activity.

10. The system of claim 7, the system further comprises: The radar is configured to direct a second transmitted signal at a second frequency different from the first frequency to the first area, The radar is configured to receive a second return signal at a second return frequency from a second harmonic tag, where the second harmonic tag is attached to a second product located in the first area and the second harmonic tag radiates the second return signal after receiving the second transmitted signal such that the second return frequency is a harmonic of the second frequency; and where execution of the executable instructions by the processor causes the processor to: Infer a second activity of using the second product based on the second return signal.

11. The system of claim 7, the system further comprises: The radar is configured to receive a second return signal at the first frequency from a user of a product that reflects the first transmitted signal in the first region.

Images