Integration of Tracker and Classifier in MMWAVE Radar

Abstract

Embodiments of the present invention relate to the integration of tracking and classifiers in MMWAVE radar. In one embodiment, a method for tracking a target using a millimeter-wave radar includes: receiving a radar signal using the millimeter-wave radar; generating a range-Doppler map based on the received radar signal; detecting a target based on the range-Doppler map; tracking the target using a trajectory; generating a predicted activity label based on the trajectory, where the predicted activity label indicates the actual activity of the target; generating a Doppler spectrogram based on the trajectory; generating a temporary activity label based on the Doppler spectrogram; assigning an uncertainty value to the temporary activity label, where the uncertainty value indicates the confidence level that the temporary activity label is the actual activity of the target; and generating a final activity label based on the uncertainty value.

Description

Technical Field

[0001] The present disclosure generally relates to an electronic system and method, and in particular embodiments, to the integration of a tracker with a classifier in a millimeter wave (mmWave) radar. Background Art

[0002] In the past few years, due to the rapid development of low-cost semiconductor technologies such as silicon germanium (SiGe) and fine geometry complementary metal oxide semiconductor (CMOS) processes, applications in the millimeter wave frequency range have received significant attention. The availability of high-speed bipolar and metal oxide semiconductor (MOS) transistors has led to an increase in the demand for integrated circuits for millimeter wave applications at 24 GHz, 60 GHz, 77 GHz, and 80 GHz and above 100 GHz. Such applications include, for example, static object and moving object detection and tracking.

[0003] In some radar systems, the distance between the radar and the target is determined by the following steps: transmitting a frequency modulated signal; receiving the reflection of the frequency modulated signal (also referred to as an echo); and determining the distance based on the time delay and / or frequency difference between the transmission and reception of the frequency modulated signal. Thus, some radar systems include a transmitting antenna for transmitting a radio frequency (RF) signal, a receiving antenna for receiving the RF, and associated RF circuitry for generating the transmitted signal and receiving the RF signal. In some cases, phased array technology can be used to use multiple antennas to achieve a directional beam. A multiple input multiple output (MIMO) configuration with multiple chip sets can also be used to perform coherent and non-coherent signal processing.

[0004] In some settings, static objects and moving objects coexist. For example, in an indoor setting, static objects such as furniture and walls coexist with moving objects such as humans. The indoor setting may also include objects that exhibit periodic movement, such as a fan. Doppler analysis has been used to distinguish between moving objects and static objects. Summary of the Invention

[0005] According to one embodiment, a method for tracking a target using a millimeter wave radar includes: receiving a radar signal using the millimeter wave radar; generating a range-Doppler map based on the received radar signal; detecting a target based on the range-Doppler map; tracking the target using a track; generating a predicted activity label based on the track, where the predicted activity label indicates the actual activity of the target; generating a Doppler spectrogram based on the track; generating a temporary activity label based on the Doppler spectrogram; assigning an uncertainty value to the temporary activity label, where the uncertainty value indicates the confidence level that the temporary activity label is the actual activity of the target; and generating a final activity label based on the uncertainty value.

[0006] According to one embodiment, a millimeter-wave radar system includes a millimeter-wave radar and a processor. The millimeter-wave radar is configured to transmit and receive radar signals. The processor includes: a radar processing block configured to generate a range-Doppler map based on the radar signals received by the millimeter-wave radar; a target detector block configured to detect a target based on the range-Doppler map; a tracker configured to: track the target using a trajectory and generate a predicted activity label based on the trajectory, where the predicted activity label indicates the actual activity of the target; a feature extraction block configured to generate a Doppler spectrogram based on the trajectory; a classifier configured to generate a provisional activity label based on the Doppler spectrogram; and a classification gating block configured to: receive an uncertainty value associated with the provisional activity label and generate gating data based on the uncertainty value, the predicted activity label, and the provisional activity classification, where the uncertainty value indicates the confidence level that the provisional activity label is the actual activity of the target, and where the tracker is configured to generate a final activity label based on the gating data.

[0007] According to one embodiment, a method for tracking a target using a millimeter-wave radar includes: receiving radar signals using the millimeter-wave radar; generating a range-Doppler map based on the received radar signals; detecting a target based on the range-Doppler map; using a tracker to track the target using a trajectory; using a classifier to generate a provisional activity label based on the output of the tracker; and using the tracker to generate a final activity label based on the output of the classifier, where the tracker uses state variables to track the activity label of the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] To more fully understand the present invention and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which:

[0009] Figure 1 A radar system according to an embodiment of the present invention is shown;

[0010] Figure 2 A range-Doppler map of a human in motion captured using a millimeter-wave radar system according to an embodiment of the present invention is shown;

[0011] Figure 3 A block diagram of a classifier for generating a target classification based on the output of a tracker is shown;

[0012] Figures 4 to 6 A scenario that may cause misclassification of a target when fed into a classifier is illustrated;

[0013] Figure 7Shows a human activity state transition diagram according to an embodiment of the present invention;

[0014] Figure 8A Shows a block diagram for detecting, tracking, and classifying the activities of a human target according to an embodiment of the present invention;

[0015] Figure 8B Shows a flowchart of an exemplary method for tracking a human target and corresponding activities according to an embodiment of the present invention;

[0016] Figure 9 Shows a classifier for classifying human activities according to an embodiment of the present invention;

[0017] Figure 10A Shows an example of a Doppler spectrogram of human activities and activity transitions of a human target according to an embodiment of the present invention;

[0018] Figure 10B Illustrates a diagram showing the actual activity classification results, the output of the classifier, and the output of the tracker according to an embodiment of the present invention;

[0019] Figure 10C and Figure 10D Respectively show confusion matrices for the classifier 812 and the tracker 808 according to an embodiment of the present invention; and

[0020] Figure 11 Illustrates a block diagram showing a machine learning pipeline for feature extraction and identification based on machine language, which can be used to train Figure 9 a classifier to classify human targets based on human activities.

[0021] Unless otherwise indicated, corresponding numbers and symbols in different figures generally refer to corresponding parts. The figures are drawn to clearly illustrate relevant aspects of the preferred embodiments and are not necessarily drawn to scale. Detailed Description

[0022] The making and using of the disclosed embodiments are discussed in detail below. However, it should be understood that the present invention provides many applicable inventive concepts that can be implemented in various specific contexts. The specific embodiments discussed are merely illustrative of specific ways of making and using the present invention and do not limit the scope of the present invention.

[0023] The following description sets forth various specific details in order to provide a thorough understanding of the various exemplary embodiments in accordance with this specification. Embodiments may be obtained without one or more of the specific details, or by using other methods, components, materials, etc. In other instances, well-known structures, materials, or operations have not been shown or described in detail so as not to obscure the various aspects of the embodiments. References to "an embodiment" in this specification indicate that a particular configuration, structure, or feature described with respect to the embodiment is included in at least one embodiment. Thus, phrases such as "in one embodiment" that may appear at various points in this specification do not necessarily refer to the same embodiment. Moreover, the specific forms, structures, or features may be combined in any suitable manner in one or more embodiments.

[0024] Embodiments of the present invention will be described in a particular context, a millimeter-wave radar that integrates a tracker for tracking human targets with a classifier for classifying human activities. Embodiments of the present invention may be used in other types of radars. Some embodiments may be used to track targets other than humans, such as animals (e.g., dogs or cats) or robots. Some embodiments may be used to classify targets using criteria different from human activities, such as classifying the type of the target (e.g., whether the target is human or non-human).

[0025] In an embodiment of the present invention, the millimeter-wave radar includes a tracker having an integrated classifier for classifying human target activities. The tracker implements the integrated classifier by using state variables corresponding to possible human activities. The tracker updates the state variables associated with human activities based on the output of the classifier, which generates a provisional classification based on the output of the tracker. Thus, in some embodiments, in addition to localizing human targets, the tracker is advantageously capable of tracking the corresponding human activities.

[0026] A radar (such as a millimeter-wave radar) can be used to detect and track targets, such as humans. For example, Figure 1 Fig. 100 shows a radar system 100 according to an embodiment of the present invention. The radar system 100 includes a millimeter-wave radar 102 and a processor 104. In some embodiments, the millimeter-wave radar 102 includes the processor 104.

[0027] During normal operation, the millimeter-wave radar 102 transmits a plurality of radiation pulses 106, such as chirp pulses, towards the scene 108. In some embodiments, the chirp pulse is a linear chirp pulse (i.e., the instantaneous frequency of the chirp pulse varies linearly with time).

[0028] The transmitted radiation pulses 106 are reflected by objects in the scene 108. The reflected radiation pulses ( Figure 1(not shown)(also known as echo signal) is detected by millimeter-wave radar 102 and processed by processor 104 to detect and track targets such as humans, for example.

[0029] Objects in scene 108 can include static humans, such as a lying human 110; humans exhibiting little and infrequent movement, such as a standing human 112; and humans in motion, such as running or walking humans 114 and 116. Objects 108 in the scene can also include static objects such as furniture and periodically moving devices (not shown). Other objects can also be present in scene 108.

[0030] Processor 104 uses signal processing techniques to analyze the echo data to determine the position of the human. For example, in some embodiments, range FFT is used to estimate the range component of the position of the detected human (e.g., relative to the millimeter-wave radar). Angle estimation techniques can be used to determine the azimuth component of the position of the detected human.

[0031] In some embodiments, a range-Doppler map (image) is generated from the echo data, and two-dimensional (2D) moving target indication (MTI) is performed on the range-Doppler map to detect moving targets.

[0032] Processor 104 can be implemented as a general-purpose processor, controller, or digital signal processor (DSP) including, for example, combinational circuits coupled to a memory. In some embodiments, for example, processor 104 can be implemented using an ARM architecture. In some embodiments, processor 104 can be implemented as a custom application-specific integrated circuit (ASIC). In some embodiments, processor 104 includes multiple processors each having one or more processing cores. In other embodiments, processor 104 includes a single processor having one or more processing cores. Other implementations are possible. Some embodiments can be implemented as a combination of a hardware accelerator and software running on a DSP or general-purpose microcontroller.

[0033] In some embodiments, millimeter-wave radar 102 operates as an FMCW radar, which includes a millimeter-wave radar sensor circuit, a transmit antenna, and at least two receive antennas. Millimeter-wave radar 102 transmits and receives signals in the range of 20 GHz to 122 GHz. Alternatively, frequencies outside this range can also be used, such as frequencies between 1 GHz and 20 GHz or between 122 GHz and 300 GHz.

[0034] In some embodiments, an echo signal received by a receiving antenna of the millimeter wave radar 102 is filtered and amplified using a band-pass filter (BPF), a low-pass filter (LPF), a mixer, a low-noise amplifier (LNA), and an intermediate frequency (IF) amplifier in a manner known in the art. Then, one or more analog-to-digital converters (ADCs) are used to digitize the echo signal for further processing. Other implementations are possible.

[0035] For various reasons, it may be desirable to detect and track human targets, for example, in an indoor environment. Conventional methods for tracking a target assume that the target is a single point in a range-Doppler map. In a conventional range-Doppler processing chain, the obtained detection clusters are used to obtain a single bin in the range-Doppler image to determine the range and Doppler components of the detected target. Then, such a single bin is fed into a tracker for tracking the target. For example, in conventional radar signal processing, range, Doppler, and angle of arrival can be detected for a single-point target. Then, such components are fed into a tracker for tracking purposes.

[0036] The motion model for a conventional tracker can be expressed as:

[0037]

[0038] where k represents the discrete time step, Δt is the time between each time step, px is the position of the target in the x direction, py is the position of the target in the y direction, vx is the velocity of the target in the x direction, and vy is the velocity of the target in the y direction.

[0039] In some radar systems, such as in millimeter wave radar systems, a human target can exhibit double spread in the range and Doppler bins due to receiving reflections from different parts of the human body during the movement of the human target. For example, Figure 2 shows a range-Doppler map of a moving human captured using a millimeter wave radar system according to an embodiment of the present invention. As Figure 2 shown, a human target can exhibit peaks at different positions in the range-Doppler map, and these different positions correspond to different parts of the human target body, such as the right foot 204, the left foot 202, and the torso and hand 206.

[0040] In some embodiments, the tracker may use a coordinated turn motion model, such as the coordinated turn motion model described in the co-pending U.S. Patent Application No. 16 / 570,190, entitled "Human Target Tracking System and Method," filed on September 13, 2019, which is incorporated herein by reference. For example, in some embodiments, the coordinated turn motion model used by the tracker may be given by the following equation:

[0041]

[0042] where k represents the discrete time step, Δt is the time between each time step, px is the position of the centroid of the human target in the x direction, py is the position of the centroid of the human target in the y direction, lx is the bounding box size in the x direction, ly is the bounding box size in the y direction, vlx is the rate of change of the x size of the bounding box, vly is the rate of change of the y size of the bounding box, v c is the radial velocity of the centroid of the human target, θ is the angle of arrival of the human target, and ω is the rate of change of the angle of arrival (AoA) of the human target, where the bounding box may have a rectangular shape and spread in Doppler and range around the target.

[0043] In one embodiment, the parameters px, py, lx, ly, vlx, vly, v c 、θ and / or ω represent the state of a tracker (e.g., a tracker implemented using a Kalman filter such as an unscented Kalman filter) that tracks a human target using a trajectory. These states can be obtained from the measurements of the millimeter wave radar system 100. For example, in one embodiment, the millimeter wave radar system 100 measures r (the distance between the target and the millimeter wave radar sensor), θ (the angle of arrival - the angle of the target), v c (the radial velocity of the target), L r (the bounding box size in range), and L d (the bounding box size in Doppler). These measurements can be collectively referred to as:

[0044] Zmeas = (r, θ, v c , L r , L d ) (3)

[0045] The Zmeas measurements can be converted into the state of the tracker by the following equation:

[0046]

[0047] where ω, vlx, and vly are initialized to zero.

[0048] At each time step, a new set of measurements Zmeas is collected and the trajectory is updated based on this new set of measurements Zmeas. For example, for each time step, the unscented Kalman filter (e.g., using Equation 2) computes the predicted states px, py, lx, ly, vlx, vly, v c , θ, and ω for the trajectory. Due to the Bayesian recursive method, these states can incorporate information from all measurements available from time 1:k (all measurements of the trajectory). This predicted state can be transformed into the form of a predicted measurement by:

[0049]

[0050] The Mahalanobis distance (Md) can be computed between the predicted measurement Zpred and the new set of measurements Zmeas by:

[0051]

[0052] where S is the covariance matrix between Zmeas and Zpred.

[0053] In some embodiments, if the distance Md is less than a predetermined threshold, the new set of measurements Zmeas is considered a valid measurement of the target being tracked by the trajectory. The multi-dimensional region where the distance Md is less than the predetermined threshold is referred to as the gating region or the expected region associated with the trajectory.

[0054] In many applications, it is desirable to classify the target. For example, in some embodiments, it may be advantageous to classify the target as human or non-human. For example, when the target is determined to be a human target (e.g., as opposed to a non-human target such as a dog), the lights in the room can be automatically turned on.

[0055] Other classifications can also be beneficial. For example, in some embodiments, a human target is classified based on the activity being performed (e.g., walking, running, standing, sitting, lying down, etc.). For example, if the target is determined to be a human target, some embodiments can further assign an activity classification to such a human target. This activity classification can be used to enhance the localization tracking of the human target. For example, a human target may have a higher probability of remaining in the same position when the human target is performing a sitting activity compared to when the human target is performing a running activity. This activity classification can also be used for other purposes. For example, the system can take an action based on the type of activity detected for a human being. For example, when the detected human target is performing an activity of lying on the floor or transitioning from walking to lying on the floor, the system can turn on a warning light. In other words, the obtained (e.g., activity) label can be used to further track the target, for example, by updating the state variables of the tracker based on the (e.g., activity) label.

[0056] To classify a target, a classifier can be used in an open loop. For example, Figure 3 FIG. 300 is a block diagram showing a classifier 308 that generates a target classification based on the output of a tracker 304. As Figure 3 shown, after a radar (e.g., millimeter-wave radar 102 together with a processor 104) performs target detection 302, the tracker 304 tracks such a target. A tracker localization (e.g., identifying the location of the target) is generated by the tracker 304. Feature extraction 306 is performed based on the output of the tracker 304. For classification purposes, the extracted features (e.g., Doppler spectrogram) are fed to the classifier 308. Then, the classifier classifies the target based on the received features. In some embodiments, the detection parameters (parameters associated with 302, 304, and 306) are updated once per frame, while the target classification is updated once every N frames, where N can be, for example, 8.

[0057] Possible classifications include, for example, whether the target is human or non - human. Other classifications include, for example, activity classification. For example, when the target is identified as a human target, possible activity classifications can include: running, walking, sleeping, sitting, waving, falling, cooking, working, etc.

[0058] During normal operation, misclassification of a target may occur for various reasons. For example, a radar (such as millimeter - wave radar 102) may produce an output that, when processed, may lead to misclassification. Figures 4 to 6 FIG. shows a scenario that may lead to misclassification of a target when fed into a classifier.

[0059] Figure 4FIG. 400 shows a human target that is undetected during a portion of the time - as shown by utilization area 402. During error detection, the trajectory of tracker 304 may not be associated with the detection, and thus, feature extraction may not occur during that frame, which is a source of error that can lead to misclassification.

[0060] Figure 5 FIG. 500 shows two human targets (502 and 504) in the same range bin in a radar system with two receive antennas. Typically, a radar system with only two receive antennas can locate at most one target. When there are two targets in such a system, an under - determined system of equations may result in the inability to independently extract features for each target, which is a source of error that can lead to misclassification.

[0061] Figure 6 FIG. 600 shows a human target transitioning between activities (e.g., from walking to standing to sitting). When a human target performs an activity that was not previously used to train classifier 308 (e.g., the activity is not part of the library used to train the classifier), classifier 308 may assign an incorrect classification to the target. The transients that may occur between a first activity (e.g., walking) and a second activity (e.g., standing) can also be a source of error that can lead to misclassification.

[0062] In one embodiment, the state variables of the tracker include state variables associated with human activities. Thus, the tracker predicts the next human activity of the human target based on the history of the trajectory during the prediction step. The classifier generates a provisional target classification based on the features extracted from the output of the tracker. During the prediction step, the tracker (e.g., using Bayesian recursion) assigns an uncertainty value to the provisional target classification. Then, when the classification uncertainty value is higher than a threshold, the tracker updates the human activity state variable with the predicted value, or when the uncertainty value is lower than the threshold, the tracker updates the human activity state variable with the provisional target classification. Then, the tracker generates a target classification based on the updated human activity state variable.

[0063] By using a tracker to generate the target classification, some embodiments advantageously improve the classification accuracy compared to applications that use a classifier in an open - loop configuration (without using a tracker to track human activities and / or without feeding back the output of the classifier to the tracker). For example, when the classifier produces a provisional human activity classification with a high uncertainty value (e.g., because the human activity is a new activity not used during training of the classifier), the tracker instead uses the predicted activity.

[0064] In some embodiments, other advantages include enhancing location tracking by using active classification states. For example, when a first human target is sitting and a second human target is walking near the location of the first human target, data association at the intersection of the human target paths can be enhanced by the knowledge that the first human target is sitting (and, for example, the probability that the first human target remains sitting is high) and the second human target is walking (and, for example, the probability that the second human target remains walking is high). Thus, in this example, for instance, target detections near the first and second human targets that correspond to non-moving targets are assigned to the trajectory of the first human target, while target detections near the first and second human targets that correspond to moving targets are assigned to the trajectory of the second human target.

[0065] In one embodiment, in addition to other state variables (such as the state variables shown in Equations 1 and 2), the tracker also includes state variables a0 to a n , such as shown by the following formula:

[0066]

[0067] where a1 corresponds to a first human activity, a2 corresponds to a second human activity, and a n corresponds to the nth human activity, where n is an integer greater than 1.

[0068] In some embodiments, the tracker performs a human activity prediction step given by the following formula for state variables a0 to a n :

[0069]

[0070] where k represents the discrete time step, and f is a transition matrix based on the transition probabilities between human activities. In some embodiments, the transition probabilities are not linear. In some embodiments, the prediction step is performed every N frames, where N can be, for example, 8.

[0071] Figure 7 FIG. 700 shows a human activity state transition diagram according to an embodiment of the present invention. As Figure 7 shown, only four activities are used in this embodiment. It should be understood that fewer than four activities, such as three or two, or more than four activities, such as five, eight, ten, or more, can also be used.

[0072] As a non - limiting example, in one embodiment, state a1 corresponds to waving, state a2 corresponds to walking, state a3 corresponds to standing, and state a4 corresponds to sitting. The probabilities of each state can be the same or can be different. For example, in one embodiment, the probability p 44 (e.g., for a human target that remains sitting once it has sat down) can be, for example, 0.95, and the probability p 22 (e.g., for a human target that remains walking once it has been walking) can be, for example, 0.6, and the probability p 33 (e.g., for a human target that remains standing once it has stood up) can be, for example, 0.6.

[0073] The probabilities of transitioning between states can be reciprocal or can not be reciprocal. For example, the probability p 31 (e.g., for transitioning from standing to walking) can be, for example, 0.1, while the probability p 13 (e.g., for transitioning from walking to standing) can be, for example, 0.3. And the probability p 23 (e.g., for transitioning from walking to sitting) can be, for example, 0.05, while the probability p 32 (e.g., for transitioning from sitting to walking) can be, for example, 0.05.

[0074] In some embodiments, the probability p i of the associated activity a i can be calculated by the following formula:

[0075]

[0076] where Q is the number of states, and A i can be given by the following formula:

[0077]

[0078] where W ij is the transition weight for transitioning from activity i to activity j. In other words, Equation 9 can be understood as a softmax operation that normalizes the values of A i to obtain the probability value p i of activity a i .

[0079] Figure 8A FIG. 800 shows a block diagram for detecting, tracking, and classifying the activities of a human target according to an embodiment of the present invention. Blocks 802, 804, 806, 808, 810, 812, and 814 can be implemented, for example, by a processor 104.

[0080] The radar processing block 802 generates a radar image based on the radar signals received by the millimeter wave radar 102. In some embodiments, the radar image is a range-Doppler image. In other embodiments, the radar image is a range cross-range image. For example, in some embodiments, a millimeter wave radar with two receiving antennas receives radar signals from a field of view (e.g., from reflections from a human target) and generates a corresponding range-Doppler map. A two-dimensional (2D) MTI filter is applied to the corresponding range-Doppler map, and the result is coherently integrated. After the coherent integration, a range-Doppler image is generated.

[0081] The target detector block 804 detects potential targets based on the received radar image. For example, in some embodiments, sequential statistical (OS) constant false alarm rate (CFAR) (OS-CFAR) detection is performed by the target detector 804. Such a CFAR detector generates a detection image based on, for example, the power level of a range-Doppler image, in which, for example, "one" indicates a target and "zero" indicates a non-target. For example, in some embodiments, the CFAR detector compares the power level of the range-Doppler image with a threshold and marks points above the threshold as targets and points below the threshold as non-targets. Although a target may be indicated by one and a non-target may be indicated by zero, it should be understood that other values may be used to indicate targets and non-targets.

[0082] In one embodiment, the OS-CFAR detector uses the k-th quartile / median instead of the average CA-CFAR (cell average). Using the k-th quartile / median can be advantageously more robust to outliers.

[0083] The detection gating block 806 generates a list of targets and associated parameters based on the detection image, data from the tracker 808, and, for example, equations 1 through 6. For example, in some embodiments, a density-based spatial clustering of applications with noise (DBSCAN) algorithm, for example, is used to cluster the targets present in the detection image. Then, for example, parameter estimation is performed on each clustered target using one or more of equations 1 through 4. The detection gating block 806 then uses the estimated parameters, the predicted parameters received from the tracker 808, and, for example, equation 6 to screen useful data, for example, using ellipsoidal gating. The detection gating block 806 then generates a list of targets and associated estimated and screened parameters. The parameters estimated and screened by the detection gating block 806 include, for example, the angle of arrival (AoA) for the respective target, the bounding box dimensions (e.g., lx and ly), the rate of change of the bounding box dimensions (e.g., vlx and vly), and so on.

[0084] The tracker block 808 uses the respective trajectories to track each target by performing a prediction step and an update step for each trajectory. For example, during the prediction step, the tracker 808 generates a prediction of the state variables of the tracker (for the parameters associated with each trajectory) based on the respective history of each respective trajectory (e.g., using one or more of equations 1 through 5 and equations 7 through 8). During the update step, the tracker 808 associates the detected target with the respective trajectory based on the output of the detection gating block 806.

[0085] The tracker block 808 can also generate and kill targets based on the history of the trajectories and based on newly received measurements (e.g., the detected targets and associated parameters). The tracker block 808 can also perform tracking filtering. For example, in some embodiments, the tracker block 808 is implemented using an unscented Kalman filter or a particle filter to predict one or more parameters associated with each target, such as distance, angle, and bounding box dimensions, based on the history of the trajectories and based on the newly received measurements.

[0086] The feature extraction block 810 extracts features for each target from the output of the tracker block 808. For example, in some embodiments, a Doppler spectrogram is extracted from the history of the corresponding trajectory tracked by the tracker block 808. A classifier block 812, such as a long short-term memory (LSTM) classifier, uses the Doppler spectrogram to generate a temporary target activity classification.

[0087] The classification gating block 814 receives a temporary target activity classification from the classifier 812, receives an uncertainty value associated with such temporary target activity classification from the tracker 808, and generates a gating data output based on the uncertainty associated with the temporary target activity classification to indicate whether to use the temporary target classification. For example, in some embodiments, an elliptical gating is used to determine whether a particular temporary target classification should be used (e.g., whether the uncertainty value associated with the temporary target classification is above or below a predetermined threshold). For example, in some embodiments, the classification gating block 814 determines a first (Mahalanobis) distance between the temporary target activity classification and the predicted activity classification (generated by the tracker 808). For example, in some embodiments, the Mahalanobis distance Md_act between the predicted measurement Zpred and the new set of measurements Zmeas can be calculated by the following formula:

[0088]

[0089] where is the covariance matrix between the activity Zpred_act predicted by the tracker and the activity Zmeas_act measured by the classifier.

[0090] The determination of the classification distance is also based on the uncertainty value associated with the temporary target activity classification. When the classification distance is below the threshold, the classification gating block 814 generates a gating data output to notify the tracker 808 to use the temporary target activity classification. When the classification distance is above the threshold, the classification gating block 814 generates a gating data output to notify the tracker 808 to use the predicted activity classification.

[0091] As Figure 8A shown, the tracker 808 receives the temporary target activity classification (e.g., from the classifier 812 or the classification gating block 812), and generates an uncertainty value associated with the temporary target activity classification (e.g., during the prediction step). For example, the tracker uses the classification probabilities from the previous time step to generate the uncertainty value associated with the temporary target activity classification.

[0092] The tracker block 808 also receives gating data from the classification gating block 814 and updates state variables associated with human activities based on such gating data. For example, in some embodiments, when the gating data output indicates that the classification distance is higher than a threshold, the tracker block 808 updates the human activity state variable with the value predicted by the tracker block 808, and when the classification distance is lower than the threshold, the tracker block 808 updates the human activity state variable with a temporary target activity classification. Other state variables can also be updated based on the updated human activity state variables. Then, the tracker block 808 generates a target activity classification and a target localization based on the updated state variables. In some embodiments, an argmax function is used to generate the target activity classification generated by the tracker block 808. For example, the human activity with the highest probability weight is the activity selected by the tracker 808 as the target activity.

[0093] As Figure 8A shown, the tracker block 808 and the classifier block 812 operate in a closed-loop manner. In some embodiments, because the tracker block 808 uses the assumption of Gaussian states and the associated uncertainty, the classifier block 812 benefits from the tracker block 808. Thus, using the tracker block 808 allows the classifier block 812 to provide an associated probability and an associated uncertainty with respect to the target classification (because in some embodiments, the Bayesian approach of the tracker 808 adds an uncertainty measure to the output of the classifier 812, which helps in selecting the next human activity).

[0094] In some embodiments, because the state variables track the associated activities of the target, thereby enhancing the prediction step of the tracker (e.g., because more information can be used for prediction), the tracker block 808 benefits from the classifier block 812. For example, when two targets intersect, the activities associated with each target can help the tracker block 808 associate the correct detection with the correct trajectory. In other words, in some embodiments, tracking the human activity state using the tracker block 808 increases the probability of correctly assigning the detected target to the corresponding trajectory.

[0095] Advantages of some embodiments include improving the response to target misclassification by using the target classification predicted by the tracker rather than the target classification determined by the classifier and having high uncertainty. In some embodiments, the tracker advantageously obtains results similar to Bayesian deep learning without implementing the complex neural networks typically associated with Bayesian deep learning.

[0096] Figure 8B A flowchart of an exemplary method 850 for tracking human targets and corresponding activities according to an embodiment of the present invention is shown. The method 850 can be implemented by the millimeter-wave radar 102 and the processor 104.

[0097] During step 852, the millimeter-wave radar 102 transmits a radar signal (e.g., 106) towards a scene (e.g., 108). For example, the millimeter-wave radar 102 may transmit radar frequency-modulated pulses organized in frames. During step 854, the millimeter-wave radar 102 receives the reflected radar signal.

[0098] During step 856, a radar image is generated, for example, by the radar processing block 802 based on the received reflected radar signal. During step 858, a detection image (e.g., a matrix including "1" for detected targets and "0" otherwise) is generated, for example, by the target detector block 704.

[0099] During step 860, the detected targets in the detection image are clustered, for example, by the detection gating block 806. For example, since a target human may exhibit double spreading in the range and Doppler frequency bands, it is possible that multiple detected targets in the detection image belong to a single human target. During step 860, the targets with a high probability of belonging to a single human target are clustered together.

[0100] During step 862, parameters of each clustered target are determined based on radar measurements (e.g., based on the received reflected radar signal and subsequent processing). During step 862, parameters are determined for each clustered target, such as r (the distance of the target from the millimeter-wave radar sensor), θ (the angle of arrival - the angle of the target), v c (the radial velocity of the target), L r (the bounding box size in range), and L d (the bounding box size in Doppler).

[0101] During step 864, a tracker (such as tracker 808) for tracking human targets generates predictions for the detection state variables of each tracked human target based on the history of the corresponding trajectories. The detection state variables may include, for example, one or more of the variables in equations 1 and 2.

[0102] During step 866, detection gating is performed, for example, by the detection gating block 806. For example, for each tracked target, a detection distance may be determined between the measured parameters and the predicted parameters, for example, using equation 5. When the measurement of the target is within the gating region (i.e., the detection distance is below a predetermined threshold), the tracker updates the detection state variables of such a target with the measured values during step 868. When the measurement of the target is outside the gating region (i.e., the detection distance is above a predetermined threshold), the tracker updates the detection state variables of such a target with the predicted values during step 868.

[0103] During step 870, a Doppler spectrogram is generated, for example by feature extraction block 810, based on the updated detection status variable and the radar image. During step 872, a temporary target activity classification is generated, for example by classifier 812, for each tracked human target. For example, classifier 812 can generate a vector including the probabilities of each possible activity for each tracked human target. The temporary target activity classification (activity label) for each tracked target is the activity with the highest probability of the corresponding vector.

[0104] During step 874, a predicted value of the activity status variable is generated for each track, for example using equation 8 (e.g., by tracker 808). During step 876, an uncertainty value is associated with each temporary target activity classification based on the predicted value of the corresponding activity status variable of the corresponding target, for example using Bayesian recursion.

[0105] During step 878, classification gating is performed, for example by classification gating block 814. For example, in some embodiments, for each tracked target, the classification distance can be determined by calculating the Mahalanobis distance between the temporary target activity classification and the predicted activity classification. When the temporary target activity classification of a target is within the gating region (i.e., the classification distance is below a predetermined threshold), the tracker updates the classification status variable of such a target with the temporary target activity classification during step 880. When the temporary target activity classification of a target is outside the gating region (i.e., the classification distance is above a predetermined threshold), the tracker updates the classification status variable of such a target with the predicted value during step 880.

[0106] During step 882, a final target classification is generated for each target based on the updated classification status variable of the tracker. During step 884, a localization of the target is generated for each target based on the updated detection status variable of the tracker.

[0107] In one embodiment, the classifier uses data associated with human activity state transitions to improve human target activity classification. The first level of the classifier is trained using human activity snippets. The second level of the classifier is trained using human activity transition snippets.

[0108] Figure 9 A classifier 900 for classifying human activities according to an embodiment of the present invention is shown. Classifiers 308 and 812 can be implemented as classifier 900. Classifier 900 is implemented using an activity model, which includes: a first level including a bidirectional LSTM network 902, a fully connected (FC) layer 904, and a softmax layer 906; and a second level including an FC layer 908.

[0109] During normal operation, classifier 900 (e.g., features extracted from tracker 304 or 808) receives N spectrograms x1 to x corresponding to N (consecutive) frames N as input. In some embodiments, N is an integer greater than 1, such as 4, 8, 32, or higher.

[0110] Bidirectional LSTM network 902 generates human target activity vectors based on the corresponding input spectrograms x1 to x N which, after being processed by fully connected layer 904 and softmax layer 906, produce output activity probabilities y1 to y N . Fully connected layer 908 implementing the transition matrix f from Equation 8 receives the output activity probabilities y1 to y N and generates corresponding final activity probability to vectors (in some embodiments, this vector is produced every Nth frame) that is a vector including the probability of each human activity among the considered human activities. The human activity with a higher probability in vector corresponds to Figure 3 the "target classification", and corresponds to Figure 8A the "temporary target activity classification".

[0111] Bidirectional LSTM network 902, fully connected layers 904 and 908, and softmax layer 906 can be implemented in any manner known in the art.

[0112] In some embodiments, classifier 900 is trained in two steps. The first step trains the first stage of classifier 900, while the second step trains the second stage of classifier 900.

[0113] During the first step of training classifier 900, a first spectrogram segment associated with human activity is provided as input x1 to x N , where the first spectrogram segment is a truncated Doppler spectrogram that does not include transitions between activities. (e.g., as described with respect to Figure 10A ) While monitoring the outputs y1 to y N , bidirectional LSTM network 902 and layers 904 and 906 are trained.

[0114] After training the first stage of classifier 900 during the first step of training classifier 900, the second stage of classifier 900 is trained. During the second step of training classifier 900, a second spectrogram segment associated with human activity transitions is provided as input x1 to x N . (e.g., as described with respect to Figure 10A ) While monitoring the outputs to Meanwhile, the fully-connected layer 908 is trained. During the second step of training the classifier 900, the bidirectional LSTM network 902 and the layers 904 and 906 are not modified.

[0115] Figure 10A The Doppler spectrogram 1000 of human activities and activity transitions of a human target according to an embodiment of the present invention is shown. As Figure 10A shown, the human target can transition between activities over a period of time. During the first step of training the classifier 900, the first spectrogram segment can include segments similar to the Figure 10A activity segments shown (e.g., walking, standing, sitting, and waving). During the second step of training the classifier 900, the second spectrogram segment can include segments similar to the Figure 10A transition segments shown (e.g., transition from walking to standing, transition from standing to sitting, and transition from sitting to waving).

[0116] Figure 10B FIGs. 1020, 1030, and 1040 respectively illustrate the actual activity classification result, the output of the classifier 812 (e.g., 900), and the output of the tracker 808 according to an embodiment of the present invention. Figure 10B The FIGS. 1020, 1030, and 1040 correspond to Figure 10A the Doppler spectrogram 1000. As shown by Figure 10B the output of the tracker 808 (curve 1040) is closer to the actual activity classification result (curve 1020) than the output of the classifier 812 (curve 1030). In other words, compared to using the classifier 812 in an open loop, Figure 10B it shows improved performance when using the tracker 808 to classify human activities.

[0117] Figure 10C and Figure 10D respectively show the confusion matrices for the classifier 812 and the tracker 808 according to an embodiment of the present invention. Figure 10C and Figure 10D The confusion matrices correspond to Figure 10B the FIGS. 1020, 1030, and 1040 and correspond to Figure 10A the Doppler spectrogram 1000. As Figure 10C and Figure 10D shown, in one embodiment, compared to the accuracy of the standalone classifier 812, the classifier 812 shows a total actual correct prediction of 87.4%, and the tracker 808 shows improved accuracy with a total actual correct prediction of 94.1%.

[0118] By including activity transitions in the training of a classifier, some embodiments advantageously minimize or eliminate misclassifications caused by transitions between activities.

[0119] Figure 11 FIG. 1100 is a block diagram illustrating a machine learning pipeline for machine language-based feature extraction and identification, which can be used to train a classifier 900 to classify human targets based on human activities. Figure 11 The top portion 1100 is devoted to processing training features (e.g., Doppler spectrogram segments) for configuring the first and second portions of the classifier 900. The bottom portion 1120 is devoted to processing new measurements using the trained classifier 900 (e.g., trained in the manner shown in the top portion 1100).

[0120] As Figure 11 shown in the top portion 1100 of FIG., the training data 1102 is transformed into feature vectors 1110 and corresponding labels 1112. The training data 1102 represents raw data (e.g., echoes). The feature vectors 1110 represent a set of generated vectors representative of the training data 1102. The labels 1112 represent user metadata associated with the corresponding training data 1102 and feature vectors 1110. For example, during a first step of training the classifier 900, the training data 1102 includes a first spectrogram segment associated with a human activity and a corresponding label (e.g., walking, standing, sitting, waving, etc.). During a second step of training the classifier 900, the training data 1102 includes a second spectrogram segment associated with a human activity transition as well as a human activity and a corresponding label (e.g., a transition from walking to standing, a transition from standing to sitting, a transition from sitting to waving, etc.).

[0121] As shown, the training data 1102 is transformed into feature vectors 1110 using an image formation algorithm. The data preparation block 1104 represents the initial formatting of the raw sensor data, and the data annotation block 1108 represents the state identification from the training data 1102.

[0122] During training, one or more radar images are captured in a controlled environment, such as using a millimeter-wave radar, which includes one or more static targets and moving targets (e.g., humans, moving machinery, furniture, and other moving devices). In some cases, multiple radar images are recorded to improve the accuracy of identification. The machine learning algorithm 1114 evaluates the ability of the prediction model 1130 to identify feature vectors and iteratively updates the training data 1102 to improve the classification accuracy of the algorithm. The training performance of the machine learning algorithm can be determined by calculating the cross-entropy performance. In some embodiments, the machine learning algorithm 1114 iteratively adjusts the image formation parameters for at least 90% classification accuracy. Alternatively, other classification accuracies can be used.

[0123] The machine learning algorithm 1114 can be implemented using various machine learning algorithms known in the art. For example, a random forest algorithm or a neural network algorithm (such as ResNet-18 or other neural network algorithms known in the art) can be used to classify and analyze the feature vectors 1110. During the iterative optimization of the feature vectors 1110, multiple parameters of the image formation 1106 can be updated.

[0124] Once the classifier has been trained using the reference training data 1102, the reference signature can be used for classification during normal operation. The trained classifier 900 is represented as the prediction model 1130 in the bottom portion 1120. During normal operation, new target data 1122 is received. The data preparation block 1124 (e.g., corresponding to blocks 802, 804, 806, and 808) prepares the new target data 1122 for image formation, and the image formation block 1126 (e.g., corresponding to block 810) forms the newly extracted feature vectors 1128. The prediction model 1130 (e.g., the classifier 900) generates a prediction label (e.g., activity classification) based on the extracted feature vectors 1128.

[0125] Exemplary embodiments of the present invention are outlined herein. Other embodiments can also be understood based on the entire content of the specification and claims submitted herein.

[0126] Example 1: A method for tracking a target using a millimeter-wave radar, the method comprising: receiving radar signals using the millimeter-wave radar; generating a range-Doppler map based on the received radar signals; detecting the target based on the range-Doppler map; tracking the target using a trajectory; generating a predicted activity label based on the trajectory, wherein the predicted activity label indicates the actual activity of the target; generating a Doppler spectrogram based on the trajectory; generating a temporary activity label based on the Doppler spectrogram; assigning an uncertainty value to the temporary activity label, wherein the uncertainty value indicates the confidence level that the temporary activity label is the actual activity of the target; and generating a final activity label based on the uncertainty value.

[0127] Example 2: The method according to Example 1, wherein generating the final activity label comprises: using the predicted activity label as the final activity label when the uncertainty value is higher than a predetermined threshold, and using the temporary activity label as the final activity label when the uncertainty value is lower than the predetermined threshold.

[0128] Example 3: The method according to one of Examples 1 or 2, wherein generating the final activity label comprises: using an elliptical gating to determine whether the uncertainty value is higher than the predetermined threshold or lower than the predetermined threshold.

[0129] Example 4: The method according to one of Examples 1 to 3, the method further comprising: generating the temporary activity label using an activity model, wherein the activity model comprises a first stage and a second stage, the first stage comprises a long short-term memory (LSTM) network, and the second stage comprises a fully connected layer, wherein the second stage has an input terminal coupled to the output terminal of the first stage.

[0130] Example 5: The method according to one of Examples 1 to 4, the method further comprising: training the first stage using a truncated Doppler spectrogram segment of the target activity.

[0131] Example 6: The method according to one of Examples 1 to 5, the method further comprising: training the second stage using a Doppler spectrogram segment of the transition between target activities.

[0132] Example 7: The method according to one of Examples 1 to 6, the method further comprising: training the first stage using a truncated Doppler spectrogram segment of the target activity; and after training the first stage, training the second stage using a Doppler spectrogram segment of the transition between target activities.

[0133] Example 8: The method according to one of Examples 1 to 7, wherein tracking the target comprises: tracking the target using an unscented Kalman filter.

[0134] Example 9: The method according to one of Examples 1 to 8, wherein the target is a human target.

[0135] Example 10: A method according to one of Examples 1 to 9, wherein generating a temporary activity label includes: using a long short-term memory (LSTM) classifier.

[0136] Example 11: A method according to one of Examples 1 to 10, wherein generating a predicted activity label based on a trajectory includes: generating a predicted activity label based on state variables associated with the corresponding activity.

[0137] Example 12: A method according to one of Examples 1 to 11, the method further including: determining a location of a target based on state variables associated with the corresponding activity.

[0138] Example 13: A millimeter-wave radar system, the millimeter-wave radar system including a millimeter-wave radar and a processor, the millimeter-wave radar being configured to transmit and receive radar signals, the processor including: a radar processing block configured to generate a range-Doppler map based on radar signals received by the millimeter-wave radar; a target detector block configured to detect a target based on the range-Doppler map; a tracker configured to: track the target using a trajectory and generate a predicted activity label based on the trajectory, wherein the predicted activity label indicates an actual activity of the target; a feature extraction block configured to generate a Doppler spectrogram based on the trajectory; a classifier configured to generate a temporary activity label based on the Doppler spectrogram; and a classification gating block configured to: receive an uncertainty value associated with the temporary activity label and generate gating data based on the uncertainty value, the predicted activity label, and a temporary activity classification, wherein the uncertainty value indicates a confidence level that the temporary activity label is the actual activity of the target, and wherein the tracker is configured to generate a final activity label based on the gating data.

[0139] Example 14: The system according to Example 13, wherein the tracker is configured to: use the predicted activity label as the final activity label when the uncertainty value is higher than a predetermined threshold, and use the temporary activity label as the final activity label when the uncertainty value is lower than the predetermined threshold.

[0140] Example 15: The system according to one of Examples 13 or 14, wherein the classifier includes a first stage and a second stage, the first stage including a long short-term memory (LSTM) network, and the second stage including a fully connected layer, wherein the second stage has an input terminal coupled to an output terminal of the first stage.

[0141] Example 16: The system according to one of Examples 13 to 15, wherein the first stage further comprises: a second fully connected layer having an input end coupled to the input end of the LSTM network; and a softmax layer having an input end coupled to the output end of the second fully connected layer and an output end coupled to the output end of the first stage.

[0142] Example 17: The system according to one of Examples 13 to 16, wherein the radar signal comprises a chirp pulse.

[0143] Example 18: A method for tracking a target using a millimeter-wave radar, the method comprising: receiving a radar signal using the millimeter-wave radar; generating a range-Doppler map based on the received radar signal; detecting a target based on the range-Doppler map; using a tracker to track the target using a trajectory; using a classifier to generate a temporary activity label based on the output of the tracker; and using the tracker to generate a final activity label based on the output of the classifier, wherein the tracker uses state variables to track the activity label of the target.

[0144] Example 19: The method according to Example 18, wherein when the uncertainty value associated with the temporary activity label is higher than a predetermined threshold, the tracker uses the predicted activity label as the final activity label, and when the uncertainty value is lower than the predetermined threshold, the tracker uses the temporary activity label as the final activity label, wherein the tracker generates the predicted activity label based on the state variables.

[0145] Example 20: The method according to one of Examples 18 or 19, wherein the tracker comprises an unscented Kalman filter.

[0146] Although the present invention has been described with reference to illustrative embodiments, the present specification is not intended to be construed in a limiting sense. Various modifications and combinations of the illustrative embodiments and other embodiments of the present invention will be apparent to those skilled in the art upon reference to the specification. Accordingly, the appended claims are intended to cover any such modifications or embodiments.

Claims

1. A method for tracking a target using a millimeter-wave radar, the method comprising: Receiving radar signals using the millimeter-wave radar; Generating a range-Doppler map based on the received radar signals; Detecting a target based on the range-Doppler map; Tracking the target using a trajectory; Generating a predicted activity label based on the trajectory, wherein the predicted activity label indicates the actual activity of the target; Generating a Doppler spectrogram based on the trajectory; Generating a provisional activity label based on the Doppler spectrogram; Assigning an uncertainty value to the provisional activity label, wherein the uncertainty value indicates the confidence level that the provisional activity label is the actual activity of the target; And Generating a final activity label based on the uncertainty value, wherein generating the final activity label includes: using the predicted activity label as the final activity label when the uncertainty value is higher than a predetermined threshold, and using the provisional activity label as the final activity label when the uncertainty value is lower than the predetermined threshold.

2. The method according to claim 1, wherein generating the final activity tag comprises: Using elliptical gating to determine whether the uncertainty value is higher than the predetermined threshold or lower than the predetermined threshold.

3. The method according to claim 1, the method further comprising: Using an activity model to generate the provisional activity label, wherein the activity model includes a first stage and a second stage, the first stage includes a long short-term memory (LSTM) network, and the second stage includes a fully connected layer, wherein the second stage has an input terminal coupled to the output terminal of the first stage.

4. The method according to claim 3, the method further comprising: Training the first stage using truncated Doppler spectrogram segments of target activities.

5. The method according to claim 3, wherein the method further comprises: Training the second stage using Doppler spectrogram segments of transitions between target activities.

6. The method according to claim 3, the method further comprising: Training the first stage using truncated Doppler spectrogram segments of target activities; And After training the first stage, training the second stage using Doppler spectrogram segments of transitions between target activities.

7. The method according to claim 1, wherein tracking the target comprises: Using an unscented Kalman filter to track the target.

8. The method according to claim 1, wherein the target is a human target.

9. The method according to claim 1, wherein generating the temporary activity tag comprises: Using a long short-term memory (LSTM) classifier.

10. The method according to claim 1, wherein generating the predicted activity label based on the trajectory comprises: Generating the predicted activity label based on state variables associated with the corresponding activity.

11. The method according to claim 10, the method further comprising: Determining the position of the target based on the state variables associated with the corresponding activity.

12. A millimeter-wave radar system, comprising: A millimeter-wave radar configured to transmit and receive radar signals; And A processor, comprising: A radar processing block configured to generate a range-Doppler map based on the radar signals received by the millimeter-wave radar; A target detector block configured to detect a target based on the range-Doppler map; A tracker configured to: Track the target using a trajectory, and Generate a predicted activity label based on the trajectory, wherein the predicted activity label indicates the actual activity of the target; A feature extraction block configured to generate a Doppler spectrogram based on the trajectory; A classifier configured to generate a provisional activity label based on the Doppler spectrogram; and A classification gating block configured to: receive an uncertainty value associated with the temporary activity label and generate gating data based on the uncertainty value, the predicted activity label, and the temporary activity label, wherein the uncertainty value indicates a confidence level that the temporary activity label is the actual activity of the target, and wherein the tracker is configured to generate a final activity label based on the gating data. Wherein the tracker is configured to: use the predicted activity label as the final activity label when the uncertainty value is higher than a predetermined threshold, and use the temporary activity label as the final activity label when the uncertainty value is lower than the predetermined threshold.

13. The system according to claim 12, wherein the classifier includes a first stage and a second stage, the first stage includes a long short-term memory (LSTM) network, and the second stage includes a fully connected layer, wherein the second stage has an input terminal coupled to the output terminal of the first stage.

14. The system according to claim 13, wherein the first stage further includes: A second fully connected layer having an input terminal coupled to the input terminal of the LSTM network; And A softmax layer having an input terminal coupled to the output terminal of the second fully connected layer and an output terminal coupled to the output terminal of the first stage.

15. The system according to claim 12, wherein the radar signal includes a chirp pulse.

16. A method for tracking a target using a millimeter-wave radar, the method comprising: Receiving a radar signal using the millimeter-wave radar; Generating a range-Doppler map based on the received radar signal; Detecting a target based on the range-Doppler map; Using a tracker to track the target using a trajectory; Using a classifier to generate a temporary activity label based on the output of the tracker; And Using the tracker to generate a final activity label based on the output of the classifier, wherein the tracker uses state variables to track the activity label of the target. Wherein when an uncertainty value associated with the temporary activity label is higher than a predetermined threshold, the tracker uses the predicted activity label as the final activity label, and when the uncertainty value is lower than the predetermined threshold, the tracker uses the temporary activity label as the final activity label, wherein the tracker generates the predicted activity label based on the state variables.

17. The method according to claim 16, wherein the tracker includes an unscented Kalman filter.

Images