Real-time dynamic trajectory tracking method and system for millimeter wave radar gesture recognition

Abstract

The invention discloses a real-time dynamic trajectory tracking method and system for millimeter wave radar gesture recognition, and relates to the technical field of gesture recognition tracking, and the method comprises the steps: receiving an echo signal reflected by a gesture, extracting a potential target point cloud, and carrying out the clustering generation of a gesture point cloud sequence; establishing a multi-modal motion model library, dynamically selecting an optimal motion model by adopting graph matching, and generating a prediction state in combination with a gesture point cloud sequence; on the basis of a Poisson multi-Bernoulli hybrid filtering framework, according to the signal-to-noise ratio and spatial distribution of the current gesture point cloud sequence, dynamically adjusting the observation weight, optimizing the observation point cloud, and carrying out optimal association by combining Mahalanobis distance with dynamic time warping; a multi-hypothesis tracking strategy is adopted to maintain trajectory hypothesis, an optimal trajectory is selected through a trajectory scoring mechanism, and Kalman filtering smoothing processing is performed on the optimal trajectory. According to the method, high-precision and low-delay tracking of gesture motion is realized, gesture habits of different users and complex environment interference can be adapted, and meanwhile, relatively high track precision is kept.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of gesture recognition, more particularly, to a real-time dynamic trajectory tracking method and system for millimeter wave radar gesture recognition. BACKGROUND

[0002] With the rapid development of human-computer interaction (HCI) technology, gesture recognition as a natural and intuitive interaction method has been widely applied in smart home, vehicle-mounted systems, virtual reality (VR) and augmented reality (AR) fields. Traditional camera-based gesture recognition technology is affected by lighting conditions, occlusion problems and privacy restrictions, and it is difficult to work stably in all scenarios. Millimeter wave radar has become a research hotspot in the field of gesture recognition due to its high resolution, strong penetration, immunity to environmental light interference and good privacy protection characteristics. Millimeter wave radar works in the frequency band of 30 GHz to 300 GHz, can emit frequency-modulated continuous wave (FMCW) or pulse signals, and receive the echo of the target reflection to extract the distance, speed and angle information of the target. Compared with optical sensors, millimeter wave radar can more accurately capture small movements and achieve real-time dynamic detection of gestures. However, millimeter wave radar gesture recognition still faces some technical challenges, such as dynamic target interference caused by Doppler effect, trajectory modeling of complex gestures, and high computational complexity of real-time processing.

[0003] Existing millimeter wave radar gesture recognition methods mainly include static feature classification and dynamic trajectory tracking. Static feature classification method extracts micro-Doppler features or point cloud distribution of gestures for pattern matching, but has limited generalization ability when dealing with continuous dynamic gestures. Dynamic trajectory tracking method captures the motion path of gestures in real time, combines Kalman filter, particle filter or deep learning model for trajectory prediction and optimization, so as to realize high-precision continuous gesture recognition. However, the existing methods still have room for improvement in real-time performance and anti-interference ability of complex gestures. Therefore, how to combine efficient signal processing algorithms and machine learning models to achieve high-robustness gesture trajectory tracking under low latency conditions is a problem to be solved. SUMMARY

[0004] In order to solve the above technical problems, the present application provides a real-time dynamic trajectory tracking method and system for millimeter wave radar gesture recognition, which realizes high-robustness gesture trajectory tracking under low latency conditions, can adapt to different users' gesture habits and complex environmental interference, and at the same time maintains high trajectory precision and recognition accuracy.

[0005] The first aspect of the present application provides a real-time dynamic trajectory tracking method for millimeter wave radar gesture recognition, comprising the following steps: The millimeter wave signal is transmitted by a frequency-modulated continuous wave millimeter wave radar, and a gesture reflected echo signal is received, the echo signal is preprocessed, constant false alarm rate detection is used to filter noise, potential target point clouds are extracted, the potential target point clouds are clustered, and a gesture point cloud sequence is generated; A multi-modal motion model library including a uniform speed model, an acceleration model and an arc motion model is established, an optimal motion model is dynamically selected by using graph matching, a next time state of the gesture is predicted in combination with the gesture point cloud sequence, and a predicted state is generated; Based on a Poisson multi-Bernoulli mixture filtering framework, an observation weight is dynamically adjusted according to a signal-to-noise ratio and a spatial distribution of the current gesture point cloud sequence, an observation point cloud is optimized, a Mahalanobis distance is used to calculate a matching degree of the observation point cloud and the predicted state, and optimal association is performed in combination with dynamic time warping. A multi-hypothesis tracking strategy is used to maintain a track hypothesis, an optimal track is selected through a track scoring mechanism, Kalman filtering smoothing processing is performed on the optimal track, and a gesture dynamic track is output.

[0006] In the scheme, the echo signal is preprocessed, constant false alarm rate detection is used to filter noise, and potential target point clouds are extracted, specifically: The received echo signal is mixed with a current transmission signal to obtain an intermediate frequency signal, the intermediate frequency signal is subjected to analog-to-digital conversion sampling to obtain a discrete time sequence, the discrete time sequence is organized based on a radar frame structure of the frequency-modulated continuous wave millimeter wave radar to generate a distance dimension data matrix and a speed dimension data matrix; The distance dimension data matrix is processed by using fast Fourier transform to convert a time domain signal into a frequency domain, a gesture target distance is obtained according to a frequency corresponding to a frequency domain peak value, and a second dimension fast Fourier transform is performed on the speed dimension data matrix to extract a Doppler shift of the gesture target and obtain a gesture target speed; A range-velocity matrix is formed according to the gesture target distance and the gesture target speed, an ordered statistics constant false alarm rate detection is used in the range-velocity matrix to calculate a local noise level through a sliding window, and real gesture targets are distinguished from noise; A dynamic threshold is set, units greater than the dynamic threshold are retained, false alarms are suppressed, a binary detection result is output, potential target point clouds are marked, the potential target point clouds are mapped to a three-dimensional space, and data in different dimensions are subjected to normalization processing.

[0007] In the scheme, the potential target point clouds are clustered to generate a gesture point cloud sequence, specifically: A neighborhood radius is defined according to a physical size of the gesture and a preset radar resolution, all potential target point clouds are traversed, when a point number in the neighborhood radius of a potential target point cloud is greater than a preset minimum point cloud number, the point cloud is marked as a core point cloud, and the core point cloud is used to represent a main part of the gesture; From the core point cloud, recursively merge all reachable potential target point clouds in its neighborhood to form independent clusters, and the boundary points are assigned to the nearest independent cluster, and the isolated point cloud not assigned to any independent cluster is marked as noise and removed; The motion consistency of the independent cluster generated by clustering is tested, the centroid coordinates and bounding box size of each independent cluster are calculated, the velocity distribution of the point cloud in the cluster is extracted, the cluster feature is acquired, the gesture label is generated, and the gesture point cloud sequence with the gesture label is output.

[0008] In this scheme, a multi-modal motion model library containing uniform speed model, acceleration model and arc motion model is established, and the optimal motion model is dynamically selected by graph matching, specifically: The uniform speed model suitable for smooth straight line motion, the acceleration model suitable for variable speed motion and the arc motion model suitable for rotation or arc trajectory are constructed, and a multi-modal motion model library is established based on the uniform speed model, the acceleration model and the arc motion model; The time series graph structure is constructed in a sliding window manner, each node contains a motion feature vector, a motion model prediction state and an observation matching degree, wherein the observation matching degree uses the Mahalanobis distance score of the motion model prediction trajectory and the measured point cloud; The time series graph structure includes time series edges, model competition edges and cross-frame similar edges, the time series edges are constructed by forcibly connecting adjacent frame nodes, the edge weights are calculated based on motion continuity, the model competition edges are constructed by connecting nodes with different models in the same frame, the edge weights are calculated based on the KL divergence between models, the cross-frame similar edges are constructed according to the connection between nodes with similar motion patterns in non-continuous frames, and the edge weights are calculated based on the trajectory shape DTW distance; After obtaining new frame data, a new node containing three model prediction states is generated, the graph structure is updated, a weighted bipartite graph is constructed based on the updated graph structure, the left side nodes represent candidate motion models of a specific frame, and the right side nodes correspond to observation data of a specific frame, the edge connection and edge weight of the weighted bipartite graph are generated according to the three edge structures and edge weights in the time series graph structure; The motion smoothness constraint is introduced as a penalty term to improve the Hungarian algorithm, the optimal motion model is selected by solving the maximum weight matching, and the next time state of the gesture is predicted based on the optimal motion model according to the gesture point cloud sequence, and the prediction state is generated.

[0009] In this scheme, the probability distribution is managed based on the Poisson multi-Bernoulli mixture filtering framework, the observation weight is dynamically adjusted according to the signal-to-noise ratio and spatial distribution of the current gesture point cloud sequence, and the observation point cloud is optimized, specifically The new-born gesture target is described by a Poisson point process, the spatial probability density of the new-born gesture target in the radar field of view is established, the existence probability and state distribution of the known gesture target are described by a multi-Bernoulli mixture, a Bernoulli tuple including the existence probability and the state distribution described by a Gaussian mixture model is maintained for each known gesture target; The signal-to-noise ratio of the observation point cloud in the gesture point cloud sequence is calculated, and the normalized entropy value of the observation point cloud is calculated to represent the spatial distribution, the observation weight is dynamically optimized combining the signal-to-noise ratio and the spatial distribution, and the hierarchical data association is performed through the optimized observation weight; In the new-born gesture target association, when the observation weight corresponding to the unmatched observation point cloud is greater than a preset weight threshold and the distance from the known gesture target is greater than a preset distance threshold, a new Bernoulli tuple is generated; In the known gesture target association, a noise matrix of the fusion observation weight is constructed according to the predicted state and the observation point cloud combined with the observation noise, the matching degree of the predicted state and the observation point cloud is evaluated by the weighted Mahalanobis distance, the n observation points with the smallest Mahalanobis distance from the predicted position are selected as candidate observations, the association probability of the candidate observations is calculated, and the existence probability and the state distribution are updated. The point cloud is reconstructed according to the Bernoulli tuple to obtain the optimized observation point cloud.

[0010] In the scheme, the Mahalanobis distance is used to calculate the matching degree of the observation point cloud and the predicted state, and the dynamic time warping is used for optimal association, specifically: For the optimized observation point cloud, the Mahalanobis distance is used to calculate the single-point matching degree with the predicted state, and for a trajectory segment of a preset length, a matching matrix is calculated according to the single-point matching degree. The dynamic time warping is used for optimal association, a warping path is defined according to the observation point cloud and the predicted state, a cost matrix is constructed combining the matching matrix and the kinematic constraint, the minimum cumulative cost is solved by dynamic programming, the optimal association pair set of a preset number is obtained by backtracking, and a preset number of hypothesis trajectories are generated.

[0011] In the scheme, a multi-hypothesis tracking strategy is used to maintain trajectory hypotheses, and a trajectory scoring mechanism is used to select the optimal trajectory, and the optimal trajectory is subjected to Kalman filtering smoothing processing to output the gesture dynamic trajectory, specifically: A dynamic trajectory hypothesis library is constructed and maintained by the multi-hypothesis tracking strategy, each hypothesis node includes a trajectory state, an existence probability and a historical observation matching sequence, a multi-dimensional scoring function is constructed using a matching score, a smoothing score and a persistence score, and the multi-dimensional scoring function is used for hypothesis trajectory scoring; The hypothesis trajectory with the highest score is selected as the optimal trajectory, the optimal trajectory is subjected to Kalman smoothing processing, the process noise is adjusted according to the dynamic characteristics of the gesture target, and the gesture dynamic trajectory is output.

[0012] The second aspect of the present application provides a real-time dynamic trajectory tracking system for millimeter wave radar gesture recognition, which comprises a signal acquisition and preprocessing module, a point cloud optimization and clustering module, a motion model matching module, a multi-modal tracking module and a trajectory smoothing module. The signal acquisition and preprocessing module transmits millimeter wave signals by a frequency-modulated continuous wave millimeter wave radar and receives echo signals reflected by gestures, and pre-processes the echo signals. The point cloud optimization and clustering module filters noise by constant false alarm rate detection, extracts potential target point clouds, clusters the potential target point clouds, and generates a gesture point cloud sequence. The motion model matching module establishes a multi-modal motion model library containing uniform speed models, acceleration models and arc motion models, dynamically selects the optimal motion model by graph matching, predicts the next time state of the gesture in combination with the gesture point cloud sequence, and generates a predicted state. The multi-modal tracking module is based on a Poisson multi-Bernoulli mixture filtering framework, dynamically adjusts the observation weight according to the signal-to-noise ratio and spatial distribution of the current gesture point cloud sequence, optimizes the observation point cloud, calculates the matching degree of the observation point cloud and the predicted state by Mahalanobis distance, and performs optimal association by dynamic time warping. A multi-hypothesis tracking strategy is used to maintain trajectory hypotheses, and the optimal trajectory is selected by a trajectory scoring mechanism. The trajectory smoothing module performs Kalman filter smoothing processing on the optimal trajectory and outputs a gesture dynamic trajectory.

[0013] Compared with the prior art, the present application has the following advantages: The real-time dynamic trajectory tracking method of the present application realizes high-precision and low-delay tracking of gesture motion through multi-module collaborative processing, and exhibits significant technical advantages in complex scenes.

[0014] In the signal acquisition and preprocessing stage, the system acquires gesture reflection signals by a high-resolution millimeter wave radar, combines adaptive filtering and detection algorithms, effectively extracts potential target point clouds, and ensures the reliability and stability of the original data. The point cloud optimization module uses dynamic clustering and motion consistency verification to significantly suppress environmental noise and static interference, improve the detection rate of effective gesture targets, and avoid false tracking caused by clutter.

[0015] The multi-modal tracking module is based on a PMBM filtering framework, and combines a multi-hypothesis tracking strategy to robustly manage the probability distribution of new-born targets and known targets. By dynamically adjusting the observation weight and optimizing data association, the system can maintain stable trajectory continuity and significantly reduce the identity switching frequency in complex situations such as target occlusion, intersection or temporary disappearance. The motion prediction and smoothing module uses a multi-model Kalman filter to adaptively select the optimal motion model, and eliminates trajectory jitter through a forward-backward smoothing algorithm to output a smooth and physically consistent gesture motion path.

[0016] The gesture recognition module fuses spatio-temporal and geometric features, and classifies in real time through a lightweight neural network to map the trajectory into a semantic gesture instruction. The overall system adopts a pipeline architecture and resource optimization strategy to achieve efficient real-time processing capability on an embedded platform, meeting the requirements of low latency and high response speed in human-computer interaction scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments or examples of the present application, the drawings needed to be used in the embodiments or example descriptions will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0018] Figure 1 A flowchart of a real-time dynamic trajectory tracking method for millimeter wave radar gesture recognition is shown. Figure 2 A flowchart of dynamically selecting the optimal motion model using graph matching is shown. Figure 3 A flowchart of optimizing the observation point cloud based on Poisson Multi-Bernoulli Mixture filtering is shown. Figure 4 A block diagram of a real-time dynamic trajectory tracking system for millimeter wave radar gesture recognition is shown. DETAILED DESCRIPTION

[0019] In order to more clearly illustrate the technical solutions in the embodiments or examples of the present application, the drawings needed to be used in the embodiments or example descriptions will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0020] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, however, the present application can also be implemented in other ways different from those described herein, therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below.

[0021] Figure 1A flowchart of a real-time dynamic trajectory tracking method of millimeter wave radar gesture recognition is shown.

[0022] As Figure 1 shown, the embodiment provides a real-time dynamic trajectory tracking method of millimeter wave radar gesture recognition, comprising: S102, transmitting a millimeter wave signal by a frequency-modulated continuous wave millimeter wave radar and receiving a gesture-reflected echo signal, preprocessing the echo signal, filtering noise by constant false alarm rate detection, extracting a potential target point cloud, clustering the potential target point cloud, and generating a gesture point cloud sequence; S104, establishing a multi-modal motion model library containing uniform speed models, acceleration models, and arc motion models, dynamically selecting an optimal motion model by graph matching, predicting a next time state of the gesture in combination with the gesture point cloud sequence, and generating a predicted state; S106, dynamically adjusting an observation weight based on a Poisson multi-Bernoulli hybrid filtering framework, according to a signal-to-noise ratio and a spatial distribution of the current gesture point cloud sequence, optimizing an observation point cloud, calculating a matching degree of the observation point cloud and the predicted state by Mahalanobis distance, and performing optimal association in combination with dynamic time warping; S108, maintaining a trajectory hypothesis by a multi-hypothesis tracking strategy, selecting an optimal trajectory by a trajectory scoring mechanism, performing Kalman filtering smoothing processing on the optimal trajectory, and outputting a gesture dynamic trajectory.

[0023] It should be noted that the radar transmitting end generates a frequency-modulated continuous wave (FMCW) signal, a typical frequency band is 60 GHz or 77 GHz, and it has a high resolution characteristic; a gesture target reflects part of the electromagnetic wave, which is captured by the radar receiving antenna to form an echo signal, and the echo signal carries the distance, speed, and angle information of the target due to the Doppler shift introduced by gesture motion. The received echo signal is mixed with the current transmitted signal to obtain an intermediate frequency signal, which contains target distance and speed information, and the frequency of the intermediate frequency signal is proportional to the target distance. The intermediate frequency signal is sampled by analog-to-digital conversion to obtain a discrete time sequence, and the discrete time sequence is organized based on the radar frame structure of the frequency-modulated continuous wave millimeter wave radar to generate a distance dimension data matrix and a speed dimension data matrix; the distance dimension data matrix is processed using fast Fourier transform to convert the time domain signal to the frequency domain, the gesture target distance is obtained according to the frequency corresponding to the frequency domain peak value, the second dimension fast Fourier transform is performed on the speed dimension data matrix to extract the Doppler shift of the gesture target and obtain the gesture target speed, and the frequency shift size is related to the target speed; a range-velocity matrix is formed according to the gesture target distance and the gesture target speed. In the range-velocity matrix, the ordered statistics constant false alarm rate detection is used to calculate the local noise level by a sliding window to distinguish between real gesture targets and noise; a dynamic threshold is set to retain cells greater than the dynamic threshold, suppress false alarms, output a binary detection result, and mark a potential target point cloud.

[0024] The DBSCAN clustering algorithm is used to merge adjacent points to form gesture-related point cloud clusters. The potential target point cloud is mapped to a three-dimensional space, and the azimuth information of each point is calculated through the antenna array configuration and beamforming technology of the radar. The data in different dimensions is normalized to enable clustering calculation in the same dimension. The neighborhood radius is defined according to the physical size of the gesture and the preset radar resolution, and all potential target point clouds are traversed. When the number of points within the neighborhood radius of the potential target point cloud is greater than the preset minimum point cloud number, the point cloud is marked as a core point cloud, and the core point cloud represents the main part of the gesture. Starting from the core point cloud, all reachable potential target point clouds within its neighborhood are recursively merged to form independent clusters, and the boundary points are assigned to the nearest independent cluster. The isolated point cloud that is not assigned to any independent cluster is marked as noise and is removed. The independent clusters generated by clustering are subjected to motion consistency verification, including Doppler velocity filtering, calculation of the average Doppler velocity of points in the same cluster, removal of outliers with a large velocity difference, and verification of the spatiotemporal continuity of the current cluster in combination with the clustering results of the previous frame. The centroid coordinates and bounding box size of each independent cluster are calculated, and the velocity distribution of the point cloud in the cluster is extracted to distinguish different gestures. The cluster features are obtained, the gesture label is generated, and the gesture point cloud sequence with the gesture label is output.

[0025] Figure 2 A flowchart of dynamically selecting an optimal motion model using graph matching is shown.

[0026] According to an embodiment of the present application, a multi-modal motion model library containing uniform speed models, acceleration models and arc motion models is established, and an optimal motion model is dynamically selected using graph matching. Specifically, S202, a uniform speed model suitable for smooth straight line motion, an acceleration model suitable for variable speed motion, and an arc motion model suitable for rotation or arc trajectory are constructed, and a multi-modal motion model library is established based on the uniform speed model, the acceleration model and the arc motion model; S204, a sliding window is used to construct a time series graph structure, each node contains a motion feature vector, a motion model prediction state and an observation matching degree, wherein the observation matching degree is obtained using the Mahalanobis distance score of the motion model prediction trajectory and the measured point cloud; S206, the time series graph structure contains time series edges, model competition edges and cross-frame similar edges, the time series edges are constructed by forcibly connecting adjacent frame nodes, the edge weights are calculated based on motion continuity, the model competition edges are constructed by connecting nodes in the same frame but different models, the edge weights are calculated based on the KL divergence between models, the cross-frame similar edges are constructed according to the connection between nodes with non-continuous frames but similar motion patterns, and the edge weights are calculated based on the trajectory shape DTW distance; S208, after obtaining the new frame data, a new node containing three model predicted states is generated, the graph structure is updated, a weighted bipartite graph is constructed based on the updated graph structure, the left side nodes represent the candidate motion models of a specific frame, the right side nodes correspond to the observation data of a specific frame, the edge connection and edge weight of the weighted bipartite graph are generated according to the three edge structures and edge weights in the time sequence graph structure; S210, the motion smoothness constraint is introduced as a penalty term to improve the Hungarian algorithm, the optimal motion model is selected by solving the maximum weight matching, and the next time state of the gesture is predicted based on the gesture point cloud sequence according to the optimal motion model to generate the predicted state.

[0027] It should be noted that the uniform speed model state equation adopts a linear dynamics model, and the state vector contains position and velocity; the acceleration model state equation extends the state vector to increase the acceleration component; the arc motion model state equation adopts a curvature parameterization model, and the angular velocity is introduced as a state variable. The time sequence graph structure is constructed in a sliding window manner, each node contains a motion feature vector, a motion model predicted state and an observation matching degree, and the motion feature vector contains displacement rate, curvature, velocity variance, acceleration modulus and the like. In the construction of the weighted bipartite graph, the left node set is a candidate motion model sequence, which contains all candidate model nodes of each frame, and the right node set is an observation feature sequence, which contains observation data nodes of each frame, including point cloud spatial distribution, kinematic features and signal-to-noise ratio, and the edge weight is a combination of model-observation matching degree and time sequence consistency. In the bipartite graph model, the weight of the time sequence edge is encoded as a model transition penalty term, which limits the Hungarian algorithm to only select model combinations with high time sequence edge weights; the KL divergence of the model competition edge is converted into the inherent weight of the left node of the bipartite graph, which significantly distinguishes different models under similar observations, and the cross-frame similar edge is mapped to the virtual connection on the right side of the bipartite graph, and a cross-frame consistency term is added in the Hungarian algorithm. In the graph structure, the model-observation nodes in the same frame must be connected, and the different model nodes across frames are not connected.

[0028] The motion smoothness constraint is introduced as a penalty term to improve the Hungarian algorithm, if the model types of adjacent frames are different, the total score is penalized, the Top-2 high weight edges of each frame are retained to generate a rough selection graph, the Hungarian algorithm is run on the rough selection graph, the Hungarian algorithm process is executed through row and column reduction, finding independent zero elements, and covering line test, a model sequence is output, the normalized total weight is obtained, and the optimal motion model is selected by matching. A user gesture habit library is established, the high-frequency model selection sequence of a specific user is counted, and the corresponding model is preloaded at the same gesture starting stage.

[0029] Figure 3 A flowchart for optimizing the observation point cloud based on Poisson multi-Bernoulli mixture filtering is shown.

[0030] According to the embodiment of the present application, the probability distribution is managed based on a Poisson multi-Bernoulli mixture filtering framework, the observation weight is dynamically adjusted according to the signal-to-noise ratio and the spatial distribution of the current gesture point cloud sequence, and the observed point cloud is optimized, specifically In S302, a new-born gesture target is described by a Poisson point process, the spatial probability density of the new-born gesture target in the radar field of view is established, the existence probability and the state distribution of the known gesture target are described by a multi-Bernoulli mixture, and a Bernoulli tuple is maintained for each known gesture target, the Bernoulli tuple including the existence probability and the state distribution described by a Gaussian mixture model. In S304, the signal-to-noise ratio of the observed point cloud in the gesture point cloud sequence is calculated, the normalized entropy value of the observed point cloud is calculated to represent the spatial distribution, the observation weight is dynamically optimized in combination with the signal-to-noise ratio and the spatial distribution, and hierarchical data association is performed through the optimized observation weight. In S306, in the association of the new-born gesture target, when the observation weight corresponding to the unmatched observed point cloud is greater than a preset weight threshold and the distance from the known gesture target is greater than a preset distance threshold, a new Bernoulli tuple is generated. In S308, in the association of the known gesture target, a noise matrix of the fusion observation weight is constructed according to the predicted state and the observed point cloud in combination with the observation noise, the matching degree of the predicted state and the observed point cloud is evaluated by a weighted Mahalanobis distance, the n observed points with the smallest Mahalanobis distance from the predicted position are selected as candidate observations, the association probability of the candidate observations is calculated, and the existence probability and the state distribution are updated. In S310, the point cloud is reconstructed according to the Bernoulli tuple to obtain the optimized observed point cloud.

[0031] It should be noted that for all point clouds of the current frame, two core indicators, the signal-to-noise ratio and the spatial distribution density, are calculated. The signal-to-noise ratio is obtained by comparing the difference between the target point amplitude and the background noise level, and the high signal-to-noise ratio observation usually corresponds to the real gesture reflection. The spatial distribution density is the number of point clouds in a unit volume, and the excessively sparse distribution may be caused by sensor missing detection, and the excessively dense area may be subject to multipath interference. The point cloud is divided into three categories: high-confidence observation (high signal-to-noise ratio and spatial continuity), edge observation (moderate signal-to-noise ratio or local sparsity), and suspected clutter (low signal-to-noise ratio or isolated point).

[0032] Poisson process part is dedicated to handle new appearing targets. When high SNR observation points cannot be associated with existing tracks, their weights are automatically boosted to ensure new born gesture targets are not filtered out. Meanwhile, the range of augmented region is limited to avoid over response to far end noise. For existing Bernoulli components, the quality trend of their history matched observations is analyzed. If the SNR of matched observations drops for 3 consecutive frames, the matching threshold of new observations is gradually lowered to increase the tolerance of low quality observations; otherwise, the standard is tightened to strengthen track purity. The spatial distribution characteristics of point cloud directly affect the weights. For sparse observations caused by gesture occlusion, neighborhood compensation algorithm is adopted to take the predicted position of track as the center, and the low weight points within a radius of 15 cm are gradually boosted in weight to compensate for the impact of missed detection. For abnormally dense clusters, competitive weight decay is implemented, and only the core points with the highest spatial consistency are retained. The adjusted observation weights directly affect two key probabilities of PMBM: detection probability and clutter density. High weight observations correspond to upward correction of detection probability, making them more likely to be associated with existing tracks; at the same time, the spatial clutter density parameter is dynamically adjusted to appropriately increase the expected number of clutters in low weight observation areas. This two-way adjustment enables the filter to maintain high sensitivity while effectively suppressing false alarms.

[0033] It should be noted that the high confidence target point cloud is extracted from the update step of the PMBM filter as the optimized observation point cloud, the Mahalanobis distance is used to calculate the single point matching degree with the predicted state, for a track segment of a preset length, a matching degree matrix is calculated according to the single point matching degree; dynamic time warping is used for optimal association, a warping path is defined according to the observation point cloud and the predicted state, and a cost matrix is constructed combining the matching degree matrix and kinematic constraints is represented as:

[0034] wherein is a distance weight coefficient, is a normalized Mahalanobis distance, is a predicted velocity vector of the i-th track, is an equivalent velocity vector of the j-th observation point cloud, and the spatial distance term reflects the spatial consistency of the track predicted position and the observation point cloud, and the velocity difference term punishes the matching of inconsistent motion directions.

[0035] The dynamic programming is used to solve the minimum cumulative cost, and the backtracking obtains a preset number of optimal association pair sets to generate a preset number of hypothesis tracks. A dynamic track hypothesis library is constructed and maintained through a multi-hypothesis tracking strategy, each hypothesis node contains a track state, an existence probability and a history observation matching sequence. The current track is successfully matched with the observation, the hypothesis is continued, the new track is generated for the unmatched observation, and the hypothesis is eliminated for 3 consecutive frames without matching. The matching score is a smoothing score and continuous scoring Construct a multi-dimensional scoring function, where is the total number of frames in the current evaluation window, is the timeframe index, For the t The squared weighted Mahalanobis distance of the frame characterizes the deviation between the observation and the model prediction, Respectively t Frame, t-1 The acceleration vector of the frame, is the number of survival frames of the current trajectory, is the preset maximum lifetime reference value. The exponential term of the matching score maps the Mahalanobis distance to a matching confidence level in the [0,1] interval. The smoothness score compresses acceleration changes in the infinite domain to the [0,1] interval using the hyperbolic tangent function, penalizing drastic speed changes and encouraging motion smoothness. The persistence score quantifies the trajectory's sustained tracking stability, preventing transient noise from being mistaken for a valid target. Newly initialized trajectories are given a low weight, while stable trajectories are given a high weight. The multi-dimensional scoring function is used to score the hypothesized trajectories. The highest-scoring hypothesized trajectory is selected as the optimal trajectory, which is then smoothed using Kalman processing. The process noise is adjusted based on the dynamic characteristics of the gesture target, and the gesture dynamic trajectory is output.

[0036] The second aspect of the present invention provides a real-time dynamic trajectory tracking system for millimeter wave radar gesture recognition. Figure 4 The figure shows the architecture of the real-time dynamic trajectory tracking system for millimeter-wave radar gesture recognition.

[0037] The second embodiment of the present invention provides a real-time dynamic trajectory tracking system 4 for millimeter wave radar gesture recognition, which includes: a signal acquisition and preprocessing module 401, a point cloud optimization and clustering module 402, a motion model matching module 403, a multimodal tracking module 404, and a trajectory smoothing module 405; The signal acquisition and preprocessing module transmits millimeter wave signals through a frequency-modulated continuous wave millimeter wave radar and receives echo signals reflected by gestures, and preprocesses the echo signals; The point cloud optimization and clustering module uses constant false alarm rate detection to filter noise, extract potential target point clouds, cluster the potential target point clouds, and generate gesture point cloud sequences; The motion model matching module establishes a multimodal motion model library including a uniform speed model, an acceleration model, and an arc motion model, dynamically selects the optimal motion model using graph matching, and predicts the next state of the gesture in combination with the gesture point cloud sequence to generate a predicted state; The multi-modal tracking module is based on a Poisson multi-Bernoulli mixture filtering framework, dynamically adjusts the observation weight according to the signal-to-noise ratio and spatial distribution of the current gesture point cloud sequence, optimizes the observation point cloud, calculates the matching degree of the observation point cloud and the predicted state by using the Mahalanobis distance, and performs optimal association in combination with dynamic time warping; a multi-hypothesis tracking strategy is used to maintain the track hypothesis, and the optimal track is selected through a track scoring mechanism; The trajectory smoothing module performs Kalman filtering smoothing processing on the optimal trajectory, and outputs a gesture dynamic trajectory.

[0038] The third embodiment of the present application provides a computer readable storage medium, the computer readable storage medium comprises a real-time dynamic trajectory tracking method program of millimeter wave radar gesture recognition, and the real-time dynamic trajectory tracking method program of millimeter wave radar gesture recognition is executed by a processor to realize the steps of the real-time dynamic trajectory tracking method of millimeter wave radar gesture recognition.

[0039] In several embodiments provided in the present application, it should be understood that the disclosed method and system can be implemented in other ways. The system embodiments described above are only illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division mode, such as: a plurality of units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed components can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms. In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or hardware plus software functional unit.

[0040] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, the above-mentioned program can be stored in a computer readable storage medium, and the program is executed to execute the steps including the above-mentioned method embodiments; and the above-mentioned storage medium includes: mobile storage device, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disc or optical disc and various storage program codes.

[0041] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A real-time dynamic trajectory tracking method for millimeter wave radar gesture recognition, characterized in that: The following steps are involved: The millimeter wave signal is transmitted by a frequency modulated continuous wave millimeter wave radar and an echo signal reflected by the gesture is received. The echo signal is preprocessed, and a constant false alarm rate detection is used to filter out noise, and a potential target point cloud is extracted. The potential target point cloud is clustered to generate a gesture point cloud sequence; Establish a multimodal motion model library including a uniform velocity model, an acceleration model, and an arc motion model, dynamically select the optimal motion model using graph matching, and predict the next state of the gesture in combination with the gesture point cloud sequence to generate a predicted state; Based on the Poisson-Multi-Bernoulli hybrid filtering framework, the observation weight is dynamically adjusted according to the signal-to-noise ratio and spatial distribution of the current gesture point cloud sequence, the observation point cloud is optimized, the Mahalanobis distance is used to calculate the matching degree between the observation point cloud and the predicted state, and the optimal association is achieved by combining dynamic time warping; A multi-hypothesis tracking strategy is adopted to maintain trajectory hypotheses, and the optimal trajectory is selected through a trajectory scoring mechanism. The optimal trajectory is smoothed by Kalman filtering and the gesture dynamic trajectory is output.

2. The real-time dynamic trajectory tracking method for millimeter wave radar gesture recognition according to claim 1, characterized in that: The echo signal is preprocessed, constant false alarm rate detection is used to filter out noise, and potential target point clouds are extracted, specifically: Mixing the received echo signal with the current transmitted signal to obtain an intermediate frequency signal, performing analog-to-digital conversion and sampling on the intermediate frequency signal to obtain a discrete time series, organizing the discrete time series based on the radar frame structure of the frequency modulated continuous wave millimeter wave radar, and generating a distance dimension data matrix and a velocity dimension data matrix; The distance-dimensional data matrix is ​​processed using a fast Fourier transform to convert the time domain signal into the frequency domain, and the gesture target distance is obtained according to the frequency corresponding to the frequency domain peak. The velocity-dimensional data matrix is ​​subjected to a second-dimensional fast Fourier transform to extract the Doppler frequency shift of the gesture target and obtain the gesture target velocity. forming a distance-speed matrix based on the gesture target distance and gesture target speed, and using ordered statistical constant false alarm rate detection in the distance-speed matrix to calculate the local noise level through a sliding window to distinguish the real gesture target from the noise; A dynamic threshold is set, units larger than the dynamic threshold are retained, warnings are suppressed, binary detection results are output, potential target point clouds are marked, the potential target point clouds are mapped to three-dimensional space, and data of different dimensions are normalized.

3. The real-time dynamic trajectory tracking method for millimeter wave radar gesture recognition according to claim 2, characterized in that: The potential target point cloud is clustered to generate a gesture point cloud sequence, specifically: A neighborhood radius is defined based on the physical size of the gesture and the preset radar resolution. All potential target point clouds are traversed. If the number of points within the neighborhood radius of a potential target point cloud exceeds the preset minimum number of point clouds, the point cloud is marked as a core point cloud and used to represent the main part of the gesture. Starting from the core point cloud, all reachable potential target point clouds in its neighborhood are recursively merged to form independent clusters, and the boundary points are classified into the nearest independent cluster. Isolated point clouds that are not classified into any independent cluster are marked as noise and removed. The motion consistency test is performed on the independent clusters generated by clustering, the center of mass coordinates and bounding box size of each independent cluster are calculated, and the velocity distribution of the point cloud within the cluster is extracted. Cluster features are generated to obtain gesture labels, and a gesture point cloud sequence with gesture labels is output.

4. The real-time dynamic trajectory tracking method for millimeter wave radar gesture recognition according to claim 1, characterized in that: A multimodal motion model library including uniform velocity model, acceleration model and arc motion model is established, and the optimal motion model is dynamically selected using graph matching. Specifically: Construct a uniform speed model for smooth linear motion, an acceleration model for variable speed motion, and an arc motion model for rotation or arc trajectories. Build a multimodal motion model library based on the uniform speed model, acceleration model, and arc motion model. A time series graph structure is constructed using a sliding window approach. Each node contains a motion feature vector, a motion model prediction state, and an observation matching degree. The observation matching degree uses the Mahalanobis distance score between the motion model prediction trajectory and the measured point cloud. The temporal graph structure includes temporal edges, model competition edges, and cross-frame similarity edges. Temporal edges are constructed by forcibly connecting nodes in adjacent frames. Edge weights are calculated based on motion continuity. Model competition edges are constructed by connecting nodes of different models in the same frame. Edge weights are calculated based on KL divergence between models. Cross-frame similarity edges are constructed by connecting nodes in non-continuous frames but with similar motion patterns. Edge weights are calculated based on DTW distance of trajectory morphology. After obtaining the new frame data, a new node containing the three model prediction states is generated, and the graph structure is updated. Based on the updated graph structure, a weighted bipartite graph is constructed. The left node represents the candidate motion model of a specific frame, and the right node corresponds to the observation data of a specific frame. The edge connections and edge weights of the weighted bipartite graph are generated based on the three edge structures and edge weights in the time series graph structure. The Hungarian algorithm is improved by introducing motion smoothness constraint as a penalty term. The maximum weight matching is solved to select the optimal motion model. Based on the optimal motion model, the next state of the gesture is predicted according to the gesture point cloud sequence to generate a predicted state.

5. The real-time dynamic trajectory tracking method for millimeter wave radar gesture recognition according to claim 4, characterized in that: Based on the Poisson-multi-Bernoulli hybrid filter framework to manage probability distribution, the observation weight is dynamically adjusted according to the signal-to-noise ratio and spatial distribution of the current gesture point cloud sequence to optimize the observation point cloud. Specifically: New gesture targets are described using a Poisson point process, and the spatial probability density of new gesture targets is established within the radar field of view. Multi-Bernoulli mixtures are used to characterize the existence probability and state distribution of known gesture targets. Each known gesture target maintains a Bernoulli tuple, which includes the existence probability and the state distribution described by the Gaussian mixture model. Calculate the signal-to-noise ratio of the observation point cloud in the gesture point cloud sequence, and calculate the normalized entropy value of the observation point cloud to represent the spatial distribution. Combine the signal-to-noise ratio and spatial distribution to dynamically optimize the observation weights. Perform hierarchical data association based on the optimized observation weights. In the new gesture target association, when the observation weight corresponding to the unmatched observation point cloud is greater than the preset weight threshold and the distance to the known gesture target is greater than the preset distance threshold, a new Bernoulli tuple is generated; In the known gesture target association, a noise matrix of fused observation weights is constructed based on the predicted state and the observed point cloud combined with the observation noise. The matching degree between the predicted state and the observed point cloud is evaluated by weighted Mahalanobis distance. The n observed point clouds with the smallest Mahalanobis distance to the predicted position are selected as candidate observations. The association probability of the candidate observations is calculated, and the existence probability and state distribution are updated. The point cloud is reconstructed according to the Bernoulli tuple to obtain the optimized observation point cloud.

6. The real-time dynamic trajectory tracking method for millimeter wave radar gesture recognition according to claim 5, characterized in that: The Mahalanobis distance is used to calculate the matching degree between the observed point cloud and the predicted state, and the optimal association is performed in combination with dynamic time warping. Specifically: For the optimized observation point cloud, the Mahalanobis distance is used to calculate the single-point matching degree with the predicted state. For trajectory segments of preset length, the matching degree matrix is ​​calculated based on the single-point matching degree. Dynamic time warping is used for optimal association. A regularized path is defined based on the observed point cloud and the predicted state. A cost matrix is ​​constructed by combining the matching matrix and kinematic constraints. Dynamic programming is used to solve the minimum cumulative cost. A preset number of optimal association pair sets are obtained by backtracking, and a preset number of hypothetical trajectories are generated.

7. The real-time dynamic trajectory tracking method for millimeter wave radar gesture recognition according to claim 1, characterized in that: A multi-hypothesis tracking strategy is used to maintain trajectory hypotheses, and the optimal trajectory is selected through a trajectory scoring mechanism. The optimal trajectory is smoothed by Kalman filtering and the gesture dynamic trajectory is output as follows: A dynamic trajectory hypothesis library is built and maintained through a multi-hypothesis tracking strategy. Each hypothesis node contains the trajectory state, existence probability, and historical observation matching sequence. A multi-dimensional scoring function is constructed using matching score, smoothing score, and persistence score, and the hypothesis trajectory is scored using the multi-dimensional scoring function. The hypothesis trajectory with the highest score is selected as the optimal trajectory, Kalman smoothing is applied to the optimal trajectory, and process noise is adjusted according to the dynamic characteristics of the gesture target to output the gesture dynamic trajectory.

8. A real-time dynamic trajectory tracking system for millimeter wave radar gesture recognition, characterized in that: A real-time dynamic trajectory tracking method for implementing millimeter-wave radar gesture recognition according to any one of claims 1 to 7, the system comprising a signal acquisition and preprocessing module, a point cloud optimization and clustering module, a motion model matching module, a multimodal tracking module, and a trajectory smoothing module; The signal acquisition and preprocessing module transmits millimeter wave signals through a frequency-modulated continuous wave millimeter wave radar and receives echo signals reflected by gestures, and preprocesses the echo signals; The point cloud optimization and clustering module uses constant false alarm rate detection to filter noise, extract potential target point clouds, cluster the potential target point clouds, and generate gesture point cloud sequences; The motion model matching module establishes a multimodal motion model library including a uniform speed model, an acceleration model, and an arc motion model, dynamically selects the optimal motion model using graph matching, and predicts the next state of the gesture in combination with the gesture point cloud sequence to generate a predicted state; The multimodal tracking module is based on a Poisson-Multi-Bernoulli hybrid filtering framework. It dynamically adjusts observation weights and optimizes the observed point cloud according to the signal-to-noise ratio and spatial distribution of the current gesture point cloud sequence. It uses Mahalanobis distance to calculate the matching degree between the observed point cloud and the predicted state, and combines dynamic time warping for optimal association. It uses a multi-hypothesis tracking strategy to maintain trajectory hypotheses and selects the optimal trajectory through a trajectory scoring mechanism. The trajectory smoothing module performs Kalman filtering smoothing on the optimal trajectory and outputs a gesture dynamic trajectory.

Images