A gesture recognition method based on sparse millimeter wave radar point cloud

Abstract

The application discloses a gesture recognition method based on sparse millimeter wave radar point cloud, and relates to the field of gesture recognition, and comprises the following steps: acquiring millimeter wave radar data of various gesture actions, and constructing a sparse space-time point cloud data set; constructing a space-time graph structure data comprising a space graph and a time graph; performing gesture recognition model training, combining a composite loss function and a hierarchical learning rate strategy to optimize model parameters; the gesture recognition model comprises a graph feature coding module, a space-time feature fusion module and a time series modeling and classification module connected in sequence; millimeter wave radar data of a gesture to be recognized is processed into space-time graph structure data, input into the trained gesture recognition model, and a corresponding gesture category recognition result is output. Through the space-time graph neural network modeling method and the training strategy specially designed for the sparse point cloud, the precision and robustness problem caused by the space-time sparsity of the point cloud in the rapid gesture recognition of the millimeter wave radar is effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of gesture recognition, and more particularly to a gesture recognition method based on sparse millimeter wave radar point cloud. BACKGROUND

[0002] Non-contact gesture recognition plays an increasingly important role in human-computer interaction, especially in scenarios such as intelligent car cabin, AR / VR interaction, and smart home control, where users expect to communicate with devices through more natural and intuitive body movements.

[0003] However, existing gesture recognition schemes based on millimeter wave radar mostly rely on high-channel, high-cost industrial-grade radar platforms; such platforms can generate dense and stable point cloud data, but in consumer-level or embedded application scenarios where cost and volume are strictly limited, low-cost radar chips with very few channels are generally used; such hardware has inherent defects such as low resolution, limited frame rate, and inability to detect pitch angles, and the original point cloud generated by such hardware is extremely sparse in spatial dimension.

[0004] The more serious challenge is that when the user performs fast and instantaneous dynamic gestures, the hardware defects are dramatically magnified; high-speed motion causes the number of effective reflection points in a single frame to decrease sharply, and the point cloud distribution is scattered and unbearable; at the same time, the limited frame rate cannot capture continuous motion trajectories, causing target loss and association break between frames, resulting in a large number of empty or invalid frames. Existing methods usually use simple interpolation, fitting or direct rejection of abnormal frames to handle such problems, but these methods rely heavily on the continuity and regularity of point cloud distribution, and have little effect in extremely sparse and broken scenarios, and even introduce noise and errors.

[0005] From the perspective of algorithm model, whether it is traditional machine learning or deep learning model based on dense point cloud, the design premise is to process structured and continuous data. It is difficult to effectively learn to extract discriminative features from extremely sparse and unstructured point cloud sequences, and direct application of these models will result in information loss, feature confusion, and ultimately a significant decrease in classification accuracy and insufficient model robustness. In addition, these models usually have high computational complexity and large parameter quantity, making it difficult to deploy and infer in real time on embedded end-side devices with limited computing power.

[0006] Therefore, how to design a gesture recognition method based on sparse millimeter wave radar point cloud, which can adapt to the hardware limitations of low-cost millimeter wave radar and effectively deal with the extremely spatio-temporal sparse point cloud data generated when interacting with fast gestures of users, to realize high-precision, high-robustness real-time gesture recognition is a problem that needs to be solved by those skilled in the art. SUMMARY

[0007] Therefore, the present application provides a gesture recognition method based on sparse millimeter wave radar point cloud, aiming to solve the problem of spatio-temporal sparsity of point cloud data caused by the inherent defects of hardware and the rapid gesture action of users, and finally realize high-accuracy and high-real-time gesture recognition on resource-constrained embedded devices to meet the needs of practical application scenarios such as intelligent cockpit and smart home.

[0008] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0009] A gesture recognition method based on sparse millimeter wave radar point cloud, comprising the following steps:

[0010] S1, acquiring millimeter wave radar data of multiple gesture actions, and constructing a sparse spatio-temporal point cloud dataset;

[0011] S2, based on the point cloud sequence in the dataset, constructing spatio-temporal graph structure data including space graph and time graph;

[0012] S3, using the spatio-temporal graph structure data to train a gesture recognition model, combining a compound loss function and a hierarchical learning rate strategy to optimize model parameters; the gesture recognition model includes a graph feature encoding module, a spatio-temporal feature fusion module, and a time series modeling and classification module connected in sequence;

[0013] S4, processing the millimeter wave radar data of the gesture to be recognized into spatio-temporal graph structure data, inputting the trained gesture recognition model, and outputting the corresponding gesture category recognition result.

[0014] Preferably, the S1 comprises:

[0015] Using a MIMO millimeter wave radar system to collect original data of multiple categories of preset gesture actions, and collecting a preset number of sample data for each gesture category, and constructing the sparse spatio-temporal point cloud dataset;

[0016] Wherein, each sample data is a point cloud sequence including multiple frames of point clouds, and each radar point cloud data point in each frame of point cloud contains its spatial coordinates, distance information, velocity information and reflection intensity information.

[0017] Preferably, in the S2, the space graph construction comprises: for the point cloud in each time frame, calculating the two-dimensional Euclidean distance according to the spatial coordinates of the points, and using the K nearest neighbor algorithm to establish a spatial connection edge for each point with K nearest neighbor points, and the feature of the spatial edge is the coordinate difference vector of the two connected points;

[0018] The time graph construction comprises: for the point clouds of adjacent time frames, calculating the spatial distance between all point pairs, and establishing a time connection edge for the point pairs with a distance lower than a preset threshold τ, and the feature of the time edge is the coordinate difference vector of the two connected points in the adjacent frames.

[0019] Preferably, the graph feature encoding module is used for separate encoding and fusion of multi-modal features, and outputs high-dimensional unified node features; comprising:

[0020] A feature splitting unit divides the multi-modal features of the input node into spatial position features [x, y] and radar parameter features [distance, Doppler velocity, echo intensity];

[0021] A first feature encoding unit internally built-in first multi-layer perception is used for encoding the spatial position features and mapping to a 128-dimensional feature space;

[0022] A second feature encoding unit internally built-in second multi-layer perception is used for encoding the radar parameter features and mapping to a 128-dimensional feature space;

[0023] A feature fusion unit concatenates the two 128-dimensional feature vectors output by the first and second feature encoding units in the channel dimension to obtain a 256-dimensional fused node feature.

[0024] Preferably, the spatio-temporal feature fusion module is used for aggregating spatial neighborhood information and fusing temporal connections of the fused node features output by the graph feature encoding module, and outputs spatio-temporally enhanced graph-level features; comprising:

[0025] A graph isomorphism network unit divides the input 256-dimensional node features into two parts in the channel dimension, the former 128-dimensional features are input into a graph isomorphism network convolutional layer for neighborhood information aggregation, the latter 128-dimensional features are directly jumped, finally the processed features and the jumped features are concatenated and linearly transformed and normalized to output 256-dimensional node-level features;

[0026] A graph attention network unit receives the 256-dimensional node-level features output by the graph isomorphism network unit, divides them into two parts, the former 128-dimensional features are input into a 4-head graph attention network convolutional layer for neighborhood information aggregation, the latter 128-dimensional features are directly jumped, finally the processed features and the jumped features are concatenated and linearly transformed and normalized to output 256-dimensional node-level features enhanced by space and time;

[0027] A graph pooling unit receives the 256-dimensional node-level features enhanced by space and time, respectively performs global average pooling and global maximum pooling on the graph data of each time frame, concatenates the obtained two 256-dimensional vectors to output a 512-dimensional graph-level feature.

[0028] Preferably, the time series modeling and classification module is used for modeling the time series dependency relationship of the graph-level features output by the spatio-temporal feature fusion module and completing classification, and outputs gesture class probability; comprising:

[0029] The dual-path time sequence modeling unit receives a time sequence composed of T 512-dimensional graph-level features, and performs time sequence modeling through an LSTM path and a TCN path respectively;

[0030] The feature fusion and attention unit splices the output features of the dual paths, and then inputs a multi-head self-attention mechanism to enhance the information of key time steps.

[0031] The classifier maps the sequence features output by the self-attention mechanism to the same dimension as the number of gesture categories through a multi-layer perceptron classifier, and outputs the probability of each category through a Softmax function.

[0032] Preferably, the LSTM path adopts a bidirectional LSTM network, and the hidden layer dimension is set to 512; the TCN path is composed of three time sequence convolution blocks cascaded, and the expansion rates thereof are set to 1, 2 and 4 respectively; and the number of heads of the multi-head self-attention mechanism is set to 16 heads.

[0033] Preferably, the classifier is composed of three linear layers, each of which is followed by a LeakyReLU activation function, a Dropout layer and a batch normalization layer.

[0034] Preferably, in the S3, the composite loss function L total is expressed as:

[0035]

[0036] wherein, L focal-smooth represents a label smoothing focal loss, L separation represents an inter-class separation loss, and λ is a weight coefficient.

[0037] Preferably, in the S3, the hierarchical learning rate strategy comprises:

[0038] The first learning rate δ1=1×10 -3 is set for the parameters in the graph feature encoding module and the spatio-temporal feature fusion module.

[0039] The second learning rate δ2=5×10 -4 is set for the parameters in the time sequence modeling and classification module.

[0040] The AdamW optimizer is used, the weight decay coefficient is set to 1×10 -5 , and the global gradient clipping is adopted, and the threshold is set to 1.0.

[0041] According to the technical solution, compared with the prior art, the technical solution has the following beneficial effects:

[0042] 1. The method converts sparse and unstructured point cloud sequences into regular graph data by constructing a space-time graph structure, and extracts the spatio-temporal features of gestures using a feature encoding and fusion module designed specifically for graph data. It can directly process extremely sparse data with few points in each frame and empty frames between frames, overcoming the dependence of traditional methods on high-quality and dense point cloud data and improving the accuracy of gesture recognition on low-cost hardware platforms.

[0043] 2. The gesture recognition model is designed modularly, with feature encoding, spatio-temporal fusion, and temporal classification performed sequentially, resulting in a clear and efficient structure. The combination of CSP-style design, hierarchical pooling, dual-path temporal modeling, and attention mechanisms reduces the demand for computational resources and parameter quantity. The use of hierarchical learning rates and gradient clipping optimizes the model training, making it stable and efficient in converging.

[0044] 3. Training with a compound loss function that combines label-smoothed focal loss and inter-class separation loss helps alleviate overfitting caused by limited training samples, improving model generalization. By explicitly constraining the distribution of different classes in the feature space and increasing the inter-class distance, the model's ability to distinguish similar gestures is enhanced, resulting in high-precision gesture classification in high-sparse and small-sample scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0045] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings without creative labor based on the provided drawings.

[0046] Figure 1 A gesture recognition method based on sparse millimeter wave radar point cloud is provided for the embodiments of the present application.

[0047] Figure 2 A gesture recognition model structure diagram is provided for the embodiments of the present application.

[0048] Figure 3 A graph feature encoding module structure diagram is provided for the embodiments of the present application.

[0049] Figure 4 A spatio-temporal feature fusion module structure diagram is provided for the embodiments of the present application.

[0050] Figure 5 A temporal modeling and classification module structure diagram is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0051] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0052] As shown in Figure 1 The present embodiment provides a gesture recognition method based on sparse millimeter wave radar point cloud, comprising the following steps:

[0053] S1, acquiring millimeter wave radar data of multiple gesture actions, and constructing a sparse space-time point cloud data set;

[0054] S2, based on the point cloud sequence in the data set, constructing space-time graph structure data including space graph and time graph;

[0055] S3, using the space-time graph structure data to train a gesture recognition model, combining a compound loss function and a hierarchical learning rate strategy to optimize model parameters; the gesture recognition model includes a graph feature encoding module, a space-time feature fusion module, and a time series modeling and classification module connected in turn;

[0056] S4, processing the millimeter wave radar data of the gesture to be recognized into space-time graph structure data, inputting the trained gesture recognition model, and outputting the corresponding gesture category recognition result.

[0057] This method effectively solves the recognition problem of point cloud space-time sparsity caused by low-cost millimeter wave radar hardware limitations and fast gestures, through constructing a space-time graph structure, using a special graph neural network model of multi-modal feature encoding and fusion, and combining a compound loss function and a hierarchical optimization strategy, finally realizing high-precision, high-robustness real-time gesture recognition on an embedded platform, significantly improving the application feasibility in intelligent cockpit, AR / VR and other harsh scenarios.

[0058] The following will further detail each step in the above method;

[0059] In the present embodiment S1, millimeter wave radar data of multiple gesture actions is acquired, and a sparse space-time point cloud data set is constructed; comprising: using a MIMO millimeter wave radar system to collect original data of multiple categories of preset gesture actions, and collecting a preset number of sample data for each gesture category, and constructing the sparse space-time point cloud data set; wherein each sample data is a point cloud sequence including multiple frames of point clouds, and each radar point cloud data point in each frame of point cloud contains its spatial coordinates, distance information, velocity information and reflection intensity information;

[0060] Specifically, data is collected by low-cost MIMO millimeter wave radar TIAWR1642, and 6 common gesture categories (such as waving, sliding, pinching, etc.) are collected, about 1500 sample data for each category, thereby constructing a sparse space-time point cloud data set of about 9000 samples in total; each sample is a point cloud sequence, the sequence length is not more than 10 frames, and the average number of points per frame is not more than 8, and each point contains 6-dimensional physical information including x, y, z, distance, Doppler velocity and reflection intensity, which truly reflects the sparse characteristics under fast gestures.

[0061] In this embodiment S2, based on the point cloud sequence in the data set, a space-time graph structure data including a space graph and a time graph is constructed; wherein the space graph construction includes: for the point cloud in each time frame, calculating the two-dimensional Euclidean distance according to the spatial coordinates of the points, and using the K nearest neighbor algorithm to establish a spatial connection edge between each point and K nearest neighbor points, and the feature of the spatial edge is the coordinate difference vector of the two connected points;

[0062] The time graph construction includes: for the point clouds of adjacent time frames, calculating the spatial distance between all point pairs, and establishing a time connection edge for the point pairs with a distance lower than a preset threshold τ, and the feature of the time edge is the coordinate difference vector of the two connected points in adjacent frames.

[0063] In this step, the space graph part calculates the Euclidean distance based on the x and y two-dimensional coordinates for each frame of point cloud, and the K value in the K nearest neighbor algorithm is adaptively selected according to the point cloud density, and is usually 3-5, so as to establish a local spatial structure; the feature of the spatial edge is defined as the coordinate difference vector of Δx and Δy, and the relative position relationship is retained. The time graph construction calculates the two-dimensional distance between all point pairs for adjacent frames, sets a distance threshold τ (usually 0.2-0.5 meters) to filter noise and non-associated points, and only establishes a time edge for the point pairs lower than the threshold, and the feature is the coordinate difference vector (Δx, Δy) between the two frames, which is used to describe the inter-frame motion trajectory of the target.

[0064] In this embodiment S3, the space-time graph structure data is used for gesture recognition model training, and a composite loss function and a hierarchical learning rate strategy are combined to optimize the model parameters; as shown in Figure 2 The gesture recognition model includes a graph feature coding module, a space-time feature fusion module, and a time series modeling and classification module connected in sequence;

[0065] As shown in Figure 3 The graph feature coding module is used for separate coding and fusion of multi-modal features, and outputs high-dimensional unified node features; including:

[0066] The feature splitting unit divides the multimodal features of the input node into spatial location features [x, y] and radar parameter features [range, Doppler velocity, echo intensity]. The first feature encoding unit, which incorporates a first multilayer perceptron, encodes the spatial location features and maps them to a 128-dimensional feature space. The second feature encoding unit, which incorporates a second multilayer perceptron, encodes the radar parameter features and maps them to a 128-dimensional feature space. The feature fusion unit concatenates the two 128-dimensional feature vectors output by the first and second feature encoding units along the channel dimension to obtain a 256-dimensional fused node feature.

[0067] In the graph feature encoding module, the feature splitting unit clearly divides the 5-dimensional input features of the nodes (x, y, distance, Doppler velocity, echo intensity) into spatial location features (x, y) and radar parameter features (the latter three dimensions). The first and second feature encoding units both adopt a two-layer MLP structure, each layer containing linear transformation, LeakyReLU activation and batch normalization operations, and finally outputting 128-dimensional features respectively. The feature fusion unit merges the two 128-dimensional vectors into a 256-dimensional fused feature by concatenation, providing a unified high-dimensional node representation for subsequent graph convolution processing.

[0068] like Figure 4 As shown, the spatiotemporal feature fusion module is used to aggregate spatial neighborhood information and fuse temporal connections of the fusion node features output by the graph feature encoding module, outputting spatiotemporally enhanced graph-level features; including:

[0069] The graph isomorphic network unit divides the input 256-dimensional node features into two parts in the channel dimension. The first 128-dimensional features are input into a graph isomorphic network convolutional layer for neighborhood information aggregation, and the latter 128-dimensional features are directly skipped. Finally, the processed features are concatenated with the skipped features and subjected to linear transformation and normalization to output 256-dimensional node-level features.

[0070] The graph attention network unit receives 256-dimensional node-level features output from the graph isomorphic network unit, divides them into two parts, inputs the first 128-dimensional features into a 4-head graph attention network convolutional layer for neighborhood information aggregation, and directly skips the connection of the latter 128-dimensional features. Finally, the processed features are concatenated with the skip-connected features and subjected to linear transformation and normalization to output 256-dimensional node-level features with spatiotemporal enhancement.

[0071] The graph pooling unit receives 256-dimensional node-level features that have undergone spatiotemporal enhancement. It performs global average pooling and global max pooling on the graph data of each time frame, concatenates the two 256-dimensional vectors, and outputs a 512-dimensional graph-level feature.

[0072] In the space-time feature fusion module, the graph isomorphism network unit aggregates the neighborhood of the first 128-dimensional space features, and the GIN convolution layer is implemented using a multi-layer perceptron, including two linear layers and a ReLU activation. The last 128-dimensional features are directly connected to avoid gradient vanishing. The graph attention network unit further divides the GIN output features, and the first 128-dimensional input is connected to four GAT convolution layers, which aggregate the neighbor information through attention weights. The last 128-dimensional features are connected directly. The outputs of the two units are fused and stabilized by linear layers, LeakyReLU and batch normalization. The graph pooling unit performs global average pooling and global maximum pooling on the node-level features of each time frame, respectively, and concatenates the obtained two 256-dimensional vectors into 512-dimensional graph-level features to retain the global statistical information of each frame of graph.

[0073] As shown in Figure 5 The time series modeling and classification module is used for modeling the time series dependence of the graph-level features output by the space-time feature fusion module and completing classification, and outputs gesture class probabilities. It includes:

[0074] The dual-path time series modeling unit receives a time series composed of T 512-dimensional graph-level features, and performs time series modeling through LSTM and TCN paths, respectively. The feature fusion and attention unit concatenates the output features of the dual-path, and then inputs a multi-head self-attention mechanism to enhance the information of key time steps. The classifier maps the sequence features output by the self-attention mechanism to the same dimension as the number of gesture classes through a multi-layer perceptron classifier, and outputs the probability of each class through the Softmax function.

[0075] Further, the LSTM path adopts a bidirectional LSTM network with a hidden layer dimension of 512. The TCN path is composed of three time series convolution blocks with expansion rates of 1, 2 and 4, respectively. The number of heads of the multi-head self-attention mechanism is set to 16. The classifier is composed of three linear layers, each followed by a LeakyReLU activation function, a Dropout layer and a batch normalization layer. Specifically, the first layer maps the input dimension to 512, the second layer maps to 256, and the third layer outputs to the number of classes. After each linear transformation, a LeakyReLU activation, a Dropout layer and a batch normalization layer are sequentially connected to enhance the non-linear expression ability, prevent overfitting and stabilize the training process.

[0076] In the temporal modeling and classification module, the dual-path temporal modeling unit receives a 512-dimensional feature sequence of T time steps: the LSTM path uses a bidirectional two-layer LSTM with a hidden layer dimension of 512 to fully learn the temporal dependencies between time steps; the TCN path consists of three cascaded temporal convolutional blocks with dilation rates of 1, 2, and 4, respectively, to gradually expand the receptive field to capture multi-scale temporal patterns; the feature fusion and attention unit concatenates the output features of the two paths and inputs them into a 16-head self-attention mechanism to calculate the dependency weights between time steps, highlighting the role of keyframes; the final classifier uses a three-layer MLP to map the features of the last frame of the sequence after attention to the dimension of the number of classes, and outputs a probability distribution through Softmax.

[0077] Furthermore, the composite loss function L total Represented as:

[0078]

[0079] The composite loss function L at this point total In the middle, L focal-smooth To smooth the focus loss of the label: label smoothing parameter ε = 0.1, focus loss adjustment factor γ = 2.0, balance factor α = 0.25; L separation The inter-class separation loss is calculated based on the cosine similarity of the average predicted probability vectors of each class within the batch, encouraging orthogonality of inter-class features; the weight coefficient λ is set to 0.1 to balance the contributions of the two losses, and the total loss guides the model to take into account both hard sample learning and class discriminability.

[0080] Furthermore, tiered learning rate strategies include:

[0081] The first learning rate δ1 = 1 × 10 is set for the parameters in the graph feature encoding module and the spatiotemporal feature fusion module. -3 Set the second learning rate δ2 = 5 × 10 for the parameters in the time series modeling and classification module. -4 Using the AdamW optimizer, the weight decay factor is set to 1×10. -5 Global gradient pruning is employed, with a threshold set to 1.0.

[0082] In the hierarchical learning rate strategy, higher learning rates are set for the parameters in the graph feature encoding module and the spatiotemporal fusion module to promote rapid convergence of spatial features; lower learning rates are set for the parameters in the temporal modeling and classification modules to ensure fine learning of temporal relationships; the optimizer uses AdamW with a weight decay of 1×10⁻⁶. -5 To achieve decoupling of weight decay, global gradient pruning is used during training to prevent gradient explosion and improve training stability.

[0083] In this embodiment S4, the millimeter wave radar data of the gesture to be recognized is processed into spatio-temporal graph structure data, input into the trained gesture recognition model, and the corresponding gesture category recognition result is output;

[0084] Specifically, it is applied to the interaction scene of the intelligent automobile cabin. When the driver or passenger makes a pre-defined gesture such as a quick hand wave, slide or tap in the radar sensing area in the cabin, the vehicle-mounted low-cost millimeter wave radar will collect the original radar signals generated by the gesture in real time. First, these original signals are processed into a multi-frame sparse point cloud sequence, and then a spatio-temporal graph structure is constructed according to the rules defined in S2. This graph structure accurately depicts the spatial distribution and motion trajectory of the gesture in a short time, forming a standardized input that the model can understand.

[0085] The spatio-temporal graph data is then input into the gesture recognition model that has been fully trained and deployed on the embedded platform of the vehicle-mounted infotainment system. The model performs forward inference on the input data, and the graph encoding, spatio-temporal fusion and time series modeling modules within the model work together to finally output a probability distribution vector representing the probabilities of each category from the classifier. The category with the highest probability (for example, the probability value corresponding to “wave right hand” is 0.92) is finally selected as the final result of this recognition.

[0086] The recognition result is not just a simple text label. The result will be transmitted as a clear control instruction through the vehicle-mounted system's high-speed bus CAN or Ethernet to the corresponding function execution unit in real time. For example, after recognizing the “wave right hand” gesture, the system will immediately trigger the “next song” music playback instruction; recognizing the “slide up” gesture corresponds to “increase volume”; the entire process from completing the gesture to triggering the function is completed within milliseconds, providing users with a smooth, touch-free natural interaction experience.

[0087] The gesture recognition method based on sparse millimeter wave radar point cloud in this embodiment effectively overcomes the challenges of sparse point cloud, low signal-to-noise ratio, etc. under fast gestures by constructing a sparse spatio-temporal point cloud dataset, designing a special graph neural network model that integrates multi-modal feature encoding and spatio-temporal graph structure, and optimizing training with a compound loss function and a hierarchical learning rate strategy. This method realizes high-precision, low-latency real-time gesture recognition on an embedded platform, has good generalization ability and deployment feasibility, and can be widely applied to natural interaction scenes such as intelligent cabins, AR / VR, smart homes, etc., providing users with a smooth and intuitive touch-free operation experience.

[0088] Various embodiments are described herein with reference to a particular software embodiment. A person skilled in the art will recognize that embodiments of the application can be implemented using other software systems or platforms, and that changes can be made to the software without departing from the spirit and scope of the application. Where a software embodiment is described, this description is intended to encompass not only the described embodiment but also other software systems that might be implemented using the principles described herein.

[0089] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Numerous modifications to these embodiments can be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not to be limited to the implementations described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A gesture recognition method based on sparse millimeter wave radar point cloud, characterized in that, The method comprises the following steps: S1, acquiring millimeter wave radar data of multiple gesture actions, and constructing a sparse space-time point cloud data set; S2, based on the point cloud sequence in the data set, constructing a space-time graph structure data including a space graph and a time graph; S3, using the space-time graph structure data to train a gesture recognition model, combining a composite loss function and a hierarchical learning rate strategy to optimize the model parameters; the gesture recognition model comprises a graph feature encoding module, a space-time feature fusion module, and a time series modeling and classification module connected in sequence; The space-time feature fusion module is used for aggregating space neighborhood information and fusing time connection of the fusion node features output by the graph feature encoding module, and outputting space-time enhanced graph level features; comprising: a graph isomorphism network unit, which divides the input 256-dimensional node features into two parts in the channel dimension, inputs the first 128-dimensional features into a graph isomorphism network convolution layer for neighborhood information aggregation, directly jumps the second 128-dimensional features, splices the processed features and the jump connection features, and performs linear transformation and normalization to output 256-dimensional node level features; a graph attention network unit, which receives the 256-dimensional node level features output by the graph isomorphism network unit, divides them into two parts, inputs the first 128-dimensional features into a 4-head graph attention network convolution layer for neighborhood information aggregation, directly jumps the second 128-dimensional features, splices the processed features and the jump connection features, and performs linear transformation and normalization to output 256-dimensional node level features enhanced by space-time; a graph pooling unit, which receives the 256-dimensional node level features enhanced by space-time, respectively performs global average pooling and global maximum pooling on the graph data of each time frame, splices the obtained two 256-dimensional vectors, and outputs a 512-dimensional graph level feature; S4, processing the millimeter wave radar data of the gesture to be recognized into space-time graph structure data, inputting the trained gesture recognition model, and outputting the corresponding gesture category recognition result.

2. The gesture recognition method based on sparse millimeter wave radar point cloud according to claim 1, characterized in that, The S1 comprises: using a MIMO millimeter wave radar system to collect original data of multiple categories of preset gesture actions, and collecting a preset number of sample data for each gesture category to construct the sparse space-time point cloud data set; wherein each sample data is a point cloud sequence including multiple frames of point clouds, and each radar point cloud data point in each frame of point clouds contains its spatial coordinates, distance information, velocity information and reflection intensity information.

3. The gesture recognition method based on sparse millimeter wave radar point cloud according to claim 1, characterized in that, In the S2, the space graph construction comprises: for the point cloud in each time frame, calculating the two-dimensional Euclidean distance according to the spatial coordinates of the points, and using a K nearest neighbor algorithm to establish a spatial connection edge between each point and K nearest neighbor points, and the feature of the spatial edge is the coordinate difference vector of the two connected points; the time graph construction comprises: for the point clouds of adjacent time frames, calculating the spatial distance between all point pairs, and establishing a time connection edge for the point pairs with a distance lower than a preset threshold τ, and the feature of the time edge is the coordinate difference vector of the two connected points in adjacent frames.

4. The gesture recognition method based on sparse millimeter wave radar point cloud according to claim 1, characterized in that, The graph feature encoding module is used for separate encoding and fusion of multi-modal features, and outputs high-dimensional unified node features; comprising: a feature splitting unit, which splits the multi-modal features of an input node into spatial position features [x, y] and radar parameter features [distance, Doppler velocity, echo intensity]; a first feature encoding unit, which is internally provided with a first multi-layer perception machine and is configured to encode the spatial position features and map them to a 128-dimensional feature space; a second feature encoding unit, which is internally provided with a second multi-layer perception machine and is configured to encode the radar parameter features and map them to a 128-dimensional feature space; a feature fusion unit, which concatenates the two 128-dimensional feature vectors output by the first and second feature encoding units in a channel dimension to obtain a 256-dimensional fusion node feature.

5. The gesture recognition method based on sparse millimeter wave radar point cloud according to claim 1, characterized in that, The time series modeling and classification module is configured to model time series dependency of the graph-level features output by the spatio-temporal feature fusion module and complete classification, and output gesture category probabilities; and includes: a dual-path time series modeling unit, which receives a time series composed of T 512-dimensional graph-level features, and respectively models time series through an LSTM path and a TCN path; a feature fusion and attention unit, which concatenates the output features of the dual paths, and then inputs a multi-head self-attention mechanism to enhance the information of key time steps; a classifier, which maps the sequence features output by the self-attention mechanism to a dimension same as the number of gesture categories through a multi-layer perception classifier, and outputs the probabilities of each category through a Softmax function.

6. The gesture recognition method based on sparse millimeter wave radar point cloud according to claim 5, characterized in that, The LSTM path adopts a bidirectional LSTM network, and the dimension of a hidden layer is set to 512; the TCN path is composed of three time series convolution blocks cascaded with each other, and the expansion rates thereof are set to 1, 2 and 4 respectively; and the number of heads of the multi-head self-attention mechanism is set to 16 heads.

7. The gesture recognition method based on sparse millimeter wave radar point cloud according to claim 5, characterized in that, The classifier is composed of three linear layers, each of which is followed by a LeakyReLU activation function, a Dropout layer and a batch normalization layer.

8. The gesture recognition method based on sparse millimeter wave radar point cloud according to claim 1, characterized in that, In the S3, the composite loss function L total is expressed as: ； wherein L focal-smooth represents the label smooth focal loss, L separation represents the inter-class separation loss, and λ is a weight coefficient.

9. The gesture recognition method based on sparse millimeter wave radar point cloud according to claim 1, characterized in that, In the S3, the hierarchical learning rate strategy includes: The first learning rate δ1=1×10-4 is set for parameters in the feature coding module and the spatio-temporal feature fusion module. -3 ; a second learning rate δ2= 5 x 10 -4 for the parameters in the timing modeling and classification module ; Using AdamW optimizer, weight decay coefficient is set to 1 x 10 -5 and global gradient clipping with threshold set to 1.0.

Images