Open-collection gesture recognition method and device based on FMCW radar

Abstract

This invention discloses an open-set gesture recognition method and device based on FMCW radar, solving the problem that traditional closed-set recognition cannot handle unknown gestures. The method includes acquiring raw gesture data from FMCW radar, filtering out static clutter using phasor mean cancellation, and extracting range-Doppler and time-frequency features through two-dimensional Fourier transform and short-time Fourier transform. An embedding vector is generated by fusing features using a dual-branch ResNet-18 network, incorporating cross-domain human behavior data and combining it with triplet loss training to construct the embedding vector space. Preset / unpreset gestures are divided based on a distance threshold; preset gestures are classified using k-nearest neighbors, while unpreset gestures are clustered into unknown categories using the DBSCAN algorithm as outer set samples. This invention improves feature representation and generalization capabilities, achieving high-precision classification of preset gestures and effective identification of unknown gestures. It is adaptable to real-world open-set interaction scenarios and can be widely applied in human-computer interaction fields such as smart homes and smart vehicles.

Description

Technical Field

[0001] This invention belongs to the field of radar signal processing technology, specifically relating to an open set gesture recognition method and device based on FMCW radar. Background Technology

[0002] With the continuous development of radar technology, its application scope has gradually expanded from the military field to the civilian field, and it has been widely used in areas such as assisted driving, smart homes, healthcare, and human-computer interaction. In the field of human-computer interaction, radar equipment can detect human hand gestures and may use different types of radar transmission waveforms, including continuous wave (CW) radar, pulse ultra-wideband radar, and frequency modulated continuous wave (FMCW) radar. Different radar systems have different advantages and disadvantages. CW radar directly measures phase difference and angle, but cannot measure distance; anomaly removal cannot utilize distance information. Pulse ultra-wideband radar has strong anti-interference capabilities, but requires high hardware and a high sampling rate. FMCW radar can detect the distance, Doppler, and angle information of hand gestures, and has rich echo signal information, but its signal processing is relatively complex.

[0003] FMCW radar can receive intermediate frequency (IF) signals obtained by mixing echo and transmitted signals. Leveraging its high frequency and high resolution characteristics, it can accurately capture subtle motion features of hand gestures by analyzing multi-dimensional features such as range, velocity, and angle within the IF signal. Existing methods have achieved high accuracy in closed-set recognition. Traditional closed-set methods can only classify preset gestures after training. However, in real-world applications, user gestures are unpredictable, requiring the system to identify unknown gestures. Traditional closed-set recognition methods cannot handle interference from open sets in real-world scenarios, making it difficult to meet practical application needs. Summary of the Invention

[0004] Therefore, the present invention provides an open-set gesture recognition method and apparatus based on FMCW radar to solve or partially solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A first aspect provides an open-set gesture recognition method based on FMCW radar, comprising:

[0006] Obtain the raw gesture data collected by the FMCW radar to obtain the raw radar signal; Static clutter filtering is performed on the original radar signal. Two-dimensional Fourier transform and short-time Fourier transform are then performed on the clutter-filtered signal to extract range-Doppler features and time-frequency features. The distance-Doppler features and the time-frequency features are fused using a dual-branch ResNet-18 network to generate a compact embedding vector; The dual-branch ResNet-18 network is trained using triplet loss to construct an embedding vector space; Based on the distance threshold of vectors in the embedded vector space, the gestures to be recognized are divided into preset gestures and non-preset gestures. For preset gestures, the following is applied... k The nearest neighbor classification method completes the category classification. Unpredictable gestures are first identified as outlier samples, and then clustering algorithms are used to classify them into different unknown gesture categories.

[0007] As a preferred scheme for the open-set gesture recognition method based on FMCW radar, the static clutter filtering adopts the phasor mean cancellation algorithm, specifically: Subtracting the average value of all pulses within the same distance cell from the data of a single pulse within a frame at that distance cell eliminates static data that does not change over time, thus filtering out static clutter.

[0008] As a preferred scheme for open-set gesture recognition methods based on FMCW radar, the formula for calculating the two-dimensional Fourier transform of the signal after clutter filtering is as follows:

[0009] In the formula, These are the eigenvalues ​​of the distance-Doppler feature map. This is the two-dimensional data matrix of the original radar signal after static clutter filtering. For the transformation index of the Doppler dimension, For the transformation index of the distance dimension, For the sampling index of the Doppler dimension, For the sampling index of the distance dimension, The number of sampling points for the distance. The number of sampling points in the Doppler dimension. The imaginary unit, This is the modulus operation for complex numbers.

[0010] As a preferred scheme for open-set gesture recognition methods based on FMCW radar, the formula for calculating the short-time Fourier transform of the signal after clutter filtering is as follows:

[0011] In the formula, These are the time-frequency eigenvalues ​​after the short-time Fourier transform. For frequency point indices in the frequency domain, For time frame indexing, This refers to the radar one-dimensional signal data after static clutter filtering and one-dimensional Fourier transform. For the time sampling point index of the signal, This represents the total number of time sampling points for the one-dimensional signal. For length is The sliding window function, This is the length of the window function in the short-time Fourier transform. It is the imaginary unit.

[0012] As a preferred scheme for the open-set gesture recognition method based on FMCW radar, during the training phase of the dual-branch ResNet-18 network, human behavior feature maps are introduced as cross-domain data, which together with the feature maps of the original gesture data to construct a cross-domain dataset, thereby expanding the types of datasets for network training. The feature fusion process of the distance-Doppler features and the time-frequency features using a dual-branch ResNet-18 network is as follows: One branch processes the time-frequency features and generates the corresponding time-frequency feature vector, while the other branch processes the distance-Doppler features and generates the corresponding distance-Doppler feature vector. The two feature vectors are then merged in a serial splicing manner to form the compact embedding vector.

[0013] As a preferred scheme for open-set gesture recognition based on FMCW radar, the specific method for training the dual-branch ResNet-18 network using triplet loss is as follows: Define anchor samples, positive samples of the same class as anchor samples, and negative samples of different classes from anchor samples. Training triples are composed of anchor samples, positive samples, and negative samples. During training, the vector distance between anchor point samples and positive samples, and the vector distance between anchor point samples and negative samples are kept at a preset margin. The feature distribution in the embedding vector space is optimized to achieve compact clustering of similar gesture features and widening of the spacing between dissimilar gesture features.

[0014] As a preferred scheme of the open-set gesture recognition method based on FMCW radar, the judgment expression for classifying the gesture to be recognized into preset gestures and non-preset gestures based on the distance threshold of the vectors in the embedded vector space is as follows:

[0015] In the formula, The preset distance threshold, Gesture sample to be recognized With the Individual gesture category samples Euclidean distance in the embedded space The dimension of the embedded vector. Gesture sample to be recognized The dimensional vector value, Samples of preset gesture categories The Dimensional vector value.

[0016] As a preferred scheme for open-set gesture recognition methods based on FMCW radar, the preset gestures are... k The specific process of nearest neighbor classification to complete category classification is as follows: Calculate the vector distance between the preset gesture sample to be classified and each preset category sample in the embedding vector space, and find the closest one. k A preset category of samples, based on this k Each sample votes on its category, and the final category of the preset gesture sample to be classified is determined by the voting results.

[0017] As a preferred scheme for open-set gesture recognition methods based on FMCW radar, the clustering algorithm is the density-based noisy applied spatial clustering algorithm DBSCAN, mathematically expressed as:

[0018] In the formula, For sample points of Neighborhood sample set This is the set of samples in the embedding space for gesture samples that are not pre-defined. , outer set of samples Any two sample vectors in the dataset, For sample points and vector distance, The neighborhood radius of the DBSCAN algorithm. To adjust the scaling factor of clustering, This is the distance threshold in open set recognition; The specific process of using the DBSCAN algorithm to cluster unpreset gestures is as follows: By examining the sample density of the outer set sample point in its ε-neighborhood, we divide the set of all sample points whose distance to the sample point is within the radius of the neighborhood, and group the sample points that satisfy the neighborhood reachability condition into the same set. Each independent set of sample points is an unknown gesture category.

[0019] Secondly, the present invention provides an open-set gesture recognition device based on FMCW radar, employing the open-set gesture recognition method based on FMCW radar according to the first aspect or any possible implementation thereof, including: The data acquisition module is used to acquire the raw gesture data collected by the FMCW radar to obtain the raw radar signal; The feature extraction module is used to perform static clutter filtering on the original radar signal, and to perform two-dimensional Fourier transform and short-time Fourier transform on the signal after clutter filtering to extract range-Doppler features and time-frequency features. The feature fusion module is used to fuse the distance-Doppler features and the time-frequency features through a dual-branch ResNet-18 network to generate a compact embedding vector; The network training module is used to train the dual-branch ResNet-18 network using triplet loss to construct an embedding vector space; The gesture recognition module is used to classify the gesture to be recognized into preset gestures and non-preset gestures based on a distance threshold of vectors in the embedded vector space, and to apply a method for the preset gestures. k The nearest neighbor classification method completes the category classification. Unpredictable gestures are first identified as outlier samples, and then clustering algorithms are used to classify them into different unknown gesture categories.

[0020] Thirdly, the present invention provides an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the processor, when executing the program or instructions, implements the open-set gesture recognition method based on FMCW radar, as described in the first aspect or any possible implementation thereof.

[0021] Fourthly, the present invention provides a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the FMCW radar-based open-set gesture recognition method of the first aspect or any possible implementation thereof.

[0022] The present invention has the following advantages: This invention filters out static clutter in the original radar signal using a phasor mean cancellation algorithm, and extracts range-Doppler features and time-frequency features by combining two-dimensional Fourier transform and short-time Fourier transform, respectively, thus achieving accurate extraction of multi-dimensional features of gestures. The two types of features complement each other to describe the temporal changes and spatial motion information of gestures, providing a comprehensive and effective feature foundation for recognition.

[0023] This invention employs a dual-branch ResNet-18 network to fuse distance-Doppler features and time-frequency features, generating compact embedding vectors through serial splicing. This fully leverages the gesture representation value of different features, enhancing feature recognition and representation capabilities.

[0024] This invention utilizes triplet loss to train a dual-branch network, which compactly clusters the embedding vectors of similar gestures and significantly expands the spacing between the embedding vectors of dissimilar gestures, thus constructing a highly discriminative embedding vector space. This lays a solid spatial feature foundation for pre-defined gesture classification and non-pre-defined gesture identification.

[0025] This invention introduces human behavior feature maps as cross-domain data during the network training phase, which expands the categories of the training dataset, effectively optimizes the generalization ability of the embedding space, makes the clustering of vectors of the same category stronger, and significantly improves the recognition accuracy of preset gestures. After testing, the recognition accuracy of preset gestures increased from 97.5% to 99.6% after adding cross-domain data.

[0026] This invention designs a hybrid decision recognition mechanism based on a distance threshold. First, gestures are divided into preset gestures and non-preset gestures using a vector distance threshold. Then, the preset gestures are... k The nearest neighbor algorithm achieves accurate classification. After determining that the unpreset gestures are out-of-set samples, the DBSCAN density clustering algorithm is used to classify them into different unknown categories. This not only ensures the classification accuracy of the preset gestures, but also achieves effective identification and clustering of unknown gestures, giving the system open set recognition capability.

[0027] The DBSCAN clustering algorithm of this invention sets the neighborhood radius based on a distance threshold, realizing unsupervised clustering of unpreset gestures. This enables unpreset gestures to form independent clusters with clear boundaries in the embedding space, which can adapt to the unpredictability of user gestures in real-world scenarios and effectively avoid the misclassification problem of traditional closed-set recognition. Attached Figure Description

[0028] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0029] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0030] Figure 1 This is a schematic diagram of the open set gesture recognition method based on FMCW radar provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the technical route of the open set gesture recognition method based on FMCW radar provided in the embodiments of the present invention; Figure 3This is a schematic diagram of the confusion matrix result before adding cross-domain data, provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the confusion matrix result after adding cross-domain data, provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the t-SNE dimensionality reduction visualization gesture clustering effect provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the architecture of the open set gesture recognition device based on FMCW radar provided in an embodiment of the present invention; Figure 7 This is an electronic device architecture diagram provided in an embodiment of the present invention. Detailed Implementation

[0031] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] With the rapid development of radar technology, its application areas have gradually expanded from traditional military fields to civilian fields such as assisted driving, smart homes, healthcare, and human-computer interaction. In the field of human-computer interaction, radar equipment, with its non-contact, highly adaptable to various environments, and unaffected by factors such as lighting, has become an important means of detecting human gestures. Currently, the radar systems used for gesture detection mainly include continuous wave (CW) radar, pulse ultra-wideband radar, and frequency modulated continuous wave (FMCW) radar. Due to differences in technical characteristics, different radar systems exhibit different advantages and disadvantages in gesture recognition applications. CW radar can directly detect phase difference and angle information, but it lacks distance measurement capabilities and cannot use distance information for anomaly removal, thus limiting the accuracy of gesture detection. Pulse ultra-wideband radar has strong anti-interference capabilities, but it has stringent hardware requirements, necessitating high-sampling-rate acquisition modules, resulting in high equipment costs and significant engineering challenges. FMCW radar can simultaneously detect multi-dimensional information of gestures, including distance, Doppler, and angle. Its echo signals contain rich gesture motion features and possess the technical advantages of high frequency and high resolution, enabling it to capture subtle changes in gesture movement. This makes it the preferred radar system in the field of gesture recognition, but its signal processing flow is relatively complex, requiring higher standards for feature extraction and recognition algorithm design.

[0033] FMCW radar detects gestures by analyzing multi-dimensional features such as distance, speed, and angle obtained from the intermediate frequency signal obtained by mixing the received echo signal and the transmitted signal. Existing gesture recognition methods based on FMCW radar have achieved high recognition accuracy in closed-set recognition scenarios. However, closed-set gesture recognition can only classify preset gestures that have been trained, and its recognition range is limited to gesture types within the training set, which cannot adapt to real-world human-computer interaction application scenarios. In actual use, user gestures are highly unpredictable. In addition to preset gestures, a large number of unpreset gestures that were not included in the model training will also occur. At this time, the system needs to have the ability to identify and reject unknown gestures. However, traditional closed-set recognition methods lack the ability to process open sets and are easily affected by unknown gestures in real-world scenarios, resulting in misclassification and distorted recognition results, which cannot meet the needs of actual human-computer interaction.

[0034] To address the technical challenges of FMCW radar gesture recognition in open-set scenarios, there is an urgent need to design an open-set gesture recognition method that can effectively distinguish between preset and non-preset gestures and can reasonably cluster non-preset gestures. This would improve the adaptability and accuracy of FMCW radar gesture recognition in real-world applications and promote the further application of FMCW radar in the field of human-computer interaction. The following are the specific details of an embodiment of this invention.

[0035] See Figure 1 and Figure 2 This invention provides an open-set gesture recognition method based on FMCW radar, comprising the following steps: S1. Obtain the raw gesture data collected by the FMCW radar to obtain the raw radar signal; S2. Perform static clutter filtering on the original radar signal, and perform two-dimensional Fourier transform and short-time Fourier transform on the signal after clutter filtering to extract range-Doppler features and time-frequency features. S3. The distance-Doppler features and the time-frequency features are fused using a dual-branch ResNet-18 network to generate a compact embedding vector; S4. The dual-branch ResNet-18 network is trained using triplet loss to construct an embedding vector space; S5. Based on the distance threshold of vectors in the embedded vector space, the gesture to be recognized is divided into preset gestures and non-preset gestures. For preset gestures, the following steps are taken: k The nearest neighbor classification method completes the category classification. Unpredictable gestures are first identified as outlier samples, and then clustering algorithms are used to classify them into different unknown gesture categories.

[0036] In one possible embodiment, in step S1, test data from a TI-based IWR1642 in TDM one-transmit-four-receive mode is used to verify the effectiveness of the open-set gesture recognition method based on FMCW radar. The radar frequency sweep starts at 77 GHz, the sweep slope is 36.355 MHz / µs, the ADC sampling point count is 256, and the ADC data acquisition rate is 5 MHz. That is, the wavelength is 0.0039 m, the system bandwidth is 4 GHz, the range resolution is 0.0375 m, the number of chirps per frame is 128, the chirp period is 200 µs, and the velocity resolution is 0.076 m / s. The gesture is performed in front of the radar panel, with a radial movement distance ranging from 0.15 meters to 1 meter.

[0037] In one possible embodiment, in step S2, the static clutter filtering employs a phasor mean cancellation algorithm, specifically: Subtracting the average value of all pulses within the same distance cell from the data of a single pulse within a frame at that distance cell eliminates static data that does not change over time, thus filtering out static clutter.

[0038] Specifically, the raw signals acquired by the FMCW radar include not only the dynamic echo signal of the gesture but also static clutter from the environmental background, such as walls, tabletops, and the radar itself. This clutter's signal data remains largely unchanged over time across different pulses within the same range cell within a single frame, while the dynamic signal of the gesture changes significantly with each pulse acquisition. Using the same range cell within a frame as the calculation unit, the average value of all pulse data within that range cell is first calculated. This average value serves as the static clutter reference value for that range cell, containing all static interference information at that location that does not change over time. Subtracting this reference average value from the raw data of a single pulse directly cancels out the static clutter component, retaining only the dynamic echo signal of the gesture that changes over time. This improves the accuracy of subsequent feature extraction and avoids interference from static clutter on gesture features.

[0039] In one possible embodiment, in step S2, the formula for calculating the two-dimensional Fourier transform of the signal after filtering out clutter is as follows:

[0040] In the formula, These are the eigenvalues ​​of the distance-Doppler feature map. This is the two-dimensional data matrix of the original radar signal after static clutter filtering. For the transformation index of the Doppler dimension, For the transformation index of the distance dimension, For the sampling index of the Doppler dimension, For the sampling index of the distance dimension, The number of sampling points in the distance dimension. The number of sampling points in the Doppler dimension. The imaginary unit, This is the modulus operation for complex numbers.

[0041] Specifically, the radar echo signal is a complex signal. A two-dimensional Fourier transform is used to analyze the signal in both the range and Doppler dimensions through two separate Fourier transforms. The first Fourier transform targets the time-delay sampling data in the range dimension, converting the signal from the time-delay domain to the range domain to obtain the relative distance information between the gesture and the radar. The second Fourier transform targets the time-sampling data in the Doppler dimension, converting the signal from the time domain to the Doppler frequency domain, and calculating the gesture's velocity using the Doppler frequency shift. The formula takes the modulus value after transforming the complex signal because the modulus represents the signal amplitude. Amplitude information reflects the signal strength of the gesture in the corresponding range-Doppler coordinates. The range-Doppler graph (RDM) composed of the modulus values ​​can intuitively present the spatial location and velocity characteristics of the gesture. This is achieved using a two-dimensional data matrix. As input, match distance dimension Doppler The number of sampling points, determined by changing the index. , Traversing the entire transformation domain, the final result is The feature values ​​for each coordinate point of the distance-Doppler feature map are used to fully characterize the distance-velocity two-dimensional motion features of the gesture.

[0042] In one possible embodiment, in step S2, the formula for calculating the short-time Fourier transform of the signal after filtering out clutter is as follows:

[0043] In the formula, These are the time-frequency eigenvalues ​​after the short-time Fourier transform. For frequency point indices in the frequency domain, For time frame indexing, This refers to the radar one-dimensional signal data after static clutter filtering and one-dimensional Fourier transform. For the time sampling point index of the signal, This represents the total number of time sampling points for the one-dimensional signal. For length is The sliding window function, This is the length of the window function in the short-time Fourier transform. It is the imaginary unit.

[0044] Specifically, the Short-Time Fourier Transform (SFT) is a time-frequency analysis algorithm designed to address the non-stationarity of radar signals. It extracts the joint features of gesture signals in both the time and frequency domains, overcoming the limitation of the Fourier Transform in reflecting frequency changes over time. Gesture movements are dynamic, and the corresponding radar echo signals are non-stationary, with their frequency characteristics changing over time. Traditional Fourier Transform only provides the overall frequency characteristics and cannot reflect temporal variations. The SFT addresses this issue through windowing and framing. It first performs a one-dimensional Fourier Transform on the clutter-filtered signal to complete the initial domain transformation, obtaining a one-dimensional signal. Then through a length of The window function performs frame segmentation on a one-dimensional signal, dividing the continuous non-stationary signal into multiple short-term stationary signal segments. The signal within each window segment can be approximated as having invariant frequency characteristics. A Fourier transform is performed on the signal within each window segment to obtain the corresponding frequency characteristics. The window function is then traversed across the entire time axis, combined with frequency point indices. The final result By using the feature values ​​under each time-frequency coordinate, a time-frequency feature map is formed, thereby characterizing the frequency changes of the gesture at different times and reflecting the temporal dynamic motion characteristics of the gesture.

[0045] In one possible embodiment, in step S3, during the training phase of the dual-branch ResNet-18 network, human behavior feature maps are introduced as cross-domain data, and together with the feature maps of the original gesture data, a cross-domain dataset is constructed to expand the types of datasets for network training. The feature fusion process of the distance-Doppler features and the time-frequency features using a dual-branch ResNet-18 network is as follows: One branch processes the time-frequency features and generates the corresponding time-frequency feature vector, while the other branch processes the distance-Doppler features and generates the corresponding distance-Doppler feature vector. The two feature vectors are then merged in a serial splicing manner to form the compact embedding vector.

[0046] Specifically, human behavior feature maps and gesture feature maps are cross-domain, same-source data. Both are collected by FMCW radar and have the same signal characteristics and data distribution patterns. Moreover, human behavior features are richer in dimensions and have more categories. Adding them as cross-domain data for training can expand the category and feature diversity of the dataset, allowing the network to learn more general radar signal feature extraction rules, avoiding overfitting the network to a limited number of gesture samples, and optimizing the feature distribution in the embedding space, making the vectors of the same category of gestures more compact and improving the recognition accuracy.

[0047] ResNet-18 is a lightweight deep convolutional network with residual connections, which avoids the gradient vanishing problem common in deep networks and is suitable for radar feature extraction and mapping. The dual-branch structure is designed because range-Doppler features and time-frequency features are complementary features of different dimensions. A single-branch network cannot simultaneously ensure the extraction accuracy of both types of features. The dual-branch network can perform independent convolution, pooling, and fully connected processing on each type of feature, maximizing the extraction of their respective feature values ​​and generating targeted time-frequency feature vectors and range-Doppler feature vectors. Serial concatenation connects the two feature vectors dimensionally to form a higher-dimensional embedding vector. This method completely preserves all information from both types of features, avoiding information loss during feature fusion. The resulting compact embedding vector is a comprehensive representation of the multi-dimensional features of gestures, providing a high-quality feature foundation for the construction of the post-implementation space.

[0048] In one possible embodiment, in step S4, the specific method for training the dual-branch ResNet-18 network using triplet loss is as follows: Define anchor samples, positive samples of the same class as anchor samples, and negative samples of different classes from anchor samples. Training triples are composed of anchor samples, positive samples, and negative samples. During training, the vector distance between anchor point samples and positive samples, and the vector distance between anchor point samples and negative samples are kept at a preset margin. The feature distribution in the embedding vector space is optimized to achieve compact clustering of similar gesture features and widening of the spacing between dissimilar gesture features.

[0049] Specifically, anchor samples serve as baseline samples. Positive samples and anchor samples belong to the same gesture category, while negative samples and anchor samples belong to different gesture categories. This triplet construction allows the training process to directly optimize the distance relationship between categories, rather than the category label prediction of traditional classification loss. This is more suitable for gesture classification based on distance thresholds in open set recognition. During training, the distance between the anchor point and the positive sample, along with a preset margin, is constrained to be less than the distance between the anchor point and the negative sample. This margin prevents the model from optimizing the distance between positive and negative samples to zero, controlling the width of the classification boundary and preventing excessive dispersion of similar samples and excessive proximity of dissimilar samples. Through iterative training with triplet samples, the network continuously adjusts its parameters, causing the embedding vectors of similar gestures to continuously approach each other in space (intra-class compactness) and the embedding vectors of different gestures to continuously move away (inter-class separation). Ultimately, this constructs a highly discriminative embedding vector space. The distance between vectors in this space directly reflects the similarity of gesture categories, providing an intuitive basis for judging preset / unpreset gesture classification based on distance thresholds.

[0050] In one possible embodiment, in step S5, based on the distance threshold of vectors in the embedded vector space, the determination expression for classifying the gesture to be recognized into preset gestures and non-preset gestures is as follows:

[0051] In the formula, The preset distance threshold, Gesture sample to be recognized With the Individual gesture category samples Euclidean distance in the embedded space The dimension of the embedded vector. Gesture sample to be recognized The dimensional vector value, Samples of preset gesture categories The Dimensional vector value.

[0052] Specifically, the formula uses Euclidean distance to calculate the sample to be identified. Compared with preset category samples The vector distance, specifically the Euclidean distance, characterizes the similarity between two samples in a high-dimensional embedding vector space. A smaller distance indicates greater similarity in features between the two samples, suggesting a higher probability that they belong to the same category. Threshold. The critical value obtained by statistically analyzing the intra-class and inter-class distances of preset gesture samples in the embedding space based on the training set is a reasonable upper limit for the maximum intra-class distance of the preset gesture, ensuring that the distance between the sample vector of the preset gesture and samples of the same class is less than 1. The distance between the sample vector of the unpreset gesture and all the sample vectors of the preset gesture is greater than 1. .

[0053] Among them, the judgment formula Indicates "sample to be identified" A With any pre-defined category sample The Euclidean distance is less than α If this condition is met, it means A If the features of the sample to be identified are highly similar to the features of a preset gesture, it is determined to be the preset gesture; if the sample to be identified... A The Euclidean distance to all preset category samples is greater than α If the feature is located outside the preset gesture in the embedding space, it is determined to be a non-preset gesture (outer set sample).

[0054] In one possible embodiment, in step S5, a preset gesture is applied. k The specific process of nearest neighbor classification to complete category classification is as follows: Calculate the vector distance between the preset gesture sample to be classified and each preset category sample in the embedding vector space, and find the closest one. k A preset category of samples, based on this kEach sample votes on its category, and the final category of the preset gesture sample to be classified is determined by the voting results.

[0055] Specifically, k Nearest Neighbors (kNN) is an instance-based lazy learning algorithm that requires no additional training and directly utilizes the distance relationships in the embedding vector space for classification. The kNN algorithm does not have an explicit training process; it only calculates the distance between the sample to be classified and known samples during the classification phase. It is well-suited to the high-discrimination embedding vector space already constructed in this method, and can directly utilize the distance features of the space for accurate classification, avoiding the complexity brought by additional model training. k The number of nearest neighbor samples selected. k The selection of values ​​needs to balance classification accuracy and robustness against interference: k If the value is too small, it is easily affected by individual abnormal samples; k If the value is too large, it can easily introduce samples from other categories, reducing classification accuracy. In this method... k The values ​​are selected reasonably based on the embedded spatial distribution of preset gestures to ensure that the selection is appropriate. k The nearest neighbor samples can truly reflect the category affiliation of the sample to be classified; The majority voting mechanism is adopted because, in the compact embedding space within a class and separate between classes, the nearest neighbor of the sample to be classified is determined by the class. k If all samples are likely to belong to the same pre-defined category, the final category can be determined by voting, which can effectively improve the robustness of classification and avoid classification errors caused by distance errors of a single sample.

[0056] In one possible embodiment, the clustering algorithm is the density-based noisy applied spatial clustering algorithm DBSCAN, mathematically expressed as:

[0057] In the formula, For sample points of Neighborhood sample set This is the set of samples in the embedding space for gesture samples that are not pre-defined. , outer set of samples Any two sample vectors in the dataset, For sample points and vector distance, The neighborhood radius of the DBSCAN algorithm. To adjust the scaling factor of clustering, This is the distance threshold in open set recognition; The specific process of using the DBSCAN algorithm to cluster unpreset gestures is as follows: By examining the sample density of the outer set sample point in its ε-neighborhood, we divide the set of all sample points whose distance to the sample point is within the neighborhood radius, and group the sample points that satisfy the neighborhood reachability condition into the same set. Each independent set of sample points is an unknown gesture category.

[0058] Specifically, neighborhood radius Based on the preset distance threshold α With proportionality coefficient The product is determined, rather than set independently, ensuring that the spatial decision scale for clustering is consistent with the preset / unpreset gesture partitioning scale, thus guaranteeing the uniformity of the open set recognition decision logic. It can be flexibly adjusted according to the density distribution of the outer set samples in the embedding space to adapt to different gesture collection scenarios. as sample points The neighborhood set, the number of sample points it contains, directly reflects the sample density at that location. Density is the core basis for determining whether samples belong to the same class. Samples of the same unknown gesture exhibit high-density clustering characteristics in the embedding space, while low-density blank areas exist between different unknown gestures.

[0059] Wherein, neighborhood reachability means that if the sample point exist Within the ε neighborhood, sample points Again Within its ε-neighborhood, then and Neighborhood reachability allows all consecutive high-density sample points in the embedding space to be grouped into the same set through transitivity, overcoming the limitations of direct distance-based determination and capturing clusters of unpredictable gesture samples with similar shapes. The outer set of samples for all unpredictable gestures is traversed point by point. First, the ε-neighborhood density of an individual sample point is determined. Then, through neighborhood reachability, consecutive high-density sample points are aggregated into independent sets. Each independent set corresponds to an unknown gesture category, while isolated low-density sample points are identified as noise samples and removed. This achieves adaptive classification of unpredictable gestures, meeting the needs of open-set recognition of unpredictable user gestures in real-world scenarios.

[0060] See Figure 3 and Figure 4 This is a schematic diagram of the confusion matrix results before and after adding cross-domain data, provided in an embodiment of the present invention. Before adding cross-domain data, the model's recognition accuracy in the preset gesture recognition test was only 97.5%, while after adding cross-domain data, the recognition accuracy in the preset gesture recognition test reached 99.6%. After the embedding space was enhanced by cross-domain data, the accuracy of preset gesture classification was significantly improved.

[0061] See Figure 5 This is a visualization of the gesture clustering effect of t-SNE dimensionality reduction in this embodiment. Figure 5It can be clearly seen that different out-of-domain data exhibit different shapes and intra-class distances. Unpreset gestures form independent clusters in the embedding space, with clear boundaries with the pre-set categories, confirming the strengthening effect of cross-domain data on the generalization ability of the feature space.

[0062] The application scenarios of this invention are as follows: Smart Home Scenarios: The FMCW radar module can be integrated into smart home devices such as smart speakers, smart lights, smart curtains, and air conditioners, enabling contactless gesture control. Users can control devices, switch modes, and adjust parameters without touching them, using preset gestures such as waving, swiping, or clenching their fists. The system can also identify meaningless gestures and other unpreset actions to prevent accidental triggering, improving the convenience and intelligence of smart home interactions. It can also be applied to the remote operation of smart appliances, such as controlling range hoods and microwave ovens with greasy hands in the kitchen, balancing safety and ease of use.

[0063] Intelligent in-vehicle scenarios: FMCW radar components can be installed on the center console, steering wheel, and in-vehicle display screen of a car to enable gesture interaction of the in-vehicle system. While driving, the driver does not need to look down and touch the screen, but can complete operations such as starting navigation, adjusting volume, answering and hanging up calls, and changing songs through simple preset gestures. At the same time, the system can effectively recognize the driver's driving movements and other unpreset gestures to avoid accidental operation while driving and improve driving safety. It can also be applied to gesture adjustment of car seats and rearview mirrors to adapt to the usage habits of different drivers.

[0064] Smart Terminal Scenarios: Millimeter-wave radar modules can be integrated into smart terminals such as smartphones, tablets, laptops, and smart tablets to enable air interaction and gesture control. Users can perform operations such as turning on the screen, turning pages, zooming, and taking screenshots through preset gestures. The system can accurately identify and reject non-preset gestures such as random hand raises to avoid accidental operations. It can also be applied to functions such as screen-off wake-up and gesture unlocking on smart terminals, enhancing the diversity and fun of terminal interaction.

[0065] Industrial control scenarios: Applicable to intelligent control consoles, industrial robots, and automated production equipment in industrial production, enabling contactless gesture control in industrial settings. Workers wearing protective gloves and with hands contaminated with oil or dust can use preset gestures to perform operations such as starting and stopping equipment, switching processes, adjusting parameters, and issuing robot action commands. The system can identify unpreset gestures from workers, preventing accidental equipment triggering. This improves the convenience of industrial operations while ensuring production safety. Furthermore, it adapts to complex operating environments in industrial settings, unaffected by factors such as light, dust, or oil.

[0066] Smart healthcare scenarios: These systems can be integrated into medical equipment, smart ward terminals, and rehabilitation training equipment to achieve contactless human-computer interaction in medical settings. In sterile medical environments such as operating rooms and sterilization rooms, medical staff can control medical instruments, retrieve patient medical records, and adjust equipment parameters using preset gestures, avoiding cross-infection caused by contact operations. In rehabilitation training scenarios, the system can recognize patients' rehabilitation training gestures to assist rehabilitation equipment in adjusting training modes, while also identifying non-training gestures to ensure the accuracy of rehabilitation training. It can also be applied to intelligent call systems in geriatric wards and intensive care units, allowing patients to complete call operations with simple preset gestures, catering to the needs of patients with limited mobility.

[0067] Public intelligent device scenarios: Intelligent inquiry machines, self-service terminals, and advertising screens can be installed in public places such as shopping malls, subway stations, and airports to achieve contactless gesture interaction in public settings. Users can complete actions such as querying information, switching interfaces, and confirming operations through preset gestures. The system can identify unpreset gestures such as people raising their hands randomly or passing by and obstructing the view, avoiding accidental operation and improving the efficiency of public equipment use. At the same time, contactless operation reduces contact contamination of public equipment, balancing hygiene and ease of use.

[0068] It should be noted that the method of this embodiment can also be applied to distributed scenarios, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the open-set gesture recognition method based on FMCW radar.

[0069] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0070] See Figure 6 This invention also provides an open-set gesture recognition device based on FMCW radar, and an open-set gesture recognition method based on FMCW radar using the above embodiments or any possible implementation thereof, including: The data acquisition module 100 is used to acquire the raw gesture data collected by the FMCW radar to obtain the raw radar signal; The feature extraction module 200 is used to perform static clutter filtering on the original radar signal, and to perform two-dimensional Fourier transform and short-time Fourier transform on the signal after clutter filtering to extract range-Doppler features and time-frequency features. The feature fusion module 300 is used to fuse the distance-Doppler features and the time-frequency features through a dual-branch ResNet-18 network to generate a compact embedding vector; The network training module 400 is used to train the dual-branch ResNet-18 network using triplet loss to construct an embedding vector space; The gesture recognition module 500 is used to classify the gesture to be recognized into preset gestures and non-preset gestures based on a distance threshold of vectors in the embedded vector space, and to apply a method for the preset gestures. k The nearest neighbor classification method completes the category classification. Unpredictable gestures are first identified as outlier samples, and then clustering algorithms are used to classify them into different unknown gesture categories.

[0071] The system described above is used to implement the open set gesture recognition method based on FMCW radar in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0072] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the open set gesture recognition method based on FMCW radar as described in any of the above embodiments.

[0073] Figure 7 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 610, a memory 620, an input / output interface 630, a communication interface 640, and a bus 650. The processor 610, memory 620, input / output interface 630, and communication interface 640 are interconnected internally via the bus 650.

[0074] The processor 610 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0075] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 620 and is called and executed by the processor 610.

[0076] The input / output interface 630 is used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0077] The communication interface 640 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0078] Bus 650 includes a pathway for transmitting information between various components of the device, such as processor 610, memory 620, input / output interface 630, and communication interface 640.

[0079] It should be noted that although the above-described device only shows the processor 610, memory 620, input / output interface 630, communication interface 640, and bus 650, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0080] The electronic devices described above are used to implement the open-set gesture recognition method based on FMCW radar in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0081] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the open set gesture recognition method based on FMCW radar as described in any of the above embodiments.

[0082] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0083] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the open set gesture recognition method based on FMCW radar as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0084] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention is limited to these examples; within the framework of the invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity.

[0085] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of the invention, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of the invention, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of the invention will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of the invention, it will be apparent to those skilled in the art that the embodiments of the invention may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0086] Although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., DRAM) may use the embodiments discussed.

[0087] The embodiments of this invention are intended to cover all such substitutions, modifications, and variations falling within the scope of the claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this invention should be included within the scope of protection of this invention.

Claims

1. An open-set gesture recognition method based on FMCW radar, characterized in that, include: Obtain the raw gesture data collected by the FMCW radar to obtain the raw radar signal; Static clutter filtering is performed on the original radar signal. Two-dimensional Fourier transform and short-time Fourier transform are then performed on the clutter-filtered signal to extract range-Doppler features and time-frequency features. The distance-Doppler features and the time-frequency features are fused using a dual-branch ResNet-18 network to generate a compact embedding vector; The dual-branch ResNet-18 network is trained using triplet loss to construct an embedding vector space; Based on the distance threshold of vectors in the embedded vector space, the gestures to be recognized are divided into preset gestures and non-preset gestures. For preset gestures, the following is applied... k The nearest neighbor classification method completes the category classification. Unpredictable gestures are first identified as outlier samples, and then clustering algorithms are used to classify them into different unknown gesture categories.

2. The open-set gesture recognition method based on FMCW radar according to claim 1, characterized in that, The static clutter filtering employs a phasor mean cancellation algorithm, specifically: Subtracting the average value of all pulses within the same distance cell from the data of a single pulse within a frame at that distance cell eliminates static data that does not change over time, thus filtering out static clutter.

3. The open-set gesture recognition method based on FMCW radar according to claim 1, characterized in that, The formula for calculating the two-dimensional Fourier transform of the signal after filtering out clutter is as follows: ； In the formula, These are the eigenvalues ​​of the distance-Doppler feature map. This is the two-dimensional data matrix of the original radar signal after static clutter filtering. For the transform index of the Doppler dimension, For the transformation index of the distance dimension, For the sampling index of the Doppler dimension, For the sampling index of the distance dimension, The number of sampling points for the distance. The number of sampling points in the Doppler dimension. The imaginary unit, This is the modulus operation for complex numbers.

4. The open-set gesture recognition method based on FMCW radar according to claim 1, characterized in that, The formula for calculating the short-time Fourier transform of the signal after filtering out clutter is as follows: ； In the formula, These are the time-frequency eigenvalues ​​after the short-time Fourier transform. For frequency point indices in the frequency domain, For time frame indexing, This refers to the radar one-dimensional signal data after static clutter filtering and one-dimensional Fourier transform. For the time sampling point index of the signal, This represents the total number of time sampling points for the one-dimensional signal. For length is The sliding window function, This is the length of the window function in the short-time Fourier transform. It is the imaginary unit.

5. The open-set gesture recognition method based on FMCW radar according to claim 1, characterized in that, During the training phase of the dual-branch ResNet-18 network, human behavior feature maps are introduced as cross-domain data, which together with the feature maps of the original gesture data to construct a cross-domain dataset, expanding the types of datasets for network training. The feature fusion process of the distance-Doppler features and the time-frequency features using a dual-branch ResNet-18 network is as follows: One branch processes the time-frequency features and generates the corresponding time-frequency feature vector, while the other branch processes the distance-Doppler features and generates the corresponding distance-Doppler feature vector. The two feature vectors are then merged in a serial splicing manner to form the compact embedding vector.

6. The open-set gesture recognition method based on FMCW radar according to claim 1, characterized in that, The specific method for training the dual-branch ResNet-18 network using triplet loss is as follows: Define anchor samples, positive samples of the same class as anchor samples, and negative samples of different classes from anchor samples. Training triples are composed of anchor samples, positive samples, and negative samples. During training, the vector distance between anchor point samples and positive samples, and the vector distance between anchor point samples and negative samples are kept at a preset margin. The feature distribution in the embedding vector space is optimized to achieve compact clustering of similar gesture features and widening of the spacing between dissimilar gesture features.

7. The open-set gesture recognition method based on FMCW radar according to claim 1, characterized in that, Based on the distance threshold of vectors in the embedded vector space, the judgment expression for classifying the gesture to be recognized into preset gestures and non-preset gestures is as follows: ； In the formula, The preset distance threshold, Gesture sample to be recognized With the Individual gesture category samples Euclidean distance in the embedded space The dimension of the embedded vector. Gesture sample to be recognized The dimensional vector value, Samples of preset gesture categories The Dimensional vector value.

8. The open-set gesture recognition method based on FMCW radar according to claim 1, characterized in that, Use preset gestures k The specific process of nearest neighbor classification to complete category classification is as follows: Calculate the vector distance between the preset gesture sample to be classified and each preset category sample in the embedding vector space, and find the closest one. k A preset category of samples, based on this k Each sample votes on its category, and the final category of the preset gesture sample to be classified is determined by the voting results.

9. The open-set gesture recognition method based on FMCW radar according to claim 8, characterized in that, The clustering algorithm is the density-based noisy applied spatial clustering algorithm DBSCAN, mathematically expressed as: ； In the formula, For sample points of Neighborhood sample set This is the set of samples in the embedding space for gesture samples that are not pre-defined. , outer set of samples Any two sample vectors in the dataset, For sample points and vector distance, The neighborhood radius of the DBSCAN algorithm. To adjust the scaling factor of clustering, This is the distance threshold in open set recognition; The specific process of using the DBSCAN algorithm to cluster unpreset gestures is as follows: By examining the sample density of the outer set sample point in its ε-neighborhood, we divide the set of all sample points whose distance to the sample point is within the radius of the neighborhood, and group the sample points that satisfy the neighborhood reachability condition into the same set. Each independent set of sample points is an unknown gesture category.

10. An open-set gesture recognition device based on FMCW radar, employing the open-set gesture recognition method based on FMCW radar as described in any one of claims 1 to 9, characterized in that, include: The data acquisition module is used to acquire the raw gesture data collected by the FMCW radar to obtain the raw radar signal; The feature extraction module is used to perform static clutter filtering on the original radar signal, and to perform two-dimensional Fourier transform and short-time Fourier transform on the signal after clutter filtering to extract range-Doppler features and time-frequency features. The feature fusion module is used to fuse the distance-Doppler features and the time-frequency features through a dual-branch ResNet-18 network to generate a compact embedding vector; The network training module is used to train the dual-branch ResNet-18 network using triplet loss to construct an embedding vector space; The gesture recognition module is used to classify the gesture to be recognized into preset gestures and non-preset gestures based on a distance threshold of vectors in the embedded vector space, and to apply a method for the preset gestures. k The nearest neighbor classification method completes the category classification. Unpredictable gestures are first identified as outlier samples, and then clustering algorithms are used to classify them into different unknown gesture categories.

Images