A millimeter wave radar gesture recognition method and device, electronic equipment and medium

Abstract

The application provides a millimeter wave radar gesture recognition method and device for multiple mobile modes, electronic equipment and medium. The method comprises the following steps: after obtaining a gesture point cloud sequence, a spatial Laplace matrix is obtained by constructing a spatial graph based on k-nearest neighbors in each frame, and a time Laplace matrix is obtained by constructing a chain time graph and expanding between adjacent frames; a disturbance constrained by a space-time structure is applied to the point cloud sequence based on the two Laplace matrices, and an enhanced point cloud sequence is generated; a self-supervised pre-training is performed on an encoder-decoder network by using the enhanced data; a domain adversarial fine-tuning is performed by reserving the encoder and combining a small amount of real labeled data, mobile mode related information is stripped through a gradient reversal layer, a mobile decoupled gesture feature representation is obtained to realize gesture recognition. Under the condition of only a small amount of real labeled samples, the application significantly improves the accuracy and robustness of gesture recognition under multiple mobile modes, and can be applied to mobile interaction scenes such as smart home and virtual reality.

Description

Technical Field

[0001] This invention relates to the field of millimeter-wave radar sensing technology, and in particular to a millimeter-wave radar gesture recognition method, device, electronic device, and storage medium for various movement modes. Background Technology

[0002] Millimeter-wave radar has shown great application potential in the field of gesture recognition in recent years due to its non-contact sensing capabilities, robustness to lighting conditions, and privacy protection features stemming from its non-image-collecting nature. In scenarios such as smart home control and virtual reality interaction, millimeter-wave radar-based gesture recognition is considered a promising human-computer interaction method.

[0003] Most current millimeter-wave radar gesture recognition methods are designed with the user in a stationary state in mind, or only consider a single, fixed movement pattern. However, in actual use, it is common for users to perform gesture interactions while moving, such as adjusting smart home devices while walking. When the user switches from a stationary state to walking, or changes their walking direction and speed, the recognition performance of existing methods often drops significantly. How to maintain the stability of gesture recognition under multiple movement modes has become a technically significant problem in this field.

[0004] To address the aforementioned issues, several technical approaches have been explored in the industry. One type of method employs supervised learning to model the distribution of gesture features. The recognition performance of this type of method is highly dependent on the training data acquisition paradigm. Training data is typically acquired under specific movement conditions, and when the user's movement pattern changes, the difference in distribution between the test data and the training data can lead to a significant decline in recognition accuracy.

[0005] Another approach incorporates transfer learning strategies, attempting to improve the model's adaptability to different users, environments, or locations through fine-tuning with a small number of target domain samples. While transfer learning mitigates the impact of data distribution shifts to some extent, in scenarios with multiple movement modes, different movement methods introduce significant spatiotemporal distribution shifts. Radar point cloud data contains substantial body motion interference mixed with relatively subtle gesture features. Superficial fine-tuning with only a small number of samples is usually insufficient to effectively bridge the deep feature differences between different movement modes.

[0006] Another type of approach addresses the issue from the signal processing level, compensating for interference caused by the device's own movement. For example, it improves signal quality by eliminating Doppler shift caused by sensor movement. This type of method primarily focuses on compensating for the motion effects of the device itself. It lacks systematic modeling and processing of the complex changes that occur during free human movement—such as torso swaying during walking and the coupling effect of arm swinging on hand movements. Therefore, its adaptability to gesture recognition tasks with multiple movement modes remains insufficient.

[0007] Overall, existing technologies for millimeter-wave radar gesture recognition across multiple mobility modes generally face challenges such as insufficient generalization ability across mobility modes and a high degree of dependence on large-scale labeled data. Given the high cost of acquiring real labeled samples and the difficulty in covering all combinations of mobility modes, achieving stable gesture recognition across multiple mobility modes using limited labeled data remains a key technical challenge to be solved. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, this invention provides a millimeter-wave radar gesture recognition method, device, electronic device, and storage medium for multiple mobility modes. By constructing a motion-robust feature learning and recognition mechanism, and combining large-scale augmented data with a small amount of labeled data, this invention achieves stable gesture recognition across multiple mobility modes, effectively solving the problem of performance degradation in complex mobility scenarios of existing methods. It can be applied to interactive scenarios such as smart homes and virtual reality.

[0009] The technical solution adopted by this invention to solve its technical problem is:

[0010] According to one aspect of this application, a millimeter-wave radar gesture recognition method for multiple mobility modes is provided, including:

[0011] Acquire a gesture point cloud sequence collected by millimeter-wave radar, the point cloud sequence containing T frames of point cloud, each frame of point cloud consisting of multiple three-dimensional points;

[0012] Constructing a spatiotemporal topology based on the point cloud sequence includes: constructing a spatial graph within each frame of point cloud based on the k-nearest neighbor algorithm to obtain a spatial Laplacian matrix describing the local geometric relationships within the frame; constructing a chained temporal graph between adjacent frames and extending the frame-level temporal relationships to the point level to obtain a temporal Laplacian matrix.

[0013] Based on the spatial Laplacian matrix and the temporal Laplacian matrix, a perturbation constrained by spatiotemporal structure is applied to the point cloud sequence to generate an enhanced point cloud sequence that balances intra-frame topology preservation and inter-frame continuity.

[0014] The encoder-decoder network is pre-trained in a self-supervised manner using the enhanced point cloud sequence. The encoder is used to extract the spatiotemporal features of the point cloud sequence, and the decoder is used to reconstruct the point cloud based on the spatiotemporal features. During the pre-training process, the consistency between the reconstructed point cloud and the input point cloud is used as a constraint to learn a gesture representation that is insensitive to changes in movement patterns.

[0015] The pre-trained encoder is used as a feature extractor. The feature extractor, gesture classifier and mobile domain discriminator are jointly fine-tuned by combining real labeled data. The real labeled data includes gesture labels and mobile domain labels. During the fine-tuning process, adversarial training is performed through a gradient inversion layer to obtain a gesture feature representation that is decoupled from movement.

[0016] Gesture recognition is performed on the gesture point cloud sequence to be recognized based on the gesture features.

[0017] In one optional implementation, constructing a spatial graph based on the k-nearest neighbor algorithm within each frame of the point cloud includes:

[0018] For the Any point in a frame point cloud Determine its k nearest neighbors in the current frame. Calculate the connection weights between the point and each nearest neighbor. :

[0019]

[0020] Where σ is a parameter that controls the width of the Gaussian kernel;

[0021] Construct the weighted adjacency matrix of the frame based on the connection weights. Sum-degree matrix And calculate the spatial Laplacian matrix of the frame. ;

[0022] Stacking the spatial Laplacian matrices of all frames diagonally yields the sequence-level spatial Laplacian matrix. .

[0023] In one optional implementation, extending the frame-level temporal relationship to the point level to obtain the temporal Laplace matrix includes:

[0024] Constructing the temporal adjacency matrix The connection weight between adjacent frames is 1, and the rest are 0;

[0025] Calculate the time degree matrix based on the time adjacency matrix. And calculate the frame-level temporal Laplacian matrix. ;

[0026] The time Laplacian matrix With size identity matrix Taking the Kronecker product, we obtain the point-level time Laplacian matrix. Each frame contains 100 points. .

[0027] In one optional implementation, a perturbation constrained by spatiotemporal structure is applied to the point cloud sequence to generate an enhanced point cloud sequence that balances intra-frame topology preservation and inter-frame continuity, including:

[0028] For a point in frame t Calculate its Laplace difference relative to each point in its spatial neighborhood. ;

[0029] The adaptive perturbation coefficient is determined based on the Laplace difference. And apply Gaussian noise The initial perturbation point is obtained:

[0030]

[0031] By applying joint spatiotemporal regularization constraints to the initial perturbation points of each frame, the enhanced point cloud sequence is obtained by solving the following optimization problem. :

[0032]

[0033] in The sequence formed by the initial perturbation points. and Let be the spatial regularization coefficient and the temporal regularization coefficient, respectively, and let their values ​​satisfy . This prioritizes ensuring the smoothness of inter-frame timing while maintaining spatial structure.

[0034] In one alternative implementation, the optimization problem is solved using closed-form equations:

[0035]

[0036] in, It is an identity matrix.

[0037] In one alternative implementation, the encoder-decoder network includes:

[0038] The encoder selects anchor points by sampling from the farthest point, constructs a spatiotemporal neighborhood across frames for each anchor point, encodes the relative offset within the neighborhood through a shared multilayer perceptron, and after feature aggregation at multiple levels, models the global spatiotemporal dependency using a multi-head self-attention mechanism.

[0039] The decoder, corresponding to the encoder's layers, progressively upsamples coarse-grained features to the original resolution through nearest-neighbor interpolation. It then performs skip connections and fusion of features from the corresponding encoder layers at each layer, and finally reconstructs the three-dimensional coordinates of the point cloud through the output head.

[0040] In one optional implementation, during the joint fine-tuning process, the training loss of the gesture classifier is the cross-entropy loss of the gesture labels, the training loss of the mobile domain discriminator is the cross-entropy loss of the mobile domain labels, and the adversarial training loss of the feature extractor is the negative of the cross-entropy loss of the mobile domain labels. A gradient inversion layer forces the feature extractor to optimize in the direction of maximizing the loss of the mobile domain discriminator. The strength of the adversarial training is determined by a coefficient. control, During the fine-tuning process, a scheduling strategy that monotonically increases with the training progress is adopted, gradually increasing from the initial value to the preset upper limit.

[0041] According to another aspect of this application, the present invention provides a millimeter-wave radar gesture recognition device for multiple mobility modes, comprising:

[0042] The point cloud acquisition module is used to acquire the gesture point cloud sequence collected by millimeter-wave radar. The point cloud sequence contains multiple frames of point cloud, and each frame of point cloud is composed of multiple three-dimensional points.

[0043] The spatiotemporal topology enhancement module includes: a spatial topology modeling unit, used to construct a spatial graph within each frame's point cloud based on the k-nearest neighbor algorithm, obtaining a spatial Laplacian matrix describing the local geometric relationships within the frame; a temporal correlation modeling unit, used to construct a chained temporal graph between adjacent frames and extend the frame-level temporal relationships to the point level, obtaining a temporal Laplacian matrix; and a spatiotemporal manifold perturbation unit, used to apply a perturbation constrained by spatiotemporal structure to the point cloud sequence based on the spatial Laplacian matrix and the temporal Laplacian matrix, generating an enhanced point cloud sequence that balances intra-frame topology preservation and inter-frame continuity.

[0044] The pre-training module is used to perform self-supervised pre-training on the encoder-decoder network using the enhanced point cloud sequence. The encoder is used to extract the spatiotemporal features of the point cloud sequence, and the decoder is used to reconstruct the point cloud based on the spatiotemporal features. During the pre-training process, the consistency between the reconstructed point cloud and the input point cloud is used as a constraint to learn gesture representations that are insensitive to changes in movement patterns.

[0045] The domain adaptation fine-tuning module includes a gesture classifier and a mobile domain discriminator. It is used to use the pre-trained encoder as a feature extractor and to jointly fine-tune the feature extractor, gesture classifier, and mobile domain discriminator with real labeled data. The real labeled data includes gesture labels and mobile domain labels. During the fine-tuning process, adversarial training is performed through a gradient inversion layer to suppress the movement pattern related information contained in the features extracted by the feature extractor, so as to obtain a movement-decoupled gesture feature representation.

[0046] The gesture recognition and output module is used to perform gesture recognition on the gesture point cloud sequence to be recognized based on the gesture feature representation, and output the recognition result.

[0047] According to another aspect of this application, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the above-described method.

[0048] According to another aspect of this application, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the above-described method.

[0049] The positive effects of this invention:

[0050] This invention proposes a millimeter-wave radar gesture recognition scheme for multiple mobility modes (mobile domain). Its core idea is to achieve stable gesture recognition across multiple mobility modes through spatiotemporal topology-preserving data augmentation and adversarial fine-tuning with motion decoupling, requiring only a minimal number of real-world labeled samples. Compared with existing technologies, this scheme demonstrates significant advantages in the following aspects:

[0051] (1) In the data augmentation stage, this application does not use conventional random translation, rotation, or noise addition methods. Instead, it introduces joint constraints of spatial and temporal maps to maintain the local geometric structure of the point cloud in a single frame and the temporal continuity between consecutive frames during the generation of augmented samples. This approach makes the augmented data closer to the real physical scene, avoiding structural damage that may be caused by random perturbations and preventing the loss of details due to excessive smoothing. Experimental results show that the average Hausdorff distance between the generated augmented data and the real data is 0.57 meters, and the 1-nearest neighbor accuracy reaches 0.61, which is close to the ideal value of 0.5, indicating that the augmented samples have good consistency with the real data in terms of distribution.

[0052] (2) In the design of the recognition network, this application adopts a two-stage framework of self-supervised pre-training plus adversarial fine-tuning. In the pre-training stage, large-scale augmented data is used to enable the encoder to learn a gesture spatiotemporal representation that is insensitive to changes in movement patterns. In the fine-tuning stage, a small amount of real labeled data is used to further remove residual movement pattern-related information from the features through adversarial training between the gesture classifier and the movement domain discriminator. This design enables the model to achieve stable recognition performance even with a small number of samples. Experimental data show that only 10 real samples are needed for fine-tuning for each movement pattern to achieve a gesture recognition accuracy of 90.57%; when the number of samples increases to 20, the accuracy increases to 96.24%; and with 30 samples, it further reaches 98.78%. This provides a feasible solution to the problems of high labeling costs and difficult data collection in actual deployment.

[0053] (3) Regarding cross-domain generalization ability, the model obtained in this application shows good robustness in the face of user differences, radar position changes, movement speed changes, and different scene switching. The average recognition accuracy is 96.62% under different user and radar position conditions, 97.96% under different movement speeds, and 96.49% in different scenes. This shows that the feature representation after motion decoupling processing effectively suppresses environmental and motion interference factors that are not related to gesture semantics.

[0054] (4) From the perspective of system implementation, this application organically integrates data augmentation, self-supervised pre-training, and adversarial domain adaptation into a complete processing flow, with clear functions and close connections between each module. The augmenter is responsible for increasing the amount of data at the data level, the pre-training is responsible for extracting basis representations that are insensitive to movement changes at the feature level, and the fine-tuning is responsible for decoupling domain-related features at the task level. This hierarchical design gives the entire solution certain expansion potential when facing unknown movement patterns, and also provides technical accumulation for the practical application of millimeter-wave radar in mobile interaction scenarios such as smart homes and virtual reality.

[0055] In summary, this application effectively improves the accuracy and stability of millimeter-wave radar gesture recognition in various movement modes while reducing reliance on large-scale real-world labeled data, and effectively solves the problem of significant performance degradation of existing methods in complex movement scenarios. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0057] Figure 1 This is a schematic diagram of the overall system architecture provided in the embodiments of the present invention;

[0058] Figure 2 This is a schematic diagram of the gesture types provided in the embodiments of the present invention;

[0059] Figure 3 This is a schematic diagram of the movement mode provided in an embodiment of the present invention;

[0060] Figure 4 This is a curve showing the change in recognition accuracy as a function of the number of fine-tuned samples, provided in an embodiment of the present invention.

[0061] Figure 5 This is a block diagram of the gesture recognition device provided in an embodiment of the present invention. Detailed Implementation

[0062] The prior art and the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed description. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0064] This invention employs a millimeter-wave radar gesture recognition method for multiple mobility modes. This method can stably and accurately recognize the gestures performed by the user when the user is in different mobility states.

[0065] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0066] Reference Figure 1 , Figure 1 The diagram illustrates a system architecture of a millimeter-wave radar gesture recognition method for multiple mobility modes according to an embodiment of the present invention. It mainly includes two parts: a spatiotemporal topology preservation enhancer and a mobility-robust gesture recognition network.

[0067] The spatiotemporal topology-preserving enhancer comprises three sub-modules: a spatial topology modeling module, a temporal correlation modeling module, and a spatiotemporal manifold perturbation module. Considering the sparse, irregular, and easily disturbed characteristics of millimeter-wave radar point clouds in moving scenes, and the smooth temporal evolution of gesture actions between consecutive frames, this application first constructs a k-nearest neighbor-based graph structure within each frame of the point cloud using the spatial topology modeling module to characterize the local geometric relationships between points. Subsequently, the temporal correlation modeling module constructs a chain-like temporal graph between adjacent frames to describe the cross-frame motion continuity during gesture execution. Based on this, a spatiotemporal manifold perturbation module is designed to apply perturbations constrained by the spatiotemporal structure to the point cloud sequence, generating an enhanced point cloud sequence that balances intra-frame topology preservation and inter-frame continuity. The resulting enhanced data exhibits good physical rationality and diversity, and can be used for pre-training of subsequent recognition networks.

[0068] The mobile-robust gesture recognition network comprises a motion-invariant representation pre-training module and a mobility decoupling domain adaptation module. In the motion-invariant representation pre-training module, self-supervised learning is performed using augmented millimeter-wave radar point cloud sequences, constructing a hierarchical encoder-decoder structure. The encoder aggregates the local geometry and short-term motion information of the point cloud layer by layer through farthest-point sampling, spatiotemporal neighborhood construction, and local spatiotemporal embedding, and then models global spatiotemporal dependencies using a multi-head self-attention mechanism. The decoder recovers the point cloud representation at the original resolution through layer-by-layer feature propagation and interpolation, and uses reconstruction loss to constrain the consistency between the predicted point cloud and the real point cloud, thereby learning gesture representations insensitive to changes in movement patterns. The amount of augmented data used for pre-training is three times the amount of generated data.

[0069] In the mobility decoupling domain adaptation module, the pre-trained encoder is retained as a feature extractor and fine-tuned using a small amount of real-world labeled data. Specifically, a gesture classifier, composed of two multilayer perceptrons, is used to extract discriminative semantic features of gestures. The mobility domain discriminator predicts the movement mode of a sample; the discriminator and classifier are jointly trained with a 1:1 training step ratio. Adversarial training is implemented through a gradient inversion layer, enabling the encoder to suppress movement mode-related features and retain essential semantic features of gestures during optimization, thus obtaining a representation that is both discriminative and motion-robust. Finally, stable gesture recognition across multiple movement modes is achieved using this robust representation.

[0070] It should be noted that the following is one possible sequence of steps, and it is not strictly required to follow this order. Some steps can be performed in parallel without interdependence.

[0071] Specifically, according to an embodiment of the gesture recognition method based on heterogeneous graphs of this application, the method includes:

[0072] S101. Acquire a gesture point cloud sequence collected by millimeter-wave radar, wherein the point cloud sequence contains T frames of point cloud, and each frame of point cloud consists of multiple three-dimensional points.

[0073] In this embodiment, the millimeter-wave radar uses a frequency-modulated continuous wave radar sensor, specifically the TI IWR1443BOOST. The operating frame rate is set to 10fps, with each frame containing 128 chirp signals. A 256-point Fast Fourier Transform (FFT) is used for the range dimension, and a 128-point FFT is used for the Doppler dimension. The system has a range resolution of 3.75cm, a velocity resolution of 0.0163m / s, a maximum detection range of 2.5m, and a maximum detectable radial velocity of 4.16m / s.

[0074] The raw millimeter-wave radar intermediate frequency signal is first subjected to a two-dimensional fast Fourier transform to construct the range-Doppler spectrum. Then, a constant false alarm rate (CFAR) algorithm is used to suppress background noise and select effective target points. Finally, spatial angle information is recovered using an angle-of-arrival estimation method to generate a three-dimensional sparse millimeter-wave radar point cloud. The above signal processing flow is implemented in the Matlab environment and stored uniformly as a ".mat" format data file.

[0075] During the data acquisition process, the radar is positioned at the origin. It should be noted that in this embodiment, the raw point cloud output by the radar is not filtered, normalized, or cropped for gesture region detection; the raw point cloud output by the radar board is used directly. Millimeter-wave radar point clouds are typically sparse and sensitive to environmental conditions, while human movements exhibit smooth and continuous dynamic characteristics even when the user is moving.

[0076] The point cloud sequence contains T frames of point clouds. The time dimension is fixed: the point cloud sampling rate is 10 frames per second. Statistically, each gesture lasts approximately 1 to 2 seconds, with a maximum of 3 seconds. Therefore, the time window is uniformly set to 30 frames, with sequences shorter than 30 frames padded with zeros. Based on the settings of the millimeter-wave radar board used, each frame can output a maximum of 64 points, i.e., the number of points contained in each frame. It is 64.

[0077] The data collection for the real dataset was as follows: Ten users were recruited, seven males and three females, aged 23 to 29, with heights ranging from 162 to 183 cm and weights from 50 to 100 kg. Users performed gestures while moving within approximately 3 meters of the millimeter-wave radar. Five types of gestures were set, such as... Figure 2 As shown, the gestures are pulling, pushing, drawing circles, raising, and tapping, each lasting about 1 to 2 seconds.

[0078] In this embodiment, seven movement modes are set, meaning the user can perform the above-mentioned gesture actions while walking in different directions at a normal walking speed. The seven movement directions are as follows: Figure 3 As shown.

[0079] Each movement pattern was performed 10 times for each type of gesture. The final real-world movement data size was: 7 movement patterns × 10 gesture executions × 5 types of gestures × 10 volunteers = 3500 samples.

[0080] S102. Constructing a spatiotemporal topology based on the point cloud sequence includes: constructing a spatial graph within each frame of point cloud based on the k-nearest neighbor algorithm to obtain a spatial Laplacian matrix describing the local geometric relationships within the frame; constructing a chained temporal graph between adjacent frames and extending the frame-level temporal relationships to the point level to obtain a temporal Laplacian matrix.

[0081] Specifically, constructing a spatial graph based on the k-nearest neighbor algorithm within each frame of the point cloud includes:

[0082] Will contain T frames and each frame has The initial gesture sequence for each point is represented as follows:

[0083]

[0084] Among them, the first The point set of the frame is:

[0085]

[0086] Within each frame, a spatial graph is constructed based on the k-nearest neighbor algorithm. , where the node set Edge set and weighted adjacency matrix These represent points in the point cloud, local neighborhood connectivity, and pairwise similarity between points, respectively.

[0087] For any two points Its connection weight The calculation formula is as follows:

[0088]

[0089] in, Point The set of k nearest neighbors, where σ is a parameter controlling the width of the Gaussian kernel.

[0090] The neighborhood size *k* controls the scope of local topology modeling, and the Gaussian kernel width *σ* controls the decay rate of neighboring edge weights with distance between points. In this embodiment, the parameters were empirically searched on the validation set. The candidate range for *k* was 4 to 12, and the candidate range for *σ* was 0.1m to 0.5m. Ultimately, *k*=8 and *σ*=0.25m were set. When *k* or *σ* is too small, the graph structure is more sensitive to local noise, resulting in insufficient stability of the enhancement results; when the values ​​are too large, excessive smoothing is easily introduced, weakening local geometric details and action distinguishability.

[0091] Based on the above connection weights, the weighted adjacency matrix of each frame is obtained. and define the degree matrix. The calculation formula is as follows:

[0092]

[0093] Each diagonal element of the degree matrix represents the total connectivity strength between a node and all its neighbors. The standard graph Laplacian operator is computed for each frame:

[0094]

[0095] This operator captures the local structure of the point cloud and serves as the basis for subsequent Laplacian difference calculations and topology-preserving regularization. To maintain intra-frame topological consistency and constrain geometric deformation, all frame-by-frame spatial Laplacian operators are stacked diagonally to form a sequence-level spatial Laplacian matrix:

[0096]

[0097] This spatial map captures relative geometric relationships that are insensitive to global offsets and environment-dependent distortions, enabling augmented perturbations to adapt to different perceptual environments while preserving the geometric features of gestures.

[0098] Furthermore, the extension of the frame-level temporal relationship to the point level to obtain the time Laplace matrix specifically includes:

[0099] Building upon spatial topology modeling, temporal correlation is introduced to ensure consistency between consecutive frames. The temporal dimension is represented as a one-dimensional chain graph, where each frame is directly connected to its neighboring frames to limit abrupt changes between frames. A temporal adjacency matrix is ​​constructed. The connection weight between adjacent frames is 1, and the rest are 0, i.e.:

[0100]

[0101] Calculate the time degree matrix based on the time adjacency matrix. :

[0102]

[0103] Calculate the frame-level temporal Laplacian matrix:

[0104]

[0105] To extend frame-level temporal modeling to the point level, the time Laplacian matrix is... With size identity matrix Take the Kronecker product to map the temporal relationships to all nodes:

[0106]

[0107] Each frame contains the following number of points: This point-level operator enforces temporal continuity and suppresses abrupt inter-frame changes.

[0108] S103. Based on the spatial Laplacian matrix and the temporal Laplacian matrix, apply a perturbation constrained by spatiotemporal structure to the point cloud sequence to generate an enhanced point cloud sequence that balances intra-frame topology preservation and inter-frame continuity.

[0109] Specifically, applying a perturbation constrained by spatiotemporal structure to the point cloud sequence to generate an enhanced point cloud sequence that balances intra-frame topology preservation and inter-frame continuity includes:

[0110] Based on the spatial Laplace matrix and time Laplace matrix This step aims to generate enhanced samples that preserve spatial and temporal continuity. Random perturbations can disrupt structural integrity, while excessive smoothing tends to obscure discernible details.

[0111] First, calculate the Laplace difference on the spatial map, using the measurement points. The degree of deviation from its neighborhood:

[0112]

[0113] in, Indicates the first Midpoint of frame The spatial neighborhood.

[0114] Subsequently, the adaptive perturbation coefficients are determined based on the Laplace difference. And apply Gaussian noise The initial perturbation point is obtained:

[0115]

[0116] Among them, the adaptive perturbation coefficient It is not a manually set fixed constant, but rather adaptively determined based on the degree of deviation of the local structure at a point. Specifically, it is based on the magnitude of the local Laplace difference. This measures the degree of deviation of a point from its neighborhood and scales the perturbation intensity accordingly: points with greater local deviation are assigned larger perturbation amplitudes, while points with smaller local deviations are only subject to weaker perturbations.

[0117] Gaussian noise It follows a zero-mean Gaussian distribution with its variance limited to a small range and set relative to the spatial scale of the point cloud. The noise intensity occupies only a small fraction of the point cloud's spatial range to ensure that only minor local perturbations are introduced without disrupting the original topology and temporal continuity. Positive It amplifies local geometric changes, while negative values ​​promote smoothing.

[0118] Since the perturbations mentioned above are applied independently to each frame, they may introduce temporal inconsistencies. Therefore, a joint spatiotemporal regularization constraint is applied to the initial perturbation points of each frame, and the following optimization problem is solved to obtain the enhanced point cloud sequence. :

[0119]

[0120] in, The sequence formed by the initial perturbation points. and These are the spatial regularization coefficient and the temporal regularization coefficient, respectively. The first term in the formula is used to preserve the initial perturbation, and the second term is used to... Enforcing the coherence of intra-frame structure, the third item is passed Punish sudden inter-frame variations. The smoothing energy term encourages coherent deformation between connected nodes.

[0121] This embodiment performs an empirical search on the validation set and finally sets... = 0.1、 = 0.2, and the value satisfies < This prioritizes ensuring the smoothness of inter-frame timing while maintaining spatial structure.

[0122] because and It is a symmetric positive semi-definite matrix, and the objective function is a strictly convex quadratic problem with a unique global minimum. Setting its derivative to zero, the optimization problem is solved using closed-form equations:

[0123]

[0124] in, It is the size of × The identity matrix. This optimization step ensures the final enhanced sequence. While introducing physically reasonable changes, global spatial fidelity and temporal smoothness are maintained.

[0125] Regarding the scale and strategy for generating augmented data: Three augmentation variants are generated for each real sample. During each augmentation process, the perturbation coefficients for each point are... Independent sampling, with signs uniformly selected from {+1, -1} and amplitudes uniformly sampled from the preset interval [0, 0.3]; Gaussian noise term. Independent sampling is performed using a zero-mean distribution.

[0126] S104. The encoder-decoder network is pre-trained in a self-supervised manner using the enhanced point cloud sequence. The encoder is used to extract the spatiotemporal features of the point cloud sequence, and the decoder is used to reconstruct the point cloud based on the spatiotemporal features. During the pre-training process, the consistency between the reconstructed point cloud and the input point cloud is used as a constraint to learn a gesture representation that is insensitive to changes in movement patterns.

[0127] Specifically, the encoder-decoder network includes:

[0128] The encoder employs a hierarchical structure to extract spatiotemporal features from millimeter-wave radar sequences. In the first... Each encoding level, the encoder independently applies the farthest point sampling (FPS) algorithm to select each frame. There are several anchor points. For each anchor point... Construct a spatiotemporal neighborhood by collecting the nearest points from the current frame and adjacent frames. Specifically, in this embodiment, adjacent frames refer to the frame before and the frame after the current frame, and the number of nearest points is set to 4. The relative offset within this neighborhood is encoded using a shared multilayer perceptron (MLP):

[0129]

[0130] This operation captures the local spatial geometry and short-term temporal evolution. The resulting features form a tensor:

[0131]

[0132] in and These are the number of frames and the number of anchor points, respectively. It is the embedded dimension.

[0133] In this embodiment, after three layers of layered aggregation, the parameters of each layer are set as follows: Layer 1 N1=32, C1=64; Layer 2 N2=16, C2=128; Layer 3 N3=8, C3=256.

[0134] The spatiotemporal features aggregated from the three layers are flattened and a potential global label is appended. Then, a multi-head self-attention mechanism is used to model the global spatiotemporal dependencies. The number of attention heads h=4, with a single-head dimension. =64, global feature dimension C3=256. The input token has a dimension of 256, and Q, K, and V are obtained through linear mapping, satisfying C3 = h × The formula for calculating attention is:

[0135]

[0136] They are obtained by linear projection of the features, respectively. For the attention dimension.

[0137] The decoder corresponds hierarchically to the encoder, progressively restoring features to their original spatial and temporal resolution in a top-down manner. The decoder comprises a three-layer recovery structure corresponding to the encoder: the... The decoding layer and the first The first coding layer forms a skip connection, and the first coding layer forms a skip connection. Coarse-grained features of layer upsampled to the th On the point set of layer -1.

[0138] At each decoding level, features are derived from a coarser set via three nearest neighbor interpolation. Upsampling to a denser point set For point , its in The three nearest neighbors The normalized inverse distance weights are calculated as follows:

[0139]

[0140] The feature propagation process is calculated as follows:

[0141]

[0142] in, Indicates the first hierarchical Features at the location. To recover fine-grained information, the propagated features are concatenated and fused channel-by-channel with the encoder features from the same level of skip connections:

[0143]

[0144] The fused representation is refined using a lightweight multilayer perceptron incorporating batch normalization and ReLU activation functions.

[0145]

[0146] After repeating the above process, the decoder eventually restores the original point resolution. Finally, a shared output head is used to perform coordinate regression on the features of each point. This output head consists of two 1×1 convolutional layers with batch normalization and ReLU activation functions inserted in between to predict the 3D coordinate offset point by point, ultimately outputting a reconstructed point cloud. .

[0147] For supervised pre-training, this application uses Earth Mover's Distance (EMD) to measure the predicted point set. Minimum matching cost between the true point set P_t and the actual point set:

[0148]

[0149] in The bijective function is used to ensure that each predicted point is precisely matched with a real point. Represents the prediction point set The number of points. This loss function enforces alignment between geometric fidelity and the global distribution.

[0150] S105. Using the pre-trained encoder as a feature extractor, the feature extractor, gesture classifier, and motion domain discriminator are jointly fine-tuned in conjunction with real labeled data. The real labeled data includes gesture labels and motion domain labels. During the fine-tuning process, adversarial training is performed through a gradient inversion layer to suppress motion mode-related information contained in the features extracted by the feature extractor, thereby obtaining a motion-decoupled gesture feature representation.

[0151] Specifically, after self-supervised pre-training, the encoder is retained. As a feature extractor, and utilizing a small number of real gesture-labeled features. and mobile domain tags sequence Fine-tuning is then performed. The encoder maps the input sequence to a latent representation. .

[0152] To supervise gesture recognition, a gesture classifier is used. Use the cross-entropy loss function to extract features with gesture discrimination:

[0153]

[0154] Because radar data exhibit drastically different motion distortions depending on the movement pattern, classifiers might use movement patterns as shortcuts. To prevent this, a movement domain discriminator is introduced. Predict the movement pattern of the underlying representation, whose parameters are obtained by minimizing the cross-entropy loss. Conduct training:

[0155]

[0156] To enforce the decoupled representation, an adversarial learning layer is employed using gradient inversion. The gradient inversion layer maintains computational invariance during forward propagation but multiplies the gradient by a negative scalar during backward propagation. This forces the encoder to maximize the discriminator's loss during the optimization process. The adversarial training loss of the feature extractor is the negative of the mobile domain label cross-entropy loss, i.e.:

[0157]

[0158] Therefore, the encoder learns to confuse the discriminator from the representation. Remove movement-related information. The joint optimization objective for the entire phase is:

[0159]

[0160] in, These are the parameters for the encoder, gesture classifier, and motion discriminator, respectively.

[0161] The intensity of the adversarial training is determined by a coefficient. control, During fine-tuning, a scheduling strategy that monotonically increases with training progress is employed, gradually increasing from the initial value to a preset upper limit. Specifically, A smooth scheduling strategy that monotonically increases with the training progress is adopted and only takes effect during the fine-tuning phase.

[0162] A sigmoid growth strategy based on training progress is adopted: at the start of fine-tuning, Initialize to 0; then gradually increase according to the current fine-tuning progress; approach the preset upper limit of 0.1 when the fine-tuning ends.

[0163] The learned feature representations possess both high-precision gesture discrimination and successful decoupling from movement patterns, enabling robust recognition across multiple unknown movement patterns with minimal fine-tuning using very few real samples.

[0164] S106. Based on the gesture feature representation, perform gesture recognition on the gesture point cloud sequence to be recognized, and output the recognition result.

[0165] Specifically, real samples that were not used in training are input into the fine-tuned model for testing. The model outputs the corresponding gesture recognition results based on the learned motion decoupling gesture feature representation.

[0166] The specific implementation process and experimental results of the method described in the embodiments of this application will be further explained below.

[0167] Overall methodology and process:

[0168] First, millimeter-wave radar is used to collect real point cloud sequences of user gestures under various movement modes, and signal processing is performed to obtain corresponding real samples. Then, based on the collected real samples, a spatial topology map and a temporal correlation map are constructed. Augmented point cloud data for various movement modes are generated through constrained perturbation and joint regularization optimization. Next, the generated augmented data is used for self-supervised pre-training to learn a spatiotemporal representation with movement invariance. Based on this, fine-tuning is performed using a small amount of real-collected labeled data, and the effectiveness of the few-sample adaptation is verified under different fine-tuning sample sizes. Finally, real samples not used in training are input into the fine-tuned model for testing, and the corresponding gesture recognition results are output.

[0169] Experimental setup and hyperparameters:

[0170] During the pre-training phase, a cosine annealing learning rate was used, the batch size was set to 32, and the number of iterations was 300. The quality of the augmented data and the real data was evaluated using Hausdorff distance and 1-nearest neighbor accuracy as metrics.

[0171] Experimental results:

[0172] (1) Enhance data quality assessment

[0173] Compared with the real data, the augmented data generated in this application has an average Hausdorff distance of 0.57 meters and a 1-nearest neighbor accuracy of 0.61, which is close to the ideal value of 0.5. This indicates that the generated augmented data has good consistency with the real data in terms of distribution.

[0174] (2) Performance of few-sample gesture recognition

[0175] Regarding gesture recognition accuracy, such as Figure 4 As shown, the recognition accuracy steadily improves with the increase in the number of fine-tuned samples. Each movement mode requires only 10 real samples to achieve an accuracy of 90.57%, 20 samples to achieve 96.24%, and 30 samples to achieve 98.78%.

[0176] (3) Comparison with existing methods

[0177] When fine-tuned with 30 samples for each gesture in each movement mode, the method of this application achieves an average recognition accuracy of 98.78% across all movement modes. Compared with existing methods, this is 6.63%, 8.07%, and 7% higher than mTransSee, Tesla-Rapture, and DGR-Net, respectively. The accuracies for the seven movement modes (d1 to d7) are 97.52%, 96.97%, 98.63%, 99.18%, 99.74%, 99.74%, and 99.18%, respectively.

[0178] (4) Cross-domain robustness assessment

[0179] In the cross-domain robustness evaluation, the average recognition accuracy of this application reached 96.62%, 97.96% and 96.49% in different user and radar positions, different movement speeds and different scenarios, respectively, indicating that the model has good robustness and generalization ability.

[0180] Figure 5 A block diagram is shown of a millimeter-wave radar gesture recognition device for multiple mobility modes according to an embodiment of this application. This device can be used to perform the steps in the foregoing method embodiments, such as... Figure 5 As shown, the device may include:

[0181] The point cloud acquisition module is used to acquire the gesture point cloud sequence collected by millimeter-wave radar. The point cloud sequence contains T frames of point cloud, and each frame of point cloud is composed of multiple three-dimensional points.

[0182] The spatiotemporal topology enhancement module is further divided into three processing units:

[0183] a. Spatial topology modeling unit, used to construct a spatial graph within each frame of point cloud based on the k-nearest neighbor algorithm, to obtain a spatial Laplacian matrix describing the local geometric relationships within the frame;

[0184] b. Temporal correlation modeling unit, used to construct a chained time graph between adjacent frames and extend the frame-level temporal relationship to the point level to obtain the temporal Laplacian matrix;

[0185] c. A spatiotemporal manifold perturbation unit, used to apply a perturbation constrained by spatiotemporal structure to the point cloud sequence based on the spatial Laplacian matrix and the temporal Laplacian matrix, to generate an enhanced point cloud sequence that takes into account both intra-frame topology preservation and inter-frame continuity;

[0186] The pre-training module is used to perform self-supervised pre-training on the encoder-decoder network using the enhanced point cloud sequence. The encoder is used to extract the spatiotemporal features of the point cloud sequence, and the decoder is used to reconstruct the point cloud based on the spatiotemporal features. During the pre-training process, the consistency between the reconstructed point cloud and the input point cloud is used as a constraint to learn gesture representations that are insensitive to changes in movement patterns.

[0187] The domain adaptation fine-tuning module includes a gesture classifier and a mobile domain discriminator. It is used to use the pre-trained encoder as a feature extractor and to jointly fine-tune the feature extractor, gesture classifier, and mobile domain discriminator with real labeled data. The real labeled data includes gesture labels and mobile domain labels. During the fine-tuning process, adversarial training is performed through a gradient inversion layer to suppress the movement pattern related information contained in the features extracted by the feature extractor, so as to obtain a movement-decoupled gesture feature representation.

[0188] The gesture recognition and output module is used to perform gesture recognition on the gesture point cloud sequence to be recognized based on the gesture feature representation, and output the recognition result.

[0189] It should be noted that, Figure 5 The illustrated device block diagram and the connections between its modules are merely illustrative and schematic representations. In actual deployment, these modules can be integrated into the same computing device or distributed across different hardware units. The coupling between modules can be direct electrical connection, communication bus connection, or data interaction via a network. For example, the point cloud acquisition module can be deployed inside a millimeter-wave radar sensor or on an edge computing node connected to it, while the pre-training module and domain adaptation fine-tuning module can run on a server or in the cloud. This application does not limit the specific hardware deployment method or the specific connections between the modules.

[0190] The specific methods by which each module performs operations in the above device embodiments have been described in detail in the foregoing method embodiments, and will not be repeated here.

[0191] An electronic device for gesture recognition according to an embodiment of this application may be a terminal and may include a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the electronic device provides computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a gesture recognition method. The display screen of the electronic device may be a liquid crystal display screen or an e-ink display screen. The input device of the electronic device may be a touch layer covering the display screen, or buttons, a trackball, or a touchpad provided on the casing of the electronic device, or an external keyboard, touchpad, or mouse, etc.

[0192] Those skilled in the art will understand that the structure of the above-described electronic device is only a partial structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than the aforementioned components, or combine certain components, or have different component arrangements.

[0193] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the gesture recognition method as described in the embodiments of this application.

[0194] In an exemplary embodiment, a storage medium is also provided, which, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the gesture recognition method in the embodiments of this application.

[0195] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the gesture recognition method in the embodiments of this application.

[0196] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0197] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0198] The above description is only a preferred embodiment of the present invention. It should be understood that the above description of the embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention, and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, etc. made within the idea and principle of the present invention should be included within the scope of protection of the present invention.

Claims

1. A millimeter-wave radar gesture recognition method for multiple mobility modes, characterized in that, include: Acquire a gesture point cloud sequence collected by millimeter-wave radar, the point cloud sequence containing T frames of point cloud, each frame of point cloud consisting of multiple three-dimensional points; Constructing a spatiotemporal topology based on the point cloud sequence includes: constructing a spatial graph within each frame of point cloud based on the k-nearest neighbor algorithm to obtain a spatial Laplacian matrix describing the local geometric relationships within the frame; constructing a chained temporal graph between adjacent frames and extending the frame-level temporal relationships to the point level to obtain a temporal Laplacian matrix. Based on the spatial Laplacian matrix and the temporal Laplacian matrix, a perturbation constrained by spatiotemporal structure is applied to the point cloud sequence to generate an enhanced point cloud sequence that balances intra-frame topology preservation and inter-frame continuity. The encoder-decoder network is pre-trained in a self-supervised manner using the enhanced point cloud sequence. The encoder is used to extract the spatiotemporal features of the point cloud sequence, and the decoder is used to reconstruct the point cloud based on the spatiotemporal features. During the pre-training process, the consistency between the reconstructed point cloud and the input point cloud is used as a constraint to learn a gesture representation that is insensitive to changes in movement patterns. The pre-trained encoder is used as a feature extractor. The feature extractor, gesture classifier and mobile domain discriminator are jointly fine-tuned by combining real labeled data. The real labeled data includes gesture labels and mobile domain labels. During the fine-tuning process, adversarial training is performed through a gradient inversion layer to obtain a gesture feature representation that is decoupled from movement. Gesture recognition is performed on the gesture point cloud sequence to be recognized based on the gesture features.

2. The millimeter-wave radar gesture recognition method for multiple mobility modes according to claim 1, characterized in that, The construction of a spatial graph based on the k-nearest neighbor algorithm within each frame of point cloud includes: For the Any point in a frame point cloud Determine its k nearest neighbors in the current frame. Calculate the connection weights between the point and each nearest neighbor. : ， Where σ is a parameter that controls the width of the Gaussian kernel; Construct the weighted adjacency matrix of the frame based on the connection weights. Sum-degree matrix And calculate the spatial Laplacian matrix of the frame. ; Stacking the spatial Laplacian matrices of all frames diagonally yields the sequence-level spatial Laplacian matrix. .

3. The millimeter-wave radar gesture recognition method for multiple mobility modes according to claim 1, characterized in that, Extending the frame-level temporal relationship to the point level to obtain the time Laplace matrix includes: Constructing the temporal adjacency matrix The connection weight between adjacent frames is 1, and the rest are 0; Calculate the time degree matrix based on the time adjacency matrix. And calculate the frame-level temporal Laplacian matrix. ; The time Laplacian matrix With size identity matrix Taking the Kronecker product, we obtain the point-level time Laplacian matrix. Each frame contains 100 points. .

4. The millimeter-wave radar gesture recognition method for multiple mobility modes according to claim 3, characterized in that, Applying a perturbation constrained by spatiotemporal structure to the point cloud sequence to generate an enhanced point cloud sequence that balances intra-frame topology preservation and inter-frame continuity includes: For a point in frame t Calculate its Laplace difference relative to each point in its spatial neighborhood. ; The adaptive perturbation coefficient is determined based on the Laplace difference. And apply Gaussian noise The initial perturbation point is obtained: ， By applying joint spatiotemporal regularization constraints to the initial perturbation points of each frame, the enhanced point cloud sequence is obtained by solving the following optimization problem. : ， in The sequence formed by the initial perturbation points. and Let be the spatial regularization coefficient and the temporal regularization coefficient, respectively, and let their values ​​satisfy . This prioritizes ensuring the smoothness of inter-frame timing while maintaining spatial structure.

5. A millimeter-wave radar gesture recognition method for multiple mobility modes according to claim 4, characterized in that, The optimization problem is solved using closed-loop equations: ， in, For size × The identity matrix.

6. The millimeter-wave radar gesture recognition method for multiple mobility modes according to claim 1, characterized in that, The encoder-decoder network includes: The encoder selects anchor points by sampling from the farthest point, constructs a spatiotemporal neighborhood across frames for each anchor point, encodes the relative offset within the neighborhood through a shared multilayer perceptron, and after feature aggregation at multiple levels, models the global spatiotemporal dependency using a multi-head self-attention mechanism. The decoder, corresponding to the encoder's layers, progressively upsamples coarse-grained features to the original resolution through nearest-neighbor interpolation. It then performs skip connections and fusion of features from the corresponding encoder layers at each layer, and finally reconstructs the three-dimensional coordinates of the point cloud through the output head.

7. The millimeter-wave radar gesture recognition method for multiple mobility modes according to claim 1, characterized in that: In the joint fine-tuning process, the training loss of the gesture classifier is the cross-entropy loss of the gesture labels, the training loss of the mobile domain discriminator is the cross-entropy loss of the mobile domain labels, and the adversarial training loss of the feature extractor is the negative value of the cross-entropy loss of the mobile domain labels. A gradient inversion layer forces the feature extractor to optimize in the direction of maximizing the loss of the mobile domain discriminator. The strength of the adversarial training is determined by a coefficient. control, During the fine-tuning process, a scheduling strategy that monotonically increases with the training progress is adopted, gradually increasing from the initial value to the preset upper limit.

8. A millimeter-wave radar gesture recognition device for multiple mobility modes, characterized in that: include: The point cloud acquisition module is used to acquire the gesture point cloud sequence collected by millimeter-wave radar. The point cloud sequence contains multiple frames of point cloud, and each frame of point cloud is composed of multiple three-dimensional points. The spatiotemporal topology enhancement module includes: a spatial topology modeling unit, used to construct a spatial graph within each frame's point cloud based on the k-nearest neighbor algorithm, obtaining a spatial Laplacian matrix describing the local geometric relationships within the frame; a temporal correlation modeling unit, used to construct a chained temporal graph between adjacent frames and extend the frame-level temporal relationships to the point level, obtaining a temporal Laplacian matrix; and a spatiotemporal manifold perturbation unit, used to apply a perturbation constrained by spatiotemporal structure to the point cloud sequence based on the spatial Laplacian matrix and the temporal Laplacian matrix, generating an enhanced point cloud sequence that balances intra-frame topology preservation and inter-frame continuity. The pre-training module is used to perform self-supervised pre-training on the encoder-decoder network using the enhanced point cloud sequence. The encoder is used to extract the spatiotemporal features of the point cloud sequence, and the decoder is used to reconstruct the point cloud based on the spatiotemporal features. During the pre-training process, the consistency between the reconstructed point cloud and the input point cloud is used as a constraint to learn gesture representations that are insensitive to changes in movement patterns. The domain adaptation fine-tuning module includes a gesture classifier and a mobile domain discriminator. It is used to use the pre-trained encoder as a feature extractor and to jointly fine-tune the feature extractor, gesture classifier, and mobile domain discriminator with real labeled data. The real labeled data includes gesture labels and mobile domain labels. During the fine-tuning process, adversarial training is performed through a gradient inversion layer to suppress the movement pattern related information contained in the features extracted by the feature extractor, so as to obtain a movement-decoupled gesture feature representation. The gesture recognition and output module is used to perform gesture recognition on the gesture point cloud sequence to be recognized based on the gesture feature representation, and output the recognition result.

9. An electronic device, comprising: processor; A memory for storing processor-executable instructions; characterized in that the processor is configured to perform the method of any one of claims 1-7.

10. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method of any one of claims 1-7.

Images