SSA-GNN-based gesture recognition method

Abstract

The invention relates to the technical field of gesture recognition, in particular to a gesture recognition method based on SSA-GNN, and the method comprises the steps: constructing a gesture recognition point cloud picture; constructing an improved GNN network, wherein the improved GNN network comprises a plurality of CBAM modules and a GNN network; the output feature maps of the CBAM modules are added and then input into a GNN network, and the GNN network comprises an input layer, a multi-layer perceptron (MLP) and a one-dimensional convolution layer. According to the method, the problems of low accuracy, high calculation complexity and insufficient real-time processing capability when an existing method is used for processing complex gestures and dynamic scenes are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of gesture recognition, and in particular to a gesture recognition method based on SSA-GNN. Background Art

[0002] Gesture recognition has received high attention in fields such as smart home and smart driving; radar-based gesture recognition has gradually become a research hotspot in the field of human-computer interaction technology.

[0003] The paper Long-range gesture recognition using millimeter wave radar proposed a long-range gesture recognition method and built a customized multi-branch convolutional neural network to process feature information and recognize gestures. The paper dynamic continuous hand gesture recognition using FMCW radar sensor combined 3D convolutional neural network and long short-term memory network to extract spatiotemporal features from gesture data sequences. At the same time, the authors used a connectionist temporal classification algorithm to recognize diverse dynamic continuous gestures.

[0004] The paper Hand-gesture Recognition Based on Parallelism CNN and Multi-domain Representation for mmWave Radar uses pulse compression and Capon beamforming algorithms to process radar signals, obtains two representations in different domains, and uses multi-channel parallel convolutional neural networks to extract independent features from two types of radar images and optimize the features. Finally, feature fusion is performed to make the feature expression more complete.

[0005] The above methods have achieved certain results in gesture recognition, but when dealing with complex gestures and dynamic scenes, there are still problems such as reduced accuracy, high computational complexity and insufficient real-time processing capabilities. Summary of the invention

[0006] In view of the shortcomings of the existing methods, the present invention solves the problems of low accuracy, high computational complexity and insufficient real-time processing capability when the existing methods process complex gestures and dynamic scenes.

[0007] The technical solution adopted by the present invention is: a gesture recognition method based on SSA-GNN comprises the following steps:

[0008] Step 1: Construct a gesture recognition point cloud map;

[0009] As a preferred embodiment of the present invention, gesture recognition includes: double tap, tap once, slide to the left and slide to the right.

[0010] Step 2: Build an improved GNN network. The improved GNN network includes: several CBAM modules and a GNN network. Add the output feature maps of several CBAM modules and input them into the GNN network. The GNN network includes an input layer, a multi-layer perceptron MLP and a one-dimensional convolutional layer.

[0011] As a preferred implementation of the present invention, an improved SSA algorithm is used to optimize the hyperparameters of the improved GNN network.

[0012] As a preferred embodiment of the present invention, the hyperparameters include: learning rate, batch size, number of layers, and number of neurons in each layer of the GNN network.

[0013] As a preferred embodiment of the present invention, the improved SSA algorithm includes:

[0014] Step 1: Initialize the sparrow population;

[0015] Step 2: Update the speed and position of the sparrow;

[0016] Step 3: Update the location of the sparrow;

[0017] Step 4: Introduce adaptive weights to improve the location update of the sparrow finder;

[0018]

[0019] Among them, L is the step size; R2 is the random factor used to increase the randomness of the search process; T S is the sth iteration; α is a random number; X ij (t) represents the current sparrow position; ε is the adaptive weight;

[0020] As a preferred embodiment of the present invention, the formula of adaptive weight is:

[0021]

[0022] Among them, ε0 is the initial weight value; C is a constant; T max is the maximum number of iterations.

[0023] Step 5: Update the location of the sparrow joiner;

[0024] As a preferred embodiment of the present invention, Levy flight is introduced to improve the position update of the sparrow joiner, and the formula is:

[0025]

[0026] Among them, N refers to the total number of iterations, X best refers to the optimal position among the finders, X worst Refers to the worst position occupied by all sparrows, Levy flies for Levy.

[0027] As a preferred embodiment of the present invention, the formula of Levy flight is:

[0028]

[0029]

[0030] Among them, N LF 、M LF is a random number that obeys Gaussian distribution; β is a constant; is the gamma function.

[0031] Step 6: In each iteration, calculate the fitness of each sparrow and select the sparrow with the highest fitness to update the global optimal position;

[0032] Step 7: Check whether the iterative process has reached the maximum number of iterations or has found the optimal solution. If the conditions are met, stop the algorithm, otherwise continue to iterate.

[0033] As a preferred embodiment of the present invention, a gesture recognition system based on SSA-GNN includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement a gesture recognition method based on SSA-GNN.

[0034] As a preferred embodiment of the present invention, a computer readable medium stores a computer program code, and when the computer program code is executed by a processor, the SSA-GNN-based gesture recognition method is implemented.

[0035] Beneficial effects of the present invention:

[0036] 1. The GNN network is proposed for point cloud processing. The SSA algorithm is introduced into the GNN network for feature combination optimization. The position update of the sparrow finder and the sparrow joiner of the SSA algorithm is improved, which greatly enhances the global search capability and the premature convergence problem of the SSA algorithm.

[0037] 2. Introducing the CBAM module into the GNN network, combined with the channel attention and spatial attention mechanisms, can highlight the key features in gesture recognition, thereby improving the accuracy and efficiency of recognition;

[0038] 3. We designed an improved GNN + improved SSA and tested it on the MMGesture dataset, an innovative dataset that uses millimeter-wave radar for gesture recognition for the first time. We finally achieved an accuracy rate of 95.74% and the time consumption was significantly shorter than that of existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a structural diagram of the improved SSA-improved GNN network of the present invention;

[0040] Figure 2 It is a structural diagram of the improved GNN network of the present invention. DETAILED DESCRIPTION

[0041] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0042] like Figure 1 As shown, a gesture recognition method based on SSA-GNN includes the following steps:

[0043] Step 1: Build a gesture recognition dataset, where gesture recognition is a point cloud map;

[0044] The point cloud image is a gesture feature vector; the gesture feature vector is used as the input of the GNN network, and the corresponding gesture action is output; the GNN network is trained;

[0045] Gesture actions include: double tap, tap once, swipe left, and swipe right;

[0046] like Figure 2 ,Step 2, build an improved GNN network;

[0047] The improved GNN network includes: several CBAM modules and a GNN network, in which the number of CBAM modules is 3 in this embodiment; the output feature maps of the CBAM modules are added and then input into the GNN network, and the GNN network includes an input layer, a multi-layer perceptron MLP and a one-dimensional convolutional layer; the input layer, i.e., the output feature maps of the three CBAM modules, are used as the input of the GNN network; the multi-layer perceptron includes MLP-h for calculating state changes, MLP-f for processing edge features, MLP-r for calculating edge weights, and MLP-g for updating states; MLP-h, MLP-f, MLP-r and MLP-g use convolutional layers for feature extraction respectively; the output layer of the one-dimensional convolutional layer is composed of several linear layers and a ReLu activation function; the GNN network iteratively updates the state in multiple time steps through forward propagation, and uses a dynamically generated adjacency matrix to capture the relationship between points, thereby enhancing the feature expression capability.

[0048] The construction of the CBAM module includes:

[0049] 1. Construct channel attention module Given the feature map F, perform global average pooling and global maximum pooling operations, which are expressed as:

[0050]

[0051]

[0052] Among them, F avg represents the global average pooling value of the feature map F, F max Represents the global maximum pooling value of the feature map F;

[0053] The pooled results are input into a shared multi-layer perceptron to generate channel attention weights.

[0054] 2. Build a spatial attention module and generate the spatial attention weight M through convolution operation:

[0055] M=σ(f(F)) (3)

[0056] Among them, M represents the spatial attention weight, f() represents the convolution operation, and σ() represents the sigmoid activation function.

[0057] The CBAM module combines channel and spatial attention, allowing the GNN network to adaptively adjust the weights of different parts of the input feature map, making the model pay more attention to features related to gesture actions.

[0058] The GNN network is used to learn the global features of gestures. The GNN network construction includes:

[0059] 1. Construct a node representation update mechanism. The GNN network captures the interaction between nodes by iteratively updating the representations between nodes. The representation of each node is obtained by aggregating the information of its neighboring nodes.

[0060] The formula for information aggregation is:

[0061]

[0062] in, represents the representation of node v at the l+1th layer, N(v) represents the set of neighbor nodes of node v, c vu represents the connection weight between node v and node u, W (l) represents the weight matrix of the lth layer, and σ() represents the activation function.

[0063] 2. Construct an edge representation update mechanism. In addition to updating node representations, GNN can also update edge representations. The formula for edge representation is:

[0064]

[0065] in, represents the representation of the edge in the l+1th layer, represents the edge weight matrix of the l+1th layer, The relationship between node v and node u represents splicing.

[0066] Step 3: Use the improved SSA algorithm to optimize the hyperparameters of the improved GNN network;

[0067] The hyperparameters for improving the GNN network include: learning rate, batch size, number of layers, and number of neurons in each layer.

[0068] Improvements to the SSA algorithm include:

[0069] Step 1: Initialize the sparrow population. In order to effectively use the SSA algorithm to optimize the feature combination, the population must be initialized first, including: the initial position Hp of each sparrow generated randomly 0 i and speed V 0 i ;

[0070] Step 2: Update the speed and position of the sparrow. The speed update formula of the sparrow is:

[0071]

[0072] Among them, V i (t+1) is the speed of the i-th sparrow in the t+1th generation, It is the best mahjong position at present. is the position of another randomly selected sparrow, ω is the inertia weight, α and β are acceleration factors, and R1 and R2 are random numbers in the range [0,1];

[0073] Step 3: The formula for updating the position of the sparrow is:

[0074]

[0075] in, is the position of the i-th sparrow in the t+1th generation.

[0076] Step 4: Update the location of the sparrow finder;

[0077] Adaptive weights are introduced to improve the position update of the sparrow finder. The improved position update formula of the sparrow finder is:

[0078]

[0079] Among them, L is a step size, which is a parameter for adjusting the search range and can be dynamically adjusted as needed; R2 is a random factor used to increase the randomness of the search process; T S is the sth iteration; α is a random number between 0 and 1; Represents the current sparrow's position.

[0080] The formula for the adaptive weight ε is:

[0081]

[0082] Among them, ε0 is the initial weight value; C is a constant, which is 0.8; T max is the maximum number of iterations.

[0083] The present invention improves the sparrow finder, enhances the global search capability through adaptive weight and maximum iteration times.

[0084] Step 5: Introduce Levy flight in the position update of the sparrow participant, and the improved position update formula of the sparrow participant is:

[0085]

[0086] Among them, N refers to the total number of iterations, X best refers to the optimal position among the finders, X worst Refers to the worst position among all sparrows.

[0087] The formula for Levy flight is:

[0088]

[0089]

[0090] Among them, N LF 、M LF is a random number that follows a Gaussian distribution; β is a constant, 1.5; is the gamma function.

[0091] The present invention introduces the Levy flight mechanism to overcome the possible premature convergence problem of the SSA algorithm and increase population diversity.

[0092] Step 6: In each iteration, calculate the fitness of each sparrow and select the sparrow with the highest fitness to update the global optimal position;

[0093] Step 7: Check whether the iterative process has reached the maximum number of iterations or has found the optimal solution. If the conditions are met, stop the algorithm, otherwise continue to iterate.

[0094] The initial position of the sparrow Hp 0i and speed V 0 i Considered as a hyperparameter that needs to be optimized in the GNN network; position Hp 0 i It is expressed as the initial value of the feature vector and weight parameter of each node in the improved GNN network; through the randomly generated positions, the algorithm explores different feature vector combinations or weight configurations; the speed V 0 i It represents the moving speed in the feature space, reflecting the ability of the SSA algorithm to adjust the feature combination during the search process; the speed update formula (6) reflects how to adjust the search direction according to the current optimal solution and other randomly selected solutions.

[0095] Eigenvector Hp 0 i Represents the initial features of each node. In GNN, the feature vector of the node affects the learning effect of the model. By randomly generating different combinations of feature vectors, different feature representations can be explored to find the optimal feature configuration. This is closely related to hyperparameters (such as learning rate, batch size, etc.), because hyperparameters affect how the model uses these features during training; weight parameters are what need to be optimized during model learning. The choice of initial weights (i.e. (Hp_{0i})) affects the convergence speed and final performance of the model. Hyperparameters such as learning rate directly affect the weight update amplitude; speed (V_{0i}) represents the rate of movement in the feature space, which can be understood as the sensitivity to hyperparameter adjustment during optimization. Higher speed means fast exploration in the feature space, while lower speed may lead to more detailed local search. Hyperparameters such as batch size and learning rate affect the effect of this movement, and thus affect the training process of the model.

[0096] The process of optimizing the improved GNN network using the improved SSA algorithm is:

[0097] SSA optimizes CBAM and GNN, improves node and edge state updates, and achieves accurate gesture recognition; the SSA algorithm can optimize the channel attention module and spatial attention module of CBAM to improve the performance of feature extraction. The combination formula is as follows:

[0098] X ssa-cbam (t) = X s (t)⊙X ij (t) (13)

[0099] Among them, X ssa-cbam (t) represents the feature map after the CBAM module and the SSA algorithm are combined at time t, X s (t) represents the feature map of the CBAM module at time t, X ij(t) represents the feature graph of the SSA algorithm at time t, and ⊙ represents element multiplication.

[0100] The processed X ssa-cbam (t) is used as the input of the GNN algorithm; the nodes of the GNN network are updated. The specific formula is as follows:

[0101]

[0102] in, Represents the hidden state of node v at layer l+1, AGGREGATE (l) represents the aggregation function of node v at layer l, N(v) represents the set of neighbor nodes of node v, and the node update function is:

[0103]

[0104] in, Represents the message of node v at layer l, COMBINE (l) represents the combination function of the lth layer;

[0105] The edge update formula is as follows:

[0106]

[0107] in, represents the hidden state of the l+1th layer edge (v,w), Represents the message of the edge (v,w) at level l, UPDATE (l) represents the layer update function of the l-th layer edge (v,w);

[0108] The layer prediction formula is:

[0109]

[0110] Among them, y represents the graph-level prediction result, Represents the hidden state of the last layer node v, V represents the set of all nodes in the graph, and READOUT represents the readout function.

[0111] The hidden states of nodes and edges of the GNN network are updated, and graph-level prediction results are output to achieve accurate recognition of complex gestures.

[0112] Experimental process:

[0113] The dataset uses the MMGesture dataset collected by Mani Srivastava's team at the University of California, Los Angeles. The TI IWR1443BOOST radar is installed on a 1.3-meter-high tripod. The data from the radar is sent to a laptop via USB through ROS (Robot Operating System) messages; it is developed and described in the ROS node running on the laptop; in order to record and store data, another ROS package rosbag is used; finally, these rosbags are converted to .txt files.

[0114] Data was collected from 144 users who performed 4 different activities in front of the radar: double tap, tap once, swipe left, swipe right, and data was collected for a total of 1,357 minutes.

[0115] Table 1 Details of the dataset

[0116]

[0117] In order to verify the effectiveness of the present invention, it is compared with existing deep learning-based methods, including the method used in this dataset and other methods.

[0118] Table 2 Accuracy of activity recognition classifier on the MMGesture dataset

[0119]

[0120] It can be seen from Table 1 that the present invention is the best among all the methods, and ZLSSA represents the introduction of adaptive weights and the introduction of Levy flight.

[0121] Table 3 Time statistics of activity recognition classifier on MMGesture dataset

[0122]

[0123]

[0124] The present invention proposes a GNN network for point cloud processing, introduces an SSA algorithm into the GNN network for feature combination optimization, and improves the position update of the sparrow finder and sparrow joiner of the SSA algorithm, which greatly enhances the global search capability and the premature convergence problem of the SSA algorithm.

[0125] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A gesture recognition method based on SSA-GNN, characterized in that: The following steps are involved: Step 1: Construct a gesture recognition point cloud map; Step 2: Build an improved GNN network, which includes: several CBAM modules and a GNN network; add the output feature maps of several CBAM modules and input them into the GNN network; The GNN network consists of an input layer, a multi-layer perceptron MLP, and a one-dimensional convolutional layer.

2. The gesture recognition method based on SSA-GNN according to claim 1, characterized in that: The improved SSA algorithm is used to optimize the hyperparameters of the improved GNN network.

3. The gesture recognition method based on SSA-GNN according to claim 2, characterized in that: Improvements to the SSA algorithm include: Step 1: Initialize the sparrow population; Step 2: Update the speed and position of the sparrow; Step 3: Update the location of the sparrow; Step 4: Introduce adaptive weights to improve the position update of the sparrow finder. The formula is: Where L is the step size; R2 is the random factor; T S is the sth iteration; α is a random number; X ij (t) represents the current sparrow position; ε is the adaptive weight; Step 5: Update the location of the sparrow joiner; Step 6: In each iteration, calculate the fitness of each sparrow and select the sparrow with the highest fitness to update the global optimal position; Step 7: Check whether the iterative process has reached the maximum number of iterations or has found the optimal solution. If the conditions are met, stop the algorithm, otherwise continue to iterate.

4. The gesture recognition method based on SSA-GNN according to claim 3 is characterized in that: The formula for adaptive weight is: Among them, ε0 is the initial weight value; C is a constant; T max is the maximum number of iterations.

5. The gesture recognition method based on SSA-GNN according to claim 3, characterized in that: Levy flight is introduced to improve the position update of sparrow joiners. The formula is: Among them, N refers to the total number of iterations, X best refers to the optimal position among the finders, X worst Refers to the worst position occupied by all sparrows, Levy flies for Levy.

6. The gesture recognition method based on SSA-GNN according to claim 5, characterized in that: The formula for Levy flight is: Among them, N LF 、M LF is a random number that obeys Gaussian distribution; β is a constant; is the gamma function.

7. The gesture recognition method based on SSA-GNN according to claim 1, characterized in that: Gesture recognition includes: double tap, tap once, swipe left, and swipe right.

8. The gesture recognition system based on SSA-GNN is characterized by: include: a memory for storing instructions executable by a processor; A processor, configured to execute instructions to implement the SSA-GNN-based gesture recognition method as described in any one of claims 1 to 7.

9. A computer readable medium storing a computer program code, characterized in that: When the computer program code is executed by a processor, the SSA-GNN-based gesture recognition method is implemented as described in any one of claims 1 to 7.

Images