Dynamic gesture recognition method based on hand key points and frame number attention mechanism

Abstract

The present application relates to the technical field of dynamic gesture recognition, and particularly relates to a dynamic gesture recognition method based on hand key points and frame number attention mechanism.The present application comprises the following steps: S1, hand key point data acquisition and standardization; S2, construction of hand key point attention mechanism; S3, LSTM long short-term memory neural network dynamic recognition: using LSTM long short-term memory neural network to extract data time characteristics, and finally using a full connection layer and normalization to output recognition results.The present application uses a dynamic sampling method based on the degree of change of the spatial distance of hand key points, which reduces repeated and less changed data, ensures that the sampling process maximally retains more important information, and improves the quality of model training data; based on hand key point attention and frame number attention mechanism, the model pays more attention to important hand key points and frame numbers, fully excavates data characteristics, and improves the recognition speed and accuracy of the model.

Description

Technical Field

[0001] This invention relates to the field of dynamic gesture recognition technology, specifically to a dynamic gesture recognition method based on hand key points and frame number attention mechanism. Background Technology

[0002] Traditional image processing techniques that rely on manually defined features and simple logical judgments are cumbersome to implement and have low recognition accuracy. Current deep learning-based dynamic gesture recognition methods mostly use graph convolutional neural networks for hand key point sequences and 3D convolutional neural networks for video sequences. While these methods offer high recognition accuracy, they also involve large numbers of parameters and complex network structures, resulting in slow model operation and long training times, which cannot meet the real-time requirements of dynamic gesture recognition.

[0003] Meanwhile, the recognition performance of dynamic gesture recognition network models is positively correlated with the quality of the network input data. In dynamic gesture video stream data, considering the data repetition and redundancy, not all data frame information is valid. Therefore, long gesture video streams need to be sampled before input. Traditional fixed frame number sampling methods or average frame number sampling methods have relatively fixed sampling sequences and cannot achieve sampling based on the importance of data. This may lead to the loss of key data information and affect the model's recognition accuracy. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a dynamic gesture recognition method based on the attention mechanism of hand key points and frame number, so that the model pays more attention to important hand key points and frame number, fully explores data features, and improves the recognition speed and accuracy of the model.

[0005] The technical solution of this invention is as follows:

[0006] A dynamic gesture recognition method based on hand key points and frame number attention mechanism includes the following steps:

[0007] S1. Hand Key Point Data Acquisition and Standardization: Custom hand key point data is defined, and the key point coordinates of all frames of dynamic gestures are collected as recognition data. At the same time, dynamic sampling is performed according to the degree of change of the recognition data, including the following steps:

[0008] S11. Custom hand key point recognition data: including the position coordinate format of hand key points (x hj ,y hj ,z hj ), where h represents the left and right hand numbers, h = l for the left hand, h = r for the right hand, and j is the number of the j-th hand key point;

[0009] S12. Collect the key point coordinates of all frames of the dynamic gesture: A dynamic gesture is divided into 30 frames, so the data for each complete dynamic gesture consists of 42×30 position coordinates. The key points of each hand are numbered as follows:

[0010] L0=(x l0 ,y l0 ,z l0 L1=(x l1 ,y l1 ,z l1 )……L 20 =(x l20 ,y l20 ,z l20 )

[0011] R0=(x r0 ,y r0 ,z r0 R1=(x r1 ,y r1 ,z r1 )……R 20 =(x r20 ,y r20 ,z r20 )

[0012] Where L j R j These represent the coordinates of the j-th key point on the left and the j-th key point on the right, respectively.

[0013] S13. Dynamic sampling based on the degree of change in recognition data: By analyzing the distance relationship between the spatial coordinate values ​​of the hand key points in each frame and the corresponding hand key points in the previous frame, the importance of the coordinate information in each frame is evaluated, i.e.:

[0014]

[0015] in This represents the distance between the j-th keypoint in the i-th frame and the j-th keypoint in the previous frame; This represents the x-coordinate of the j-th keypoint in the i-th frame. Represents the x-coordinate of the j-th keypoint in the (i-1)-th frame; Define The original sampled frame index, Represents the total score of the i-th frame;

[0016]

[0017] S131. Select the first sampling frame, and The frame ranked first in the index is selected as the original sampling frame;

[0018] S132. Define sampling frame metrics:

[0019]

[0020] Where l is the frame number that was not selected as a sampling frame, and l-1 is the closest selected sampling frame whose frame number is before the l-th frame.

[0021]

[0022] Represents the total score of the l-th frame, selected The first-ranked frame is the sampled frame;

[0023] S133. Repeat the operation, recalculating the index after each selected sampling frame. Then select a sampling frame, and continue until 30 sampling frames are selected, at which point sampling ends and this operation is terminated;

[0024] S2. Constructing a hand keypoint attention mechanism: Since hand keypoint attention and hand frame count attention have different impacts on recognition performance, a convolutional neural network is used to extract hand keypoint features and hand frame count features. The corresponding weights are then learned by a fully connected layer, including the following steps:

[0025] S21. Construction of the hand keypoint attention mechanism: A CNN feature extraction layer is used to extract features of each keypoint across all frames in the original data. A fully connected layer and normalization processing are used to extract the weight C of each hand keypoint. h0 C h1 ...C h20 The resulting weighting coefficients are multiplied by the corresponding original data to obtain hand coordinate data based on the hand key point attention mechanism;

[0026] S22. Construction of the hand frame-by-frame attention mechanism: A one-dimensional convolutional neural network is used to extract all hand keypoint features in each frame vertically from the original data. Fully connected layers and normalization processing are used to extract the weight C of each hand keypoint. f0 C f1 ...C f49 The resulting weighting coefficients are multiplied by the corresponding original data to obtain hand coordinate data based on the hand frame attention mechanism;

[0027] S23, Fusion of hand key point attention and hand frame number attention: CNN feature extraction module where N2=2, that is, two CNN feature extraction modules are connected in series, namely the first CNN feature extraction module and the second CNN feature extraction module. Each CNN feature extraction module has 30 convolutional layers, and the input data dimension of each convolutional layer is 21×1.

[0028] S3, LSTM Long Short-Term Memory Neural Network Dynamic Recognition: The LSTM long short-term memory neural network is used to extract the time features of the data, and finally the recognition result is output with a fully connected layer and normalization.

[0029] Preferably, in the dynamic gestures of step S12, the sampling rule for dividing the dynamic gestures into 30 frames is as follows: in dynamic gesture video data with more than 30 frames, the data is sampled into 30 frames to achieve standardization of the input model data format.

[0030] Considering that each dynamic gesture in actual sampling is not the standard 30 frames, for data less than 30 frames, zero data is used to pad to 30 frames; for data greater than 30 frames, the frame number is sampled to 30 frames.

[0031] Preferably, in the dynamic sampling of step S13, the importance of the hand key point coordinate data is related to the similarity of the hand key point coordinate data in the previous sampling frame. The greater the similarity between adjacent frame data, that is, the greater the data repetition, the less important the data.

[0032] Preferably, in the construction of the hand keypoint attention mechanism in step S21, each CNN feature extraction module contains 21 convolutional layers, and each convolutional layer includes a one-dimensional valid convolution, a non-linear activation function, and a one-dimensional pooling layer. The output of the feature extraction module is...

[0033] in In this context, n represents the nth feature extraction module. Here, * represents the weights of the v-th convolutional kernel in the n-th feature extraction module, and * denotes the convolution operation. This is the offset of the v-th convolution operation in the n-th feature extraction module;

[0034]

[0035] The fully connected layer consists of an input layer, a hidden layer, and an output layer. The input layer is a one-dimensional output of global pooling. The hidden layer has 21 neurons. To avoid overfitting, a dropout layer is added to the linear layer. The output layer has 21 neurons.

[0036] Preferably, in the construction of the hand frame-count attention mechanism in step S22, the CNN feature extraction module includes N2 convolutional layers, each of which is a combination of a one-dimensional valid convolution, a non-linear activation function, and a one-dimensional pooling layer; the output of the feature extraction module is...

[0037] in In this context, n represents the nth feature extraction module. Here, * represents the weights of the v-th convolutional kernel in the n-th feature extraction module, and * denotes the convolution operation. This is the offset of the v-th convolution operation in the n-th feature extraction module;

[0038]

[0039] The fully connected layer consists of an input layer, a hidden layer, and an output layer. The input layer has a one-dimensional global pooling output, the hidden layer has 30 neurons, and a dropout layer is added to the linear layer to avoid overfitting. The output layer has 21 neurons.

[0040] Preferably, in the fusion of hand key point attention and hand frame number attention in step S23, the input data dimension in the convolutional layer of the first CNN feature extraction module is 21×1, the one-dimensional convolutional neural network includes 8 7×1 convolutional kernels with a stride of 1; the one-dimensional max pooling size is 3×1; the activation function is the ReLU function, and the output dimension is 5×8.

[0041] In the convolutional layer of the second CNN feature extraction module, the input data dimension is 5×8, the one-dimensional convolutional neural network includes 16 3×8 convolutional kernels with a stride of 1; the one-dimensional max pooling size is 3×1; the activation function is the ReLU function, and the output dimension is 1×16.

[0042] The size of the one-dimensional global average pooling GAP is 1×16, the output dimension is 1×1, and the output is each X_i, where i is the sequence number of a dynamic gesture frame, and j∈[0,29].

[0043] Preferably, the LSTM long short-term memory neural network dynamic recognition in step S3 includes the following sub-steps:

[0044] The data weighted by the key point attention and frame attention mechanisms are summed. To reduce the weakening of the temporal features of the weighted data, the original data is summed again with the data from the two attention mechanisms. The summed data is then input into the LSTM network module to extract the temporal features from the data. The output is processed by the fully connected layer and normalized to obtain the gesture prediction result.

[0045] Where D i Let represent the coordinates of all hand keypoints in the i-th frame, i∈[0,29]. The input data dimension in the LSTM module is 30, and the hidden layer dimension is 30.

[0046] The fully connected layer has an input layer dimension of 30, a hidden layer with 30 neurons, and an output layer dimension of 30. The output layer is connected to the Softmax classification layer and outputs the model's classification result.

[0047] Preferably, the LSTM long short-term memory neural network dynamic recognition in step S23 further includes the following steps:

[0048] In the dynamic recognition part of the LSTM long short-term memory neural network, the fully connected layer input layer has a dimension of 21, the hidden layer has 21 neurons, and the output layer has 21 neurons. The model uses the Softmax function as the classification function.

[0049] During training, at least 1000 complete dynamic gesture training data are collected for each gesture. The collected samples are divided into training set, validation set and test set in a ratio of 6:2:2 for training. The number of epochs is set to be no less than one-fifth and no more than one-half of the data volume. If the number of epochs is too high, the training time will increase and the model recognition accuracy will no longer improve. The learning rate of the Adam optimization algorithm is 0.001. When the batch size is 100, the optimal solution of the model is calculated using the cross-entropy loss function.

[0050] Compared with the prior art, the present invention has the following advantages:

[0051] (1) Based on the recognition of key points of hand skeleton, the coordinates of key points of hand in all frames of a dynamic gesture are taken as the original data. An attention mechanism based on key points of hand and an attention mechanism based on frame number are proposed, so that the model pays more attention to key points of hand and frame number that are more important to dynamic gesture recognition, thereby making the model recognition more accurate and efficient.

[0052] (2) A dynamic sampling method based on the degree of data change is proposed. The number of sampling frames is determined according to the degree of change between the previous frame data and the sampled frame data, so that the model samples more important data, reduces the repetition and less changed data, and ensures that the sampling process retains more important information to the maximum extent, thereby improving the quality of model training data and indirectly improving the model's recognition efficiency. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the principle connection of the present invention.

[0055] Figure 2 This is an example diagram of the key points of the hand in this invention.

[0056] Figure 3 This is an example diagram of the standardized data for the left and right hands of the dynamic gestures of this invention.

[0057] Figure 4 This is an example diagram of the sampling process of the present invention.

[0058] Figure 5 This is an example diagram of the sampling process of the present invention.

[0059] Figure 6 This is a structural diagram of the hand key point attention mechanism feature extraction of the present invention.

[0060] Figure 7 This is a structural diagram of the feature extraction using the frame-number attention mechanism of this invention.

[0061] Figure 8 This is a simplified diagram of the network model of this invention.

[0062] Figure 9 This is a graph showing the relationship between the number of frames and the recognition accuracy of this invention.

[0063] Figure 10 This is a graph showing the relationship between the number of frames and the detection time in this invention.

[0064] Figure 11 This is a graph showing the relationship between frame rate and efficiency metrics in this invention. Detailed Implementation

[0065] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0066] Example 1

[0067] This invention proposes a dynamic sampling method based on the distance variation relationship of hand key points and a dynamic gesture recognition method based on hand key point attention and hand frame number attention mechanisms. First, the original dynamic gesture data frame count is fixed at 30 frames. Dynamic gesture data with more than 30 frames is sampled. Distance evaluation indicators are calculated and sorted according to the magnitude of the distance variation of hand key points. This process is repeated iteratively. Based on the evaluation indicators, the 30 sampled frames with the largest information content are selected sequentially to obtain standard-format dynamic gesture training data. Second, the Mediapipe algorithm is used to identify hand key points and return their coordinates. In the recognition task, different hand key points and frame numbers have different impacts on the recognition effect. A convolutional neural network is used to extract hand key point features and hand frame number features, and a fully connected layer learns their corresponding weights. Finally, an LSTM (Long Short-Term Memory) neural network is used to identify the dynamic gesture category, thus achieving dynamic gesture recognition.

[0068] The present invention has the following advantages: (1) The present invention takes into account the problem of information duplication and redundancy in the original data sequence. The traditional fixed frame number sampling method has a relatively fixed sampling number, and the sampling process may lead to the loss of key data information in the data. The present invention proposes a dynamic sampling method based on the physical distance of hand key points, which ensures that the sampling frame sequence with the most information is sampled to the maximum extent during the sampling process, improves the quality of model training data, and indirectly improves the model recognition efficiency; (2) The present invention converts image pixel data processing into less hand key point coordinate data processing, which greatly reduces the amount of data processed. The model can achieve high accuracy dynamic gesture recognition with a simple structure and a small number of parameters, and the training speed and recognition speed of the model are fast; (3) The present invention takes into account that each hand key point and frame number has different importance to the dynamic gesture recognition task. The data changes of some important hand key points and frame numbers play a more important role in the recognition task. Therefore, the present invention proposes a key point attention mechanism and a frame number attention mechanism to make the model pay more attention to the data changes of some important hand key points and frame numbers, thereby improving the recognition speed and recognition accuracy of the model.

[0069] like Figure 1 As shown, this embodiment provides a dynamic gesture recognition method based on hand key points and frame number attention mechanism, including the following steps:

[0070] S1. Hand Key Point Data Acquisition and Standardization: Custom hand key point data is defined, and the key point coordinates of all frames of dynamic gestures are collected as recognition data. At the same time, dynamic sampling is performed according to the degree of change of the recognition data, including the following steps:

[0071] S11. Custom hand key point recognition data: including the position coordinate format of hand key points (x hj ,y hj ,z hj ), where h represents the left and right hand numbers, h = l for the left hand, h = r for the right hand, and j is the number of the j-th hand key point;

[0072] S12. Collect the key point coordinates of all frames of the dynamic gesture: A dynamic gesture is divided into 30 frames, so the data for each complete dynamic gesture consists of 42×30 position coordinates. The key points of each hand are numbered as follows:

[0073] L0=(x l0 ,y l0 ,z l0 L1=(x l1 ,y l1 ,z l1 )……L 20 =(x l20 ,y l20 ,z l20 )

[0074] R0=(x r0 ,y r0 ,z r0 R1=(x r1 ,y r1 ,z r1 )……R 20 =(x r20 ,y r20 ,z r20 )

[0075] Where L j R j These represent the coordinates of the j-th key point on the left and the j-th key point on the right, respectively.

[0076] S13. Dynamic sampling based on the degree of change in recognition data: By analyzing the distance relationship between the spatial coordinate values ​​of the hand key points in each frame and the corresponding hand key points in the previous frame, the importance of the coordinate information in each frame is evaluated, i.e.:

[0077]

[0078] in This represents the distance between the j-th keypoint in the i-th frame and the j-th keypoint in the previous frame; This represents the x-coordinate of the j-th keypoint in the i-th frame. Represents the x-coordinate of the j-th keypoint in the (i-1)-th frame; Define The original sampled frame index, Represents the total score of the i-th frame;

[0079]

[0080] S131. Select the first sampling frame, and The frame ranked first in the index is selected as the original sampling frame;

[0081] S132. Define sampling frame metrics:

[0082]

[0083] Where l is the frame number that was not selected as a sampling frame, and l-1 is the closest selected sampling frame whose frame number is before the l-th frame.

[0084]

[0085] Represents the total score of the l-th frame, selected The first-ranked frame is the sampled frame;

[0086] S133. Repeat the operation, recalculating the index after each selected sampling frame. Then select a sampling frame, and continue until 30 sampling frames are selected, at which point sampling ends and this operation is terminated;

[0087] S2. Constructing a hand keypoint attention mechanism: Since hand keypoint attention and hand frame count attention have different impacts on recognition performance, a convolutional neural network is used to extract hand keypoint features and hand frame count features. The corresponding weights are then learned by a fully connected layer, including the following steps:

[0088] S21. Construction of the hand keypoint attention mechanism: A CNN feature extraction layer is used to extract features of each keypoint across all frames in the original data. A fully connected layer and normalization processing are used to extract the weight C of each hand keypoint. h0 C h1 ...C h20 The resulting weighting coefficients are multiplied by the corresponding original data to obtain hand coordinate data based on the hand key point attention mechanism;

[0089] S22. Construction of the hand frame-by-frame attention mechanism: A one-dimensional convolutional neural network is used to extract all hand keypoint features in each frame vertically from the original data. Fully connected layers and normalization processing are used to extract the weight C of each hand keypoint. f0 C f1 ...C f49 The resulting weighting coefficients are multiplied by the corresponding original data to obtain hand coordinate data based on the hand frame attention mechanism;

[0090] S23, Fusion of hand key point attention and hand frame number attention: CNN feature extraction module where N2=2, that is, two CNN feature extraction modules are connected in series, namely the first CNN feature extraction module and the second CNN feature extraction module. Each CNN feature extraction module has 30 convolutional layers, and the input data dimension of each convolutional layer is 21×1.

[0091] S3, LSTM Long Short-Term Memory Neural Network Dynamic Recognition: The LSTM long short-term memory neural network is used to extract the time features of the data, and finally the recognition result is output with a fully connected layer and normalization.

[0092] like Figure 1As shown, this invention proposes a hand keypoint attention mechanism and a frame number attention mechanism based on the coordinates of key hand points in dynamic gestures. This enables the model to pay more attention to important hand key points and frame number data changes during the recognition process, improving the accuracy and speed of dynamic gesture recognition. Finally, the data weighted by the attention mechanism is processed through an LSTM (Long Short-Term Memory) neural network to extract temporal features and identify the type of dynamic gesture. The dynamic gesture recognition method of this invention mainly includes three parts: first, data acquisition and standardization; second, implementation of the attention mechanism; and third, implementation of dynamic hand recognition.

[0093] like Figure 2 As shown, this invention uses the Mediapipe algorithm to identify 21 key points on the hand and obtain their position coordinates. The format for obtaining the position coordinates of the key points is (x... hj ,y hj ,z hj ), where h represents the left and right hand numbers, h=l for the left hand, h=r for the right hand, and j is the number of the j-th hand key point. Therefore, each frame will return 42 sets of left and right hand key point data if a hand is detected. If some key points are not detected, they will be padded with 0. For example:

[0094] (x l0 ,y l0 ,z l0 (x) l1 ,y l1 ,z l1 (x) l2 ,y l2 ,z l2 )……(x l20 ,y l20 ,z l20 ),

[0095] (x r0 ,y r0 ,z r0 (x) r1 ,y r1 ,z r1 (x) r2 ,y r2 ,z r2 )……(x r20 ,y r20 ,z r20 ).

[0096] like Figure 3 As shown, a dynamic gesture is divided into 30 frames. Therefore, the data for each complete dynamic gesture consists of 42*30 position coordinates. For ease of representation, each key point is numbered as follows:

[0097] L0=(xl0 ,y l0 ,z l0 L1=(x l1 ,y l1 ,z l1 )……L 20 =(x l20 ,y l20 ,z l20 ),

[0098] R0=(x r0 ,y r0 ,z r0 R1=(x r1 ,y r1 ,z r1 )……R 20 =(x r20 ,y r20 ,z r20 ).

[0099] L j R j These represent the coordinates of the j-th key point on the left and the j-th key point on the right, respectively.

[0100] like Figure 4 As shown, dynamic gesture video data is collected. Standardized data is defined as 30 frames per complete dynamic gesture action. However, considering that each dynamic gesture may not actually have 30 frames in actual sampling, data with fewer than 30 frames is padded with zeros to reach 30 frames. For data with more than 30 frames, the frame number is sampled to 30 frames. The data sampling rules are as follows: In dynamic gesture video data with more than 30 frames, the data is sampled to 30 frames to standardize the input model data format. Different gestures have different key information frame numbers. Traditional fixed frame number sampling methods or average frame number sampling methods have relatively fixed sampling numbers and cannot achieve sampling based on the importance of the data, which may lead to the loss of key data information and affect the model's recognition accuracy. This invention proposes a sampling method for dynamic frame numbering based on the coordinates of key points in dynamic gestures. By using the coordinate data of the hand key points of the dynamic gestures identified in each frame, an evaluation index is quantified based on the distance between the hand key point data in each frame and the hand key point data in the previous frame. The sampling is filtered by the evaluation index. This sampling method can reduce data with little change relationship between adjacent frames, that is, reduce the sampling of repeated frames, filter out the information that is more important to the model recognition, minimize the loss of data information, and improve the quality of model training data.

[0101] like Figure 5As shown, the importance of hand keypoint coordinate data is related to the similarity of hand keypoint coordinate data in the previous sampled frame. The greater the similarity between adjacent frames, i.e., the greater the data redundancy, the lower the importance of the data. Evaluation indicators were quantified by the distance relationship between adjacent frames, and the 30 most important sets of hand keypoint coordinate data were sampled. The importance of each frame's coordinate information was evaluated by the change in spatial coordinate values ​​between the hand keypoints in each frame and the corresponding hand keypoints in the previous frame.

[0102]

[0103] in This represents the distance between the j-th keypoint in the i-th frame and the j-th keypoint in the previous frame. This represents the x-coordinate of the j-th keypoint in the i-th frame. This represents the x-coordinate of the j-th keypoint in the (i-1)-th frame (the previous frame). Definition The original sampled frame index, This represents the total score for the i-th frame.

[0104]

[0105] Simply using the above Using indicators to evaluate the importance of data only reflects the relationship between the current data and the previous frame. However, the previous frame may not be a sampling frame, which could lead to some slowly changing data showing little change from the previous frame but significant change from the sampling frame. This could result in the loss of crucial data. Therefore, relying solely on these indicators to determine the importance of coordinate data has some problems. To solve these problems, firstly, the first sampling frame is selected... The frame ranked first in terms of metrics is selected as the original sampled frame. Next, the sampled frame metrics are defined:

[0106]

[0107] Where l is the frame number that was not selected as a sampling frame, and l-1 is the closest selected sampling frame whose frame number is before the l-th frame.

[0108]

[0109] Represents the total score of the l-th frame, selected The first-ranked frame is the sampled frame. Finally, repeat the above operation, recalculating the index after each sampled frame is selected. Then select a sampling frame, and continue until 30 sampling frames are selected, at which point the sampling ends and the operation terminates.

[0110] This invention employs an adaptive sampling method based on the importance of data. Compared to traditional fixed frame number sampling methods or average frame number sampling methods, the sampling number in this method is not a fixed value. Furthermore, sampling is performed according to the importance of the data, ensuring that the sampling process is maximized, retaining more important information, improving the quality of model training data, and indirectly improving the model's recognition efficiency.

[0111] like Figure 6 As shown, to improve the accuracy of model recognition, this invention considers the varying importance of different hand key points in a dynamic gesture and proposes a hand key point attention mechanism. This mechanism makes the model pay more attention to changes in hand key points that are more important for dynamic gesture recognition, thereby improving the model's recognition accuracy. A CNN feature extraction layer is used to extract features from all frames for each key point in the original data. Fully connected layers and normalization processing are used to extract the weight C of each hand key point. h0 C h1 ...C h20 The resulting weighting coefficients are multiplied by the corresponding original data to obtain hand coordinate data based on the hand key point attention mechanism.

[0112] Each CNN feature extraction module contains 21 convolutional layers, each including a one-dimensional valid convolution, a non-linear activation function, and a one-dimensional pooling layer. The output of the feature extraction module is... in In this context, n represents the nth feature extraction module. Here, * represents the weights of the v-th convolutional kernel in the n-th feature extraction module, and * denotes the convolution operation. This is the offset (i.e., stride) of the v-th convolutional operation in the n-th feature extraction module.

[0113]

[0114] The fully connected layer consists of an input layer, a hidden layer, and an output layer. The input layer is a one-dimensional output of global pooling. The hidden layer has 21 neurons. To avoid overfitting, a dropout layer is added to the linear layer. The output layer has 21 neurons.

[0115] A network structure for the hand keypoint attention mechanism is constructed, consisting of a CNN feature extraction module with N1=3, meaning three CNN feature extraction modules connected in series: the first CNN feature extraction module, the second CNN feature extraction module, and the third CNN feature extraction module. Each CNN feature extraction module has 21 convolutional layers, and the input data dimension of each convolutional layer is 30×1.

[0116] In the convolutional layer of the first CNN feature extraction module, the input data dimension is 30×1, the one-dimensional convolutional neural network includes 8 7×1 convolutional kernels with a stride of 1; the one-dimensional max pooling size is 2×1; the activation function is the ReLU function, and the output dimension is 12×8.

[0117] In the convolutional layer of the second CNN feature extraction module, the input data dimension is 12×8, the one-dimensional convolutional neural network includes 16 5×8 convolutional kernels with a stride of 1; the one-dimensional max pooling size is 2×1; the activation function is the ReLU function, and the output dimension is 4×16.

[0118] In the convolutional layer of the third CNN feature extraction module, the input data dimension is 4×16, the one-dimensional convolutional neural network includes 32 3×16 convolutional kernels with a stride of 1; the one-dimensional max pooling size is 2×1; the activation function is the ReLU function, and the output dimension is 1×32.

[0119] The one-dimensional global average pooling gap size is 1×32, and the output dimension is 1×1. The output is... Figure 6 In the figure, X_j is the index of the 21 key points of the hand, j∈[0,20].

[0120] like Figure 7 As shown, to improve the accuracy of model recognition, this invention considers the varying importance of different frames in a dynamic gesture for the recognition task. Therefore, a frame-number attention mechanism is proposed, making the model pay more attention to data changes in key frames, thereby improving the model's recognition accuracy. A one-dimensional convolutional neural network is used to extract all hand keypoint features from each frame vertically from the original data. Fully connected layers and normalization processing are used to extract the weight C of each hand keypoint. f0 C f1 ...C f49 The resulting weighting coefficients are multiplied by the corresponding original data to obtain hand coordinate data based on the hand key point attention mechanism.

[0121] The CNN feature extraction module contains N² convolutional layers, each of which is a combination of a one-dimensional valid convolution, a non-linear activation function, and a one-dimensional pooling layer; the output of the feature extraction module is... in In this context, n represents the nth feature extraction module. Here, * represents the weights of the v-th convolutional kernel in the n-th feature extraction module, and * denotes the convolution operation. This is the offset (i.e., stride) of the v-th convolutional operation in the n-th feature extraction module.

[0122]

[0123] The fully connected layer consists of an input layer, a hidden layer, and an output layer. The input layer has a one-dimensional global pooling output, the hidden layer has 30 neurons, and a dropout layer is added to the linear layer to avoid overfitting. The output layer has 21 neurons.

[0124] This invention proposes a dynamic gesture recognition method based on hand keypoints and frame number attention mechanisms. Building upon the coordinates of hand keypoints in dynamic gestures, it introduces hand keypoint attention and frame number attention mechanisms. This allows the model to focus more on important hand keypoints and frame number data changes during recognition, improving the accuracy and speed of dynamic gesture recognition. Finally, the attention-weighted data is processed through an LSTM (Long Short-Term Memory) neural network to extract temporal features and identify the type of dynamic gesture. The dynamic gesture recognition method of this invention mainly consists of three parts, see... Figure 1 First, data acquisition and standardization; second, implementation of the attention mechanism; third, implementation of dynamic hand recognition.

[0125] The network structure for the hand keypoint attention mechanism includes a CNN feature extraction module, where N² = 2, meaning two CNN feature extraction modules are connected in series: the first CNN feature extraction module and the second CNN feature extraction module. Each CNN feature extraction module has 30 convolutional layers, and the input data dimension of each convolutional layer is 21×1.

[0126] In the convolutional layer of the first CNN feature extraction module, the input data dimension is 21×1, the one-dimensional convolutional neural network includes 8 7×1 convolutional kernels with a stride of 1; the one-dimensional max pooling size is 3×1; the activation function is the ReLU function, and the output dimension is 5×8.

[0127] In the convolutional layer of the second CNN feature extraction module, the input data dimension is 5×8, the one-dimensional convolutional neural network includes 16 3×8 convolutional kernels with a stride of 1; the one-dimensional max pooling size is 3×1; the activation function is the ReLU function, and the output dimension is 1×16.

[0128] The one-dimensional global average pooling gap size is 1×16, and the output dimension is 1×1. Figure 7 In each X_i, i is a dynamic gesture frame number, j∈[0,29].

[0129] like Figure 8 As shown, the data weighted by keypoint attention and frame attention mechanisms are summed. To reduce the weakening of temporal features in the weighted data, the original data is summed again with the data from both attention mechanisms. The summed data is then input into the LSTM network module to extract temporal features. The gesture prediction result is obtained from the output of the fully connected layer after normalization. Where D... iLet represent the coordinates of all hand keypoints in the i-th frame, i∈[0,29]. The input data dimension in the LSTM module is 30, and the hidden layer dimension is 30. The fully connected layer has an input layer dimension of 30, 30 hidden layer neurons, and an output layer dimension of 30. The output layer is connected to the Softmax classification layer to output the model classification result.

[0130] The LSTM (Long Short-Term Memory) neural network for dynamic recognition has a fully connected input layer with 21 dimensions, 21 hidden neurons, and 21 output neurons. The model uses the Softmax function for classification. During training, at least 1000 complete dynamic gesture training data points are collected for each gesture. The collected samples are divided into training, validation, and test sets in a 6:2:2 ratio. The number of epochs is set to be no less than one-fifth and no more than half the data volume. Excessive epochs will increase training time and no longer improve model recognition accuracy. The Adam optimization algorithm has a learning rate of 0.001, and with a batch size of 100, the optimal solution is calculated using the cross-entropy loss function.

[0131] Example 2

[0132] This invention focuses on using distance relationships to filter key frames, employing a one-dimensional convolutional network to extract hand keypoint features and frame number features, and an LSTM network to extract temporal features, finally outputting the gesture result. These two approaches are fundamentally different. This invention also focuses on dynamic sampling of video frames for dynamic gestures and the attention mechanism for dynamic gestures, thereby improving the accuracy of the model's dynamic gesture recognition. These two approaches are also fundamentally different. Starting from two-dimensional data, this invention separately proposes horizontal and vertical data features, and introduces hand keypoint attention mechanisms and frame number attention mechanisms. The attention mechanisms first change the weights of the input data before performing the recognition task.

[0133] The choice of the number of frames for a dynamic gesture has a significant impact on the recognition efficiency of the network model. More frames for a dynamic gesture result in higher recognition accuracy, but also increase the network's structural parameters and the number of frames to be processed, thus slowing down the recognition speed. Conversely, fewer frames result in lower recognition accuracy, but fewer network parameters and fewer frames to process, leading to faster recognition speed. This invention aims to find the optimal balance between the number of frames for dynamic gestures, selecting the optimal number of frames for dynamic gesture recognition to achieve both high recognition accuracy and good recognition speed—that is, maximizing the model's recognition efficiency.

[0134] like Figure 9As shown, in the frame rate and recognition rate test, the recognition accuracy continuously improves with the increase of the frame rate of a dynamic gesture; however, the model's recognition accuracy shows a significant improvement when the frame rate is around 2 to 35 frames, while the improvement is smaller between 35 and 60 frames. The highest recognition accuracy is 97.1% at 60 frames.

[0135] like Figure 10 As shown, in the frame rate and detection time test, as the frame rate increases, the amount of data that the model needs to detect to recognize dynamic gestures increases, and the model parameters also increase accordingly, resulting in a corresponding increase in the detection time. Overall, the increase is approximately linear, but the detection time is relatively small between 20 and 40 frames. The maximum detection time is 4.14 seconds.

[0136] Therefore, efficiency metrics are defined as follows:

[0137]

[0138] like Figure 11 As shown, the efficiency metric is calculated by dividing the number of frames by the detection time to get the detection time per frame, and multiplying the detection time per frame by the accuracy rate to define efficiency. It can be seen that the overall trend of the efficiency metric is that it first rises to a peak and then declines, reaching its highest point around 30 frames. That is, when each dynamic gesture is taken as 30 frames, the model recognition accuracy is 93.1%, and the model detection time is 1.68 seconds. This represents the highest efficiency for dynamic gesture frames while maintaining high recognition accuracy and model recognition speed.

[0139] This invention establishes a dynamic gesture recognition model based on keypoint attention and frame-number attention mechanisms. Based on the recognition of key points of the hand skeleton, it collects the keypoint coordinates of all frames of a dynamic gesture as recognition data. Simultaneously, it proposes a dynamic sampling method based on the degree of data change, employing dynamic sampling for data exceeding the standard frame number to retain key information to the greatest extent. A one-dimensional convolutional neural network is used to extract the frame-wide change features of each keypoint and the keypoint change features of each frame, respectively. Subsequently, an LSTM (Long Short-Term Memory) neural network is used to extract the temporal features of the data. Finally, a fully connected layer and normalization are used to output the recognition results.

[0140] This invention has the following two advantages: (1) This invention proposes a dynamic sampling method based on the degree of change in the spatial distance of key hand points. This method reduces repetitive and less varied data, ensures that the sampling process retains as much important information as possible, improves the quality of model training data, and indirectly improves the model's recognition efficiency. (2) This invention proposes a dynamic gesture recognition method based on the hand key point attention and frame number attention mechanism, which makes the model pay more attention to important hand key points and frame number, fully explores data features, and improves the model's recognition speed and accuracy.

[0141] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the invention should also be covered within the protection scope of the invention. Therefore, the protection scope of the invention should be determined by the scope of the claims.

Claims

1. A dynamic gesture recognition method based on hand key points and frame number attention mechanism, characterized in that, Comprise the following steps: S1, hand key point data acquisition and standardization: custom recognition hand key point data, collect all frames of dynamic gesture key point coordinates as recognition data, at the same time, according to the dynamic sampling of recognition data variation degree, including the following steps: S11, self-defined recognition hand key point data: including hand key point position coordinate format (x hj ,y hj ,z hj ), wherein h represents left and right hand number, h=l is left hand, h=r is right hand, and j is the number of the jth hand key point; S12, collect all frames of dynamic gesture key point coordinates: divide a dynamic gesture into 30 frames, so the data of each complete dynamic gesture is 42*30 position coordinates, each hand key point number is: L0 = (x l0 ,y l0 ,z l0 ) L1 = (x l1 ,y l1 ,z l1 ) … L 20 = (x l20 ,y l20 ,z l20 ) R0 = (x r0 ,y r0 ,z r0 ) R1 = (x r1 ,y r1 ,z r1 )... R 20 = (x r20 ,y r20 ,z r20 ) wherein L j , R j respectively represent all coordinates of the left-hand jth key point, all coordinates of the right-hand jth key point; S13, dynamic sampling according to the recognition data variation degree: through the distance size relationship of the spatial coordinate value change of each frame hand key point and the corresponding hand key point of the last frame, the importance of each frame coordinate information is evaluated, that is: wherein represents the distance between the jth keypoint of the ith frame and the jth keypoint of the previous frame; represents the x-coordinate of the jth keypoint of the ith frame, represents the x-coordinate of the jth keypoint of the ith-1 frame; defined as is the original sampling frame index, represents the total score of the ith frame; S131, select the first sampling frame, and The frame with the highest index is selected as the original sampling frame. S132, define sampling frame index: Wherein l is the frame number of the frame not selected as the sampling frame, l-1 is the selected sampling frame closest to the frame number of the frame, representing the total score of the first frame, selecting the first ranked frame as the sample frame; S133, repeat the operation, after selecting one sample frame, recalculate the index and select one sample frame, until 30 sample frames are selected, the sampling is completed, and the operation is ended. S2, construct hand key point attention mechanism: because the influence of hand key point attention and hand frame number attention on recognition effect is different, convolutional neural network is used to extract hand key point feature and hand frame number feature, and the corresponding weight is learned by full connection layer, including the following steps: S21, construction of hand key point attention mechanism: using CNN feature extraction layer to extract features of all frame numbers of each key point in the original data transversely, using full connection layer and normalization processing to extract the weight C of each hand key point h0 C h1 … h20 , the obtained weighting coefficient is multiplied by the corresponding original data to obtain hand coordinate data based on hand key point attention mechanism; S22, construction of hand frame number attention mechanism: all hand key point features of each frame are extracted from the original data using a one-dimensional convolutional neural network, and the weight C of each hand key point is extracted using a fully connected layer and normalization processing f0 C f1 … f49 The obtained weighting coefficient is multiplied by the corresponding original data to obtain hand coordinate data based on the hand frame number attention mechanism. S23, fusion of hand key point attention and hand frame number attention: CNN feature extraction module, wherein N2=2, that is, two CNN feature extraction modules are connected in series, which are first CNN feature extraction module and second CNN feature extraction module, and each CNN feature extraction module has 30 convolution layers, and the input data dimension of each convolution layer is 21*1; S3, LSTM long short term memory neural network dynamic recognition: use LSTM long short term memory neural network to extract data time feature, and finally use full connection layer and normalization to output recognition result.

2. The dynamic gesture recognition method based on hand key points and frame number attention mechanism according to claim 1, wherein, In the dynamic gesture of step S12, the sampling rule of 30 frames of dynamic gesture is as follows: in the dynamic gesture video data greater than 30 frames, the data is sampled to 30 frames to realize the standardization of input model data format; Considering that each dynamic gesture in actual sampling is not standard 30 frames, for data less than 30 frames, all 0 data is filled to 30 frames; for data greater than 30 frames, the frame number is sampled to 30 frames.

3. The dynamic gesture recognition method based on hand key points and frame number attention mechanism according to claim 2, wherein, In the dynamic sampling of step S13, the importance of hand key point coordinate data is related to the similarity of hand key point coordinate data of the last sampling frame, the greater the similarity of adjacent frame data, that is, the greater the data repetition, the smaller the importance of data.

4. The dynamic gesture recognition method based on hand key points and frame number attention mechanism according to claim 1, wherein, In the construction of the hand key point attention mechanism of the step S21, each CNN feature extraction module contains 21 convolution layers, each convolution layer includes one-dimensional valid convolution, a nonlinear activation function, and one-dimensional pooling layer, and the output of the feature extraction module is wherein n is the nth feature extraction module, is the weight of the vth convolution kernel of the nth feature extraction module, and * is a convolution operation, is the offset of the vth convolution operation of the nth feature extraction module. The full connection layer includes input layer, hidden layer and output layer, the input layer dimension is one-dimensional global pooling output, the number of neurons in the hidden layer is 21, in order to avoid overfitting, dropout layer is added to the linear layer, and the number of neurons in the output layer is 21.

5. The dynamic gesture recognition method based on hand key points and frame number attention mechanism according to claim 4, wherein, The CNN feature extraction module in the construction of the hand frame number attention mechanism in step S22 comprises N2 convolution layers, each of which is a combination of one-dimensional valid convolution, a nonlinear activation function and one-dimensional pooling. wherein n is the nth feature extraction module, is the weight of the vth convolution kernel of the nth feature extraction module, and * is a convolution operation, is the offset of the vth convolution operation of the nth feature extraction module. The full connection layer includes input layer, hidden layer and output layer, the input layer dimension is one-dimensional global pooling output, the number of neurons in the hidden layer is 30, in order to avoid overfitting, dropout layer is added to the linear layer, and the number of neurons in the output layer is 21.

6. The dynamic gesture recognition method based on hand key points and frame number attention mechanism according to claim 5, wherein, In the fusion of the hand key point attention and the hand frame number attention of the step S23, in the convolution layer of the first CNN feature extraction module, the input data dimension is 21x1, the one-dimensional convolutional neural network includes 8 7x1 convolution kernels with a step of 1, the one-dimensional maximum pooling size is 3x1, the activation function is a ReLu function, and the output dimension is 5x8; In the convolution layer of the second CNN feature extraction module, the input data dimension is 5x8, the one-dimensional convolutional neural network includes 16 3x8 convolution kernels with a step of 1, the one-dimensional maximum pooling size is 3x1, the activation function is a ReLu function, and the output dimension is 1x16; The one-dimensional global average pooling GAP size is 1x16, the output dimension is 1x1, and the output is each X_i, i is a dynamic gesture frame number sequence number, and j is in [0, 29].

7. The dynamic gesture recognition method based on hand key points and frame number attention mechanism according to claim 1, wherein, In the LSTM long short-term memory neural network dynamic recognition of the step S3, the following small steps are included: The data weighted by the key point attention and the frame attention mechanism are added together, in order to reduce the weakening of the time features of the weighted data, therefore, the original data and the data of the two attention mechanisms are added together again, the added data is input into the LSTM network module, the time features of the data are extracted, the gesture prediction result is obtained through normalization processing of a full connection layer; wherein D i is all hand key point coordinates of the i-th frame, i ∈ [0, 29], the input data dimension in the LSTM module is 30, and the hidden layer dimension is 30; The full connection layer input layer dimension is 30, the number of hidden layer neurons is 30, the output layer dimension is 30, the output layer is connected with a Softmax classification layer, and a model classification result is output.

8. The dynamic gesture recognition method based on hand key points and frame number attention mechanism according to claim 7, wherein, The LSTM long short-term memory neural network dynamic recognition of the step S23 further includes the following steps: In the LSTM long short-term memory neural network dynamic recognition part, the full connection layer input layer dimension is 21, the number of hidden layer neurons is 21, the number of output layer neurons is 21, and a Softmax function is used as a classification function for model classification; During training, no less than 1000 complete dynamic gesture training data are collected for each gesture, the collected samples are divided into a training set, a verification set and a test set according to a ratio of 6:2:2, the number of Epochs is set to be no less than one fifth of the data amount and no more than one half of the data amount, the training time increases and the model recognition accuracy no longer improves when the number of Epochs is too high, the learning rate of the Adam optimization algorithm is 0.001, the batch size is 100, and the optimal solution of the model is calculated by using a cross-entropy loss function.

Images