Gesture recognition method and system based on multi-head attention and spatial-temporal feature fusion

Abstract

The invention belongs to the field of gesture recognition, and provides a multi-head attention and spatial-temporal feature fusion-based gesture recognition method and system, and the method comprises the steps: building a neural network model structure through a deep learning frame, importing a gesture sequence of hand articulation points containing time sequence features, and carrying out the recognition of the gesture sequence; and performing feature extraction, position coding and attention calculation, feature fusion and feature mapping on the gesture sequence data by using a proposed gesture recognition algorithm, and outputting a gesture classification result with higher accuracy. A new network topology structure is adopted, and a local and global multi-head attention module is utilized to capture a complex relationship in space and time in two dimensions of time and space, so that the problem that a current model is insufficient in capturing short-term and long-term dependency relationships when coping with complex time sequence data is solved; and the fusion algorithm is more flexible and efficient when processing different types of gesture sequence data.

Description

Technical Field

[0001] The present invention belongs to the technical field of gesture recognition, and in particular relates to a gesture recognition method and system based on multi-head attention and spatiotemporal feature fusion. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, with the continuous development of deep learning technology, the accuracy and efficiency of gesture recognition have been significantly improved. However, existing models still face many challenges in processing complex time series data, especially in capturing short-term and long-term dependencies.

[0004] Gesture recognition tasks usually involve a large amount of dynamically changing time series data, the characteristics of which are highly nonlinear and complex in the time dimension. Some improved models such as the spatiotemporal attention convolutional model (DG-STA) introduce attention mechanisms to enhance the ability to capture dependencies. However, such models may have insufficient sensitivity to changes in time series. This makes the information of important features often ignored in fast-changing gestures or long time series, reducing the accuracy of recognition. At the same time, the actions in the gesture sequence not only depend on the input at the current moment, but also are closely related to the previous state. Therefore, effectively capturing short-term and long-term dependencies is crucial to improving gesture recognition performance. At present, many existing methods such as the spatial-temporal graph convolutional network (ST-GCN) capture the dynamic characteristics of gestures by considering both spatial and temporal convolution operations. Although this method has advantages in capturing spatial-temporal dependencies, it may face the problem of information loss for long sequence data. The key to gestures lies in the relative motion dependencies between different joints. Ignoring this information may cause the model to be unable to accurately capture the essence of the action, especially when distinguishing between complex or similar gestures or modeling complex multi-joint gestures. The dependencies between the relative positions of the joints may not be fully considered, affecting the final recognition effect. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a gesture recognition method and system based on multi-head attention and spatiotemporal feature fusion, which effectively captures the short-term and long-term dependencies in complex time series data, and improves the modeling ability of relative position relationships in gesture sequences through position encoding; through multi-level feature extraction and fusion, not only the sensitivity to the dynamic changes of gestures is improved, but also the understanding of the relative position dependence between joints is enhanced, thereby providing higher accuracy and robustness for gesture recognition tasks.

[0006] According to some embodiments, a first solution of the present invention provides a gesture recognition method based on multi-head attention and spatiotemporal feature fusion, which adopts the following technical solutions:

[0007] The gesture recognition method based on multi-head attention and spatiotemporal feature fusion includes:

[0008] Get gesture sequence and preprocess it;

[0009] Based on the preprocessed gesture sequence, the trained gesture recognition network model is used to perform gesture category recognition, specifically:

[0010] Based on the preprocessed gesture sequence, the multi-sequence frame time features and the hand key point position features in each frame are extracted for encoding and feature extraction;

[0011] The extracted features are sent to the temporal feature module and the spatial feature module respectively to perform feature reconstruction using self-selected frequency domain convolution. Then, the key information of the reconstructed features is extracted according to the relative position encoding. The local global multi-head attention is used to capture the spatiotemporal relationship in the time series data. Finally, the channel aggregation convolution operation is performed.

[0012] The computational output results of the two modules of time and space are fused, and the gesture category results are output through feature mapping through the enhanced feedforward network.

[0013] Furthermore, the gesture sequence after preprocessing is encoded and feature extracted by extracting the time features of multiple sequence frames and the position features of the key points of the hand in each frame, specifically:

[0014] Determine a learnable parameter matrix according to the product of the length of the preprocessed gesture sequence and the number of joints and the feature dimension;

[0015] Generate a vector based on the position of each hand key point in the preprocessed gesture sequence, extract a set number of rows of position information based on the learnable parameter matrix and add them to the preprocessed gesture sequence to obtain the position encoding corresponding to the time feature of each frame in the preprocessed gesture sequence;

[0016] Temporal convolution is used to extract features from positional encoding information.

[0017] Furthermore, the self-selected frequency domain convolution is used to perform feature reconstruction, specifically:

[0018] Convert the extracted features from one dimension to two-dimensional representation, and then perform feature segmentation on the feature map of the two-dimensional representation;

[0019] Apply local convolution processing to a part of the feature map after segmentation, and merge the result of the local convolution with another part of the feature map after segmentation to obtain a reconstructed feature map;

[0020] The reconstructed feature map is flattened back into a one-dimensional representation, the one-dimensional feature map is processed by a linear layer and then averagely split, a part of the feature map is subjected to a depthwise separable convolution operation and then multiplied with another part of the feature map;

[0021] The result after the above multiplication is linearly transformed to restore the original dimension, and a residual connection is performed with the input extracted features to obtain the output result of the self-selected frequency domain convolution.

[0022] Furthermore, the depthwise separable convolution operation includes processing of a feature rearrangement layer, a depthwise separable convolution, and a flattening layer.

[0023] Furthermore, the local global multi-head attention is used to capture the spatiotemporal relationship in the time series data, specifically:

[0024] Based on the features after relative position encoding, calculate the local attention results and the global attention results respectively;

[0025] The score between the query and the key in the local attention is calculated according to the local mask and converted into the local attention weight, and the score between the query and the key in the global attention is calculated according to the global mask and converted into the global attention weight;

[0026] The local attention weights are used to weight the sum of the values ​​to obtain the final local attention output, and the global attention weights are used to weight the sum of the values ​​to obtain the final global attention output;

[0027] The local attention output and the global attention output are combined to obtain the temporal features of the gesture sequence.

[0028] Furthermore, the gesture recognition network model includes a position encoding module and a temporal convolution feature extraction module connected in sequence;

[0029] The temporal convolution feature extraction module is respectively connected to the temporal feature fusion module and the spatial feature fusion module;

[0030] The outputs of the temporal feature fusion module and the spatial feature fusion module are fused and output to the enhanced feedforward network module;

[0031] The enhanced feedforward network module outputs a gesture category recognition prediction result.

[0032] Furthermore, the temporal feature fusion module and the spatial feature fusion module both include a self-selected frequency domain convolution layer, a relative position encoding layer, a local global attention module and a channel aggregation convolution layer.

[0033] According to some embodiments, the second solution of the present invention provides a gesture recognition system based on multi-head attention and spatiotemporal feature fusion, which adopts the following technical solutions:

[0034] The gesture recognition system based on multi-head attention and spatiotemporal feature fusion includes:

[0035] A gesture sequence processing module, configured to obtain a gesture sequence and perform preprocessing;

[0036] The gesture recognition module is configured to perform gesture category recognition based on the preprocessed gesture sequence using the trained gesture recognition network model, specifically:

[0037] Based on the preprocessed gesture sequence, the multi-sequence frame time features and the hand key point position features in each frame are extracted for encoding and feature extraction;

[0038] The extracted features are sent to the temporal feature module and the spatial feature module respectively to perform feature reconstruction using self-selected frequency domain convolution. Then, the key information of the reconstructed features is extracted according to the relative position encoding. The local global multi-head attention is used to capture the spatiotemporal relationship in the time series data. Finally, the channel aggregation convolution operation is performed.

[0039] The computational output results of the two modules of time and space are fused, and the gesture category results are output through feature mapping through the enhanced feedforward network.

[0040] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0041] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the gesture recognition method based on multi-head attention and spatiotemporal feature fusion as described in the first aspect above.

[0042] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0043] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the gesture recognition method based on multi-head attention and spatiotemporal feature fusion as described in the first aspect above are implemented.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention constructs a gesture recognition network model based on the local-global multi-head attention mechanism and the fusion of spatiotemporal features. The neural network model structure is built using the framework of deep learning, and the gesture sequence of hand joints containing temporal features is imported. The proposed gesture recognition algorithm is used to perform feature extraction, position encoding and attention calculation, feature fusion and feature mapping on the gesture sequence data, and output gesture classification results with higher accuracy. The new network topology structure is adopted and the local-global multi-head attention module is used to capture the complex relationship in space and time in the two dimensions of time and space, which solves the problem that the current model does not capture short-term and long-term dependencies sufficiently when dealing with complex temporal data, making the fusion algorithm more flexible and efficient when processing different types of gesture sequence data. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0047] Figure 1 This is an overall flow chart of a gesture recognition method based on multi-head attention and spatiotemporal feature fusion in an embodiment of the present invention.

[0048] Figure 2 is a structural diagram of a gesture recognition network model in an embodiment of the present invention;

[0049] Figure 3 It is a structural diagram of a self-selected frequency domain convolution layer in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0051] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0052] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0053] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0054] Embodiment 1

[0055] This embodiment provides a gesture recognition method based on multi-head attention and spatiotemporal feature fusion. This embodiment uses the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a terminal, a server, and a system, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In this embodiment, the method includes the following steps:

[0056] Get gesture sequence and preprocess it;

[0057] Based on the preprocessed gesture sequence, the trained gesture recognition network model is used to perform gesture category recognition, specifically:

[0058] Based on the preprocessed gesture sequence, the multi-sequence frame time features and the hand key point position features in each frame are extracted for encoding and feature extraction;

[0059] The extracted features are sent to the temporal feature module and the spatial feature module respectively to perform feature reconstruction using self-selected frequency domain convolution. Then, the key information of the reconstructed features is extracted according to the relative position encoding. The local global multi-head attention is used to capture the spatiotemporal relationship in the time series data. Finally, the channel aggregation convolution operation is performed.

[0060] The computational output results of the two modules of time and space are fused, and the gesture category results are output through feature mapping through the enhanced feedforward network.

[0061] like Figure 1 As shown, the overall process of the method described in this embodiment is specifically as follows:

[0062] Step 1: Get gesture sequences and key point data from the set datasets DHG14 / 28 and Shrec'17, which contain 14 sets of gesture data. We extract 2660 sets of data as training sets and 140 sets of data as test sets from the two datasets through the written extraction scripts.

[0063] It can be explained that, in actual application, this embodiment acquires a gesture image and then pre-processes the gesture image.

[0064] Step 2: Preprocess the acquired gesture sequence and skeleton data. The specific approach of the present invention is, first, accept the video and label data, the input video length, and the flag of whether to use data enhancement during initialization. When obtaining a single data sample, extract the skeleton data and perform data enhancement through scaling, translation, noise addition, and time interpolation. We randomly sample a specified number of frames from the complete gesture sequence and normalize the skeleton data. In the present invention, the number of frames is set to 22 and the number of hand key points is 22. Finally, the processed skeleton data is converted into the Tensor format of PyTorch, and the samples and labels are returned to ensure that the data can be effectively used for model training. The overall process improves the diversity of data and the generalization ability of the model through random sampling and data enhancement.

[0065] The gesture image is preprocessed. First, the hand contour is extracted based on the gesture image. Then, frames are collected on the hand contour according to the hand joint data points (22 hand bones) to obtain a gesture sequence with skeleton data.

[0066] Step 3: Construct a gesture recognition network model, perform position encoding and temporal convolution on the preprocessed gesture sequence; then use self-selected frequency domain convolution for feature extraction, relative position encoding and attention calculation, feature fusion and feature mapping. Feature extraction and relative position encoding add position encoding to sequence data to retain time and space information and enhance the model's understanding of temporal and spatial position features; the feature fusion part maps the input features into query, key, and value vectors, and uses the local and global masks embedded in the multi-head attention to calculate the local and global attention respectively. The local and global attention modules capture temporal features and dependencies of different scales, return the corresponding results, and then send them to the channel aggregation convolution layer for calculation. The spatial features and temporal features are spliced ​​in the last dimension, and the spliced ​​features are mapped to the original feature dimension through the linear layer. The output features are added to the input features to form the final feature representation, retaining the original information; the feature mapping maps the fused features to the final number of categories through a linear layer, and outputs the prediction results.

[0067] like Figure 2 As shown, this embodiment proposes a gesture recognition network model based on a local-global multi-head attention mechanism and spatiotemporal feature fusion, including a position encoding module and a temporal convolution feature extraction module connected in sequence;

[0068] The temporal convolution feature extraction module is respectively connected to the temporal feature fusion module and the spatial feature fusion module;

[0069] The outputs of the temporal feature fusion module and the spatial feature fusion module are fused and output to the enhanced feedforward network module;

[0070] The enhanced feedforward network module outputs a gesture category recognition prediction result.

[0071] The temporal feature fusion module and the spatial feature fusion module both include a self-selected frequency domain convolution layer, a relative position encoding layer, a local global attention module, and a channel aggregation convolution layer.

[0072] The gesture recognition network model encodes and extracts features by extracting the time features of multiple sequence frames and the position features of the hand key points in each frame, and then sends the features to the time and space feature modules for self-selected frequency domain convolution, relative position encoding, attention calculation and channel aggregation convolution operations. The calculated output results of the two time and space modules are then feature fused, and finally the gesture category results are output through feature mapping through an enhanced feedforward network.

[0073] The method based on the preprocessed gesture sequence is to extract the time features of multiple sequence frames and the position features of the key points of the hand in each frame for encoding and feature extraction, specifically:

[0074] Position encoding is performed on the preprocessed gesture sequence according to the product of the reference sequence and the learnable position encoding parameter matrix;

[0075] And adding the position coding information to the preprocessed gesture sequence to obtain the gesture sequence position coding;

[0076] Temporal convolution is used to extract features from gesture sequence position encoding.

[0077] Positional encoding is a technique used to introduce positional information in sequence models, especially when there is no order information. By generating a vector for each position in the input sequence, the model is able to understand the relative and absolute positions at different time steps.

[0078] The implementation uses a learnable positional encoding parameter matrix P emb , whose shape is [P len ,F s ], where the reference sequence P len is the length of the reference sequence (22 frame sequences) multiplied by the number of reference joints (22 joints), F s It is the feature dimension, and position encoding is performed, specifically:

[0079]

[0080] In the forward propagation, the position code P enc Added to the input tensor x to enhance the position feature, the formula is as follows:

[0081] O enc =x+Penc

[0082] In the formula, let the input preprocessed gesture sequence be Where B is the batch size, T is the number of time steps, D is the feature dimension, and P enc is a learnable positional encoding corresponding to each time step of the input sequence.

[0083] The specific topological structure of the gesture recognition network model consists of four layers: position encoding, temporal convolution layer, spatiotemporal feature fusion module, and enhanced feedforward network. The spatiotemporal feature module structure includes a self-selected frequency domain convolution layer, a relative position encoding layer, a local global attention module, and a channel aggregation convolution layer.

[0084] A convolutional layer with a self-selected frequency-domain kernel is used, which consists of two linear layers and a depthwise separable convolution. The first linear layer is followed by an activation function (leaky ReLU by default) that expands the input dimension by two times and then splits it into two equal-width parts. The depthwise separable convolution uses group convolution so that each input channel has an independent filter. The second linear layer maps the hidden dimension back to the input dimension. A partial convolution is also included, which only acts on a portion of the input feature map.

[0085] The self-selected frequency domain convolution is used to perform feature reconstruction, specifically:

[0086] Convert the extracted features from one dimension to two-dimensional representation, and then perform feature segmentation on the feature map of the two-dimensional representation;

[0087] Apply local convolution processing to a part of the feature map after segmentation, and merge the result of the local convolution with another part of the feature map after segmentation to obtain a reconstructed feature map;

[0088] The reconstructed feature map is flattened back into a one-dimensional representation, the one-dimensional feature map is processed by a linear layer and then averagely split, a part of the feature map is subjected to a depthwise separable convolution operation and then multiplied with another part of the feature map;

[0089] The result after the above multiplication is linearly transformed to restore the original dimension, and a residual connection is performed with the input extracted features to obtain the output result of the self-selected frequency domain convolution.

[0090] like Figure 2 As shown in the figure, the structure of the self-selected frequency domain convolution layer is specifically to set the feature data after feature extraction by the temporal convolution layer as the input tensor of the self-selected frequency domain convolution layer in the spatiotemporal feature fusion module Where B represents the batch size, HW represents the product of the height and width of the feature map, and C represents the number of input channels. Define γ as the split ratio, where γ∈(0,1), and set the dimensions of the input tensor X in different directions to i and j represent the subscripts of the convolution kernel in the feature map, and k represents the number of feature dimensions to be processed locally.

[0091] During the forward propagation process, spatial reconstruction is performed, and a linear normalization layer is used to convert the input tensor X from one dimension to a two-dimensional feature map, namely:

[0092] X spatial [q]=X[p,k]

[0093] Then the feature map of the two-dimensional representation of the local convolutional layer is segmented according to a certain ratio γ:

[0094]

[0095]

[0096] here represents the integer part of Cγ, that is, the first γ ratio of the number of channels C;

[0097] Then, partial convolution (PConv) is applied to a part of the feature map after segmentation. PConv allows the network to use only the information of the valid area, which is particularly useful for repairing the missing part because the missing area does not affect the calculation. The result of the local convolution is merged with another part of the feature map after segmentation that has not been processed to obtain the reconstructed feature map:

[0098] X'1[q]=Σ m，n X1[q']·ω[m,n,l,k]

[0099] Among them, ω is the convolution kernel weight, m ​​and n are the offsets of the convolution kernel, and l is the input channel index. In this case, q′=B,i+m-1,j+n-1,l.

[0100] The feature map after the above reconstruction operation is flattened back into a one-dimensional representation:

[0101]

[0102] Apply the first linear layer and activation function, specifically:

[0103] Y[p]=act(Σ k X′[p,k]·o[k,j]+b[j]

[0104] Among them, act represents the activation function.

[0105] The transformed tensor linear layer output is evenly split using the feature segmentation layer, specifically:

[0106]

[0107]

[0108] Among them, r represents the scale factor of the internal feature dimension of the depth-separable layer, and hidden_dim represents the internal feature dimension of the depth-separable layer (usually called the hidden layer dimension)

[0109] Half of the output features Y1[p] are used for depthwise separable convolution operations, where the depthwise separable convolution operations include feature rearrangement layers (rearrange), depthwise separable convolutions, and flattening layers. Specifically,

[0110] Y′ [p] =act(Σ m Y1[p']·ω[m,k])

[0111] Where p′=B,i+m-1,j,k.

[0112] And multiply the other half of the output Y2[p] with the output features processed by the depthwise separable convolution operation:

[0113] Z[p]=Y'1[p]·Y2[p]

[0114] The result of the above multiplication is applied to the original dimension restored by the second linear transformation, and the residual connection and summation with the feature data-input tensor X after feature extraction through the temporal convolution layer are performed to obtain the output result of the self-selected frequency domain convolution layer:

[0115] O[p]=Σ k Z[p,k]·ω[k,j]+b[j]

[0116] The residual connection results obtained above are used in the time and space domains to obtain the attention calculation features of the input gesture sequence using relative position encoding, local-global multi-head attention mechanism, channel aggregation convolution and Softmax operation, and then the features of the two dimensions of time and space are fused through the Concat technology, and then the fused features are sent to the feature mapping-enhanced feedforward network module;

[0117] Relative position encoding is used to capture the relative position relationship between elements in a sequence. This encoding method is particularly important when dealing with sequences with dynamic changes, because it allows the model to consider the relative distance between elements when calculating attention, rather than just the absolute position. By generating a learnable relative position encoding parameter matrix, whose shape is [M len ,F s ], relative benchmark sequence M len is the length of the reference sequence (22 frame sequences) multiplied by the number of reference joints (22 joints), F sis the feature dimension. During forward propagation, the corresponding relative position encoding is selected based on the length of the input sequence and combined with the attention score to enhance the understanding of the relative position relationship.

[0118] The local global multi-head attention is used to capture the spatiotemporal relationship in the time series data, specifically:

[0119] Based on the features after relative position encoding, calculate the local attention results and the global attention results respectively;

[0120] The score between the query and the key in the local attention is calculated according to the local mask and converted into the local attention weight, and the score between the query and the key in the global attention is calculated according to the global mask and converted into the global attention weight;

[0121] The local attention weights are used to weight the sum of the values ​​to obtain the final local attention output, and the global attention weights are used to weight the sum of the values ​​to obtain the final global attention output;

[0122] The local attention output and the global attention output are combined to obtain the temporal features of the gesture sequence.

[0123] The encoded output data is sent to the time and space modules based on the local-global multi-head attention mechanism for attention calculation. First, a linear mapping is performed to convert the input tensor X r Mapping to query Q, key K, value V. For local attention Q loacl and global attention Q global , respectively define:

[0124]

[0125] Among them, W Q , W K , W V is the corresponding weight matrix, X r is the data tensor after relative position encoding. Then the attention score is calculated. For local attention, the score between the query and the key is calculated:

[0126]

[0127] Among them, M loacl is a local mask, used to ensure that attention is only focused on the elements within the current window, d K represents the dimension of the key K, and T is the transpose.

[0128] When performing specific numerical calculations, the feature dimension F can be s As part of d K , for example, when F s =128, dK It can be 32 or 64, etc.; set the feature dimension F s As part of d K , represents the dimension of key K, and the total number of attention heads is h, then Scaling Factor Used to prevent the dot product from being too large and ensure the stability of the gradient.

[0129] For global attention, calculate the score:

[0130]

[0131] Among them, M global is a global mask, usually a matrix of all 1s, allowing attention to the entire sequence, and T is the transpose. Then the attention weights are calculated, and the scores are converted to weights using the softmax function:

[0132] A loacl =ssoftmax(S loacl )

[0133] A global =softmax(S global )

[0134] The weighted sum is used to get the output, and the values ​​are weighted summed using the attention weights to get the final local and global attention outputs, and the local and global attention outputs are combined to get the final result:

[0135]

[0136] Among them, V loacl is the local attention calculation value, V global is the global attention calculation value.

[0137] The above attention calculation result O lg As the input of the channel aggregation convolution layer, the channel aggregation convolution layer structure is two layers, where the first layer consists of a fully connected layer FC1, a depth-separable convolution layer, an activation function layer, and a Dropout layer, and the second layer consists of a 1x1 convolution layer, a fully connected layer FC2, and a Dropout layer. lg The feature is expanded through the fully connected layer FC1 in the first layer, and then the spatial information is fused through the depthwise separable convolution layer, followed by the activation function and the Dropout layer, and then enters the 1x1 convolution layer in the second layer for feature decomposition, and then the feature is compressed back to the original dimension through the fully connected layer FC2, and the Dropout layer is applied again, and finally the processed feature data is output.

[0138] The feature mapping uses the Enhanced Feedforward Network EnhancedFFN module, which first passes the input data through the first linear layer to map the dimension from the input to the hidden layer, performs a nonlinear transformation through the activation function Leaky ReLU, uses Dropout to randomly mask a part of the neurons to enhance the generalization ability of the model, and then passes it through the second linear layer for processing. The above process is repeated until it passes through the last linear layer, at which time the features are mapped to the output layer to finally obtain the gesture classification result.

[0139] The present invention defines that the temporal feature fusion module and the spatial feature fusion module are composed of self-selected frequency domain convolution, relative position coding, local global multi-head attention and channel aggregation convolution modules. The core idea is to use self-selected frequency domain convolution to reconstruct the features in the time domain and space domain respectively, use relative position coding to highlight the reconstructed feature information, use local attention to capture short-term dependencies, use global attention to integrate long-term dependencies, combine multi-head mechanisms to learn different features in parallel, so as to fully understand the spatiotemporal relationship in complex gesture time series data, and use channel aggregation convolution to control the feature information flow to achieve effective information fusion. Due to the differences in sequence length and gesture changes in gesture sequence data input, different from other attention algorithms, the temporal feature fusion module and spatial feature fusion module based on local global multi-head attention fusion proposed in the present invention can effectively capture the relationship between adjacent time steps through short-term and long-term dependencies in the sequence, integrate global context information, help the model understand the instantaneous changes and global structural characteristics of data in the sequence, and achieve effective information fusion.

[0140] Step 4: Train the gesture recognition network of the present invention. The present invention adopts the following parameter settings: We use DHG14 / 28 and Shrec'17 data sets as training sets, the network training cycle is set to 300, and the batch size and learning rate are set to 32 and 10-3 respectively. The network model is trained for two data modes of 14Gestures and 28Gestures. The hardware environment of our gesture recognition network is RTX4090 / 24GB RAM, and the software environment uses windows10, python 3.9, pytorch 2.4.1. The gesture recognition network we invented trains and saves the weights of the entire network.

[0141] Step 5: Inferring gestures. Input the gesture sequence data to be recognized into the weights trained in step 5 to obtain the gesture classification result, and then end the step.

[0142] Embodiment 2

[0143] This embodiment provides a gesture recognition system based on multi-head attention and spatiotemporal feature fusion, including:

[0144] A gesture sequence processing module, configured to obtain a gesture sequence and perform preprocessing;

[0145] The gesture recognition module is configured to perform gesture category recognition based on the preprocessed gesture sequence using the trained gesture recognition network model, specifically:

[0146] Based on the preprocessed gesture sequence, the multi-sequence frame time features and the hand key point position features in each frame are extracted for encoding and feature extraction;

[0147] The extracted features are sent to the temporal feature module and the spatial feature module respectively to perform feature reconstruction using self-selected frequency domain convolution. Then, the key information of the reconstructed features is extracted according to the relative position encoding. The local global multi-head attention is used to capture the spatiotemporal relationship in the time series data. Finally, the channel aggregation convolution operation is performed.

[0148] The computational output results of the two modules of time and space are fused, and the gesture category results are output through feature mapping through the enhanced feedforward network.

[0149] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0150] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0151] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0152] Embodiment 3

[0153] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the gesture recognition method based on multi-head attention and spatiotemporal feature fusion as described in the first embodiment are implemented.

[0154] Embodiment 4

[0155] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the gesture recognition method based on multi-head attention and spatiotemporal feature fusion as described in the first embodiment are implemented.

[0156] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0157] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0158] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0160] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0161] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A gesture recognition method based on multi-head attention and spatiotemporal feature fusion, characterized in that: include: Get gesture sequence and preprocess it; Based on the preprocessed gesture sequence, the trained gesture recognition network model is used to perform gesture category recognition, specifically: Based on the preprocessed gesture sequence, the multi-sequence frame time features and the hand key point position features in each frame are extracted for encoding and feature extraction; The extracted features are sent to the temporal feature module and the spatial feature module respectively to perform feature reconstruction using self-selected frequency domain convolution. Then, the key information of the reconstructed features is extracted according to the relative position encoding. The local global multi-head attention is used to capture the spatiotemporal relationship in the time series data. Finally, the channel aggregation convolution operation is performed. The computational output results of the two modules of time and space are fused, and the gesture category results are output through feature mapping through the enhanced feedforward network.

2. The gesture recognition method based on multi-head attention and spatiotemporal feature fusion as claimed in claim 1, characterized in that: The gesture sequence after preprocessing is encoded and feature extracted by extracting the time features of multiple sequence frames and the position features of the key points of the hand in each frame, specifically: Determine a learnable parameter matrix according to the product of the length of the preprocessed gesture sequence and the number of joints and the feature dimension; Generate a vector based on the position of each hand key point in the preprocessed gesture sequence, extract a set number of rows of position information based on the learnable parameter matrix and add them to the preprocessed gesture sequence to obtain the position encoding corresponding to the time feature of each frame in the preprocessed gesture sequence; Temporal convolution is used to extract features from positional encoding information.

3. The gesture recognition method based on multi-head attention and spatiotemporal feature fusion as claimed in claim 1, characterized in that: The self-selected frequency domain convolution is used to perform feature reconstruction, specifically: Convert the extracted features from one dimension to two-dimensional representation, and then perform feature segmentation on the feature map of the two-dimensional representation; Apply local convolution processing to a part of the feature map after segmentation, and merge the result of the local convolution with another part of the feature map after segmentation to obtain a reconstructed feature map; The reconstructed feature map is flattened back into a one-dimensional representation, the one-dimensional feature map is processed by a linear layer and then averagely split, a part of the feature map is subjected to a depthwise separable convolution operation and then multiplied with another part of the feature map; The result after the above multiplication is linearly transformed to restore the original dimension, and a residual connection is performed with the input extracted features to obtain the output result of the self-selected frequency domain convolution.

4. The gesture recognition method based on multi-head attention and spatiotemporal feature fusion as claimed in claim 3, characterized in that: The depthwise separable convolution operation includes processing of a feature rearrangement layer, a depthwise separable convolution, and a flattening layer.

5. The gesture recognition method based on multi-head attention and spatiotemporal feature fusion as claimed in claim 1, characterized in that: The local global multi-head attention is used to capture the spatiotemporal relationship in the time series data, specifically: Based on the features after relative position encoding, local attention results and global attention results are calculated respectively; The score between the query and the key in the local attention is calculated according to the local mask and converted into the local attention weight, and the score between the query and the key in the global attention is calculated according to the global mask; And converted into global attention weight; The local attention weights are used to weight the sum of the values ​​to obtain the final local attention output, and the global attention weights are used to weight the sum of the values ​​to obtain the final global attention output; The local attention output and the global attention output are combined to obtain the temporal features of the gesture sequence.

6. The gesture recognition method based on multi-head attention and spatiotemporal feature fusion as claimed in claim 1, characterized in that: The gesture recognition network model includes a position encoding module and a temporal convolution feature extraction module connected in sequence; The temporal convolution feature extraction module is respectively connected to the temporal feature fusion module and the spatial feature fusion module; The outputs of the temporal feature fusion module and the spatial feature fusion module are fused and output to the enhanced feedforward network module; The enhanced feedforward network module outputs a gesture category recognition prediction result.

7. The gesture recognition method based on multi-head attention and spatiotemporal feature fusion as claimed in claim 6, characterized in that: The temporal feature fusion module and the spatial feature fusion module both include a self-selected frequency domain convolution layer, a relative position encoding layer, a local global attention module, and a channel aggregation convolution layer.

8. A gesture recognition system based on multi-head attention and spatiotemporal feature fusion, characterized in that: include: A gesture sequence processing module, configured to obtain a gesture sequence and perform preprocessing; The gesture recognition module is configured to perform gesture category recognition based on the preprocessed gesture sequence using the trained gesture recognition network model, specifically: Based on the preprocessed gesture sequence, the multi-sequence frame time features and the hand key point position features in each frame are extracted for encoding and feature extraction; The extracted features are sent to the temporal feature module and the spatial feature module respectively to perform feature reconstruction using self-selected frequency domain convolution. Then, the key information of the reconstructed features is extracted according to the relative position encoding. The local global multi-head attention is used to capture the spatiotemporal relationship in the time series data. Finally, the channel aggregation convolution operation is performed. The computational output results of the two modules of time and space are fused, and the gesture category results are output through feature mapping through the enhanced feedforward network.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the gesture recognition method based on multi-head attention and spatiotemporal feature fusion as described in any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the gesture recognition method based on multi-head attention and spatiotemporal feature fusion as described in any one of claims 1 to 7 are implemented.

Images