A gesture recognition method and system based on spatiotemporal networks

Abstract

This invention discloses a gesture recognition method and system based on a spatiotemporal network. The method constructs a spatiotemporal network including a convolutional neural network module, an attention module, and a long short-term memory (LSTM) network module for gesture recognition. The gesture image is input into the convolutional neural network and the attention module for feature extraction, yielding a first feature and a second feature respectively. The first and second features are then fused and input into the LSM network module to output the gesture recognition result. This invention combines a convolutional neural network with an LSM network, achieving efficient gesture recognition with a short response time. Furthermore, the model can be deployed on various platforms, exhibiting high scalability. The attention mechanism applies convolutional operations to the pooled feature map, focusing more on information that has already been pooled and aggregated, which is beneficial for extracting local features.

Description

Technical Field

[0001] This invention relates to a gesture recognition method and system, and more particularly to a gesture recognition method and system based on spatiotemporal networks. Background Technology

[0002] Gesture recognition technology, as a natural and intuitive interaction method, has been widely applied in smart homes, virtual reality, human-computer interaction, and gaming. Computer vision technology is one of the fundamental technologies for gesture recognition; by analyzing and processing gesture images, it can extract the feature information of gestures, enabling gesture recognition and classification. Machine learning technology is one of the core technologies for gesture recognition; by learning and training on large amounts of gesture data, it enables computers to recognize and understand gestures. Sensor technology is one of the important means of achieving gesture recognition, including technologies such as cameras, infrared sensors, gyroscopes, and accelerometers, which can acquire the dynamic information contained in gestures, achieving more accurate and precise gesture recognition.

[0003] Existing gesture recognition technologies face some challenges in practical applications, such as the accuracy of extracting gesture feature information and the real-time performance of gesture recognition, which require further research. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a gesture recognition method with short response time and high recognition accuracy; the second purpose of this invention is to provide a gesture recognition system with short response time and high recognition accuracy.

[0005] Technical solution: The gesture recognition method based on spatiotemporal networks described in this invention includes the following steps:

[0006] A spatiotemporal network is constructed, which includes a convolutional neural network module, an attention module, and a long short-term memory network module.

[0007] The gesture image is input into a convolutional neural network and an attention module for feature extraction to obtain the first feature and the second feature respectively; the first feature and the second feature are fused and input into a long short-term memory network module to output the gesture recognition result.

[0008] Furthermore, after the gesture image is input into the attention module, it is divided into two paths. The first path passes through an average pooling layer and a convolutional layer to obtain the third feature, and the second path passes through a max pooling layer and a convolutional layer to obtain the fourth feature. The third feature and the fourth feature are fused and then processed by an activation function to obtain the second feature.

[0009] Furthermore, the loss function of the spatiotemporal network is a multi-class cross-entropy loss function.

[0010] Furthermore, the experimental dataset used to train and validate the spatiotemporal network is a fusion of a local dataset and a hand dataset; the local dataset is constructed by: acquiring hand images through an image acquisition device, using a hand detection model in a lightweight recognition model to annotate key points of the acquired hand images and saving the spatial location information of each key point to obtain the local dataset.

[0011] Furthermore, the Long Short-Term Memory (LSTM) network module includes two cascaded LSM networks.

[0012] Furthermore, before inputting the gesture image into the convolutional neural network and the attention module to extract features and obtain the first and second features respectively, the process further includes preprocessing the gesture image;

[0013] The preprocessing includes image enhancement, image denoising, and image normalization, as well as using a hand detection model in a lightweight recognition model to detect key hand points in the gesture image and connect the key hand points to obtain the hand skeleton.

[0014] Based on the same inventive concept, the spatiotemporal network-based gesture recognition system of the present invention includes a spatiotemporal network model for outputting gesture recognition results based on the input gesture image;

[0015] The spatiotemporal network model includes a convolutional neural network module, an attention module, and a long short-term memory network module. After the gesture image passes through the convolutional neural network and the attention module, a first feature and a second feature are obtained. The first feature and the second feature are integrated and used as the input of the long short-term memory network module.

[0016] Furthermore, the attention module is divided into two paths: the first path includes an average pooling layer and a convolutional layer, and the second path includes a max pooling layer and a convolutional layer; the gesture image is processed through the average pooling layer and the convolutional layer to obtain the third feature, and the gesture image is processed through the max pooling layer and the convolutional layer to obtain the fourth feature; the third feature and the fourth feature are fused and then processed by an activation function to obtain the second feature.

[0017] The electronic device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the gesture recognition method based on a spatiotemporal network.

[0018] The computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the gesture recognition method based on spatiotemporal networks.

[0019] Beneficial effects: Compared with the prior art, the advantages of the present invention are: (1) The present invention combines convolutional neural network with long short-term memory network, which enhances feature extraction and data processing capabilities, realizes efficient judgment of gestures, and has the ability to judge in real time. The response time is short, and the model can be deployed on various platforms with high scalability; (2) The attention mechanism of the present invention is average pooling / max pooling, convolution, and feature fusion. Compared with the attention mechanism of average pooling / max pooling, feature fusion, and convolution, the convolution operation of the present invention is applied to the feature map after pooling, which can pay more attention to the information that has been pooled and aggregated, which is conducive to extracting some more abstract and local features. Then, feature fusion is performed on the feature maps obtained by different pooling operations to combine different types of information. Attached Figure Description

[0020] Figure 1 This is a flowchart of the gesture recognition method of the present invention;

[0021] Figure 2 This is a schematic diagram of 21 key hand points involved in the annotation of key hand points using a lightweight recognition model in an embodiment of the present invention;

[0022] Figure 3 This is a diagram of the attention module architecture of the present invention;

[0023] Figure 4 This is a diagram of the spatiotemporal network architecture of the present invention;

[0024] Figure 5 This is a schematic diagram illustrating that the gesture recognition result is category "3" in an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram illustrating that the gesture recognition result is category "10" in an embodiment of the present invention;

[0026] Figure 7 This is a schematic diagram showing that the gesture recognition result is category "right" in an embodiment of the present invention. Detailed Implementation

[0027] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0028] The basic principles for defining gesture semantics are as follows: First, gesture semantics are divided through action classification, with each gesture category corresponding to a specific action meaning. Second, action consistency requires that gestures of the same category maintain consistency in action to ensure that the model can accurately distinguish between different actions. Furthermore, spatial location is also an important criterion, as gestures with different hand or finger positions may have different meanings, and the model needs to be able to recognize and differentiate gestures from different positions. Finally, time series provide additional semantic information; information such as the sequence and duration of different gesture actions should be captured by the model to improve its understanding of gesture sequences.

[0029] like Figure 1 As shown, the gesture recognition method based on spatiotemporal networks includes the following steps:

[0030] Step 1: Construct the experimental dataset.

[0031] During the construction of the local dataset, the collected image data is adjusted to ensure it conforms to the image size, label, and other information of mainstream datasets, maintaining format consistency between the local and mainstream datasets and facilitating subsequent data processing and model training. Subsequently, the local dataset is merged with the mainstream dataset, introducing more diverse and varied samples, enabling the model to better learn and understand different features and patterns. This process includes the following steps:

[0032] Step 1.1: Determine the gesture categories to be included in the gesture set;

[0033] Step 1.2: Set the relevant parameters for image acquisition according to actual needs (such as how many images to acquire for each action, and how many frames of data are contained in one image);

[0034] Step 1.3: Turn on the image acquisition device to begin acquisition. During the acquisition process, a lightweight recognition model is used to annotate key hand points. Key hand points are as follows: Figure 2 As shown, the spatial coordinate information of each point is saved at the same time;

[0035] Step 1.4: After the data collection is completed, the local dataset is constructed. Then, the local dataset is combined with the current mainstream datasets to form the experimental dataset of this invention.

[0036] Step 2: Construct the convolutional neural network structure and attention module, using the experimental dataset as input.

[0037] Convolutional neural network structure is constructed, and the experimental dataset obtained in step 1 is used as the input of convolutional neural network module and attention module. Convolutional kernels are used to adaptively extract features. The extracted features are then pooled by max pooling layer to reduce the data dimensionality and retain the main feature information. The dimensionality-reduced feature data is fused with the features of attention module as input to long short-term memory network. At the same time, the network weights are updated and the relevant errors are calculated.

[0038] like Figure 3 As shown, the attention module (APAM), through the organic combination of average pooling and max pooling, improves the ability to extract and understand hand feature information, while also enhancing the model's performance on complex tasks and making the model's decision-making process more transparent and interpretable. Specifically, it includes the following steps:

[0039] Step 2.1: Construct the convolutional neural network structure;

[0040] Step 2.2: Use the dataset obtained in Step 1 as input to the convolutional neural network and the attention module;

[0041] Step 2.3: The convolutional neural network and the attention module perform feature extraction simultaneously. After the process is completed, the data obtained from feature extraction are integrated and used as input to the long short-term memory network.

[0042] Step 2.4: Calculate the error and update the network weights.

[0043] Step 3: Construct a Long Short-Term Memory (LSTM) network module. The network input is the data extracted in Step 2.

[0044] The Long Short-Term Memory (LSTM) network module consists of two cascaded LSM networks. It performs time-series modeling on the features from step 2, specifically including the following steps:

[0045] Step 3.1: Construct a Long Short-Term Memory (LSTM) network;

[0046] Step 3.2: Use the data information obtained in Step 2.3 after feature extraction by the convolutional neural network as the input to the Long Short-Term Memory network;

[0047] Step 3.3: Use a long short-term memory network to perform time series modeling on the features extracted by the convolutional neural network;

[0048] Step 3.4: Output the results and calculate the relevant errors;

[0049] Step 3.5, update network weights.

[0050] Step 4: Obtain the spatiotemporal network recognition model.

[0051] Traditional convolutional neural network (CNN) architectures stack multiple convolutional and pooling layers, ultimately outputting the prediction result through a fully connected layer. In contrast, such as Figure 4 As shown, the spatiotemporal network introduces a Long Short-Term Memory (LSTM) network on top of the traditional convolutional neural network (CNN) structure. Its CNN portion includes an input layer, convolutional layers, and pooling layers, as well as additional convolutional and pooling layers for further feature extraction. The LSTM portion includes an input layer, LSTM layers, and additional LSTM layers to increase the model's depth and capacity. Finally, a fully connected layer receives these features and outputs the results; specifically, the steps are as follows:

[0052] Step 4.1: Use the multi-class cross-entropy loss function to evaluate the results obtained in Step 3.4. The formula for the multi-class cross-entropy loss function is shown below:

[0053]

[0054] Where: h = [h0,…,h] c-1 ] is a probability distribution, h i Let y represent the probability that a sample belongs to the i-th class, y = [y0, ..., y]. c-1 When the sample is correctly classified into class i, y i =1, otherwise y i =0;

[0055] Step 4.2: If the accuracy requirement is met, the spatiotemporal network model will be output and used for gesture recognition;

[0056] Step 4.3: If the accuracy does not meet the requirements, update the network weights and continue training until the accuracy requirements are met.

[0057] Step 5: Use the obtained spatiotemporal network recognition model to perform gesture recognition.

[0058] Turn on the image acquisition device (e.g., a camera). The person being inspected makes a relevant gesture. The camera captures a real-time image stream and transmits it to the recognition model for processing. The model classifies and recognizes the gesture and outputs the corresponding recognition result. The process is automated. For image acquisition, a high-definition camera or a high-resolution monocular camera can be used. The minimum required resolution is 800×600. Place the camera slightly above the level of the person being inspected's head (e.g., fixed on a stand to avoid excessive downward or upward views). The camera should be tilted at an angle of approximately 30 to 45 degrees to provide a good field of view while ensuring the camera can capture sufficient detail. In actual deployment, fine-tune the camera angle based on the characteristics of the gesture and the scene.

[0059] After image acquisition, preprocessing operations such as image enhancement, denoising, and normalization are performed on the acquired gesture images to eliminate noise, blur, and distortion. This improves the reliability and consistency of image features, making subsequent feature extraction and modeling processes more accurate and stable.

[0060] Specifically, the steps include the following:

[0061] Step 5 involves using the obtained spatiotemporal network recognition model to perform gesture recognition, which includes the following steps:

[0062] Step 5.1: Based on the obtained spatiotemporal network recognition model, write a gesture recognition program;

[0063] Step 5.2, turn on the data acquisition device;

[0064] Step 5.3: Make a gesture towards the data acquisition device and wait for the recognition result.

[0065] Step 6: Output the recognition results.

[0066] Step 6.1: Use the gesture recognition module in the lightweight recognition model to detect hand position;

[0067] Step 6.2: Use the gesture recognition module mentioned in Step 6.1 to detect key points of the hand and connect them to display the complete hand skeleton;

[0068] Step 6.3: Wait for the judgment result of the gesture recognition program. The left side of the result display window will show the names of the gesture categories that need to be detected.

[0069] Step 6.4: The program returns the judgment result of gesture recognition, including the category to which the gesture belongs and the accuracy of that category. The category is the category name mentioned in step 6.3. The accuracy will be displayed as a progress bar at the bottom of each category name mentioned in step 6.3. The longer the progress bar, the higher the accuracy.

[0070] like Figures 5 to 7 The image shows three diagrams illustrating the results of gesture recognition, demonstrating the ability to recognize numbers 0-10 and left / right gestures. Figure 5 The result shown is that the gesture recognition result is category "3", and the progress bar represents the recognition accuracy. Figure 6 The result shown is category "10"; Figure 7 The result shown is the identification result for category "right".

[0071] The gesture recognition system based on spatiotemporal networks described in this invention includes a spatiotemporal network model for outputting gesture recognition results based on an input gesture image;

[0072] The spatiotemporal network model includes a convolutional neural network module, an attention module, and a long short-term memory network module. After the gesture image passes through the convolutional neural network and the attention module, a first feature and a second feature are obtained. The first feature and the second feature are integrated and used as the input of the long short-term memory network module.

[0073] The electronic device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the gesture recognition method based on a spatiotemporal network.

[0074] The computer-readable storage medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer.

[0075] The processor is used to execute a computer program stored in memory to implement the various steps in the methods described in the above embodiments.

Claims

1. A gesture recognition method based on a spatio-temporal network, characterized in that, The method comprises the following steps: real-time capture gesture image stream of the to-be-tested personnel through an image acquisition device; constructing a space-time network, the space-time network comprising a convolutional neural network module, an attention module and a long short-term memory network module; inputting the gesture image into the convolutional neural network and the attention module to respectively extract first features and second features; inputting the first features and the second features into the long short-term memory network module after fusion to output gesture recognition results; after the gesture image is input into the attention module, the gesture image is divided into two paths, the first path passes through an average pooling layer and a convolutional layer to obtain third features, and the second path passes through a maximum pooling layer and a convolutional layer to obtain fourth features; the third features and the fourth features are fused and then processed by an activation function to obtain the second features; the fusion of the third features and the fourth features comprises: respectively performing Relu activation and convolution on the third features and the fourth features and then splicing; 2. The gesture recognition method based on a spatio-temporal network according to claim 1, characterized in that, a loss function of the space-time network is a multi-class cross-entropy loss function; the multi-class cross-entropy loss function is as follows: ； where: is a probability distribution, denotes the probability that the sample belongs to the i-th class, when the sample is correctly classified to belong to the i-th class , otherwise .

3. The gesture recognition method based on spatio-temporal network according to claim 1, characterized in that, an experimental data set for training and verifying the space-time network is a fusion of a local data set and a hand data set; a construction method of the local data set is as follows: acquiring hand images through an image acquisition device, performing hand key point labeling on the acquired hand images by using a hand detection model in a lightweight recognition model, saving spatial position information of each key point, and obtaining the local data set.

4. The gesture recognition method based on a spatio-temporal network according to claim 1, characterized in that, the long short-term memory network module comprises two cascaded long short-term memory networks.

5. The gesture recognition method based on spatio-temporal network according to claim 1, characterized in that, the method further comprises pre-processing the gesture image before inputting the gesture image into the convolutional neural network and the attention module to respectively extract the first features and the second features; the pre-processing comprises image enhancement, image denoising and image normalization, and performing hand key point detection on the gesture image by using the hand detection model in the lightweight recognition model to obtain a hand skeleton by connecting the hand key points.

6. A gesture recognition system based on a spatio-temporal network, characterized in that, The method comprises the following steps: real-time capture gesture image stream of the to-be-tested personnel through an image acquisition device; constructing a space-time network, the space-time network comprising a convolutional neural network module, an attention module and a long short-term memory network module; inputting the gesture image into the convolutional neural network and the attention module to respectively extract first features and second features; inputting the first features and the second features into the long short-term memory network module after fusion to output gesture recognition results; after the gesture image is input into the attention module, the gesture image is divided into two paths, the first path passes through an average pooling layer and a convolutional layer to obtain third features, and the second path passes through a maximum pooling layer and a convolutional layer to obtain fourth features; the third features and the fourth features are fused and then processed by an activation function to obtain the second features; the fusion of the third features and the fourth features comprises: respectively performing Relu activation and convolution on the third features and the fourth features and then splicing; a loss function of the space-time network is a multi-class cross-entropy loss function; the multi-class cross-entropy loss function is as follows: an experimental data set for training and verifying the space-time network is a fusion of a local data set and a hand data set; a construction method of the local data set is as follows: acquiring hand images through an image acquisition device, performing hand key point labeling on the acquired hand images by using a hand detection model in a lightweight recognition model, saving spatial position information of each key point, and obtaining the local data set. the long short-term memory network module comprises two cascaded long short-term memory networks. the method further comprises pre-processing the gesture image before inputting the gesture image into the convolutional neural network and the attention module to respectively extract the first features and the second features; the pre-processing comprises image enhancement, image denoising and image normalization, and performing hand key point detection on the gesture image by using the hand detection model in the lightweight recognition model to obtain a hand skeleton by connecting the hand key points. The method comprises the following steps: real-time capture gesture image stream of the to-be-tested personnel through an image acquisition device; constructing a space-time network, the space-time network comprising a convolutional neural network module, an attention module and a long short-term memory network module; inputting the gesture image into the convolutional neural network and the attention module to respectively extract first features and second features; inputting the first features and the second features into the long short-term memory network module after fusion to output gesture recognition results; after the gesture image is input into the attention module, the gesture image is divided into two paths, the first path passes through an average pooling layer and a convolutional layer to obtain third features, and the second path passes through a maximum pooling layer and a convolutional layer to obtain fourth features; the third features and the fourth features are fused and then processed by an activation function to obtain the second features; the fusion of the third features and the fourth features comprises: respectively performing Relu activation and convolution on the third features and the fourth features and then splicing the fusion comprises activation, convolution and splicing operations.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer program, when loaded into the processor, implements the gesture recognition method based on the spatio-temporal network according to any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 7. The computer program, when executed by the processor, implements the gesture recognition method based on the spatio-temporal network according to any one of claims 1-5.

Images