Dynamic gesture recognition method based on 3DCNN and Transformer
The integration of 3DCNN and Transformer in gesture recognition systems addresses the limitations of existing methods by enhancing local and global feature extraction, achieving high accuracy in diverse gesture recognition.
Patent Information
- Application Number
- CN202411184616.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-20
AI Technical Summary
Existing dynamic gesture recognition systems are limited by high costs due to the use of depth cameras and lack robustness in recognizing diverse gestures, often focusing on local features while neglecting global characteristics, leading to poor recognition stability and accuracy.
A method utilizing 3DCNN for local feature extraction and Transformer for global feature representation, combined with a classifier network, to process video frame sequences, enhancing the recognition of multiple gestures by integrating local and global features through 3D convolution and contextual semantic fusion.
The method achieves over 96% accuracy in recognizing 40 different types of gestures, demonstrating improved stability and accuracy by leveraging 3DCNN and Transformer for comprehensive feature representation.
Smart Images
Figure CN119169699A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of dynamic gesture recognition, and in particular to a dynamic gesture recognition method based on 3DCNN and Transformer. Background Art
[0002] The automatic recognition of human posture and gestures based on vision can greatly improve the naturalness and efficiency of human-computer interaction, and has a wide range of application scenarios and needs. Since gestures can represent rich semantic information, such as sign language and other complex application scenarios, how to achieve dynamic gesture recognition is a hot research direction in the research field.
[0003] There are two common methods for data collection at present, namely ordinary RGB cameras and depth cameras with depth information. The data collected based on depth cameras contains depth information, which can provide better three-dimensional point cloud data and improve the accuracy of gesture recognition to a certain extent. However, the cost of depth cameras is high, which limits the scope of their application to a certain extent. With the development of machine learning, especially deep learning technology, a large number of advanced algorithms can still achieve high recognition accuracy without explicit depth measurement data, thereby reducing the requirements for acquisition equipment and greatly reducing the cost of actual applications.
[0004] Existing deep learning frameworks generally use end-to-end technology to achieve direct mapping from input to output, thereby simplifying the algorithm design process. Some existing dynamic gesture recognition solutions are too simple and can only recognize a few or a dozen fixed gestures, and the recognition stability and accuracy are poor.
[0005] Some solutions focus too much on local features and ignore the representation of global characteristics. How to achieve accurate recognition of more types of gestures and how to ensure the reliability and stability of recognition are issues that need to be urgently addressed.
[0006] To this end, a dynamic gesture recognition method based on 3DCNN and Transformer is proposed. Summary of the invention
[0007] The purpose of the present invention is to provide a dynamic gesture recognition method based on 3DCNN and Transformer to solve the problems raised in the above background technology.
[0008] To achieve the above object, the present invention provides the following technical solution: a dynamic gesture recognition method based on 3DCNN and Transformer, comprising the following steps:
[0009] S1, capturing dynamic gestures based on a camera, obtaining video frame sequence images, and preprocessing the video frame sequence images;
[0010] S2, a video frame sequence S of a specific length m is divided into k video subsequences si according to a fixed number n, where i = [1, k];
[0011] S3, for each video subsequence si, perform feature extraction operation based on a 3DCNN network;
[0012] S4. Assume that the number of channels of the feature map of each video subsequence after 3DCNN processing is p, flatten the feature maps of these p channels and connect them end to end to form a feature vector v, whose dimension is defined as d;
[0013] S5. The channel feature maps of all video subsequences are stacked together in sequence to form kp feature maps in total. Alternatively, the final output vectors v of the video subsequences can be directly stacked together in sequence to form k feature maps in total.
[0014] S6. Flatten each feature map generated in the previous step into a vector representation, and set a position code for each feature map according to its position characteristics;
[0015] S7, the feature sequence and the corresponding position code of the previous step are sent to a Transformer encoder module, the number of vectors outputted by the module is the same as the input, and the output feature sequence is flattened into a one-dimensional vector;
[0016] S8. Send the global feature vector obtained in the previous step into a classifier network for category mapping. The classifier uses a 1-2 layer fully connected neural network and finally realizes the normalized category probability output through a softmax layer.
[0017] Preferably: in step S1, the preprocessing mainly includes the following aspects:
[0018] 1. Locate the face area, and then determine a rectangular frame that can completely contain the hand activity area based on the proportion characteristics of each part of the human body and the range characteristics of the hand activity;
[0019] 2. Based on the rectangular frame of the hand activity area, crop the image frame to obtain the gesture image;
[0020] 3. Normalize the sizes of all gesture images.
[0021] Preferably: in step S2, two adjacent subsequences may overlap, and the degree of overlap is controlled by a sliding step, that is, a fixed window length n is used and a specific step length is used to slide on the sequence S to extract the subsequence.
[0022] Preferably: in the step S3, this operation is a local feature representation link, and multiple 3D convolutional layers can be set for serial connection, and a pooling layer can be added after each convolutional layer as needed.
[0023] Preferably, in step S7, the vector is the result of contextual semantic fusion of local features of the video frame sequence, reflecting the expression of global features and high-level semantics.
[0024] Compared with the prior art, the beneficial effects of the present invention are: the present invention can achieve comprehensive representation of local features and global features in video frame sequences, and has stronger feature characterization capabilities. In particular, the association of contextual semantics can be achieved by sliding the 3D convolutional network on the timeline, and the association of higher-level contextual semantics can be achieved based on Transformer. On the public dynamic gesture dataset KSU-SSL (containing 40 different types of gestures), an accuracy rate of more than 96% can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 This is the overall framework diagram of the dynamic gesture recognition method based on 3DCNN and Transformer. DETAILED DESCRIPTION
[0026] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0027] See also Figure 1 The present invention provides a technical solution: a method for implementing dynamic gesture recognition based on 3DCNN and Transformer, which can obtain local features based on an efficient convolutional network and obtain global features using a Transformer model.
[0028] In this embodiment, the preprocessing process of the original signal is completed using the integrated tool MATLAB or the open source Python algorithm package. The convolutional neural network and Transformer model are built based on Pytorch, one of the most popular deep learning frameworks.
[0029] like Figure 1 As shown, the specific implementation steps are as follows:
[0030] 1. Based on the camera, dynamic gestures are collected, video frame sequence images are obtained, and the video frame sequence images are preprocessed, which mainly includes the following aspects:
[0031] (1) Locate the face area. Based on the open source face detection algorithm package, the face area is located, and the size of the positioning rectangle is W*H.
[0032] (2) Gesture image cropping. Based on the face area, expand 150 lines upward, 250 lines downward, 100 lines leftward, and 100 lines rightward. Crop the image based on the extended rectangular frame to form a gesture image.
[0033] (3) Normalize the size of all gesture images and use linear scaling to scale the images to 128×128.
[0034] 2. Cut the input video stream with a length of 32 frames, which is defined as sequence S. Define the sliding step size as 8 and the length of the video frame subsequence as 16, then the sequence can be divided into 3 segments, and the adjacent subsequences have a 50% overlap. In this way, 3 video subsequences si are formed, where i = [1, 3].
[0035] 3. For each video subsequence si, a feature extraction operation is performed based on a 3DCNN network. This operation can be regarded as a local feature representation link. Each subsequence si has 16 pictures, which are slid based on the set 3D convolution template.
[0036] 4. Here, four convolutional layers are set, and a maximum pooling layer is added after each layer. The convolution kernel size of each convolutional layer is 3×3×16, the sliding step length stride=1, and the number of templates is 10. The parameters of the pooling layer are 2×2×1, stride=2. The activation function of the neuron uses the Relu function.
[0037] 5. Stack the feature maps of all video subsequences in order to form 3 feature maps in total.
[0038] 6. Flatten each feature map generated in the previous step into a vector representation with a dimension of 640. This forms a feature vector sequence consisting of 3 vectors. Set the position encoding for each feature map according to its position characteristics.
[0039] 7. The feature sequence and the corresponding position code in the previous step are sent to a Transformer encoder module, and its output is still a sequence of 3 vectors. The feature sequence is flattened into a one-dimensional vector. This vector can be regarded as the result of contextual semantic fusion of the local features of the video frame sequence, reflecting the expression of global features and high-level semantics.
[0040] 8. The global feature vector obtained in the above step is sent to a classifier network for category mapping. The classifier uses a 2-layer fully connected neural network and finally achieves normalized category probability output through a softmax layer.
[0041] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A dynamic gesture recognition method based on 3DCNN and Transformer, characterized in that: The following steps are involved: S1, capturing dynamic gestures based on a camera, obtaining video frame sequence images, and preprocessing the video frame sequence images; S2, a video frame sequence S of a specific length m is segmented into a fixed number n, into k video subsequences si, where i = [1, k]; S3, for each video subsequence si, perform feature extraction operation based on a 3DCNN network; S4. Assume that the number of channels of the feature map of each video subsequence after 3DCNN processing is p, flatten the feature maps of these p channels and connect them end to end to form a feature vector v, whose dimension is defined as d; S5. The channel feature maps of all video subsequences are stacked together in sequence to form kp feature maps in total. Alternatively, the final output vectors v of the video subsequences can be directly stacked together in sequence to form k feature maps in total. S6. Flatten each feature map generated in the previous step into a vector representation, and set a position code for each feature map according to its position characteristics; S7, the feature sequence and the corresponding position code of the previous step are sent to a Transformer encoder module, the number of vectors outputted by the module is the same as the input, and the output feature sequence is flattened into a one-dimensional vector; S8. Send the global feature vector obtained in the previous step into a classifier network for category mapping. The classifier uses a 1-2 layer fully connected neural network and finally realizes the normalized category probability output through a softmax layer.
2. The dynamic gesture recognition method based on 3DCNN and Transformer according to claim 1, characterized in that: In step S1, the preprocessing mainly includes the following aspects:
1. Locate the face area, and then determine a rectangular frame that can completely contain the hand activity area based on the proportion characteristics of each part of the human body and the range characteristics of the hand activity; 2. Based on the rectangular frame of the hand activity area, crop the image frame to obtain the gesture image; 3. Normalize the sizes of all gesture images.
3. The dynamic gesture recognition method based on 3DCNN and Transformer according to claim 1, characterized in that: In step S2, two adjacent subsequences may overlap, and the degree of overlap is controlled by a sliding step, that is, a fixed window length n is used and a specific step length is used to slide on the sequence S to extract the subsequence.
4. The dynamic gesture recognition method based on 3DCNN and Transformer according to claim 1, characterized in that: In step S3, this operation is a local feature representation link, and multiple 3D convolutional layers can be set for serial connection, and a pooling layer can be added after each convolutional layer as needed.
5. The dynamic gesture recognition method based on 3DCNN and Transformer according to claim 1, characterized in that: In step S7, the vector is the result of contextual semantic fusion of local features of the video frame sequence, reflecting the expression of global features and high-level semantics.