Gesture recognition method fusing continuous frame segmentation and spatial features

Abstract

The invention discloses a gesture recognition method fusing continuous frame segmentation and spatial features. The gesture recognition method comprises the steps that three-dimensional coordinates of hand joint points are obtained through Mediape; performing dual normalization processing to eliminate size and position differences; extracting point cloud convex hull features, joint point distance features and specific triangularization area combination features; fusing a plurality of types of features and then training a Transform or GCN model; carrying out automatic segmentation on the continuous gesture sequence based on double cosine similarity thresholds; and inputting the segmented stable paragraphs into the model to realize real-time identification. The method provided by the invention has high precision and strong robustness in a complex scene, and is suitable for the fields of man-machine interaction, intelligent control and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and human-computer interaction, and particularly relates to a gesture recognition method fusing continuous frame segmentation and spatial features. BACKGROUND

[0002] Gesture recognition, as an important research direction of computer science, has wide application prospects in the fields of human-computer interaction and intelligent control. Existing gesture recognition methods mainly include several categories based on data gloves, surface electromyography sensors, WIFI signals and visual sensors. Among them, the methods based on data gloves and surface electromyography sensors need to rely on special wearable devices, which limits the freedom of user's actions and has poor portability; the method based on WIFI signals performs poorly in complex gesture recognition; the method based on visual sensors mainly covers two categories based on depth images and color images, and the color image is easily affected by light changes, dynamic backgrounds and multiple people interference, and has limited recognition accuracy; although the depth image overcomes the interference of light and skin color to some extent, it still faces stability challenges in the presence of multiple people and complex scenes.

[0003] In recent years, gesture recognition methods based on skeleton data have attracted attention due to their robustness in complex environments. However, existing methods mostly rely on pre-segmented data segments, ignoring the temporal information in continuous gesture actions, and have deficiencies in hand size normalization and geometric feature representation, which limit the generalization ability and recognition accuracy of the model. SUMMARY

[0004] The present application aims to solve the problems of sensitivity to hand size, dependence on pre-segmented data and limited feature representation in existing gesture recognition methods, and proposes a gesture recognition method fusing continuous frame segmentation and spatial features.

[0005] The gesture recognition method fusing continuous frame segmentation and spatial features according to the present application comprises the following steps:

[0006] Step S1, input original skeleton point data: collect the original three-dimensional coordinate data of 21 joints of the left and right hands from a common camera through the Mediapipe framework;

[0007] Step S2, data preprocessing: perform double normalization on the original three-dimensional coordinates to eliminate the influence of different user hand sizes and spatial position differences, specifically including:

[0008] Position normalization: subtract the mean value of all joints from each joint coordinate;

[0009] Scale normalization: divide all coordinate points by the maximum distance from the joint to the center point;

[0010] Step S3, feature extraction:

[0011] constructing a point cloud object, calculating the convex hull surface area and volume;

[0012] calculating the Euclidean distance between the normalized joint points;

[0013] dividing the 21 joint points of the hand into 7 groups according to a specific combination, each group containing 3 joint points, forming a spatial triangle, calculating the area of each group of triangles respectively, and performing normalization processing to highlight the geometric differences between different gestures;

[0014] Step S4, feature fusion: the double-normalized coordinate features and the triangle area features are fused to form a final feature vector;

[0015] Step S5, model training: the fused features are input into a deep learning model (Transformer or graph convolution network GCN) for training, and the gesture recognition model with the best recognition performance is saved;

[0016] Step S6, collect continuous gesture sequence: collect a continuous gesture action sequence containing non-fixed start and end time, and obtain the original three-dimensional skeleton coordinates thereof;

[0017] Step S7, continuous sequence segmentation: based on the cosine similarity double threshold method, the continuous gesture sequence is automatically segmented to extract stable individual gesture passages, the cosine similarity between adjacent frames is calculated, and double thresholds are set for segmentation:

[0018] Threshold one: the similarity range is between 1.9995 and 2.00;

[0019] Threshold two: requires at least 30 consecutive valid frames in each segment, and merges passages with a distance of no more than 1 frame to form a stable gesture passage;

[0020] Step S8, gesture recognition: input the segmented stable gesture passage into the best gesture recognition model to output the recognition result.

[0021] The application also discloses a gesture recognition system for implementing the gesture recognition method, and the system comprises:

[0022] A data acquisition module is used to obtain hand joint point coordinates through a camera and a hand key point detection framework;

[0023] A preprocessing module is used to perform double normalization processing on the joint point coordinates;

[0024] A feature extraction and fusion module is used to construct a point cloud object, extract spatial geometric features, and perform feature fusion;

[0025] A model training module is configured to train a gesture recognition model using the fused features and save an optimal model.

[0026] A continuous sequence segmentation module is configured to automatically segment continuous gesture sequences based on cosine similarity.

[0027] An identification output module is configured to input the segmented paragraphs into the optimal model and output an identification result.

[0028] The application further discloses a computer readable storage medium having a computer program stored thereon, and the program is executed by a processor to implement the gesture recognition method.

[0029] The application has the following advantages:

[0030] The application extracts hand skeleton point data based on a Mediapipe framework and introduces a double normalization technique, effectively eliminating the interference caused by the size difference and spatial position change of the hand, and significantly improving the generalization ability and recognition accuracy of the model in different users and multiple scenarios.

[0031] For continuous action sequences in gesture recognition, the application proposes a double-threshold segmentation method based on cosine similarity, which realizes automatic segmentation of indefinite-length hand action sequences and overcomes the limitations of traditional methods that rely on manual segmentation of data segments. This method is not only suitable for gestures with strong continuity, but also effectively deals with the uncertainty and dynamic changes in gesture data. The introduction of geometric features based on point cloud triangulation area enhances the ability to capture subtle gesture differences.

[0032] In terms of feature extraction, the application method comprehensively utilizes multiple features, including Euclidean distance, convex hull volume and convex hull surface area of point cloud objects, and conducts comparative experiments combined with multiple mainstream gesture recognition models. Experimental results show that the geometric feature method based on specific triangulation area combination of point cloud objects proposed by the application has excellent recognition accuracy and strong adaptability in multiple test scenarios. Especially in the processing of complex gestures and dynamic change data, this method not only has higher accuracy, but also has stronger robustness. The optimal model achieves an accuracy of 97%.

[0033] The application also constructs a complete real-time continuous gesture recognition system that still maintains high recognition accuracy and robustness in complex scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The gesture recognition method flowchart of the application.

[0035] Figure 2 The Mediapipe three-dimensional coordinate system (a) and hand joint detection schematic diagram (b).

[0036] Figure 3 Skeletal joint for specific triangulation area feature illustration; Figure 3 (a) Triangulation illustration specific to gesture 0 and gesture 5, Figure 3 (b) Triangulation illustration specific to gesture 1 and gesture 2.

[0037] Figure 4 Flowchart for skeleton data segmentation.

[0038] Figure 5 Graph of skeleton data segmentation results.

[0039] Figure 6 Still state graph for gesture category.

[0040] Figure 7 Flowchart for gesture action training recognition. DETAILED DESCRIPTION

[0041] The present summary will be described in detail below in conjunction with the accompanying drawings and examples.

[0042] Examples:

[0043] A gesture recognition method that fuses continuous frame segmentation and spatial features, as shown in Figure 1 , the recognition method is mainly divided into two parts:

[0044] The left part is the model training process, and the specific steps include:

[0045] (1.1) Through the Mediapipe framework, use a normal camera to collect the original three-dimensional coordinate data of the 21 joints of the hand, the Mediapipe three-dimensional coordinate system and hand joint detection, as shown in Figure 2 ;

[0046] (1.2) Double normalization processing is performed on the original three-dimensional coordinates to eliminate the problem of different sizes and spatial position transformations of different subjects' hands. The specific method is as follows:

[0047] For each frame, each coordinate point is subtracted from the mean value of all joints , , , respectively, to eliminate the position change of the hand in space, and the maximum distance of all coordinate points to the center point is calculated.

[0048] Then, by dividing the coordinates of each point by the maximum distance, the is obtained, so that the differences in hand position, size, etc. of different people will not affect the subsequent analysis;

[0049] ​For example, as shown in formula

[0050]

[0051] (1.3) Extract the following features:

[0052] Point cloud convex hull surface area and volume:

[0053] Input the three-dimensional coordinates of 21 joints of a hand collected by Mediapipe, and convert these coordinates into a point cloud object using the pointcloud function in the Open3D library;

[0054] Then, use the compute_convex_hull method to calculate the smallest convex polyhedron (convex hull) that encloses these points;

[0055] Subsequently, the get_volume method is applied to calculate the volume of the convex hull, and the get_surface_area method is used to obtain the surface area of the convex hull;

[0056] Normalized joint Euclidean distance feature:

[0057] After double normalization of the skeletal data, the joint three-dimensional coordinates are used to calculate the joint distance feature, which can effectively quantify the spatial relationship between the hand joints. The specific method is as follows:

[0058] Assume that the hand has N joints, for the th joint, the normalized coordinates are , and the Euclidean distance between any two joints and in the same frame is defined as , and each group of distances is divided by the sum of all joint pair distances to obtain the normalized distance , ensuring that the distance sum is consistent between different samples, facilitating model comparison and learning, as shown in formula

[0059]

[0060] Specific triangulation area combination feature:

[0061] In the constructed point cloud object, the 21 joints of the hand are grouped into 7 groups with three points in each group, and the joint combinations of each group are shown in Table 1:

[0062] Table 1 Selected joint combination list

[0063] ​​​​ ;

[0064] The area of ​​the triangle formed by combining three joints in space for each of the seven groups is used as the feature. The grouping method in Table 1 is adopted to highlight the gesture category that will be classified in the dataset. For example, gesture 1 mainly highlights the index finger compared to other gestures, gesture 2 mainly highlights the middle finger, gesture 3 mainly highlights the ring finger, and so on.

[0065] Reference Figure 3 and Figure 6 By grouping the 21 joints into specific groups and combining them using spatial triangulation based on the gesture category, the differences between each gesture will be highlighted. For example, the combinations of joints 4, 17, 0 and 4, 12, 20 are used to emphasize the difference between gesture 5 and gesture 0, as well as the connections between their joints. Figure 3 As shown in (a); joints 8, 4, 0 and 12, 4, 0 are used to highlight the difference between gesture 1 and gesture 2, and the connection between their joints, such as... Figure 3 As shown in (b).

[0066] (1.4) The double-normalized coordinates are fused with the above features to form the final feature vector;

[0067] Using double-normalized coordinates Calculate the lengths of the three sides of a triangle, as shown in the formula. As shown:

[0068] ;

[0069] in, and The first and the Normalized 3D coordinates of each joint This represents the Euclidean distance between two points;

[0070] Next, the area of ​​the triangle formed by these three points in space is calculated using Heron's formula, as shown in equations (4) and (5):

[0071] (4);

[0072] (5);

[0073] in, These represent the lengths of the three sides of the triangle. Let be the semi-perimeter of the triangle. Let be the area of ​​the triangle;

[0074] To eliminate the scale effect caused by differences in hand size between different frames, a normalization process is introduced to calculate the volume of the convex hull of the point cloud in the current frame. The area of ​​the triangle is standardized by using the square of its cube root as a normalization factor, as shown in equation (6):

[0075] (6);

[0076] in, Indicates the first Area of ​​the triangle This represents the convex hull volume formed by the point cloud in that frame, and the normalized area feature. It can effectively eliminate the influence of the overall size of the hand on the area characteristics, making the triangle area comparable under different hand samples.

[0077] Various combinations of contrast features are shown in Table 2 below:

[0078] ;

[0079] (1.5) The Transformer and GCN network models are used to train the above features, and the best model is saved;

[0080] Identify some experimental data and results:

[0081] The gesture recognition is divided into 10 categories, from 0 to 9, such as... Figure 6 As shown, the recording was done by 10 people. Each person recorded 5 times for each type of gesture from any angle in space, resulting in a total of 500 action sequences, with each sequence consisting of 30 frames.

[0082] Table 3. Transformer recognition accuracy results for four methods: ;

[0083] Table 4. GCN recognition accuracy results for four methods:

[0084] .

[0085] The right side shows the continuous gesture recognition training process, see below. Figure 1 and Figure 7 Show:

[0086] (2.1) Read the original coordinates of the left and right hands obtained from the frame in the continuous gesture action sequence dataset, and remove the data of the first 10 frames to reduce the number of unstable frames at the beginning; (2.2) Calculate the cosine similarity between adjacent frames, set a double threshold for segmentation, and obtain stable gesture segments, such as... Figure 4 As shown;

[0087] Cosine similarity is used to quantify the relationship between adjacent frames, as shown in the equation. As shown: ;

[0088] in, This represents two one-dimensional vectors with a length of 63, representing the original one-dimensional coordinates of 21 key points in two adjacent frames;

[0089] Calculate eigenvalues , as shown As shown:

[0090] ;

[0091] A dual-threshold method is used to segment the obtained feature values:

[0092] Threshold 1: The range of feature values, set between 1.9995 and 2.00;

[0093] Threshold 2: Find a stable segment with at least 30 consecutive valid frames, and merge stable segments that are no more than 1 frame apart as the overall stable segment of this action;

[0094] Setting dual thresholds for segmentation aims to reduce the impact of unstable frames at the edges and ensure the continuity of stable segments, tolerate slight fluctuations and noise in the data, and enhance robustness.

[0095] Segmentation flowchart, such as Figure 4 As shown, the segmentation result is as follows Figure 5 As shown, from Figure 5 It can be seen that the cosine similarity dual-threshold method can accurately and stably segment indeterminate length sequence skeletal data into independent segments, where Figure 5 The area marked by the red horizontal line represents the threshold 1 range. The area marked by the blue-green vertical line represents the threshold 2 range.

[0096] (2.3) Calculate the features of the segmented stable individual category segments according to the various features in (1.3), input them into the saved best model, and output the recognition results.

[0097] Experimental results show that the present invention achieves a maximum accuracy of 97% in recognizing 10 types of gestures, which is significantly better than traditional methods.

[0098] This invention proposes a dual normalization method to eliminate differences in hand size and spatial position among different subjects. It also proposes a cosine similarity-based dual-threshold method to segment stable segments of continuous gesture sequences with variable start and end times. Furthermore, it proposes a feature based on specific triangulation area combinations from point clouds to capture geometrical changes between hand joints, further enhancing the model's sensitivity to specific gesture details and improving accuracy. The stable segments of the continuous gesture sequence are then used as test input to the optimal network model trained based on these point cloud triangulation area combination features for recognition, thus forming a real-time system capable of recognizing continuous gesture sequences.

[0099] References to this invention:

[0100] [1]Yongfeng D, Jielong L, Wenjie Y. Dynamic Hand Gesture RecognitionBased on Signals From Specialized Data Glove and Deep Learning Algorithms[J]. IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, 2021, 70.

[0101] [2] Zhang Kai, Chen Feng. Gesture recognition based on surface electromyography signals using hybrid dilated convolution CNN and BiGRU [J]. Computer Applications and Software, 2024, 41(11):220-227.

[0102] [3] Xu Yue, Zhou Hui. Static gesture recognition based on OpenCV in simple backgrounds [J]. Computer Science, 2022, 49(S2): 393-398.

[0103] [4]Debajit S, K. ​​M B. Methods, Databases and Recent Advancement ofVision-Based Hand Gesture Recognition for HCI Systems: A Review[J]. SNComputer Science, 2021, 2(6): 436-436.

[0104] [5]Y F, Sakti S, Wu Y, et al. Make Skeleton-based Action RecognitionModel Smaller, Faster and Better. [J]. CoRR, 2019, abs / 1907. 09658。

[0105] [6]Xin, C.; Kim, S.; Cho, Y.; Park, K.S. Enhancing Human ActionRecognition with 3D Skeleton Data: A Comprehensive Study of Deep Learning andData Augmentation.Electronics2024,13, 747。

[0106] [7]Ikne O, Allaert B, Wannous H. Skeleton-based self-supervisedfeature extraction for improved dynamic hand gesture recognition[C] / / 2024IEEE 18th International Conference on Automatic Face and Gesture Recognition(FG). IEEE, 2024: 1-10。

[0107] [8]Zhao Chen Y, Zhao L, Peng X, et al. Construct dynamic graphs forhand gesture recognition via spatial-temporal attention[J]. arXiv preprintarXiv:1907. 08871, 2019。

Claims

1. A gesture recognition method integrating continuous frame segmentation and spatial features, characterized in that, Includes the following steps: Step S1: Input raw skeletal point data: Capture hand images through a camera and use a hand keypoint detection framework to extract the raw 3D coordinates of 21 joints for each hand. Step S2, Data Preprocessing: The original three-dimensional coordinates are subjected to double normalization to eliminate differences in hand size and spatial position among different subjects; Step S3, Feature Extraction: Construct a point cloud object based on the preprocessed hand joint coordinates, and extract spatial geometric features including specific triangulation area combinations; Step S4, Feature Fusion: The double-normalized coordinate features are fused with the specific triangulated area combination features to form the final input features; Step S5, Model Training: Input the fused features into the deep learning model for training, and save the gesture recognition model with the best recognition performance; Step S6: Collect continuous gesture sequences: Collect continuous gesture action sequences with variable start and end times, and obtain their original three-dimensional skeletal coordinates; Step S7: Continuous sequence segmentation: Based on the cosine similarity dual threshold method, the continuous gesture sequence is automatically segmented to extract stable individual gesture segments; Step S8, Gesture Recognition: Input the segmented stable gesture segments into the optimal gesture recognition model and output the recognition results.

2. The method according to claim 1, characterized in that, The dual normalization process in step S2 specifically includes: for the coordinates of each hand joint point in each frame, first calculate the average coordinate value of all joint points, and subtract the average value from the coordinates of each joint point to eliminate the influence of spatial position; then calculate the maximum distance from all joint points to the center point, and divide the coordinates of each point by the maximum distance to unify the scale and obtain the normalized coordinates.

3. The method according to claim 1, characterized in that, The specific triangular area combination features extracted in step S3 are obtained in the following way: the 21 joints of the hand are divided into 7 groups according to preset rules, each group contains 3 joints, forming a spatial triangle; the area of ​​each triangle is calculated, and the area is normalized based on the volume of the point cloud convex hull of the current frame to eliminate hand scale differences.

4. The method according to claim 3, characterized in that, The seven triangle combinations include at least one of the following key point number combinations: (4,17,0), (8,4,0), (12,4,0), (16,4,0), (20,4,0), (4,12,20), (8,16,0).

5. The method according to claim 1, characterized in that, The continuous sequence segmentation in step S7 specifically includes: calculating the cosine similarity between two adjacent frames of gesture data, wherein the data is a vector composed of the three-dimensional coordinates of 21 joint points; setting a first threshold to filter candidate frame regions that meet the similarity condition; setting a second threshold to extract stable segments that have no less than N consecutive valid frames and are located within the candidate region; merging adjacent segments with a spacing of no more than one frame to form the final segmented stable gesture segments.

6. The method according to claim 5, characterized in that, The first threshold is a cosine similarity value between 1.9995 and 2.00; the second threshold is a stable segment containing at least 30 consecutive valid frames.

7. The method according to claim 1, characterized in that, The spatial geometric features extracted in step S3 also include one or more of the following: the surface area of ​​the point cloud convex hull, the volume of the point cloud convex hull, and the normalized Euclidean distance between joints.

8. The method according to claim 1, characterized in that, The deep learning model in step S5 is either a Transformer model or a Graph Convolutional Network (GCN) model.

9. A gesture recognition system for implementing the method as described in any one of claims 1 to 8, characterized in that, include: The data acquisition module is used to obtain the coordinates of hand joints through a camera and a hand keypoint detection framework; The preprocessing module is used to perform double normalization on the coordinates of the joints; The feature extraction and fusion module is used to construct point cloud objects, extract spatial geometric features, and perform feature fusion. The model training module is used to train a gesture recognition model using fused features and save the best model. The continuous sequence segmentation module is used to automatically segment continuous gesture sequences based on cosine similarity. The recognition output module is used to input the segmented paragraphs into the best model and output the recognition results.

10. A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method as described in any one of claims 1 to 8.

Images