A dynamic gesture recognition method based on multi-feature confidence level hierarchical fusion

Abstract

The application relates to a dynamic gesture recognition method based on multi-feature confidence level grading fusion, and belongs to the technical field of computer vision, which comprises the following steps: acquiring a gesture video to be detected, and intercepting multiple effective images from the gesture video to be detected; inputting displacement features, length features and angle features of the gesture to be detected into corresponding classification categories to obtain feature recognition results, and meanwhile, calculating the confidence levels of the feature recognition results; according to the displacement recognition result of the gesture to be detected, the displacement recognition result confidence level, the length recognition result, the length recognition result confidence level, the angle recognition result, the angle recognition result confidence level, and the priority order and the best confidence threshold corresponding to a preset sample gesture, the final recognition result of the gesture to be detected is determined; on the basis of the existing displacement feature recognition dynamic gesture, the length and angle feature recognition are added, and the problem that the gesture angle deflection influences the recognition effect is solved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a dynamic gesture recognition method based on multi-feature confidence hierarchical fusion. Background Technology

[0002] Gestures are one of the oldest forms of human communication, and dynamic gestures have a wide range of applications in the field of human-computer interaction. Because dynamic gestures are a natural, contactless operation method, they are more suitable for the next generation of human-computer interaction logic. Therefore, dynamic gestures have become a frontier and hot topic of research at home and abroad. According to the different methods of gesture data collection, dynamic gesture recognition can be divided into non-visual and visual types.

[0003] Non-visual gesture data acquisition relies on specialized hardware, such as data gloves and capacitive sensor arrays. While non-visual gesture methods perform better when encountering occlusions, their application scenarios are limited by factors such as device size and cost, causing inconvenience to users. However, visual gesture recognition methods are more convenient, so many researchers prefer to use visual methods for gesture recognition.

[0004] Visual methods typically require specialized depth or infrared cameras to acquire gesture information. Additionally, some researchers use colored finger sleeves to locate finger positions. However, existing dynamic gesture recognition methods usually only recognize gestures based on the displacement characteristics of joints, and cannot effectively recognize dynamic gestures when the palm angle changes, resulting in low accuracy in dynamic gesture recognition. Summary of the Invention

[0005] This invention addresses the problem that existing dynamic gesture recognition methods typically rely solely on the displacement characteristics of joints, but fail to effectively recognize dynamic gestures when the palm angle deflects, resulting in low accuracy in dynamic gesture recognition.

[0006] The dynamic gesture recognition method based on multi-feature confidence hierarchical fusion of the present invention adopts the following technical solution: Obtain the gesture to be detected; where a gesture to be detected contains multiple finger joints and one palm joint; Multiple valid images are extracted from the hand gesture to be detected based on the variation amplitude of each finger joint. The displacement features of the hand gesture to be detected are obtained based on the displacement vector of each finger joint between two adjacent valid frames. The length feature of the gesture to be detected is obtained based on the straight-line distance between each finger joint and the palm joint in each valid image frame; the angle feature of the gesture to be detected is obtained based on the straight-line slope between each finger joint and the palm joint in each valid image frame. The process involves obtaining the displacement, length, and angle classification categories from the training set of sample gestures; performing KNN classification on the displacement features of the gesture to be detected to obtain the displacement recognition result relative to the displacement classification category, and calculating the confidence level of the displacement recognition result; performing KNN classification on the length features of the gesture to be detected to obtain the length recognition result relative to the length classification category, and calculating the confidence level of the length recognition result; and performing KNN classification on the angle features of the gesture to be detected to obtain the angle recognition result relative to the angle classification category, and calculating the confidence level of the angle recognition result. Based on the displacement recognition results, displacement recognition result confidence level, length recognition result, length recognition result confidence level, angle recognition result, angle recognition result confidence level, as well as the priority order and optimal confidence threshold corresponding to the preset sample gestures, the final recognition result of the gesture to be detected is determined.

[0007] Furthermore, the steps for determining the final recognition result of the gesture to be detected include: The displacement recognition result of the gesture to be detected is denoted as A1, and the confidence level of the displacement recognition result is denoted as C1; The length recognition result of the gesture to be detected is recorded as A2, and the confidence level of the length recognition result is recorded as C2. The angle recognition result of the gesture to be detected is recorded as A3, and the confidence level of the angle recognition result is recorded as C3. Traverse the recognition results (A1, C1), (A2, C2), (A3, C3). If the displacement feature of the preset sample gesture corresponding to the recognition result A1 has the highest priority and the confidence of the recognition result C1 is greater than the best confidence corresponding to the displacement, then A1 is considered reliable and the displacement recognition result A1 is taken as the final recognition result of the gesture to be detected. Otherwise, continue iterating. If the length feature of the preset sample gesture corresponding to the recognition result A2 has the highest priority and the confidence level of the recognition result C2 is greater than the best confidence level corresponding to the length, then A2 is considered reliable and the length recognition result A2 is taken as the final recognition result of the gesture to be detected. Otherwise, continue iterating. If the angle feature of the preset sample gesture corresponding to the recognition result A3 has the highest priority and the confidence level of the recognition result C3 is greater than the best confidence level corresponding to the angle, then A3 is considered reliable and the angle recognition result A3 is taken as the final recognition result of the gesture to be detected. If A1, A2, and A3 are all unreliable, then filter out duplicate results in the identification results (A1, C1), (A2, C2), and (A3, C3); If there are duplicate results in (A1, C1), (A2, C2), and (A3, C3), then the duplicate results will be used as the final recognition result of the gesture to be detected. If there are no duplicate results among (A1,C1), (A2,C2), and (A3,C3), then the search results (A1,C1), (A2,C2), and (A3,C3) will be checked to see if there is a unique highest priority. If there is a unique highest priority, the recognition result corresponding to the highest priority will be used as the final recognition result of the gesture to be detected. If there is no unique highest priority, the recognition of the gesture to be detected will fail.

[0008] Furthermore, the step of extracting multiple valid images from the gesture to be detected based on the variation amplitude of each finger joint includes: Acquire multi-frame temporal images of dynamic gesture videos; Calculate the variation amplitude of each finger joint in each frame of the time sequence image; The X-frame start window and the Y-frame end window are preset, and the change amplitude threshold is also preset. The gesture is determined to begin when the change amplitude of each finger joint in X consecutive time-series images is greater than a preset change amplitude threshold; the gesture is determined to end when the change amplitude of each finger joint in Y consecutive time-series images is less than a preset change amplitude threshold. The multiple frames between the start and end of a gesture are recorded as multiple valid frames.

[0009] Furthermore, the step of obtaining the displacement features of the gesture to be detected based on the displacement vector of each finger joint between two adjacent valid frames includes: Obtain the coordinates of each finger joint in each valid image frame; Based on the coordinates of each finger joint in two adjacent valid frames, calculate the inter-frame displacement vector of each finger joint between two adjacent valid frames. Multiply the inter-frame displacement vector and orientation reference vector of each finger joint between two adjacent valid images to obtain the encoded vector of each finger joint between two adjacent valid images. The encoded vector sequence of all finger joints between two adjacent valid frames is used as the displacement feature of the gesture to be detected.

[0010] Furthermore, the step of obtaining the length feature of the gesture to be detected based on the straight-line distance between each finger joint and the palm joint in each valid image frame includes: Obtain the coordinates of each finger joint and palm joint in each valid image frame; Calculate the straight-line distance between each finger joint point and the palm joint point in each valid image frame; The difference between the straight-line distances of each finger joint point and the palm joint point in two adjacent valid images is obtained by directly subtracting the straight-line distance difference between each finger joint point in two adjacent valid images. The sequence of linear distance differences between all finger joints in two adjacent valid frames is used as the length feature of the gesture to be detected.

[0011] Furthermore, the step of obtaining the angular features of the gesture to be detected based on the slope of the straight line between each finger joint and the palm joint in each valid image frame includes: Obtain the coordinates of each finger joint and palm joint in each valid image frame; Calculate the slope of the straight line relative to the palm joint in each valid frame of the image. The slope of the line relative to the palm joint point in each of the two adjacent valid frames is directly subtracted to obtain the difference in the slope of the line between the two adjacent valid frames for each finger joint point. The sequence of linear slope differences between all finger joints in two adjacent valid frames is used as the angular feature of the gesture to be detected.

[0012] Furthermore, the steps for obtaining the displacement classification category, length classification category, and angle classification category of the sample gesture training set include: Displacement classification categories are obtained by using the displacement features of the sample gesture training set to perform KNN classification. The length classification category is obtained by using the length features of the sample gesture training set to perform KNN classification. The angle classification category is obtained by using the angle features of the sample gesture training set to perform KNN classification.

[0013] Furthermore, the steps for performing KNN classification on the displacement features of the gesture to be detected to obtain the displacement recognition result of the relative displacement category of the gesture to be detected include: Calculate the displacement similarity distance between the displacement features of each sample in the sample gesture training set and the displacement features of the gesture to be detected; select the K samples with the closest displacement similarity distance to the gesture to be detected; obtain the category A1 to which the K samples belong most frequently, and use A1 as the displacement recognition result of the gesture to be detected; The steps for obtaining the length recognition result of the relative length classification category of the gesture to be detected by KNN classification of the length features of the gesture to be detected include: calculating the length similarity distance between the length features of each sample in the sample gesture training set and the length features of the gesture to be detected; selecting the K samples with the closest length similarity distance to the gesture to be detected; obtaining the category A2 to which the K samples belong most frequently, and using A2 as the length recognition result of the gesture to be detected; The steps for obtaining the angle recognition result of the relative angle classification category of the gesture to be detected by KNN classification of the angle features of the gesture to be detected include: calculating the angle similarity distance between the angle features of each sample in the sample gesture training set and the angle features of the gesture to be detected; selecting the K samples with the closest angle similarity distance to the gesture to be detected; obtaining the category A3 to which the K samples belong most, and using A3 as the angle recognition result of the gesture to be detected.

[0014] Furthermore, the formula for calculating the confidence level of the displacement identification result is as follows:

[0015] in, The confidence score of the displacement recognition result is represented by X, which represents the gesture to be detected; T represents the category to which the displacement recognition result of the gesture to be detected belongs; K represents the K closest samples with displacement similarity to the gesture to be detected; and Ki represents the number of the K closest samples with displacement similarity to the gesture to be detected belonging to category T. This represents the percentage of the K closest samples to the gesture being detected in terms of displacement similarity that do not belong to category T. This represents the average displacement similarity distance between the gesture to be detected and all samples included in category T.

[0016] Furthermore, it also includes: The steps for calculating the displacement similarity distance between the displacement features of each sample in the training set of the sample gesture and the displacement features of the gesture to be detected include: obtaining the encoding vector 'a' of each finger joint in the gesture to be detected between two adjacent valid images; selecting a sample J from the training set, and obtaining the encoding vector 'b' of each finger joint in sample J between two adjacent valid images according to the method for obtaining the encoding vector 'a' of each finger joint in the gesture to be detected; using the similarity distance between the encoding vector sequence 'a' and the encoding vector sequence 'b' as the displacement similarity distance between each finger joint in sample J and each finger joint in the gesture to be detected; and using the sum of the displacement similarity distances between all finger joints in sample J and the corresponding finger joints in the gesture to be detected as the displacement similarity distance between the displacement features of sample J and the displacement features of the gesture to be detected. The steps for calculating the length similarity distance between the length features of each sample in the training set of the sample gestures and the length features of the gesture to be detected include: directly subtracting the length features corresponding to each finger joint in sample J from the length features corresponding to each finger joint in the gesture to be detected to obtain the length similarity distance between each finger joint in sample J and each finger joint in the gesture to be detected; and summing the length similarity distances between all finger joints in sample J and their corresponding finger joints in the gesture to be detected as the length similarity distance between the length features of sample J and the length features of the gesture to be detected. The steps for calculating the angular similarity distance between the angular features of each sample in the training set and the angular features of the gesture to be detected are as follows: directly subtract the angular features corresponding to each finger joint in sample J from the angular features corresponding to each finger joint in the gesture to be detected to obtain the angular similarity distance between each finger joint in sample J and each finger joint in the gesture to be detected; and sum the angular similarity distances between all finger joints in sample J and their corresponding finger joints in the gesture to be detected as the angular similarity distance between the angular features of sample J and the angular features of the gesture to be detected.

[0017] The beneficial effects of this invention are: This invention provides a dynamic gesture recognition method based on multi-feature confidence hierarchical fusion. First, multiple effective images are extracted from the video of the gesture to be detected according to the change amplitude of each finger joint. The displacement features, length features, and angle features of the gesture to be detected are extracted from the multiple effective images. The method provided by this invention adds length and angle features on the basis of existing displacement features. Without affecting the original recognition effect, it can effectively deal with the problem of dynamic gesture recognition being affected by gesture deflection.

[0018] Simultaneously, this invention inputs the three features of the gesture to be detected into the corresponding classification categories to obtain the feature recognition results for each classification category, and calculates the confidence level of each feature recognition result. Based on the confidence level of each feature recognition result of the gesture to be detected, the priority order of the preset sample gestures, and the optimal confidence threshold, the final recognition result of the gesture to be detected is determined. This invention uses priority ranking and the optimal confidence threshold to determine the final recognition result of the gesture to be detected, thereby improving the recognition accuracy of the gesture to be detected. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the overall steps of an embodiment of a dynamic gesture recognition method based on multi-feature confidence hierarchical fusion according to the present invention. Figure 2 This is a schematic diagram of the eight directional reference vectors in this invention; Figure 3 This is a schematic diagram of the gesture recognition process to be detected in this invention; Figure 4 This is a schematic diagram of recall and precision curves corresponding to different screening thresholds in this invention; Figure 5 This is a schematic diagram of the confusion matrix comparison in this invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] An embodiment of the dynamic gesture recognition method based on multi-feature confidence hierarchical fusion of the present invention is as follows: Figure 1 As shown, the method includes: S1. Obtain the gesture to be detected; where a gesture to be detected includes multiple finger joints and one palm joint.

[0023] This invention first acquires a video of the gesture to be detected. The hand joints are the connection points between the bones of the fingers of animals and play a key role. They are the joints between the bones and cartilage of the finger bones of vertebrates. A point is selected from each joint as a joint point. The coordinates of the finger joint points in the video of the gesture to be detected are identified by the mediapipe, including the coordinates of 20 finger joint points and 1 palm point.

[0024] S2. Extract multiple valid images from the gesture to be detected based on the change range of each finger joint.

[0025] The steps for extracting multiple valid frames from a gesture to be detected based on the change amplitude of each finger joint include: acquiring multiple temporal images of a dynamic gesture video; calculating the change amplitude of each finger joint in each temporal image; setting an X-frame start window, a Y-frame end window, and a change amplitude threshold; determining the start of a gesture when the change amplitude of each finger joint in X consecutive temporal images is greater than the preset change amplitude threshold; determining the end of a gesture when the change amplitude of each finger joint in Y consecutive temporal images is less than the preset change amplitude threshold; and recording the multiple frames from the start to the end of the gesture as multiple valid frames.

[0026] This step involves extracting valid images from the video of the gesture to be detected. When multiple frames are continuously detected where the amplitude of finger joint changes exceeds a preset threshold, the gesture is considered to have begun; when multiple frames are continuously detected where the amplitude of finger joint changes is less than the preset threshold, the gesture is considered to have ended. If the number of frames in the extracted valid images is less than the minimum threshold, it is considered an invalid frame; otherwise, it is considered a valid gesture portion. The threshold settings are related to factors such as camera resolution, camera frame rate, and the distance between the camera and the hand. In this embodiment, the camera resolution is 640*480, the frame rate is 30fps, the camera is shooting from a first-person perspective, the distance from the hand is approximately 40-60cm, the average gesture duration is approximately 0.36s-0.9s, the preset starting window size is 4 frames (X = 4 frames), the preset stopping window size is 5 frames (Y = 5 frames), and the preset amplitude threshold is 6.

[0027] S3. Based on the displacement vector of each finger joint between two adjacent valid frames, obtain the displacement features of the gesture to be detected.

[0028] The steps for obtaining the displacement features of the gesture to be detected based on the displacement vector of each finger joint between two adjacent valid images include: obtaining the coordinates of each finger joint in each valid image frame; calculating the inter-frame displacement vector of each finger joint between two adjacent valid images based on the coordinates of each finger joint in two adjacent valid images; multiplying the inter-frame displacement vector of each finger joint between two adjacent valid images and the orientation reference vector to obtain the encoded vector of each finger joint between two adjacent valid images; and using the sequence of encoded vectors of all finger joints between two adjacent valid images as the displacement features of the gesture to be detected.

[0029] Displacement features are the encoded vectors of finger joints between two adjacent valid frames. The inter-frame displacement vector between frame t and frame t-1 is represented as: Px represents the x-coordinate of the finger joint P in the two-dimensional coordinate system, and Py represents the y-coordinate of the finger joint P in the two-dimensional coordinate system. This represents the x-coordinate of the finger joint in the t frames of valid image. This represents the x-axis coordinate of the finger joint in the effective image of frame t-1; This represents the y-coordinate of the finger joint in the valid image of frame t; This represents the y-coordinate of the finger joint in the (t-1)th frame of valid images; the encoding vector of the finger joint between two adjacent valid frames is... ,in This represents a direction reference vector. In this embodiment, there are eight direction reference vectors, such as... Figure 2 The diagram shown is a schematic diagram of the eight directional reference vectors in this invention.

[0030] S4. Based on the straight-line distance between each finger joint and the palm joint in each valid image frame, obtain the length feature of the gesture to be detected; based on the straight-line slope between each finger joint and the palm joint in each valid image frame, obtain the angle feature of the gesture to be detected.

[0031] The steps for obtaining the length feature of the gesture to be detected based on the straight-line distance between each finger joint and the palm joint in each valid image frame include: obtaining the coordinates of each finger joint and the palm joint in each valid image frame; calculating the straight-line distance between each finger joint and the palm joint in each valid image frame; directly subtracting the straight-line distances between each finger joint and the palm joint in two adjacent valid images to obtain the straight-line distance difference between each finger joint in two adjacent valid images; and using the sequence of straight-line distance differences between all finger joints in two adjacent valid images as the length feature of the gesture to be detected.

[0032] The steps for obtaining the angular features of the gesture to be detected based on the slope of the straight line of each finger joint relative to the palm joint in each effective image frame include: obtaining the coordinates of each finger joint and palm joint in each effective image frame; calculating the slope of the straight line of each finger joint relative to the palm joint in each effective image frame; directly subtracting the slope of the straight line of each finger joint relative to the palm joint in two adjacent effective images to obtain the slope difference value of the straight line of each finger joint between the two adjacent effective images; and using the sequence of slope difference values ​​of the straight line of all finger joints between the two adjacent effective images as the angular features of the gesture to be detected.

[0033] The length features of the gesture to be detected are extracted. The finger joint point P is calculated relative to the palm point. The straight-line distance; extract the angular features of the gesture to be detected. The angular feature θ is the distance between the finger joint point P and the palm center point. The arctangent of the slope of the line, i.e.:

[0034] Where Px represents the x-coordinate of the finger joint point P in the two-dimensional coordinate system, Py represents the y-coordinate of the finger joint point P in the two-dimensional coordinate system; X0 represents the x-coordinate of the palm point; Y0 represents the y-coordinate of the palm point; and D represents the distance between the finger joint point P and the palm point. The straight-line distance; θ represents the finger joint point P relative to the palm center point. The arctangent of the slope of the line; Represents the arctangent function; The obtained length and angle features are normalized to eliminate jitter errors and reduce computation. The preset length normalization threshold TD = 5 and the preset angle normalization threshold Tθ = 0.1. Values ​​greater than the preset length normalization threshold TD are retained and set to 1; the rest are considered jitter and discarded. Similarly, values ​​greater than the preset angle normalization threshold Tθ are retained and set to 1.

[0035] It should be noted that displacement, length, and angle features are all time series, not single values. A dynamic gesture contains multiple frames of valid images. Inter-frame differencing yields the displacement, length, and angle features of each joint. The displacement features are obtained by subtracting the displacement vectors between frames after approximating their encoding, while the length and angle features are obtained by directly subtracting the vectors between frames.

[0036] For example, a dynamic gesture consisting of 10 valid images with 21 key points will yield displacement, length, and angle features of size 21*9 after feature extraction. When performing DTW template matching, these features are input in time series format, as DTW's purpose is to match the similarity distance between two time series.

[0037] For example: If the dynamic gesture consists of 10 valid images, the distance (i.e., length feature) between a certain finger joint and the palm is as follows: A=(1,2,1,3,2,7,3,7,2,4) The inter-frame difference is then divided into: B=(1,-1,2,-1,5,-4,4,-5,2) Threshold filtering (removing excessively small jitter): C=(0,0,0,0,1,-1,1,-1,0) The C sequence represents the length feature of this joint point. The length features of all finger joint points can be calculated in the same way, and the angle features are calculated similarly.

[0038] S5. Obtain the displacement classification category, length classification category, and angle classification category of the sample gesture training set; perform KNN classification on the displacement features of the gesture to be detected to obtain the displacement recognition result of the gesture relative to the displacement classification category, and calculate the confidence level of the displacement recognition result; perform KNN classification on the length features of the gesture to be detected to obtain the length recognition result of the gesture relative to the length classification category, and calculate the confidence level of the length recognition result; perform KNN classification on the angle features of the gesture to be detected to obtain the angle recognition result of the gesture relative to the angle classification category, and calculate the confidence level of the angle recognition result.

[0039] The steps to obtain the displacement, length, and angle classification categories of the sample gesture training set include: using the displacement features of the sample gesture training set to perform KNN classification to obtain the displacement classification category; using the length features of the sample gesture training set to perform KNN classification to obtain the length classification category; and using the angle features of the sample gesture training set to perform KNN classification to obtain the angle classification category.

[0040] KNN (K-Nearest Neighbor) classification is a theoretically mature method and one of the simplest machine learning algorithms. The idea behind this method is very simple and intuitive: if most of the K most similar (i.e., nearest) samples in the feature space belong to a certain class, then the sample also belongs to that class. This method determines the class of a sample based solely on the class of its one or a few nearest neighbors.

[0041] The steps for performing KNN classification on the displacement features of the gesture to be detected to obtain the displacement recognition result of the gesture relative to the displacement classification category include: calculating the displacement similarity distance between the displacement features of each sample in the sample gesture training set and the displacement features of the gesture to be detected; selecting the K samples with the closest displacement similarity distance to the gesture to be detected; obtaining the category A1 to which the K samples belong most, and using A1 as the displacement recognition result of the gesture to be detected; The steps for obtaining the length recognition result of the relative length classification category of the gesture to be detected by KNN classification of the length features of the gesture to be detected include: calculating the length similarity distance between the length features of each sample in the sample gesture training set and the length features of the gesture to be detected; selecting the K samples with the closest length similarity distance to the gesture to be detected; obtaining the category A2 to which the K samples belong most frequently, and using A2 as the length recognition result of the gesture to be detected; The steps for obtaining the angle recognition result of the relative angle classification category of the gesture to be detected by KNN classification of the angle features of the gesture to be detected include: calculating the angle similarity distance between the angle features of each sample in the sample gesture training set and the angle features of the gesture to be detected; selecting the K samples with the closest angle similarity distance to the gesture to be detected; obtaining the category A3 to which the K samples belong most, and using A3 as the angle recognition result of the gesture to be detected.

[0042] The steps for calculating the displacement similarity distance between the displacement features of each sample in the training set and the displacement features of the gesture to be detected include: obtaining the encoding vector 'a' of each finger joint in the gesture to be detected between two adjacent valid images; selecting a sample J from the training set, and obtaining the encoding vector 'b' of each finger joint in sample J between two adjacent valid images, following the method used to obtain the encoding vector 'a' of each finger joint in the gesture to be detected; using the similarity distance between encoding vector 'a' and encoding vector 'b' as the displacement similarity distance between each finger joint in sample J and each finger joint in the gesture to be detected; and using the sum of the displacement similarity distances between all finger joints in sample J and their corresponding finger joints in the gesture to be detected as the displacement similarity distance between the displacement features of sample J and the displacement features of the gesture to be detected.

[0043] The formula for calculating the similarity distance between encoded vector a and encoded vector b is:

[0044] Where a represents encoding vector a; b represents encoding vector b; This represents the absolute value of the difference between encoded vector a and encoded vector b; This represents the similarity distance between encoding vector a and encoding vector b.

[0045] The steps for calculating the length similarity distance between the length features of each sample in the training set and the length features of the gesture to be detected are as follows: Taking sample J as an example, the length features corresponding to each finger joint in sample J are directly subtracted from the length features corresponding to each finger joint in the gesture to be detected to obtain the length similarity distance between each finger joint in sample J and each finger joint in the gesture to be detected; the sum of the length similarity distances between all finger joints in sample J and the corresponding finger joints in the gesture to be detected is taken as the length similarity distance between the length features of sample J and the length features of the gesture to be detected.

[0046] The steps for calculating the angular similarity distance between the angular features of each sample in the training set and the angular features of the gesture to be detected are as follows: Taking sample J as an example, the angular features corresponding to each finger joint in sample J are directly subtracted from the angular features corresponding to each finger joint in the gesture to be detected to obtain the angular similarity distance between each finger joint in sample J and each finger joint in the gesture to be detected; the sum of the angular similarity distances between all finger joints in sample J and the corresponding finger joints in the gesture to be detected is taken as the angular similarity distance between the angular features of sample J and the angular features of the gesture to be detected. That is, the length similarity distance and the angular similarity distance can be obtained by directly subtracting them.

[0047] The formula for calculating the confidence level of displacement identification results is as follows:

[0048] in, The confidence score of the displacement recognition result is represented by X, which represents the gesture to be detected; T represents the category to which the displacement recognition result of the gesture to be detected belongs; K represents the K closest samples with displacement similarity to the gesture to be detected; and Ki represents the number of the K closest samples with displacement similarity to the gesture to be detected belonging to category T. This represents the percentage of the K closest samples to the gesture being detected in terms of displacement similarity that do not belong to category T. This represents the average displacement similarity distance between the gesture to be detected and all samples included in category T. The formulas for calculating the confidence scores of length recognition and angle recognition are the same as those for displacement recognition, and will not be repeated here.

[0049] The average distance between the gesture to be detected X and the displacement similarity distance of all samples in class T is normalized so that the C value is sufficiently diffused into the (0,1) interval. Let d be the average distance of the sample to be detected X to all training set samples of class T, and let dist be the average distance to all classes. The normalization result can be represented as dnormalization.

[0050] S6. Based on the displacement recognition result, displacement recognition result confidence level, length recognition result, length recognition result confidence level, angle recognition result, angle recognition result confidence level, as well as the priority order and optimal confidence level threshold corresponding to the preset sample gestures, determine the final recognition result of the gesture to be detected.

[0051] The displacement recognition result of the gesture to be detected is denoted as A1, and the confidence level of the displacement recognition result is denoted as C1; The length recognition result of the gesture to be detected is recorded as A2, and the confidence level of the length recognition result is recorded as C2. The angle recognition result of the gesture to be detected is recorded as A3, and the confidence level of the angle recognition result is recorded as C3; Traverse the recognition results (A1, C1), (A2, C2), (A3, C3). If the displacement feature of the preset sample gesture corresponding to the recognition result A1 has the highest priority and the confidence of the recognition result C1 is greater than the best confidence corresponding to the displacement, then A1 is considered reliable and the displacement recognition result A1 is taken as the final recognition result of the gesture to be detected. Otherwise, continue iterating. If the length feature of the preset sample gesture corresponding to the recognition result A2 has the highest priority and the confidence level of the recognition result C2 is greater than the best confidence level corresponding to the length, then A2 is considered reliable and the length recognition result A2 is taken as the final recognition result of the gesture to be detected. Otherwise, continue iterating. If the angle feature of the preset sample gesture corresponding to the recognition result A3 has the highest priority and the confidence level of the recognition result C3 is greater than the best confidence level corresponding to the angle, then A3 is considered reliable and the angle recognition result A3 is taken as the final recognition result of the gesture to be detected. If A1, A2, and A3 are all unreliable, then filter out duplicate results in the identification results (A1, C1), (A2, C2), and (A3, C3); If there are duplicate results in (A1, C1), (A2, C2), and (A3, C3), then the duplicate results will be used as the final recognition result of the gesture to be detected. If there are no duplicate results among (A1,C1), (A2,C2), and (A3,C3), then the search results (A1,C1), (A2,C2), and (A3,C3) will be checked to see if there is a unique highest priority. If there is a unique highest priority, the recognition result corresponding to the highest priority will be used as the final recognition result of the gesture to be detected. If there is no unique highest priority, the recognition of the gesture to be detected will fail.

[0052] It should be noted that: The displacement recognition result of the gesture to be detected is A1, and the confidence level of the displacement recognition result is C1; the length recognition result of the gesture to be detected is A2, and the confidence level of the length recognition result is C2; the angle recognition result of the gesture to be detected is A3, and the confidence level of the angle recognition result is C3. The queries are performed in the order of A1, A2, and A3. Taking three gestures (left, right, and up) as an example, assuming the pre-trained results are as shown in Table 1: Table 1: Feature Priority and Optimal Confidence for Different Gestures

[0053] For an unknown gesture X to be detected, perform three-feature KNN recognition to obtain the recognition results: (A1,C1), (A2,C2), (A3,C3), where A1 is the displacement feature recognition result, A2 is the length feature recognition result, and A3 is the angle feature recognition result. Begin iterating through (A1,C1), (A2,C2), and (A3,C3), as follows... Figure 3 The diagram shown is a schematic flowchart of the gesture recognition process to be detected in this invention.

[0054] First, examine the recognition results of the displacement features (A1, C1): If A1 = left, query the data for the left gesture in the table above. We find that the displacement feature priority of the left gesture is 2, so we continue to traverse (A2, C2); if A1 = right, query the data for the right gesture in the table above. We find that the displacement feature priority of the right gesture is 1, so we keep it; at this point, we check if C1 is greater than 0.6. If it is, we end the recognition process directly and determine that X = right. If not, we continue to traverse (A3, C3); if A1 is up or other gestures, the same applies.

[0055] When traversing (A2, C2): If A2 = left, the table shows that the priority of the left gesture length feature is 3, so continue traversing (A3, C3); if A2 = right, the table shows that the priority of the right gesture length feature is 3, so continue traversing (A3, C3); if A2 = up, the table shows that the priority of the up gesture length feature is 1, so keep it; at this time, check if C2 is greater than 0.55. If it is, end the recognition process directly and determine X = up. If not, continue traversing (A3, C3).

[0056] Continue iterating through (A3, C3) using the same method as iterating through (A2, C2). If none of them can be identified, then start identifying duplicates and unique highest priority.

[0057] It should be further explained that the priority order and optimal confidence threshold of the sample gestures are calculated and preset in advance. The specific calculation steps are as follows: Different preset filtering thresholds Tc are used to filter the feature recognition results and confidence scores for each category. Recognition results with confidence scores greater than the filtering threshold Tc are selected. Precision and recall are calculated after filtering at different thresholds Tc. Higher thresholds Tc result in higher precision, while lower thresholds Tc result in higher recall. The highest precision scores obtained for each category at different filtering thresholds Tc are sorted in descending order; this sequence represents the feature priority. Using precision as the primary keyword and recall as the secondary keyword, confidence thresholds are sorted to obtain the optimal confidence thresholds for displacement, length, and angle features. Precision and recall are calculated again after filtering at different thresholds Tc. Higher thresholds indicate more reliable recognition results, leading to higher precision; lower thresholds mean the recognition results include as many possibilities as possible, resulting in higher recall.

[0058] Feature priority is the feature order obtained by sorting the features in descending order of the highest precision obtained under different screening thresholds.

[0059] By ranking the confidence thresholds using precision as the primary keyword and recall as the secondary keyword, the optimal confidence thresholds for the three features can be obtained.

[0060] In this embodiment, 10 common gestures in software interaction logic were designed. For each gesture, 150 gesture data points were collected from a first-person perspective using a camera at different palm deflection angles, using a dual sliding threshold window method, for a total of 1500 gesture samples. After dividing the data into a training set, the feature priority and optimal confidence of the gestures need to be calculated on the training set.

[0061] Let's take the calculation process of one gesture as an example. When the results are filtered using different thresholds, different recall and precision rates will be obtained, such as... Figure 4 The figure shows the recall and precision curves corresponding to different screening thresholds in this invention. It can be seen that the highest precision in the displacement feature curve is 1.0, while the other two features are 0.96 and 0.93 respectively. It can be concluded that for the recognition of this gesture, the displacement feature has higher reliability, and its feature priority is as follows: displacement feature, length feature, and angle feature.

[0062] The optimal confidence level is the primary criterion for deciding whether to retain the final identification result. Recall and precision represent two different evaluation metrics. Since it's necessary to select the most reliable results, precision is used as the primary criterion for selecting the optimal confidence level. Using precision as the primary keyword and recall as the secondary keyword, ranking the confidence thresholds yields the optimal confidence thresholds for the three features. (Continuing with...) Figure 4 Taking gestures as an example, it can be seen that when the confidence threshold Tc for displacement features is greater than 0.2, the precision is always 1.0; however, the recall decreases to some extent as Tc increases. Therefore, it is considered that displacement features have the best recognition effect when Tc equals 0.2, which is the optimal confidence level for displacement features. That is, if the sample to be detected is identified as the gesture in the displacement feature and the confidence level is greater than 0.2, then the result is considered reliable. Similarly, the optimal confidence levels for the other two features are 0.5 and 0.8, respectively.

[0063] This invention, based on traditional displacement features, adds length and angle features to describe dynamic gestures. For example... Figure 5 The diagram illustrates the confusion matrix comparison in this invention. In this embodiment, the dynamic gesture recognition method using multi-feature confidence-level fusion improves accuracy by 4.5% compared to existing single-feature recognition methods. Based on traditional displacement features, length and angle features are added to describe dynamic gestures. Simultaneously, KNN confidence is used to filter the recognition results, selecting the best result from the three features. This improves recognition accuracy in gestures involving hand deflection. When the hand deflects, the displacement features change, allowing the length and angle features to supplement the recognition, reducing the impact of hand deflection.

[0064] This invention provides a multi-feature fusion gesture recognition method based on confidence level. It adds length and angle features to the existing displacement features, effectively addressing the dynamic gesture recognition problem under palm deflection without affecting the original recognition effect.

Claims

1. A dynamic gesture recognition method based on multi-feature confidence hierarchical fusion, characterized in that, The method includes: Obtain the gesture to be detected; where a gesture to be detected contains multiple finger joints and one palm joint; Multiple valid images are extracted from the hand gesture to be detected based on the variation amplitude of each finger joint. The displacement features of the hand gesture to be detected are obtained based on the displacement vector of each finger joint between two adjacent valid frames. The length feature of the gesture to be detected is obtained based on the straight-line distance between each finger joint and the palm joint in each valid image frame; the angle feature of the gesture to be detected is obtained based on the straight-line slope between each finger joint and the palm joint in each valid image frame. The process involves: acquiring the displacement, length, and angle classification categories of the sample gesture training set; performing KNN classification on the displacement features of the gesture to be detected to obtain the displacement recognition result relative to the displacement classification category, and calculating the confidence level of the displacement recognition result; performing KNN classification on the length features of the gesture to be detected to obtain the length recognition result relative to the length classification category, and calculating the confidence level of the length recognition result; and performing KNN classification on the angle features of the gesture to be detected to obtain the angle recognition result relative to the angle classification category, and calculating the confidence level of the angle recognition result. Based on the displacement recognition results, displacement recognition result confidence level, length recognition result, length recognition result confidence level, angle recognition result, angle recognition result confidence level, as well as the priority order and optimal confidence level threshold corresponding to the preset sample gestures, the final recognition result of the gesture to be detected is determined. The formula for calculating the confidence level of displacement identification results is as follows: in, X represents the confidence level of the displacement recognition result; T represents the category to which the displacement recognition result of the gesture to be detected belongs; K represents the K closest samples to the displacement similarity of the gesture to be detected; Ki represents the number of the K closest samples to the displacement similarity of the gesture to be detected belonging to category T. This represents the percentage of the K closest samples to the gesture being detected in terms of displacement similarity that do not belong to category T. This represents the average displacement similarity distance between the gesture to be detected and all samples included in category T.

2. The dynamic gesture recognition method based on multi-feature confidence hierarchical fusion according to claim 1, characterized in that, The steps to determine the final recognition result of the gesture to be detected include: The displacement recognition result of the gesture to be detected is denoted as A1, and the confidence level of the displacement recognition result is denoted as C1; The length recognition result of the gesture to be detected is recorded as A2, and the confidence level of the length recognition result is recorded as C2. The angle recognition result of the gesture to be detected is recorded as A3, and the confidence level of the angle recognition result is recorded as C3. Traverse the recognition results (A1, C1), (A2, C2), (A3, C3). If the displacement feature of the preset sample gesture corresponding to the recognition result A1 has the highest priority and the confidence of the recognition result C1 is greater than the best confidence corresponding to the displacement, then A1 is considered reliable and the displacement recognition result A1 is taken as the final recognition result of the gesture to be detected. Otherwise, continue iterating. If the length feature of the preset sample gesture corresponding to the recognition result A2 has the highest priority and the confidence level of the recognition result C2 is greater than the best confidence level corresponding to the length, then A2 is considered reliable and the length recognition result A2 is taken as the final recognition result of the gesture to be detected. Otherwise, continue iterating. If the angle feature of the preset sample gesture corresponding to the recognition result A3 has the highest priority and the confidence level of the recognition result C3 is greater than the best confidence level corresponding to the angle, then A3 is considered reliable and the angle recognition result A3 is taken as the final recognition result of the gesture to be detected. If A1, A2, and A3 are all unreliable, then filter out duplicate results in the identification results (A1, C1), (A2, C2), and (A3, C3); If there are duplicate results in (A1, C1), (A2, C2), and (A3, C3), then the duplicate results will be used as the final recognition result of the gesture to be detected. If there are no duplicate results among (A1,C1), (A2,C2), and (A3,C3), then the search results (A1,C1), (A2,C2), and (A3,C3) will be checked to see if there is a unique highest priority. If there is a unique highest priority, the recognition result corresponding to the highest priority will be used as the final recognition result of the gesture to be detected. If there is no unique highest priority, the recognition of the gesture to be detected will fail.

3. The dynamic gesture recognition method based on multi-feature confidence hierarchical fusion according to claim 1, characterized in that, The steps for extracting multiple valid images from the gesture to be detected based on the variation amplitude of each finger joint include: Acquire multi-frame temporal images of dynamic gesture videos; Calculate the variation amplitude of each finger joint in each frame of the time sequence image; The X-frame start window and the Y-frame end window are preset, and the change amplitude threshold is also preset. The gesture is determined to begin when the change amplitude of each finger joint in X consecutive time-series images is greater than a preset change amplitude threshold; the gesture is determined to end when the change amplitude of each finger joint in Y consecutive time-series images is less than a preset change amplitude threshold. The multiple frames of images from the start to the end of a gesture are recorded as multiple valid frames.

4. The dynamic gesture recognition method based on multi-feature confidence hierarchical fusion according to claim 1, characterized in that, The steps for obtaining the displacement features of the gesture to be detected based on the displacement vector of each finger joint between two adjacent valid frames include: Obtain the coordinates of each finger joint in each valid image frame; Based on the coordinates of each finger joint in two adjacent valid frames, calculate the inter-frame displacement vector of each finger joint between two adjacent valid frames. Multiply the inter-frame displacement vector and orientation reference vector of each finger joint between two adjacent valid images to obtain the encoded vector of each finger joint between two adjacent valid images. The encoded vector sequence of all finger joints between two adjacent valid frames is used as the displacement feature of the gesture to be detected.

5. The dynamic gesture recognition method based on multi-feature confidence hierarchical fusion according to claim 1, characterized in that, The steps for obtaining the length features of the gesture to be detected based on the straight-line distance between each finger joint and the palm joint in each valid image frame include: Obtain the coordinates of each finger joint and palm joint in each valid image frame; Calculate the straight-line distance between each finger joint point and the palm joint point in each valid image frame; The difference between the straight-line distances of each finger joint point and the palm joint point in two adjacent valid images is obtained by directly subtracting the straight-line distance difference between each finger joint point in two adjacent valid images. The sequence of linear distance differences between all finger joints in two adjacent valid frames is used as the length feature of the gesture to be detected.

6. The dynamic gesture recognition method based on multi-feature confidence hierarchical fusion according to claim 1, characterized in that, The steps for obtaining the angular features of the hand gesture to be detected based on the slope of the straight line between each finger joint and the palm joint in each valid image frame include: Obtain the coordinates of each finger joint and palm joint in each valid image frame; Calculate the slope of the straight line relative to the palm joint in each valid frame of the image. The slope of the line relative to the palm joint point in each of the two adjacent valid frames is directly subtracted to obtain the difference in the slope of the line between the two adjacent valid frames for each finger joint point. The sequence of linear slope differences between all finger joints in two adjacent valid frames is used as the angular feature of the gesture to be detected.

7. The dynamic gesture recognition method based on multi-feature confidence hierarchical fusion according to claim 1, characterized in that, The steps to obtain the displacement classification category, length classification category, and angle classification category of the sample gesture training set include: Displacement classification categories are obtained by using the displacement features of the sample gesture training set to perform KNN classification. The length classification category is obtained by using the length features of the sample gesture training set to perform KNN classification. The angle classification category is obtained by using the angle features of the sample gesture training set to perform KNN classification.

8. The dynamic gesture recognition method based on multi-feature confidence hierarchical fusion according to claim 6, characterized in that, The steps for performing KNN classification on the displacement features of the gesture to be detected to obtain the displacement recognition result relative to the displacement category of the gesture include: Calculate the displacement similarity distance between the displacement features of each sample in the sample gesture training set and the displacement features of the gesture to be detected; select the K samples with the closest displacement similarity distance to the gesture to be detected; obtain the category A1 to which the K samples belong most frequently, and use A1 as the displacement recognition result of the gesture to be detected; The steps for obtaining the length recognition result of the relative length classification category of the gesture to be detected by KNN classification of the length features of the gesture to be detected include: calculating the length similarity distance between the length features of each sample in the sample gesture training set and the length features of the gesture to be detected; selecting the K samples with the closest length similarity distance to the gesture to be detected; obtaining the category A2 to which the K samples belong most frequently, and using A2 as the length recognition result of the gesture to be detected; The steps for obtaining the angle recognition result of the relative angle classification category of the gesture to be detected by KNN classification of the angle features of the gesture to be detected include: calculating the angle similarity distance between the angle features of each sample in the sample gesture training set and the angle features of the gesture to be detected; selecting the K samples with the closest angle similarity distance to the gesture to be detected; obtaining the category A3 to which the K samples belong most, and using A3 as the angle recognition result of the gesture to be detected.

9. A dynamic gesture recognition method based on multi-feature confidence hierarchical fusion according to claim 8, characterized in that, Also includes: The steps for calculating the displacement similarity distance between the displacement features of each sample in the training set of the sample gesture and the displacement features of the gesture to be detected include: obtaining the encoding vector 'a' of each finger joint in the gesture to be detected between two adjacent valid images; selecting a sample J from the training set, and obtaining the encoding vector 'b' of each finger joint in sample J between two adjacent valid images according to the method for obtaining the encoding vector 'a' of each finger joint in the gesture to be detected; using the similarity distance between the encoding vector sequence 'a' and the encoding vector sequence 'b' as the displacement similarity distance between each finger joint in sample J and each finger joint in the gesture to be detected; and using the sum of the displacement similarity distances between all finger joints in sample J and the corresponding finger joints in the gesture to be detected as the displacement similarity distance between the displacement features of sample J and the displacement features of the gesture to be detected. The steps for calculating the length similarity distance between the length features of each sample in the training set of the sample gestures and the length features of the gesture to be detected include: directly subtracting the length features corresponding to each finger joint in sample J from the length features corresponding to each finger joint in the gesture to be detected to obtain the length similarity distance between each finger joint in sample J and each finger joint in the gesture to be detected; and summing the length similarity distances between all finger joints in sample J and their corresponding finger joints in the gesture to be detected as the length similarity distance between the length features of sample J and the length features of the gesture to be detected. The steps for calculating the angular similarity distance between the angular features of each sample in the training set and the angular features of the gesture to be detected are as follows: directly subtract the angular features corresponding to each finger joint in sample J from the angular features corresponding to each finger joint in the gesture to be detected to obtain the angular similarity distance between each finger joint in sample J and each finger joint in the gesture to be detected; and sum the angular similarity distances between all finger joints in sample J and their corresponding finger joints in the gesture to be detected as the angular similarity distance between the angular features of sample J and the angular features of the gesture to be detected.

Images