Time sequence dynamic gesture recognition method and system

Abstract

The invention discloses a time sequence dynamic gesture recognition method and system. The time sequence dynamic gesture recognition method mainly comprises the steps of S1, hand target detection; step S2, hand target tracking; s3, outputting category information; s4, setting an information container; and S5, gesture motion prediction. According to the method, the problem of time sequence prediction of gesture actions is simplified, meanwhile, the problem of easy ID exchange during target tracking is solved, YOLO is mainly used for detecting a hand and positioning key point information of a hand skeleton, a neural network is realized based on a 1D operator, and gesture action prediction is carried out on time sequence key point information. According to the dynamic gesture recognition method based on combination of the multiple models, lightweight design is carried out on the YOLOv8 model, real-time recognition is achieved on equipment with limited resources (for example, the real-time recognition can reach 25FPS on RK3566), and the problem that ID information is likely to be exchanged when hands are waved crosswise is solved through dynamic threshold control parameters of the target tracking module.

Description

TECHNICAL FIELD

[0001] The application discloses a gesture recognition method, in particular a time sequence dynamic gesture recognition method and system, and relates to the technical field of CV algorithms. BACKGROUND

[0002] Gesture recognition is a technology that detects, tracks and understands human gesture actions through computer vision or sensor technology, and is widely used in human-computer interaction, virtual reality, intelligent device control and other fields. Traditional gesture recognition methods usually use libraries such as OpenCV to process images, and recognize hand positions and shapes through edge detection, contour extraction, skin color models and other algorithms (such as Haar cascade, template matching). At present, the mainstream gesture recognition method is to use deep learning for gesture recognition.

[0003] There are several common methods for gesture recognition using deep learning in the prior art: (1) a neural network based on 3D operators (Conv3D or 3D Transformer) predicts time sequence gesture actions; (2) a neural network based on 2D operators combined with 3D operators (two-way fusion), which uses 2D operators (such as ResNet to extract hand key points) to learn the image content of each frame and uses 3D operators (such as Conv3D or LSTM) to learn the information between the previous and next frames in the time sequence to predict time sequence gesture actions; (3) a neural network based on 1D operators (such as using MediaPipe to extract the (x, y, z) coordinates of 21 hand key points), which flattens the time sequence information into 1D data, and then uses 1D operators to learn the flattened data to predict time sequence gesture actions.

[0004] The main shortcomings of the current gesture recognition algorithm using deep learning are: (1) the difficulty of deploying the algorithm on the board. Although the neural network composed of 3D operators can learn time sequence information more easily than the neural network composed of 2D operators and 1D operators, most consumer-grade development board chips currently do not support the deployment of 3D operators, greatly increasing the difficulty of deployment. (2) When predicting and tracking the ID information of each hand, the ID information is prone to exchange. In the case of waving hands during target tracking, two targets that visually produce track intersections are prone to ID exchange, resulting in incorrect track information being fed to the neural network for recognition and obtaining incorrect results. SUMMARY

[0005] In view of the above-mentioned shortcomings in the prior art that gesture recognition algorithm using deep learning is difficult to deploy and prone to incorrect results, the present application provides a time-series dynamic gesture recognition method and system, which simplifies the problem of time-series gesture action prediction and solves the problem of ID exchange during target tracking. The present application mainly uses YOLO to detect hands and locate hand skeleton key point information, and then uses a neural network based on 1D operator to predict gesture actions for time-series key point information.

[0006] The present application solves the technical problems by adopting the technical solution of a time-series dynamic gesture recognition method, which comprises the following steps:

[0007] Step S1, hand target detection: input a video stream, and input the video stream after being disassembled into each frame of picture into a gesture detection and time-series dynamic gesture recognition algorithm for hand target detection. During hand target detection, a light YOLOv8 model is used to detect hand key points in the picture. If a hand target is detected, the time-series dynamic gesture recognition algorithm is activated to proceed to step S2. If no hand target is detected, the system enters a standby state and continuously performs hand target detection through step S1.

[0008] Step S2, hand target tracking: a multi-target tracker is used to track the hand target.

[0009] Step S3, output class information: in combination with steps S1 and S2, the time-series dynamic gesture recognition algorithm considers that the same palm is detected, and outputs corresponding class information. The bounding box position information, confidence score and class information output by the YOLOv8 model are compared with the artificially set confidence threshold and IOU threshold to determine the palm class score. If the set threshold is not reached, it is considered that it is not a palm class, and the next frame of picture is directly processed. If the set threshold is reached, subsequent hand skeleton key point processing is performed.

[0010] Step S4, set information container: set a 30-frame hand skeleton key point information container. The 30th frame processes information from the 1st to 30th frame, the 31st frame processes information from the 2nd to 31st frame, and so on. From the 30th frame, the container will always store 30 frames of hand information. Only data that meets the condition of at least 25 frames of data being palm class data is output to the next step.

[0011] Step S5, gesture action prediction: input the 30-frame hand skeleton key point information container that meets the condition into a 1D neural network to predict gesture actions. The prediction of dynamic gestures is completed in a multi-frame static gesture key point splicing manner, and finally the class with the highest confidence is output from the five classes of "left hand waving", "right hand waving", "up hand waving", "down hand waving" and "other".

[0012] A time sequence dynamic gesture recognition system, comprising:

[0013] A hand target detection unit: used for inputting a video stream, inputting the video stream into a hand gesture detection and time sequence dynamic gesture recognition algorithm after the video stream is disassembled into each frame of picture, and performing hand target detection; when the hand target detection is performed, a light YOLOv8 model is used to detect hand key points in the picture; if the hand target is detected, the time sequence dynamic gesture recognition algorithm is activated to enter a hand target tracking unit; if the hand target is not detected, a standby state is entered, and the hand target detection is continuously performed through the hand target detection unit;

[0014] A hand target tracking unit: used for tracking the hand target by using a multi-target tracker;

[0015] An output category information unit: used for combining the hand target detection unit and the hand target tracking unit, outputting corresponding category information by the time sequence dynamic gesture recognition algorithm, and outputting the detection frame position information, the confidence score and the category information by the YOLOv8 model; the palm category score is judged by comparing a manually set confidence threshold and an IOU threshold; if the set threshold is not reached, it is considered that it is not the palm category, and the next frame of picture is directly processed; if the set threshold is reached, subsequent hand skeleton key point processing is performed;

[0016] An information container setting unit: used for setting an information container of 30 frames of hand skeleton key points; the information of 1 to 30 frames is processed in the 30th frame, the information of 2 to 31 frames is processed in the 31st frame, and so on; from the 30th frame, the container will always store 30 frames of hand information; only the data that meets the condition that at least 25 frames of data are palm category data is output to the next step in the container;

[0017] A gesture action prediction unit: used for inputting the 30 frames of hand skeleton key point information container that meets the condition into a 1D neural network to predict a gesture action; the prediction of the dynamic gesture is completed in a multi-frame static gesture key point splicing manner; finally, the category with the highest confidence in five categories of "left hand waving", "right hand waving", "up hand waving", "down hand waving" and "other" is output.

[0018] The technical solution further comprises the following steps:

[0019] In the step S1, the light YOLOv8 model replaces C2f of YOLOv8 with a light Conv module; the complicated C2f module is replaced with the Conv module.

[0020] In the step S2, if the hand target is detected for the first time, the target tracking algorithm is skipped; from the second time when the hand target is detected, whether it is the same palm is judged according to the historical accumulation information.

[0021] The target tracking algorithm in step S2 is composed of Kalman filtering and Hungarian algorithm, comprising:

[0022] The hand key point information obtained by YOLOv8 is screened by a confidence threshold and an IOU threshold, and low-quality detection boxes are filtered. If the current input frame is the first frame, the center point coordinates (cx, cy) and the class information (class_id) of the first frame target are accumulated. The detection box (x1, y1, x2, y2) output by YOLOv8 is converted into the target center point position through the formula , and the class information is obtained by filtering through the confidence threshold. From the second frame data, the state vector x of the target is calculated in combination with two frame data k =(x,y,v x ,v y ), wherein x k is the state vector of the target, x and y are the position coordinates of the target center point, v x is the velocity component of the target in the x direction, and v y is the velocity component of the target in the y direction. Assuming that the target moves at a constant speed in the image, the state transition matrix F is obtained:

[0023]

[0024] Wherein, F is the state transition matrix, Δt is the time interval between adjacent state estimates or observations,

[0025] Then, the predicted target position is obtained through the state prediction equation:

[0026] x k|k-1 =F*x k-1

[0027] Wherein, k is the current frame, k-1 is the last frame, x k-1 is the actual state estimation value of the last frame, and x k|k-1 is the predicted state value of the current frame.

[0028] The Euclidean distance C is calculated by the position information predicted by the second frame and the hand target position information detected by the second frame, and if it exceeds the max_distance parameter, it is considered as not matching, and in the case of losing the target, it is accumulated, and when the number of continuous frames exceeds the max_age parameter, the same target ID is lost, that is, the ID target is not detected, the target is cleared and it is considered that the target of the same ID in the picture has left the picture range; if there is one or more matching targets, the one with the smallest Euclidean distance is considered to be the same target, in order to ensure the reliability of the Euclidean distance determination, the max_distance parameter is set as a dynamic parameter, when the hand detects that there is intersection, the max_distance parameter is reduced from the Euclidean distance value 30 to the Euclidean distance value 10, and in the case that there is no intersection, the max_distance parameter uses the default value of the Euclidean distance value 30, and the max_age parameter is set in the same way, when the number of unmatched frames exceeds max_age, it is considered that the target is lost, and the specific formula is as follows:

[0029]

[0030] In the formula, i is the serial number of the predicted position; j is the serial number of the actual position; C ij is the cost matrix, which represents the matching cost of the ith detection and the jth predicted trajectory; A ij is the assignment matrix, A ij =1 indicates that detection i matches trajectory j, otherwise 0; s.t. represents the constraint condition of the optimization problem.

[0031] The process of matching the minimum Euclidean distance is the Hungarian algorithm, which finds the global optimal matching through matrix operation, and finally updates the matching item x k and Kalman gain K k , completes the whole target tracking process:

[0032] x k = x k + K k (x k -x k|k-1 )

[0033] K k =P k|k-1 H T (HP k|k-1 H T +R) -1

[0034] P k|k-1 =FP k-1 F T +Q

[0035] Wherein, H, Q, R are set as constants, H is a linear relationship matrix of measurement and state, Q is a process noise matrix, R is a measurement noise matrix, P is an error covariance matrix, after completing the update of the Kalman gain and the matching item, the current frame target tracking is completed, and then the next cycle is entered to process the next frame of data.

[0036] In step S5, the specific steps of the self-defined 1D neural network gesture model prediction are as follows:

[0037] In the training process, data random scaling enhancement and overall displacement random variation enhancement are enabled, which reduces model overfitting and increases data diversity at the same time. In addition, gradient clipping is used in each epoch to limit gradient explosion, further stabilizing the model training.

[0038] Load data to obtain 21 key point information of the detected target hand, then build a key point container to accumulate 30 frames of key information of the hand, continuously track the same target through the target tracking algorithm, and obtain the (x, y) coordinates of each key point in each frame of data.

[0039] The input layer is composed of a 1D convolution layer, normalization, an activation function and down sampling:

[0040] input=MaxPool(ReLU(BatchNorm(Conv1d(X))))

[0041]

[0042] ReLU(X)=max(0,X)

[0043] Wherein, mu, sigma are the mean and variance of the channel; gamma, beta are learnable scaling and offset parameters, X is tensor data, indicating input feature information, and epsilon is a very small constant. BatchNorm accelerates training convergence, and ReLU activation function makes the model nonlinear.

[0044] The residual block is connected by 6 residual modules, and the formula is as follows:

[0045] Residual=ReLU(F(X)+downsample(X))

[0046] In the formula, F is two convolution layers, downsample is a jump connection layer, X is tensor data, indicating input feature information, Residual is a residual module, and ReLU is an activation function. Through the stacking of 6 residual networks, deep features are learned.

[0047] The formula of the self-attention module is as follows:

[0048] Output=V*Attention T

[0049] Attention=Softmax(Q T K)

[0050] Assume that the feature after the residual block is (C, L), where C is the number of channels, L is the sequence length, V is the value matrix, which is used to store the actual information of the position for attention weighted aggregation, T means transposition, the matrix is represented by T when transposed, K is the key matrix, which is used as the basis for matching when being "queried" by other positions, Q is the query matrix, which is used to calculate the "attention" of the current position to other positions, Q T is the transposed matrix of Q, which is used to match the dimensions when matrix multiplication, where T means transposition, the matrix is represented by T when transposed, then the projection of Query, Key and Value is as follows:

[0051] Q=W q X

[0052] K=W k X

[0053] V=W v X

[0054] Where X is the input feature matrix, which is tensor data, W q , W k and W v are trainable weight matrices, W T is the weight coefficient in the training process, the attention of each element is obtained through Q T K, then the model is normalized by row using Softmax, and finally the self-attention module is completed by fusing the V parameter.

[0055] In the hand target detection unit, the lightweight YOLOv8 model replaces the C2f of YOLOv8 with a lightweight Conv module, and the Conv module replaces the relatively complex C2f module.

[0056] In the hand target tracking unit, if it is the first time to detect the hand target, the target tracking algorithm is skipped, and from the second time to detect the hand target, whether it is the same palm is determined according to the historical accumulation information.

[0057] In the hand target tracking unit, the target tracking algorithm is composed of Kalman filtering and Hungarian algorithm, including:

[0058] The hand key point information obtained by YOLOv8 is screened by a confidence threshold and an IOU threshold, low-quality detection boxes are filtered, if the current input frame is the first frame, the detection box center point coordinates (cx, cy) and the class information (class_id) of the first frame target are accumulated, the detection box (x1, y1, x2, y2) output by YOLOv8 is converted into the target center point position through the formula , and the class information is obtained by filtering through the confidence threshold, from the second frame data, the state vector x of the target is calculated in combination with two frame data k =(x,y,v x ,v y ), in the formula, x k is the state vector of the target, x and y are the position coordinates of the target center point, v x is the velocity component of the target in the x direction, and v y is the velocity component of the target in the y direction, assuming that the target is uniformly moving in the image, the state transition matrix F is obtained:

[0059]

[0060] Wherein, F is the state transition matrix, Δt is the time interval between adjacent state estimations or observations,

[0061] Then, the predicted target position is obtained through the state prediction equation:

[0062] x k|k-1 =F*x k-1

[0063] Wherein, k is the current frame, k-1 is the last frame, x k-1 is the actual state estimation value of the last frame, and x k|k-1 is the predicted state value of the current frame.

[0064] The Euclidean distance C is calculated by the position information predicted by the second frame and the hand target position information detected by the second frame. If it exceeds the max_distance parameter, it is considered as not matching. In the case of losing the target, accumulation is performed. When the number of continuous frames exceeds the max_age parameter, the same target ID is lost, that is, the target of the ID is not detected, the target is cleared, and it is considered that the target of the ID in the picture has left the picture range. If there is one or more matching targets, the one with the minimum Euclidean distance is considered as the same target. In order to ensure the reliability of the Euclidean distance determination, the max_distance parameter is set as a dynamic parameter. When the hand detection exists intersection, the max_distance parameter is reduced from the Euclidean distance value 30 to the Euclidean distance value 10. In the case where there is no intersection, the max_distance parameter uses the default value of the Euclidean distance value 30. The max_age parameter is set in the same way. When the number of unmatched frames exceeds max_age, it is considered that the target is lost. The specific formula is as follows:

[0065]

[0066] In the formula, i is the serial number of the predicted position; j is the serial number of the actual position; C ij is the cost matrix, which represents the matching cost of the ith detection and the jth predicted trajectory; A ij is the assignment matrix, A ij =1 indicates that the detection i matches the trajectory j, otherwise 0; s.t. represents the constraint condition of the optimization problem.

[0067] The process of matching the minimum Euclidean distance is the Hungarian algorithm, which finds the global optimal matching through matrix operation. Finally, the matching item x k and the Kalman gain K k are updated to complete the whole target tracking process:

[0068] x k = x k + K k (x k -x k|k-1 )

[0069] K k = P k|k-1 H T (HP k|k-1 H T + R) -1

[0070] P k|k-1 = FP k-1 F T + Q

[0071] Wherein, H, Q, R are set as constants, H is a linear relationship matrix of measurement and state, Q is a process noise matrix, R is a measurement noise matrix, P is an error covariance matrix, after completing the update of the Kalman gain and the matching item, the current frame target tracking is completed, and then the next cycle is entered to process the next frame of data.

[0072] The gesture action prediction unit, the specific steps of the self-defined 1D neural network prediction gesture model are as follows:

[0073] In the training process, data random scaling enhancement and overall displacement random variation enhancement are enabled, which reduces the model overfitting and also increases the diversity of data, and in each epoch, gradient clipping is used to limit gradient explosion, further stabilizing the model training;

[0074] Load data to obtain 21 key point information of the detected target hand, then build a key point container to accumulate 30 frames of key information of the hand, continuously track the same target through the target tracking algorithm, and obtain the (x, y) coordinates of each key point in each frame of data;

[0075] The input layer is composed of a 1D convolution layer, normalization, an activation function and down sampling:

[0076] input=MaxPool(ReLU(BatchNorm(Conv1d(X))))

[0077]

[0078] ReLU(X)=max(0,X)

[0079] Wherein, mu, sigma are the mean and variance of the channel; gamma, beta are learnable scaling and offset parameters, X is tensor data, indicating input feature information, and epsilon is a very small constant; BatchNorm accelerates training convergence, and ReLU activation function makes the model nonlinear;

[0080] The residual block is connected by 6 residual modules, and the formula is as follows:

[0081] Residual=ReLU(F(X)+downsample(X))

[0082] In the formula, F is two convolution layers, downsample is a jump connection layer, X is tensor data, indicating input feature information, Residual is a residual module, and ReLU is an activation function; 6 residual networks are stacked to learn deeper features;

[0083] The formula of the self-attention module is as follows:

[0084] Output = V * Attention T

[0085] Attention = Softmax(Q) T K)

[0086] Assume the features after the residual block are (C, L), where C is the number of channels, L is the sequence length, V is the value matrix used to store the actual position information for attention-weighted aggregation, T stands for transpose, the matrix is ​​represented by T when transposed, K is the key matrix used as the matching basis when "queried" by other positions, and Q is the query matrix used to calculate the "attention" of the current position to other positions. T Let T be the transpose of Q, used for matching dimensions during matrix multiplication. T stands for transpose, and the matrix is ​​represented by T when transposed. The projection of Query, Key, and Value is as follows:

[0087] Q = W q X

[0088] K = W k X

[0089] V = W v X

[0090] Where X is the input feature matrix, W is the tensor data, and W is the input feature matrix. q W k and W v Both are trainable weight matrices, where W represents the weight coefficients during training, and Q is used to calculate the weights. T K obtains the attention level of each element, then uses Softmax to normalize the model row by row, and finally fuses the V parameters to complete the self-attention module.

[0091] The beneficial effects of this invention are: This invention provides a dynamic gesture recognition method that combines multiple models, and it adopts a lightweight design for the YOLOv8 model to achieve real-time recognition on devices with limited resources (such as 25 FPS on RK3566). The problem of ID information being easily swapped when waving hands is solved by using dynamic threshold control parameters of the target tracking module.

[0092] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0093] Figure 1 This is a flowchart of the algorithm of the present invention.

[0094] Figure 2 This is a flowchart of the lightweight YOLOv8 model algorithm in this invention.

[0095] Figure 3A 1D neural network gesture prediction model algorithm flow block diagram in the application.

[0096] Figure 4 A residual block model schematic diagram in the application. DETAILED DESCRIPTION

[0097] The embodiment is a preferred embodiment of the application, and other embodiments having the same or similar principles and basic structures as the embodiment are within the protection scope of the application.

[0098] Please refer to the accompanying Figure 1 to the accompanying Figure 4 , the application mainly protects a time sequence dynamic gesture recognition method, which comprises the following steps:

[0099] Step S1, hand target detection: input a video stream, and input the video stream after being disassembled into each frame of picture into a hand gesture detection and time sequence dynamic gesture recognition algorithm to perform hand target detection.

[0100] In the embodiment, when the hand target is detected, a light YOLOv8 model is used to detect the hand key points in the picture, if the hand target is detected, the time sequence dynamic gesture recognition algorithm is activated to enter step S2, if the hand target is not detected, the standby state is entered, and the hand target detection is continuously performed through step S1.

[0101] In the embodiment, the specific steps of the light YOLOv8 model are as follows:

[0102] The original YOLOv8 convolution adopts a C2f module, which has high structural complexity and contains branch, residual, fusion and other operations, the original C2f module formula is as follows:

[0103] C2f(X)=Conv 1×1 (concat(X1,Z1,...,Z n ))

[0104] X1,X2=split(X,dim=2),X1∈R H×W×C / 2 ,X2∈R H×W×C / 2

[0105] Z t =Z t-1 +Conv 3×3 (Conv 1×1 (Z t-1 ))

[0106] Conv 1×1represents a convolution kernel module, the convolution kernel traverses the window by 1x1 step, X is an input matrix (tensor) data, the form of the tensor data inside the neural network is composed according to (batch, channel number, height, width), Z1, X1, Z n are tensor data from different neural network layers, R, H, W, C respectively represent the batch, height, width, and channel number of the feature map, Zt represents the intermediate feature generated by the t-th step recursion, concat is a splicing module; split is a splitting module, which is divided into two parts here. As can be seen, the original C2f module has a large amount of calculation and is not suitable for hardware with limited resources.

[0107] In this embodiment, the C2f of YOLOv8 is replaced by a lightweight Conv module (the Conv module is a module built in the neural network for convolution), the complexity of the model is reduced by replacing the relatively complex C2f module with the Conv module, and the model interpretability is enhanced, thereby greatly reducing the time consumed by model inference.

[0108] Step S2, hand target tracking: a multi-target tracker is used to track the hand target.

[0109] In this embodiment, if the hand target is detected for the first time, the target tracking algorithm is skipped, and from the second time the hand target is detected, whether it is the same palm is judged according to the historical accumulation information, so as to complete the target tracking algorithm.

[0110] In this embodiment, the target tracking algorithm is mainly composed of Kalman filtering and Hungarian algorithm, and the following is the specific implementation process:

[0111] The hand key point information obtained by YOLOv8 is screened by a confidence threshold and an IOU threshold, and low-quality detection boxes are filtered. If the current input frame is the first frame, since there is no historical information, the detection box center point coordinates (cx, cy) and the class information (class_id) of the first frame target are mainly accumulated. This process needs to pass the detection box (x1, y1, x2, y2) (left upper corner and right lower corner coordinates) output by YOLOv8 through The formula is converted into the target center point position, and the class information is filtered by a confidence threshold (in this embodiment, the confidence score threshold is set to 0.35). From the second frame data, the state vector x k of the target can be calculated in combination with two frame data. x y k x y ​​​​The velocity component of the target in the y direction. Assuming the target moves at a constant speed in the image, the state transition matrix F can be obtained:

[0112]

[0113] Where F is the state transition matrix, and Δt is the time interval between adjacent state estimates (or observations) (unit: seconds), which is a key parameter in Kalman filtering connecting the states at different times, reflecting the rate of dynamic changes of the system.

[0114] Then, the predicted target position can be obtained by the state prediction equation:

[0115] x k|k-1 = F * x k-1

[0116] Where k is the current frame, k-1 is the previous frame, x k-1 is the actual state estimate value of the previous frame, and x k|k-1 is the predicted state value of the current frame. Through the prediction equation, the following two effects are achieved: predicting the next frame position of the target, making up for detection omissions (such as occlusion), smoothing the motion trajectory, and reducing detection jitter.

[0117] The Euclidean distance C is calculated by the predicted position information of the second frame and the hand target position information detected in the second frame. If it exceeds the max_distance parameter, it is considered not to match. In the case of missing the target, the accumulation is carried out. When the number of consecutive frames exceeds the max_age parameter, the same target ID is lost, that is, the ID target is not detected, the target is cleared and it is considered that the target with the same ID in the picture has left the picture range. If there is one or more matching targets, the one with the smallest Euclidean distance is considered to be the same target. To ensure the reliability of the Euclidean distance determination, the max_distance parameter is set as a dynamic parameter. When the hand detection exists intersection, the max_distance parameter is reduced from the Euclidean distance value 30 to the Euclidean distance value 10, and a more stringent condition is enabled. In the case where there is no intersection, the max_distance parameter uses the default value of the Euclidean distance value 30. Similarly, the max_age parameter is set. When the number of unmatched frames exceeds max_age, it is considered that the target is lost, which can effectively prevent the ID exchange of the target tracking. The specific formula is as follows:

[0118]

[0119] In the formula, i is the serial number of the predicted position; j is the serial number of the actual position; C ij is the cost matrix, indicating the matching cost (such as Euclidean distance) of the ith detection and the jth predicted trajectory; A ij is the assignment matrix, A ij=1 indicates that the detection i matches the track j, otherwise 0; s.t. is the abbreviation of "subject to", meaning "subject to", indicating the constraint condition of the optimization problem.

[0120] The process of matching the minimum Euclidean distance is the Hungarian algorithm, which mainly finds the global optimal matching through matrix operation. Finally, the matching item x k and Kalman gain K k , complete the whole target tracking process:

[0121] x k = x k + K k (x k -x k|k-1 )

[0122] K k =P k|k-1 H T (HP k|k-1 H T +R) -1

[0123] P k|k-1 =FP k-1 F T +Q

[0124] In the application, H, Q and R are all set as constants, wherein H is a linear relationship matrix of measurement and state, Q is a process noise matrix, R is a measurement noise matrix, and P is an error covariance matrix. After completing the update of Kalman gain and matching item, the current frame target tracking is completed, and then the next cycle is entered to process the next frame of data.

[0125] When Kalman filtering and the Hungarian algorithm are used to solve the cross waving hand prediction tracking ID information of each hand, the ID information is prone to exchange problems.

[0126] Step S3, output category information: combine S1, S2 steps, consider that the same palm is detected through the time sequence dynamic gesture recognition algorithm (the time sequence dynamic gesture recognition algorithm of the application is an algorithm built by 1D operator), and output the corresponding category information.

[0127] In this embodiment, the position information, confidence score and category information of the detection frame output by the YOLOv8 model are compared with the manually set confidence threshold and IOU threshold (in this embodiment, the score confidence threshold is set to 0.35, and the iou threshold is set to 0.4) to judge the palm category score. If the set threshold is not reached, it is considered not to be a palm category, and the next frame of picture processing is directly performed; if the set threshold is reached, subsequent hand skeleton key point processing is performed. The gesture categories here include: 0 (fist), 1, 2, 3, 4, 5 (palm), ok, and others. The types here are all gesture types, 0 represents a fist, 1 represents gesture 1, 2 represents gesture yes or gesture 2, 3 represents converging thumb and little finger gesture 3, 4 represents converging thumb gesture 4, 5 represents spreading palm gesture 5, ok represents converging thumb and index finger gesture ok, and others represent gestures not in the above categories.

[0128] Step S4, setting an information container: setting a 30-frame hand skeleton key point information container, the 30th frame processes information from the 1st to the 30th frame, the 31st frame processes information from the 2nd to the 31st frame, and so on. In this embodiment, the information container is a 30-frame data information container set by a person for storage, which is a list set by a person. The length of the list is 30, and the 1st to 30th frames are information of each frame in the video. Only the 30 frames are stored, and the list is accumulated if there are less than 30 frames. After 30 frames are met, the first frame is removed, and the last frame is added to form a new 30 frames. From the 30th frame, the container will always store 30 frames of hand information. Only when at least 25 frames of data in the container are 5 (palm) category data, the data in the container is transmitted to the next step.

[0129] Step S5, gesture action prediction: inputting the 30-frame hand skeleton key point information container meeting the condition into a 1D neural network to predict the gesture action. The prediction of dynamic gestures is completed by splicing multiple frames of static gesture key points, and finally the highest confidence category in the five categories of “left swing hand”, “right swing hand”, “up swing hand”, “down swing hand” and “other” is output.

[0130] In this embodiment, the specific steps of the self-defined 1D neural network gesture model are as follows:

[0131] Data random scaling augmentation and overall displacement random variation augmentation are enabled in the training process, where random augmentation means adding random factors, which may enhance or weaken. For neural networks, both enhancement and weakening are beneficial to the generalization ability of the model. Adding sample diversity is equivalent to adding sample diversity, which helps the neural network model learn complex sample features, reduces model overfitting, and increases data diversity. At the same time, gradient clipping is used to limit gradient explosion in each epoch (epoch is a key hyperparameter, which means one complete traversal of the entire training data set), further stabilizing the model training.

[0132] Loading data mainly obtains 21 key point information of the detection target hand, and then constructs a key point container to accumulate 30 frames of hand key information. The target tracking algorithm continuously tracks the same target, and the (x, y) coordinates of each key point can be obtained in each frame of data, that is, a target in a frame of picture will provide 21*2 dimensions, and 30 frames of picture will provide 30*21*2=1260 dimensions of information.

[0133] The input layer is mainly composed of 1D convolution layer, normalization, activation function and down sampling.

[0134] input=MaxPool(ReLU(BatchNorm(Conv1d(X))))

[0135]

[0136] ReLU(X)=max(0,X)

[0137] Where μ, σ are the mean and variance of the channel; γ, β are the learnable scaling and offset parameters, X is the tensor data (input feature information), and data is operated in the form of tensor in the neural network. ε is a very small constant (such as 10 -5 ), used to prevent the denominator from being zero (numerical stability). BatchNorm accelerates training convergence and alleviates the problem of gradient vanishing and explosion. The ReLU activation function makes the model nonlinear and enhances the model's expression ability.

[0138] The residual block is connected by 6 residual modules, and the formula is as follows:

[0139] Residual=ReLU(F(X)+downsample(X))

[0140] F is two convolutional layers, downsample is a skip connection layer, X is tensor data (input feature information), and data are operated in the form of tensors inside the neural network, Residual is the name of the residual module, and ReLU is the name of the activation function. Residual and ReLU are both industry-specific professional terms, and a deep feature is learned through the stacking of six residual networks.

[0141] The self-attention module formula is as follows:

[0142] Output=V*Attention T

[0143] Attention=Softmax(Q T K)

[0144] Suppose the feature after the residual block is (C, L), where C is the number of channels, L is the sequence length, V is the value matrix, which stores the actual information of the position, and is used for attention weighted aggregation, T means transposition, and the matrix is usually represented by T when transposed, K is the key matrix, which is used as the matching basis when being "queried" by other positions, Q is the query matrix, which is used to calculate the "attention" of the current position to other positions, Q T is the transposed matrix of Q, which is used to match the dimensions when matrix multiplication is performed, then the projection of Query, Key and Value is as follows:

[0145] Q=W q X

[0146] K=W k X

[0147] V=W v X

[0148] where X is an input feature matrix (tensor data), W q , W k , and W v are trainable weight matrices, W is a weight coefficient in the training process, the attention of each element is obtained through Q T K, then the model is normalized row by row using Softmax, and finally the self-attention module is completed by fusing the V parameter. The attention matrix makes the model interpretable and pays more attention to key frames.

[0149] Finally, the class information of the dynamic gesture is output through global pooling and a classification head.

[0150] The application provides a dynamic gesture recognition method in which multiple models are combined, a YOLOv8 model is designed to be lightweight, real-time recognition (such as 25 FPS on RK3566) is achieved on a device with limited resources, and the problem that ID information is easily exchanged when crossing waving hands is solved by a target tracking module dynamic threshold control parameter.

[0151] The application also protects a timing dynamic gesture recognition system, which comprises the following functional units:

[0152] A hand target detection unit is used for inputting a video stream, inputting the video stream into a hand gesture detection and timing dynamic gesture recognition algorithm after the video stream is disassembled into each frame of picture, and performing hand target detection.

[0153] In this embodiment, the hand target detection is performed by detecting the hand key points in the picture through the lightweight YOLOv8 model, if the hand target is detected, the timing dynamic gesture recognition algorithm is activated to enter step S2, and if the hand target is not detected, the standby state is entered, and the hand target detection is continuously performed through step S1.

[0154] In this embodiment, the specific steps of the lightweight YOLOv8 model are as follows:

[0155] The original YOLOv8 convolution adopts a C2f module, which has high structural complexity and contains branch, residual, fusion and other operations, the original C2f module formula is as follows:

[0156] C2f(X)=Conv 1×1 (concat(X1,Z1,...,Z n ))

[0157] X1,X2=split(X,dim=2),X1∈R H×W×C / 2 ,X2∈R H×W×C / 2

[0158] Z t =Z t-1 +Conv 3×3 (Conv 1×1 (Z t-1 ))

[0159] Wherein, Conv 1×1 represents a convolution kernel module, the convolution kernel traverses the window by 1x1 steps, X is an input matrix (tensor) data, the form of the tensor data in the neural network is composed of (batch, channel number, height, width), Z1, X1, Z nThe tensors data from different neural network layers are represented, R, H, W, and C represent the batch, height, width, and channel number of the feature map respectively, Zt represents the intermediate feature generated by the t-th step recursion, concat represents a splicing module, and split represents a splitting module, which is split into two parts in this embodiment. As can be seen, the original C2f module has a large amount of calculation and is not suitable for hardware with limited resources.

[0160] In this embodiment, the C2f of YOLOv8 is replaced by a lightweight Conv module (the Conv module is a module built in the neural network for convolution), the complexity of the model is reduced by replacing the relatively complex C2f module with the Conv module, and the model interpretability is enhanced, thereby greatly reducing the time consumed by model inference.

[0161] The hand target tracking unit is configured to track the hand target by using a multi-target tracker.

[0162] In this embodiment, if the hand target is detected for the first time, the target tracking algorithm is skipped, and from the second time when the hand target is detected, it is determined whether it is the same palm according to the historical accumulation information, so as to complete the target tracking algorithm.

[0163] In this embodiment, the target tracking algorithm is mainly composed of Kalman filtering and the Hungarian algorithm, and the specific implementation process is as follows:

[0164] The hand key point information obtained by YOLOv8 is subjected to target screening through a confidence threshold and an IOU threshold, and low-quality detection boxes are filtered. If the current input frame is the first frame, since there is no historical information, the detection box center point coordinates (cx, cy) and the class information (class_id) of the first frame target are mainly accumulated. In this process, the detection box (x1, y1, x2, y2) (left upper corner and right lower corner coordinates) output by YOLOv8 needs to be subjected to target center point position conversion, and the class information is obtained through a confidence threshold (in this embodiment, the confidence score threshold is set to 0.35). From the second frame data, the state vector x of the target can be calculated in combination with two frame data. k x y k is the state vector of the target, x and y are the position coordinates of the target center point, and V x and V y are the velocity components of the target in the x and y directions. Assuming that the target moves at a constant speed in the image, the state transition matrix F can be obtained.

[0165]

[0166] ​​​Where F is the state transition matrix, and At is the time interval between two successive state estimations (or observations) (unit: second), which is the key parameter in Kalman filter to connect the states at different time, and reflects the rate of dynamic change of the system.

[0167] Then the predicted target position can be obtained by the state prediction equation:

[0168] x k|k-1 = F * x k-1

[0169] Where k is the current frame, k-1 is the previous frame, x k-1 is the actual state estimation value of the previous frame, and x k|k-1 is the predicted state value of the current frame. Through the prediction equation, the following two effects are achieved: predicting the next frame position of the target, making up for detection omission (such as occlusion), smoothing the motion trajectory, and reducing detection jitter.

[0170] The Euclidean distance C is calculated by the position information predicted by the second frame and the hand target position information detected by the second frame. If it exceeds the max_distance parameter, it is considered that there is no match. In the case of missing the target, the accumulation is carried out. When the number of continuous frames exceeds the max_age parameter, the same target ID is lost, that is, the ID target is not detected, the target is cleared, and it is considered that the target with the same ID has left the picture range in the new picture. If there is one or more matched targets, the one with the smallest Euclidean distance is considered to be the same target. In order to ensure the reliability of the Euclidean distance determination, the max_distance parameter is set as a dynamic parameter. When the hand detection exists intersection, the max_distance parameter is reduced from the Euclidean distance value 30 to the Euclidean distance value 10, and a more stringent condition is enabled. In the case where there is no intersection, the max_distance parameter uses the default value of the Euclidean distance value 30. Similarly, the max_age parameter is set. When the number of unmatched frames exceeds max_age, it is considered that the target is lost, which can effectively prevent the ID exchange situation in target tracking. The specific formula is as follows:

[0171]

[0172] In the formula, i is the serial number of the predicted position; j is the serial number of the actual position; C ij is the cost matrix, which represents the matching cost (such as Euclidean distance) of the ith detection and the jth predicted trajectory; A ij is the assignment matrix, A ij = 1 indicates that detection i matches trajectory j, otherwise 0; s.t. is the abbreviation of "subject to", which means "subject to", indicating the constraint condition of the optimization problem.

[0173] The process of matching the minimum Euclidean distance is the Hungarian algorithm, which mainly finds the global optimal matching through matrix operation. k and Kalman gain K k , complete the whole target tracking process:

[0174] x k = x k + K k (x k -x k|k-1 )

[0175] K k = P k|k-1 H T (HP k|k-1 H T + R) -1

[0176] P k|k-1 = FP k-1 F T + Q

[0177] In the application, H, Q and R are all set as constants, wherein H is a linear relationship matrix of measurement and state, Q is a process noise matrix, R is a measurement noise matrix, and P is an error covariance matrix. After updating the Kalman gain and the matching item, the current frame target tracking is completed, and then the next cycle is entered to process the next frame of data.

[0178] When the Kalman filter and the Hungarian algorithm are used to solve the cross waving hand prediction tracking ID information of each hand, the ID information is prone to exchange.

[0179] The output category information unit is used in combination with the hand target detection unit and the hand target tracking unit, and through a time sequence dynamic gesture recognition algorithm (the time sequence dynamic gesture recognition algorithm in the embodiment is an algorithm built by a 1D operator) to consider that the same palm is detected, and corresponding category information is output.

[0180] In this embodiment, the position information, confidence score and class information of the detection box output by the YOLOv8 model are compared with the manually set confidence threshold and IOU threshold to determine the palm class score. If the set threshold is not reached, it is considered not to be a palm class, and the next frame of picture processing is directly performed. If the set threshold is reached, subsequent hand skeleton key point processing is performed. The gesture classes here include: 0 (fist), 1, 2, 3, 4, 5 (palm), ok, and others. Here, the types are all gesture gestures, 0 represents a fist, 1 represents gesture gesture 1, 2 represents gesture gesture yes or gesture gesture 2, 3 represents converging thumb and little finger gesture 3, 4 represents converging thumb gesture 4, 5 represents spreading palm gesture gesture 5, ok represents converging thumb and index finger gesture ok, and others represent gestures not in the above categories.

[0181] The information container setting unit is used to set a 30-frame hand skeleton key point information container. The 30th frame processes information from the 1st to the 30th frame, the 31st frame processes information from the 2nd to the 31st frame, and so on. In this embodiment, the information container is a 30-frame data information container set by a person for saving, which is a list set by a person. The length of the list is 30. The 1st to the 30th frames are information of each frame in the video, which is only stored in full 30 frames. If it is not full 30 frames, it will continue to accumulate in the list. After 30 frames are met, the first frame is removed and the last frame is added to form a new 30 frames. From the 30th frame, the container will always store 30 frames of hand information. Only when at least 25 frames of data in the container are 5 (palm) class data, the data in the container is transmitted to the next step.

[0182] The gesture action prediction unit is used to input the 30-frame hand skeleton key point information container that meets the condition into a 1D neural network to predict the gesture action. The prediction of dynamic gestures is completed by splicing multiple frames of static gesture key points. Finally, the highest confidence class among the 5 classes of "left swing hand", "right swing hand", "up swing hand", "down swing hand" and "others" is output.

[0183] In this embodiment, the specific steps of the self-defined 1D neural network gesture model are as follows:

[0184] In the training process, data random scaling enhancement and overall displacement random variation enhancement are enabled. Random enhancement means adding random factors, which may enhance or weaken. For a neural network, whether it is enhanced or weakened is beneficial to the generalization ability of the model. It is equivalent to adding the diversity of samples, which is helpful for the neural network model to learn complex sample features, reduces the overfitting of the model, and increases the diversity of data. At the same time, gradient clipping is used to limit gradient explosion in each epoch (epoch is a key hyperparameter, which means one complete traversal of the entire training data set), further stabilizing the training of the model.

[0185] The loading data is mainly to obtain the 21 key point information of the detection target hand, and then to construct a key point container to accumulate the key information of 30 frames of hands. By continuously tracking the same target through the target tracking algorithm, the (x, y) coordinates of each key point can be obtained for each frame of data, that is, one target in a frame of picture will provide 21*2 dimensions, and 30 frames of pictures will provide 30*21*2=1260 dimensions of information.

[0186] The input layer is mainly composed of 1D convolution layer, normalization, activation function and down sampling.

[0187] input=MaxPool(ReLU(BatchNorm(Conv1d(X))))

[0188]

[0189] ReLU(X)=max(0,X)

[0190] Wherein, μ, σ is the mean and variance of the channel; γ, β is the learnable scaling and offset parameters, X is the tensor data (representing input feature information), in the neural network, data is operated in the form of tensor, ε is a very small constant (such as 10 -5 ), used to prevent the denominator from being zero (numerical stability). BatchNorm accelerates the training convergence and alleviates the problem of gradient vanishing and explosion. ReLU activation function makes the model nonlinear and enhances the expression ability of the model.

[0191] The residual block is connected by 6 residual modules, and the formula is as follows:

[0192] Residual=ReLU(F(X)+downsample(X))

[0193] F in the formula is two convolution layers, downsample is a jump connection layer, X is tensor data (input feature), in the neural network, data is operated in the form of tensor, Residual is the name of residual module, ReLU is the name of activation function, Residual and ReLU are professional terms, and the stacking of 6 residual networks learns deeper features.

[0194] The formula of self-attention module is as follows:

[0195] Output=V*Attention T

[0196] Attention=Softmax(Q T K)

[0197] Suppose the feature after the residual block is (C, L), where C is the number of channels, L is the sequence length, V is the value matrix, which is used to store the actual information of the position, for attention weighted aggregation, T means transposition, the matrix is usually represented by T when transposed, K is the key matrix, which is used as the basis for matching when being "queried" by other positions, Q is the query matrix, which is used to calculate the "attention" of the current position to other positions, Q T is the transpose matrix of Q, which is used to match the dimensions when matrix multiplication, then projected by Query, Key, Value as follows:

[0198] Q=W q X

[0199] K=W k X

[0200] V=W v X

[0201] Where X is the input feature matrix (tensor data), W q , W k , W v are all trainable weight matrices, and W is the weight coefficient in the training process, the attention of each element is obtained through Q T K, then the model is normalized by row using Softmax, and finally the V parameter is fused to complete the self-attention module, the attention matrix is used to make the model interpretable and pay more attention to key frames.

[0202] Finally, the category information of the dynamic gesture is output through global pooling and a classification head.

[0203] The application provides a dynamic gesture recognition method in which multiple models are combined, a YOLOv8 model is designed to be lightweight, real-time recognition (such as 25FPS on RK3566) is achieved on a device with limited resources, and the problem of easy exchange of ID information when crossing waving hands is solved by using a target tracking module to dynamically control parameters.

Claims

1. A method of temporal dynamic gesture recognition, characterized in that: The method comprises the following steps: Step S1, hand target detection: input a video stream, and input the video stream after being disassembled into each frame of picture into a gesture detection and time sequence dynamic gesture recognition algorithm to perform hand target detection; when the hand target detection is performed, a light YOLOv8 model is used to detect hand key points in the picture; if the hand target is detected, the time sequence dynamic gesture recognition algorithm is activated to enter step S2; if the hand target is not detected, a standby state is entered, and the hand target detection is continuously performed through step S1; Step S2, hand target tracking: a multi-target tracker is used to track the hand target; Step S3, output category information: in combination with steps S1 and S2, the same palm is considered to be detected through the time sequence dynamic gesture recognition algorithm, and corresponding category information is output; the position information, the confidence score and the category information output by the YOLOv8 model are compared with a manually set confidence threshold and an IOU threshold to judge the palm category score; if the set threshold is not reached, it is considered that it is not the palm category, and the next frame of picture is directly processed; if the set threshold is reached, subsequent hand skeleton key point processing is performed; Step S4, set information container: a 30-frame hand skeleton key point information container is set, 1 to 30 frames of information are processed in the 30th frame, 2 to 31 frames of information are processed in the 31st frame, and so on; from the 30th frame, the container will always store 30 frames of hand information; only the data that meets the condition that at least 25 frames of data are palm category data is output to the next step; Step S5, gesture action prediction: the 30-frame hand skeleton key point information container that meets the condition is input into a 1D neural network to predict a gesture action; a multi-frame static gesture key point splicing method is used to complete the prediction of the dynamic gesture; finally, the highest confidence category in five categories of "left swing hand", "right swing hand", "up swing hand", "down swing hand" and "other" is output.

2. The method of claim 1, wherein: In step S1, the light YOLOv8 model replaces C2f of YOLOv8 with a light Conv module, and the Conv module is used to replace the relatively complex C2f module.

3. The method of claim 1, wherein: In step S2, if the hand target is detected for the first time, the target tracking algorithm is skipped; from the second time when the hand target is detected, whether it is the same palm is judged according to historical accumulation information.

4. The method of claim 1, wherein: In step S2, the target tracking algorithm is composed of Kalman filtering and the Hungarian algorithm, and comprises: The hand keypoint information acquired by YOLOv8 is used to filter targets by applying confidence and IOU thresholds, filtering out low-quality detection boxes. If the current input frame is the first frame, the center point coordinates (cx, cy) and class information (class_id) of the detection boxes of the targets in the first frame are accumulated. The detection boxes (x1, y1, x2, y2) output by YOLOv8 are then processed... The formula is transformed into the target center point location, and category information is obtained by filtering through a confidence threshold. Starting from the second frame of data, the target's state vector x is calculated by combining the two frames of data. k = (x, y, v) x v y In the formula, x k Let x and y be the state vector of the target, and v be the position coordinates of the target's center point. x Let v be the velocity component of the target in the x-direction. y Let F be the velocity component of the target in the y-direction. Assuming the target moves at a constant velocity in the image, the state transition matrix F is obtained: Wherein, F is a state transition matrix, and Δt is a time interval between adjacent state estimates or observations, Then, the predicted target position is obtained through a state prediction equation: x k|k-1 = F * x k-1 wherein k is the current frame, k-1 is the previous frame, x k-1 is the actual state estimate value of the previous frame, x k|k-1 is the predicted state value of the current frame; Calculate the Euclidean distance C between the position information predicted by the second frame and the hand target position information detected by the second frame. If it exceeds the max_distance parameter, it is considered not to match. In the case of missing the target, accumulate when the number of consecutive frames exceeds the max_age parameter, and if the same target ID is missing, the target is cleared and it is considered that the target of the same ID in the picture has left the picture range. If there is one or more matching targets, the one with the smallest Euclidean distance is considered to be the same target. To ensure the reliability of the Euclidean distance determination, the max_distance parameter is set as a dynamic parameter. When the hand detection exists intersection, the max_distance parameter is reduced from the Euclidean distance value 30 to the Euclidean distance value 10. In the case where there is no intersection, the max_distance parameter uses the default value of the Euclidean distance value 30. Similarly, the max_age parameter is set. When the number of unmatched frames exceeds max_age, it is considered that the target is lost. The specific formula is as follows: where i is the index of the predicted position; j is the index of the actual position; C ij is the cost matrix, which represents the matching cost of the ithdetection and the jthpredicted trajectory; A ij is the assignment matrix, which indicates that A ij = 1 means that the detection i matches the trajectory j, otherwise 0; s.t. means the constraint condition of the optimization problem; The process of matching the minimum Euclidean distance is the Hungarian algorithm, which finds the global optimal matching through matrix operation, and finally updates the matching item x k and Kalman gain K k , complete the entire target tracking process: x k = x k + K k (x k - x k|k-1 ) K k = P k|k-1 H T (HP k|k-1 H T + R) -1 P k|k-1 = FP k-1 F T + Q Wherein, H, Q, R are all set as constants, H is the linear relationship matrix of measurement and state, Q is the process noise matrix, R is the measurement noise matrix, P is the error covariance matrix, after completing the update of Kalman gain and matching item, the current frame target tracking is completed, then it will enter the next cycle to process the next frame data.

5. The method of claim 1, wherein: In step S5, the specific steps of customizing the 1D neural network gesture prediction model are as follows: In the training process, data random scaling enhancement and overall displacement random variation enhancement are enabled to reduce model overfitting while increasing data diversity. In addition, gradient clipping is used in each epoch to limit gradient explosion, further stabilizing model training. Load data to obtain 21 key point information of the detected target hand, then build a key point container to accumulate 30 frames of key information of the hand, and continuously track the same target through the target tracking algorithm to obtain the (x, y) coordinates of each key point in each frame of data. The input layer is composed of 1D convolution layer, normalization, activation function and down sampling: input=MaxPool(ReLU(BatchNorm(Conv1d(X)))) ReLU(X)=max(0,X) Wherein, μ, σ are the mean and variance of the channel; γ, β are the learnable scaling and offset parameters, X is the tensor data, indicating the input feature information, and ε is a very small constant. BatchNorm accelerates training convergence, and ReLU activation function makes the model nonlinear. The residual block is connected by 6 residual modules, and the formula is as follows: Residual=ReLU(F(X)+downsample(X)) In the formula, F is two convolution layers, downsample is a skip connection layer, X is tensor data, indicating input feature information, Residual is a residual module, and ReLU is an activation function. Through the stacking of 6 residual networks, deep-level features are learned. The self-attention module formula is as follows: Output = V * Attention T Attention = Softmax(Q T K) Assume the features after the residual block are (C, L), where C is the number of channels, L is the sequence length, V is the value matrix, which stores the actual information of the position for attention weighted aggregation, T means the transpose, the matrix is denoted by T when transposed, K is the key matrix, which is the basis for matching when being "queried" by other positions, Q is the query matrix, which is used to calculate the "attention" of the current position to other positions, Q T is the transpose matrix of Q, which is used to match the dimensions when matrix multiplication, where T means the transpose, the matrix is denoted by T when transposed, then the projection of Query, Key, Value is as follows: Q = W q X K = W k X V = W v X Wherein, X is an input feature matrix, is a tensor data, W q , W k and W v are all trainable weight matrices, W is a weight coefficient of the training process, and the attention degree of each element is obtained through Q T , and then the model is normalized by row using Softmax, and finally the self-attention module is completed by fusing the V parameter.

6. A temporal dynamic gesture recognition system characterized by: The system comprises: The hand target detection unit is configured to input a video stream, input the video stream into a gesture detection and time sequence dynamic gesture recognition algorithm after the video stream is disassembled into each frame of picture, and perform hand target detection. In the hand target detection, a light YOLOv8 model is used to detect hand key points in the picture. If a hand target is detected, the time sequence dynamic gesture recognition algorithm is activated to enter the hand target tracking unit. If no hand target is detected, the hand target detection unit is kept in a standby state. The hand target tracking unit is configured to track the hand target by using a multi-target tracker. The output category information unit is configured to combine the hand target detection unit and the hand target tracking unit, output corresponding category information by the time sequence dynamic gesture recognition algorithm, and output the detection frame position information, the confidence score and the category information by the YOLOv8 model. The palm category score is determined by comparing a manually set confidence threshold and an IOU threshold. If the set threshold is not reached, it is determined that it is not a palm category, and the next frame of picture is directly processed. If the set threshold is reached, subsequent hand skeleton key point processing is performed. The information container setting unit is configured to set an information container of 30 frames of hand skeleton key points. The information of 1st to 30th frames is processed in the 30th frame, the information of 2nd to 31st frames is processed in the 31st frame, and so on. From the 30th frame, the container will always store 30 frames of hand information. Only when at least 25 frames of data in the container are palm category data, the data in the container is transmitted to the next step. The gesture action prediction unit is configured to input the 30 frames of hand skeleton key point information container meeting the condition into a 1D neural network to predict a gesture action. The prediction of the dynamic gesture is completed by using a multi-frame static gesture key point splicing method. Finally, the highest confidence category in the five categories of "left swing hand", "right swing hand", "up swing hand", "down swing hand" and "other" is output.

7. The system of claim 6, wherein: In the hand target detection unit, the light YOLOv8 model replaces the C2f of YOLOv8 with a light Conv module. The light Conv module replaces the relatively complex C2f module.

8. The system of claim 6, wherein: In the hand target tracking unit, if the hand target is detected for the first time, the target tracking algorithm is skipped. From the second time when the hand target is detected, whether it is the same palm is determined according to the historical accumulation information.

9. The system of claim 6, wherein: In the hand target tracking unit, the target tracking algorithm is composed of Kalman filtering and the Hungarian algorithm, including: The hand key point information obtained by YOLOv8 is screened by a confidence threshold and an IOU threshold, low-quality detection boxes are filtered, if the current input frame is the first frame, the detection box center point coordinates (cx, cy) and the class information (class_id) of the first frame target are accumulated, the detection box (x1, y1, x2, y2) output by YOLOv8 is converted into the target center point position through the formula , and the class information is obtained by filtering through the confidence threshold, from the second frame data, the state vector x of the target is calculated in combination with two frame data k =(x, y, v x , v y ), in the formula, x k is the state vector of the target, x and y are the position coordinates of the target center point, v x is the velocity component of the target in the x direction, and v y is the velocity component of the target in the y direction, it is assumed that the target moves at a constant speed in the image, and the state transition matrix F is obtained: wherein F is a state transition matrix, and Δt represents a time interval between adjacent state estimates or observations, Then, the predicted target position is obtained by a state prediction equation: x k|k-1 = F * x k-1 wherein k is the current frame, k-1 is the previous frame, x k-1 is the actual state estimate value of the previous frame, x k|k-1 is the predicted state value of the current frame; Calculate the Euclidean distance C between the position information predicted by the second frame and the hand target position information detected by the second frame. If it exceeds the max_distance parameter, it is considered not to match. In the case of missing the target, accumulate when the number of continuous frames exceeds the max_age parameter, and when the same target ID is missing, it is considered that the target of the same ID is not detected, the target is cleared and it is considered that the target of the same ID in the picture has left the picture range. If there is one or more matching targets, the one with the smallest Euclidean distance is considered to be the same target. To ensure the reliability of the Euclidean distance judgment, the max_distance parameter is set as a dynamic parameter. When the hand is detected to exist intersection, the max_distance parameter is reduced from the Euclidean distance value 30 to the Euclidean distance value 10. In the case where there is no intersection, the max_distance parameter uses the default value of the Euclidean distance value 30. Similarly, the max_age parameter is set. When the number of unmatched frames exceeds max_age, it is considered that the target is lost. The specific formula is as follows: where i is the index of the predicted position; j is the index of the actual position; C ij is the cost matrix, which represents the matching cost of the ithdetection and the jthpredicted trajectory; A ij is the assignment matrix, A ij = 1 indicates that the detection i matches the trajectory j, otherwise 0; s.t. represents the constraint conditions of the optimization problem; The process of matching the minimum Euclidean distance is the Hungarian algorithm, which finds the global optimal matching through matrix operation, and finally updates the matching item x k and Kalman gain K k , complete the entire target tracking process: x k = x k + K k (x k - x k|k-1 ) K k = P k|k-1 H T (HP k|k-1 H T + R) -1 P k|k-1 = FP k-1 F T + Q where H, Q, R are set as constants, H is a linear relationship matrix of measurement and state, Q is a process noise matrix, R is a measurement noise matrix, P is an error covariance matrix, after completing the update of Kalman gain and matching items, the current frame target tracking is completed, and then the next cycle is entered to process the next frame data.

10. The system of claim 1, wherein: In the gesture action prediction unit, the specific steps of defining a 1D neural network prediction gesture model are as follows: In the training process, data random scaling enhancement and overall displacement random variation enhancement are enabled to reduce model overfitting and increase data diversity. In addition, gradient clipping is used in each epoch to limit gradient explosion, further stabilizing model training. Load data to obtain 21 key point information of the detected target hand. Then, a key point container is constructed to accumulate 30 frames of key information of the hand. The same target is continuously tracked by a target tracking algorithm, and the (x, y) coordinates of each key point are obtained in each frame of data. The input layer is composed of 1D convolution layer, normalization, activation function and down sampling: input=MaxPool(ReLU(BatchNorm(Conv1d(X)))) ReLU(X)=max(0,X) Where, μ, σ are the mean and variance of the channel; γ, β are the learnable scaling and offset parameters, X is the tensor data, which represents the input feature information, and ε is a very small constant. BatchNorm accelerates training convergence, and ReLU activation function makes the model nonlinear. The residual block is connected by 6 residual modules, and the formula is as follows: Residual=ReLU(F(X)+downsample(X)) In the formula, F is two convolution layers, downsample is a jump connection layer, X is tensor data, which represents input feature information, Residual is a residual module, and ReLU is an activation function. Through the stacking of 6 residual networks, deep features are learned. The self-attention module formula is as follows: Output = V * Attention T Attention = Softmax(Q T K) Assume the features after the residual block are (C, L), where C is the number of channels, L is the sequence length, V is the value matrix, which stores the actual information of the position for attention weighted aggregation, T means the transpose, the matrix is denoted by T when transposed, K is the key matrix, which is the basis for matching when being "queried" by other positions, Q is the query matrix, which is used to calculate the "attention" of the current position to other positions, Q T is the transpose matrix of Q, which is used to match the dimensions when matrix multiplication, where T means the transpose, the matrix is denoted by T when transposed, then the projection of Query, Key, Value is as follows: Q = W q X K = W k X V = W v X where X is an input feature matrix, is a tensor data, W q , W k , and W v are all trainable weight matrices, W is a weight coefficient of the training process, and the attention degree of each element is obtained by Q T , and then the model is normalized by row using Softmax, and finally the self-attention module is completed by fusing the V parameter.

Images