Dynamic gesture recognition method based on hand key points and double-layer bidirectional LSTM network

Abstract

The application relates to a dynamic gesture recognition method based on hand key points and a double-layer bidirectional LSTM network, and comprises the following steps: S1, acquiring a hand key point dataset to obtain an input sequence; S2, performing time flip data enhancement processing on the input sequence obtained in the last step to generate a flip sequence, and taking the input sequence and the flip sequence as network model input layers; S3, constructing a double-layer bidirectional LSTM network model to perform sign language dynamic recognition; S4, processing the flip reverse output sequence by using an attention layer, respectively calculating forward and reverse attention weighted outputs, and splicing the forward and reverse attention outputs; and S5, mapping the attention weighted output to an output dimension by using a full connection layer. According to the application, hand key points are collected by processing and collecting a sign language video dataset, a double-layer bidirectional LSTM network model is constructed to recognize sign language based on key point information, the key point information is fully mined and utilized, and the speed and accuracy of sign language recognition are improved.

Description

Technical fields:

[0001] This invention relates to the field of dynamic gesture recognition technology, specifically to a dynamic gesture recognition method based on hand key points and a two-layer bidirectional LSTM network. Background technology:

[0002] Sign language is a widely used communication medium among hearing-impaired and speech-impaired individuals worldwide, effectively breaking down communication barriers between them and society. Sign language possesses independent grammar, semantic organization, and specific gestural logic. It conveys semantic meaning through various information such as hand shape, direction, and position, as well as body posture, arm gestures, eye contact, lip movements, and facial expressions, without relying on any form of spoken communication. This makes it difficult for non-hearing-impaired individuals to understand the meaning of sign language. The development of sign language recognition technology facilitates effective communication between non-hearing-impaired individuals and those with hearing or speech impairments.

[0003] Currently, sign language recognition can be divided into two categories: isolated symbol recognition (i.e., static gesture recognition) and continuous symbol recognition (i.e., dynamic gesture recognition). Isolated symbol recognition refers to fine-grained recognition of independent sign language actions, while continuous symbol recognition refers to mapping coherent sign language videos to a sequence of corresponding annotations. Static gesture recognition, representing characters without context or with zero sentence structure, has certain limitations in everyday use. Dynamic gestures possess the ability to remember and express information and are context-aware, capable of representing temporal dependencies; therefore, dynamic gesture recognition (from raw modal sequences to discrete labeled streams) has greater development potential. Based on differences in temporal feature extraction methods, dynamic gesture recognition can be divided into traditional machine learning models and popular deep learning models.

[0004] In the field of deep learning, researchers have achieved some results: Yang Guanci et al. proposed an LSTM model based on a global attention mechanism, which considers all hidden states of the encoder when inferring context vectors, and realizes the recognition of long sequence sign language; Sharma and Kumar proposed a spatiotemporal information tracking model that integrates 3D-CNN and motion history images, which can extract dynamic spatial attention and its related attributes to improve recognition accuracy; Dwivedi proposed a temporal multi-channel visual transformer model (TMC-ViT), which uses EMG signals to decode the motion of objects in dexterity hand manipulation tasks, and realizes multi-channel gesture tracking; Wang Shuai et al. proposed a Transformer model for continuous sign language recognition, which uses linear high-dimensional mapping to proportionally increase the number of attention heads and embedding dimensions to achieve overall internal frame mining of long sign language sequences. However, sign language recognition still faces some challenges: gesture artifacts, hand motion blur, low resolution, and complex and varied backgrounds can lead to recognition difficulties and limit the model's fitting ability. At the same time, occlusion of the main structure, overexposure, and underexposure of lighting result in a lack of diversity in the format and linguistic granularity of sign language annotations. These factors make sign language recognition more difficult.

[0005] It should be noted that the above content falls within the inventor's technical knowledge and does not necessarily constitute prior art. Summary of the Invention:

[0006] The purpose of this invention is to solve the problems existing in the prior art and provide a dynamic gesture recognition method based on hand key points and a two-layer bidirectional LSTM network. The method collects key point information from sign language video datasets and uses a two-layer bidirectional LSTM network model to recognize sign language based on key point information, thereby fully mining and utilizing key point information and improving the speed and accuracy of sign language recognition.

[0007] The present invention achieves the above objectives by adopting the following technical solutions:

[0008] A dynamic gesture recognition method based on hand key points and a two-layer bidirectional LSTM network includes the following steps:

[0009] S1. Obtain the hand key point dataset.

[0010] The process involves acquiring sign language video data from a sign language video dataset and then sequentially initializing the Mediapipe human pose model, sampling the matrix with given axes, initializing the Holistic model, reading each frame of the video in a loop, performing keypoint detection and saving the detected keypoint information data to a list, drawing hand pose keypoints and connecting lines, converting the keypoint list into a NumPy array, and sampling the keypoint list to obtain the input sequence.

[0011] S2, Input Layer of Generative Network Model

[0012] The input sequence obtained in the previous step is subjected to time-flipped data augmentation to generate a flipped sequence. The input sequence and the flipped sequence are used as the input layer of the network model.

[0013] S3. Construct a two-layer bidirectional LSTM network model for dynamic sign language recognition.

[0014] The two-layer bidirectional LSTM network model consists of two sequentially designed bidirectional LSTM layers. Each bidirectional LSTM layer includes a forward LSTM layer and a backward LSTM layer. The input sequence passes through the forward LSTM layer to obtain the forward hidden state and cell state. The forward hidden state and cell state are passed and updated within the forward LSTM layer. The reversed sequence passes through the backward LSTM layer to obtain the reversed hidden state and cell state. The reversed hidden state and cell state are passed and updated within the backward LSTM layer. The input layer is processed by the two-layer bidirectional LSTM network model to extract the temporal features of the data and obtain the reversed output sequence.

[0015] S4. Use an attention layer to process the flipped and reversed output sequence, calculate the attention-weighted outputs for the forward and reverse directions respectively, and concatenate the attention outputs for the forward and reverse directions.

[0016] S5. Use a fully connected layer to map the attention-weighted output to the output dimension.

[0017] In step S1, during the hand keypoint data collection phase, a capture object is instantiated to access and decode consecutive video frames. Hand tracking is implemented using the MediaPipe hand model, identifying and locating keypoints of the hand from the video frames. Changes in keypoint positions between consecutive frames are compared, and a NumPy set of keypoint coordinates from the dynamic gesture video stream is collected. Priority empirical playback sampling is performed based on the sparsity of the temporal data. The specific steps of priority empirical playback are as follows:

[0018] S11. Calculate the TD error. For each empirical example, calculate its TD error:

[0019]

[0020] In the formula, TD error (t) is the TD error at time step t, R t+1 It is the actual reward obtained at time step t+1, γ is the discount factor used to calculate the current estimate of future rewards, and π(a|S t+1 π(a|S) is the probability of choosing action a in state t+1. t+1 ) and Q(S t+1, a; θ) is the state-action value function, which calculates the expected future reward given a state and action, and θ represents the weights of the network;

[0021] S12. Assign priority: Use the TD error to assign priority to each sample, and add a positive number ε to the TD error to ensure that every experience has the possibility of being sampled: p(t)=|TD error (t)|+ε

[0022] In the formula, p(t) is the priority of time step t, |TD error (t)| represents the absolute value of the TD error at time step t, and the positive number ε is used to ensure that each experience has the possibility of being sampled;

[0023] S13, Calculate the sampling probability

[0024] The probability of each experience being sampled is calculated based on the priority of the experience and the hyperparameter α:

[0025]

[0026] In the formula, P(t) is the probability that the experience is sampled at time step t, P is the priority of the experience, and α is a hyperparameter between 0 and 1 used to control the degree of influence of the priority.

[0027] S14, Weight Update

[0028] After sampling and learning using experience, importance sampling weights are calculated to adjust the impact of that experience when updating the Q-value:

[0029]

[0030] In the formula, N is the size of the experience pool, P(t) is the probability that the experience is sampled, and β is a hyperparameter between 0 and 1:

[0031]

[0032] In the formula, ρ is the decay factor, and θ is the learning rate. It is the moving average of the squared gradient over the past period. It is the gradient of the loss function with respect to θ.

[0033] In step S1, during keypoint detection, pixel-level masks and deformation fluctuation transforms are introduced to guide the network's attention and track fine-grained gesture changes; the continuous form expression of the pixel-level mask is:

[0034]

[0035] The discrete form of a pixel-level mask is expressed as follows:

[0036]

[0037] In the formula, E(u) is the energy of the deformation vector field u; Ω is the definition of the region, which is a two-dimensional or three-dimensional space; u1 and u2 are the components of u, which represent the changes of u in the x and y directions in two-dimensional space. These four terms represent the gradient of u; dx is the infinitesimal element of the integral, representing a small region in space; λ is the regularization parameter, which controls the contribution of the magnitude of u to the energy; i and j are the coordinates in the image or a discrete grid; u i+1，j u i-1，j u i，j+1 u i，j-1 These four terms represent the differences of u in various directions within a discrete grid; n and m represent the size of the image or discrete grid.

[0038] The deformation wave transformation expression is:

[0039]

[0040] I(x, y) = w R ·R(x,y)+w G ·G(x,y)+w B ·B(x,y)

[0041] In the formula, A(x, y) is the attention weight of pixel (x, y), I(x, y) is the pixel value of the input image at (x, y), and the summation operation in the denominator is performed on the entire image; R(x, y), G(x, y), and B(x, y) represent the intensities of the red, green, and blue channels of pixel (x, y), respectively, while w R w G and w B It represents the weight of the corresponding channel.

[0042] In step S2, for RGB channel time series containing high-dimensional features at each time step, time flip enhancement is used by combining time shifting and time noise injection. Each RGB channel is treated as an independent sequence, and the sequence of each channel is flipped. The time shifting involves moving the entire time series forward or backward by a certain time unit, reducing the model's dependence on the absolute time of events and focusing more on the relative order or pattern of events. Specific steps are as follows:

[0043] Suppose the input data is a tensor X of size t×m×n×3, X∈R T×H×W×C Where t is the number of time steps, m and n are the image height and width in each time step, 3 represents the three RGB color channels, and each element x of the tensor ijklLet X represent the pixel value at time i, height j, width k, and color channel l; select a translation step s, move tensor X along the time axis by s steps to obtain a new tensor X′, allocate the corresponding space for the new tensor X′, which can be represented as:

[0044] X′∈R T×H×W×C

[0045] Each element x′i,j,k,l of the new tensor X′ can be represented as:

[0046] x′i,j,k,l=x (i+s)modT，j，k，l

[0047] In the formula, (i+s)modT represents shifting i along the time axis, and modT ensures that the index is within the valid range. This process can be represented by the following equation:

[0048]

[0049] For each j∈[0, H), k∈[0, W), and l∈[0, C), the above system of equations holds;

[0050] X′=x (i+s)modt，j，k，l In the formula, mod represents the modulo operation, which ensures that the index is within the valid range.

[0051] The time noise injection involves adding random noise to the time series to simulate measurement errors or other forms of uncertainty that may occur in the real world, making the model more robust and capable of handling noisy data in practical applications. Specific steps include:

[0052] Suppose the input data is a tensor X of size t×m×n×3. Noise injection is performed on each element of tensor X to obtain a new tensor X′:

[0053] X′=x i，j，k，l +noise(i, j, k, l)

[0054] In the formula, noise(i, j, k, l) is the Gaussian noise generation function;

[0055] Suppose there is a four-dimensional tensor X∈R T×H×W×C Where T is the number of time steps, H and W are the height and width of the image, respectively, and C is the number of color channels. A noise generation function g also needs to be defined, whose generated noise follows a Gaussian distribution. A new tensor X′ is defined, whose elements x′... i，j，k，l Add the generated noise to the corresponding elements of the original tensor:

[0056] x′ i，j，k，l =x i，j，k，l +g(0, σ)

[0057] In the formula, g(0, σ) represents Gaussian noise with a mean of 0 and a standard deviation of σ. The above process is represented by a system of equations:

[0058] x′0,j,k,l=x0,j,k,l+g(0,σ)

[0059] x′1,j,k,l=x1,j,k,l+g(0,σ)

[0060] x'T-1,j,k,l=x(T-1),j,k,l+g(0,σ)

[0061] The above system of equations holds for every j∈[0, H), k∈[0, W), and l∈[0, C).

[0062] In step S3, the forward LSTM layer and the backward LSTM layer each include multiple LSTM units. Each LSTM unit includes an input gate, a forget gate, an output gate, a cell state, and a hidden state. The input gate is used to control the weights of the input data and the flow of information into the cell state. The forget gate is used to control the information that needs to be forgotten in the cell state. The output gate is used to control the flow of information from the cell state to the hidden state. The cell state is the memory part of the LSTM unit, responsible for storing and transmitting information. The hidden state is the output of the LSTM unit and is the input of the next time step.

[0063] In step S3, the first bidirectional LSTM layer receives and processes the sign language keypoint sequence input x∈R. input_size By simultaneously considering both positive and negative information in the data, LSTM captures the temporal characteristics of sign language by characterizing the dynamic changes in the keypoint sequence.

[0064] h t =f(W*[h t-1 x t ]+b)

[0065] In the formula, h t It is in a hidden state, x t The input key points are W and b, which are the weights and biases learned, and f is the activation function.

[0066] For each time step t, the first layer of the forward LSTM performs the following calculations:

[0067] i 1，t =σ(W i1 *x t +U i1 *h 1，t-1 +b i1 )

[0068] f1，t =σ(W f1 *x t +U f1 *h 1，t-1 +b f1 )

[0069] o 1，t =σ(W o1 *x t +U o1 *h 1，t-1 +b o1 )

[0070] g 1，t =tanh(W g1 *x t +U g1 *h 1，t-1 +b g1 )

[0071] c 1，t =f 1，t *c 1，t-1 +i 1，t *g 1，t

[0072] h 1，t =o 1，t *tanh(c 1，t )

[0073] In the formula, i, f, o, g represent the candidate states of the input gate, forget gate, output gate, and memory unit, respectively; h and c represent the hidden state and memory state, respectively; W represents the weight between the input and the hidden state, U represents the weight between the input and the candidate memory unit, b represents the bias vector; σ and tanh represent the sigmoid function and the hyperbolic tangent function, respectively; the subscript 1 indicates the first layer, and t represents the time step;

[0074] LSTM adds a reverse information flow on top of this. The reverse LSTM calculation is the same as above, but the time step is reversed. Let the dimension of the hidden layer be hidden_size, then the calculation of the forward and reverse information can be represented as:

[0075]

[0076]

[0077] In the formula, → and ← represent forward and backward propagation, respectively. This is the weight matrix. For bias terms;

[0078] Since LSTM is bidirectional, its output dimension is 2*hidden_size. To enhance the model's expressive power, residual blocks and skip connections are introduced. For the output h of each layer... t Adding skip connections allows it to propagate directly to subsequent layers, i.e.:

[0079]

[0080] In the formula, F(·) represents the function mapping of layer l, and W l The weights of this layer are used to increase the receptive field of the network without increasing model parameters and computational complexity, enabling the model to capture long-distance dependencies in the input data.

[0081] In choosing the activation function, parameterized ReLU is used, that is:

[0082] f(x) = max(0, x) + amin(0, x)

[0083] In the formula, α is a learnable parameter. By introducing more nonlinearity, the model can learn more complex mapping relationships. PReLU is more flexible than ReLU because it introduces a learnable parameter, allowing the network to adaptively adjust the shape of the activation function during training. To improve the training stability and speed of the model, batch normalization is used.

[0084]

[0085] In the formula, μ and σ 2 Let x be the mean and variance, respectively, and let ∈ be a very small number to prevent division by zero. Batch standardization can make the input of each layer approximately zero mean and unit variance, which can accelerate the convergence of the model and also help prevent overfitting and improve the generalization ability of the model.

[0086] In step S3, the second bidirectional LSTM layer receives the output of the first bidirectional LSTM layer and performs further processing to obtain richer dynamic features. The calculation process is as follows:

[0087] i 2，t =σ(W i2 *h 1，t +U i2 *h 2，t-1 +b i2 )

[0088] f 2,t =σ(W f2 *h 1,t +U f2 *h 2，t-1 +b f2 )

[0089] o 2，t =σ(W o2 *h 1，t +U o2 *h 2,t-1 +b o2 )

[0090] g 2,t =tanh(W g2 *h 1，t +U g2 *h 2，t-1 +b g2 )

[0091] c 2,t =f 2，t *c 2,t-1 +i 2,t *g 2,t

[0092] h 2,t =o 2,t *tanh(c 2,t )

[0093] In the formula, i, f, o, g represent the candidate states of the input gate, forget gate, output gate, and memory unit, respectively; h and c represent the hidden state and memory state, respectively; W represents the weight between the input and the hidden state, U represents the weight between the input and the candidate memory unit, b represents the bias vector; σ and tanh represent the sigmoid function and the hyperbolic tangent function, respectively; the subscript 2 indicates the second layer, and t represents the time step;

[0094] Its output dimension is 2*hidden_size. Since this layer also needs to consider forward and backward information, an attention mechanism is introduced to further improve model performance. The attention mechanism allows the model to assign different attention weights to different input elements when processing sequence data, thereby better capturing important information in the sequence. Scaled Dot-Product Attention is used.

[0095]

[0096] In the formula, Q, K, and V represent the query, key, and value, respectively, and d k The dimension is the key; during the calculation, the dot product of the query and the key is first calculated, and then divided by... The values ​​are scaled, then converted into a probability distribution using the softmax function, and finally weighted and summed using this distribution.

[0097] Furthermore, a normalization operation is applied to the output of the second bidirectional LSTM layer, ensuring that each dimension of the output has the same scale, thereby improving the model's stability and accelerating training.

[0098]

[0099] In the formula, x i It is the input, μ B and These are the mean and variance of the batch, respectively. is the normalized input, γ and β are learnable scaling factors and offset terms, and ∈ is a very small number to prevent division by zero.

[0100] In step S4, the attention layer assigns weights to each element in the input, and the assigned weights reflect the importance of that element. The attention layer includes three sub-attention layers. The attention mechanism is used to weight the output of the second bidirectional LSTM layer, allowing the model to focus more on key gestures. The attention weight calculation formula is as follows:

[0101]

[0102]

[0103]

[0104] In the formula, e ij It is the unnormalized attention weight, α ij It is the normalized attention weight, c i It is a context vector, h j These are the hidden states; W, U, and V are the learned weights of the three sub-attention layers; and tanh and exp are the activation functions.

[0105] The computation of the three sub-attention layers is completed by three linear layers, W, U, and V, respectively, as follows:

[0106] S41. Calculate attention weights

[0107] The attention weights are calculated by passing the output of the bidirectional LSTM through two linear layers, W and U. The calculation steps are as follows:

[0108] Process the output of W using the tanh activation function:

[0109]

[0110] Process the output of U using the σ activation function:

[0111]

[0112] Multiply the results of the above two steps element-wise to obtain the attention weights:

[0113]

[0114] S42. Calculate attention scores

[0115] The attention scores are obtained by passing the attention weights calculated in the previous step through layer V, as expressed by:

[0116] s ij =V T ·e ij

[0117] S43. Calculate attention probability

[0118] Applying the softmax function to the attention scores normalizes all scores to the range of 0 to 1, and sums them to 1, thus obtaining the attention probability at each position, as shown in the following formula:

[0119]

[0120] S44. Introducing an attention mechanism

[0121] The attention probabilities obtained in the previous step are applied to the output of the LSTM to calculate the attention-weighted output. This process can be viewed as multiplying the output of each position by its corresponding attention probability, and then summing the results for all positions to obtain the final attention output.

[0122]

[0123] S45, Combine forward and backward attention outputs

[0124] The forward and backward attention outputs are concatenated to obtain the final output:

[0125]

[0126]

[0127] In step S5, the fully connected layer integrates the local feature information extracted by each of the aforementioned convolutional or pooling layers and fuses them into a global feature description. This global information not only contains the details of each individual feature but also embeds the relationships between features, injecting richer information into subsequent classification tasks.

[0128] By introducing a multilayer perceptual feedforward mechanism in the hidden layer, the number of neurons can be dynamically adjusted according to the complexity of the task and the characteristics of the dataset.

[0129] Let the input be Where n (0)It is the number of neurons in the input layer;

[0130] The weight matrix and bias vector of hidden layer l are W, respectively. (l) and b (l) Then the output h of the l-th hidden layer (l) for:

[0131] h (l) =f(W (l)T h (l-1) +b (l) )

[0132] In the formula, when l = 0, h (0) That is, the initial input x, and f(·) is the activation function;

[0133] Let W be the weight matrix of the output layer. (L+1) The bias matrix is ​​b (L+1) Then the output y of the output layer is:

[0134] y = g(W (L+1)T h (L) +b (L+1) )

[0135] In the formula, g(·) is the activation function of the output layer;

[0136] By finding an ideal balance point, the model can effectively reveal complex patterns in the data, while avoiding the introduction of too many parameters to prevent overfitting.

[0137] Residual connections are used to optimize model output:

[0138] z (l) =W (l) a (l-1) +b (l)

[0139] a (l) =φ(z) (l) )+a (l-1)

[0140] In the formula, z (l) It is the weighted input of the l-th layer, a (l-1) It's the activation of the previous layer, W (l) and b (l) These are the weights and bias parameters, and φ is the activation function;

[0141] To enhance the fully connected layers, a batch normalization layer is introduced. This layer normalizes each batch of data, making the output mean close to 0 and the variance close to 1, thereby improving the network's stability during training. This method reduces the network's sensitivity to weight initialization choices and helps prevent the vanishing gradient problem.

[0142] Calculate the mean and variance of each batch of data:

[0143]

[0144] Standardize the input by subtracting the mean and dividing by the standard deviation:

[0145]

[0146] Introduce scaling factor γ and offset factor β:

[0147]

[0148] Embedding layers are introduced to process data with high-dimensional discrete features. An embedding layer maps each discrete feature to a low-dimensional real-valued vector space, ensuring that similar inputs are mapped to similar embedding vectors, thereby effectively capturing and utilizing the relationships between input features.

[0149] Suppose each hand keypoint sequence has M keypoints, and each keypoint can be any one of k possible locations. Initialize the embedding matrix E for each possible keypoint location. keypoint Its size is K×D, where D is the dimension of the embedding vector.

[0150] For each hand keypoint sequence, find the corresponding row in the embedding matrix to obtain its embedding vector, and then concatenate the embedding vectors of all keypoints to obtain the embedding vector representing the entire keypoint sequence:

[0151] e i，m =E keypoint [k i，nv :]

[0152]

[0153] In the formula, e i，m It is the embedding vector of the m-th keypoint in the i-th hand keypoint sequence, k i，m It is the location index of the key point, E keypoint It is the keypoint embedding matrix, e i It is the embedding vector of the entire i-th hand keypoint sequence.

[0154] Based on this method, the embedding of each keypoint position can be learned, and an embedding vector representing the overall keypoint sequence can be obtained, which can better capture the relationship between keypoints in the hand keypoint sequence.

[0155] A recurrent layer is added to process the sequential data. This recurrent layer preserves information from previous time steps, allowing the model to capture the time-series nature of the input data. The hidden state h...t It is based on the hidden state h of the previous moment. t-1 and the current input x t The calculation yielded the following results:

[0156] h t =φ(W hh h t-1 +W xh x t +b h )

[0157] In the formula, W hh and W xh It is the weight matrix, b h φ is the bias term, and φ is the non-linear activation function. This structure allows the network to remember past information and use that information to influence the current output when necessary.

[0158] The present invention, by employing the above method, can bring the following beneficial effects:

[0159] First, the sign language video dataset is converted into key points. Gesture recognition is then performed based on these key points, allowing the network model to focus more on important hand landmarks, thus improving accuracy and efficiency. Designing a two-layer bidirectional LSTM layer further enhances the extraction and retention of key information, improving the quality of training data and ultimately increasing the network model's recognition efficiency. Attached image description:

[0160] Figure 1 This is a schematic diagram of the two-layer bidirectional LSTM network model of the present invention;

[0161] Figure 2 This is a flowchart illustrating how the present invention obtains the input sequence; Detailed implementation method:

[0162] To more clearly illustrate the overall concept of this invention, a detailed description is provided below with reference to the accompanying drawings and examples.

[0163] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0164] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0165] like Figure 1-2 As shown, the dynamic gesture recognition method based on hand key points and a two-layer bidirectional LSTM network includes the following steps:

[0166] S1. Obtain the hand key point dataset.

[0167] The process involves acquiring sign language video data from a sign language video dataset and then sequentially initializing the Mediapipe human pose model, sampling the matrix with given axes, initializing the Holistic model, reading each frame of the video in a loop, performing keypoint detection and saving the detected keypoint information data to a list, drawing hand pose keypoints and connecting lines, converting the keypoint list into a NumPy array, and sampling the keypoint list to obtain the input sequence.

[0168] S2, Input Layer of Generative Network Model

[0169] The input sequence obtained in the previous step is subjected to time-flipped data augmentation to generate a flipped sequence. The input sequence and the flipped sequence are used as the input layer of the network model.

[0170] S3. Construct a two-layer bidirectional LSTM network model for dynamic sign language recognition.

[0171] The dual-layer bidirectional LSTM network model includes two sequentially designed bidirectional LSTM layers. Each bidirectional LSTM layer includes a forward LSTM layer and a backward LSTM layer. The input sequence passes through the forward LSTM layer to obtain the forward hidden state and cell state. The forward hidden state and cell state are passed and updated within the forward LSTM layer. The reversed sequence passes through the backward LSTM layer to obtain the reversed hidden state and cell state. The reversed hidden state and cell state are passed and updated within the backward LSTM layer. The input layer is processed by the dual-layer bidirectional LSTM network model to extract the temporal features of the data and obtain the reversed output sequence.

[0172] S4. Use an attention layer to process the flipped and reversed output sequence, calculate the attention-weighted outputs for the forward and reverse directions respectively, and concatenate the forward and reverse attention outputs.

[0173] S5. A fully connected layer is used to map the attention-weighted output to the output dimension. First, the sign language video dataset is converted into key points. Gesture recognition is then performed based on these key points, allowing the network model to focus more on important hand key points, which helps improve recognition accuracy and efficiency. By designing a two-layer bidirectional LSTM layer, key information can be more fully extracted and retained, improving the quality of the network model's training data and increasing the network model's recognition efficiency.

[0174] In step S1, the acquisition of hand keypoint data involves instantiating a capture object to access and decode consecutive video frames. The MediaPipe hand model is used for gesture tracking, identifying and locating keypoints in the video frames. Changes in keypoint positions between consecutive frames are compared, and a NumPy set of keypoint coordinates from the dynamic gesture video stream is collected. Priority empirical playback sampling is performed based on the sparsity of the temporal data. The specific steps of priority empirical playback are as follows:

[0175] S11. Calculate TD error

[0176] For each experience, calculate its TD error:

[0177]

[0178] In the formula, TD error (t) is the TD error at time step t, R t+1 It is the actual reward obtained at time step t+1, γ is the discount factor used to calculate the current estimate of future rewards, and π(a|S t+1 π(a|S) is the probability of choosing action a in state t+1. t+1 ) and Q(S t+1 , a; θ) is the state-action value function, which calculates the expected future reward given a state and action, and θ represents the weights of the network;

[0179] S12, Assign Priority

[0180] The TD error is used to prioritize each sample, and a positive number ε is added to the TD error to ensure that every experience has the potential to be sampled.

[0181] p(t) = |TD error (t)|+ε

[0182] In the formula, p(t) is the priority of time step t, |TD error (t)| represents the absolute value of the TD error at time step t, and the positive number ε is used to ensure that each experience has the possibility of being sampled;

[0183] S13, Calculate the sampling probability

[0184] The probability of each experience being sampled is calculated based on the priority of the experience and the hyperparameter α:

[0185]

[0186] In the formula, P(t) is the probability that the experience is sampled at time step t, P is the priority of the experience, and α is a hyperparameter between 0 and 1 used to control the degree of influence of the priority.

[0187] S14, Weight Update

[0188] After sampling and learning using experience, importance sampling weights are calculated to adjust the impact of that experience when updating the Q-value:

[0189]

[0190] In the formula, N is the size of the experience pool, P(t) is the probability that the experience is sampled, and β is a hyperparameter between 0 and 1:

[0191]

[0192] In the formula, ρ is the decay factor, and θ is the learning rate. It is the moving average of the squared gradient over the past period. It is the gradient of the loss function with respect to θ.

[0193] In step S1, during the key point detection process, pixel-level masks and deformation fluctuation transformations are introduced to guide the network's attention and track fine-grained changes in gestures, taking into account the deformation of non-rigid objects such as hands, dynamic backgrounds, and the variability of high internal and low interval categories.

[0194] The continuous form expression of the pixel-level mask is:

[0195]

[0196] The discrete form expression of the pixel-level mask is:

[0197]

[0198] In the formula, E(u) is the energy of the deformation vector field u; Ω is the definition of the region, which is a two-dimensional or three-dimensional space; u1 and u2 are the components of u, which represent the changes of u in the x and y directions in two-dimensional space. These four terms represent the gradient of u; dx is the infinitesimal element of the integral, representing a small region in space; λ is the regularization parameter, which controls the contribution of the magnitude of u to the energy; i and j are the coordinates in the image or a discrete grid; u i+1，j u i-1，j u i，j+1 u i，j-1 These four terms represent the differences of u in various directions within a discrete grid; n and m represent the size of the image or discrete grid.

[0199] The deformation wave transformation expression is:

[0200]

[0201] I(x, y) = wR ·R(x,y)+w G ·G(x,y)+w B ·B(x,y)

[0202] In the formula, A(x, y) is the attention weight of pixel (x, y), I(x, y) is the pixel value of the input image at (x, y), and the summation operation in the denominator is performed on the entire image; R(x, y), G(x, y), and B(x, y) represent the intensities of the red, green, and blue channels of pixel (x, y), respectively, while W... R W G and w B It represents the weight of the corresponding channel.

[0203] In step S2, for the RGB channel time series containing high-dimensional features at each time step, time flip enhancement is used by combining time translation and time noise injection. Each RGB channel is treated as an independent sequence, and the sequence of each channel is flipped.

[0204] The time shift refers to moving the entire time series forward or backward by a certain number of time units, reducing the model's dependence on the absolute time of events and focusing more on the relative order or pattern of events. Specific steps include:

[0205] Suppose the input data is a tensor X of size t×m×n×3, X∈R T×H×W×C Where t is the number of time steps, m and n are the image height and width in each time step, 3 represents the three RGB color channels, and each element x of the tensor ijkl Let X represent the pixel value at time i, height j, width k, and color channel l; select a translation step s, move tensor X along the time axis by X steps to obtain a new tensor X′, allocate the corresponding space for the new tensor X′, which can be represented as:

[0206] X′∈R T×H×W×C

[0207] Each element x′i,j,k,l of the new tensor X′ can be represented as:

[0208] x′i,j,k,l=x (i+s)modT，j，k，l

[0209] In the formula, (i+s)modT represents shifting i along the time axis, and modT ensures that the index is within the valid range. This process can be represented by the following equation:

[0210]

[0211] For each j∈[0, H), k∈[0, W), and l∈[0, C), the above system of equations holds;

[0212] X′=x (i+s)modt，j，k，l

[0213] In the formula, mod represents the modulo operation, which ensures that the index is within the valid range;

[0214] The time noise injection involves adding random noise to the time series to simulate measurement errors or other forms of uncertainty that may occur in the real world, making the model more robust and capable of handling noisy data in practical applications. Specific steps include:

[0215] Suppose the input data is a tensor X of size t×m×n×3. Noise injection is performed on each element of tensor X to obtain a new tensor X′:

[0216] X′=x i，j，k，l +noise(i, j, k, l)

[0217] In the formula, noise(i, j, k, l) is the Gaussian noise generation function;

[0218] Suppose there is a four-dimensional tensor X∈R T×H×W×C Where T is the number of time steps, H and W are the height and width of the image, respectively, and C is the number of color channels. A noise generation function g also needs to be defined, whose generated noise follows a Gaussian distribution. A new tensor X′ can be defined, whose elements x′... i，j，k，l Add the generated noise to the corresponding elements of the original tensor:

[0219] x′i,j,k,l=xi,j,k,l+g(0,σ)

[0220] In the formula, g(0, σ) represents Gaussian noise with a mean of 0 and a standard deviation of σ. This process can be expressed as the following system of equations:

[0221] x′0,j,k,l=x0,j,k,l+g(0,σ)

[0222] x′1,j,k,l=x1,j,k,l+g(0,σ)

[0223] x'T-1,j,k,l=x(T-1),j,k,l+g(0,σ)

[0224] The above system of equations holds for every j∈[0, H), k∈[0, W), and l∈[0, C).

[0225] In step S3, the forward LSTM layer and the backward LSTM layer each include multiple LSTM units. Each LSTM unit includes an input gate, a forget gate, an output gate, a cell state, and a hidden state. The input gate is used to control the weights of the input data and the flow of information into the cell state. The forget gate is used to control the information that needs to be forgotten in the cell state. The output gate is used to control the flow of information from the cell state to the hidden state. The cell state is the memory part of the LSTM unit, responsible for storing and transmitting information. The hidden state is the output of the LSTM unit and is the input of the next time step.

[0226] In step S3, the first bidirectional LSTM layer receives and processes the sign language keypoint sequence input x∈R. input_size By simultaneously considering both positive and negative information in the data, LSTM captures the temporal characteristics of sign language by characterizing the dynamic changes in the keypoint sequence.

[0227] h t =f(W*[h t-1 x t ]+b)

[0228] In the formula, h t It is in a hidden state, x t The input key points are W and b, which are the weights and biases learned, and f is the activation function.

[0229] For each time step t, the first layer of the forward LSTM performs the following calculations:

[0230] i 1，t =σ(W i1 *x t +U i1 *h 1,t-1 +b i1 )

[0231] f 1,t =σ(W f1 *x t +U f1 *h ，t-1 +b f1 )

[0232] o 1,t =σ(W o1 *x t +U o1 *h 1,t-1 +b o1 )

[0233] g 1，t =tanh(W g1 *x t +U g1 *h 1,t-1+b g1 )

[0234] c 1，t =f 1，t *c 1，t-1 +i 1，t *g 1，t

[0235] h 1，t =o 1，t *tanh(c 1，t )

[0236] In the formula, i, f, o, g represent the candidate states of the input gate, forget gate, output gate, and memory unit, respectively; h and c represent the hidden state and memory state, respectively; W represents the weight between the input and the hidden state, U represents the weight between the input and the candidate memory unit, b represents the bias vector; σ and tanh represent the sigmoid function and the hyperbolic tangent function, respectively; the subscript 1 indicates the first layer, and t represents the time step;

[0237] LSTM adds a reverse information flow on top of this. The reverse LSTM calculation is the same as above, but the time step is reversed. Let the dimension of the hidden layer be hidden_size, then the calculation of the forward and reverse information can be represented as:

[0238]

[0239]

[0240] In the formula, → and ← represent forward and backward propagation, respectively. This is the weight matrix. For bias terms;

[0241] Since LSTM is bidirectional, its output dimension is 2*hidden_size. To enhance the model's expressive power, residual blocks and skip connections are introduced. For the output h of each layer... t Adding skip connections allows it to propagate directly to subsequent layers, i.e.:

[0242]

[0243] In the formula, F(·) represents the function mapping of layer l, and W l The weights of this layer are used to increase the receptive field of the network without increasing model parameters and computational complexity, enabling the model to capture long-distance dependencies in the input data.

[0244] In choosing the activation function, parameterized ReLU is used, that is:

[0245] f(x) = max(0, x) + amin(0, x)

[0246] In the formula, 'a' is a learnable parameter. By introducing more nonlinearity, the model can learn more complex mapping relationships. PReLU is more flexible than ReLU because it introduces a learnable parameter, which allows the network to adaptively adjust the shape of the activation function during training.

[0247] To improve the training stability and speed of the model, batch normalization is used:

[0248]

[0249] In the formula, μ and σ 2 Let x be the mean and variance, respectively, and let ∈ be a very small number to prevent division by zero. Batch standardization can make the input of each layer approximately zero mean and unit variance, which can accelerate the convergence of the model and also help prevent overfitting and improve the generalization ability of the model.

[0250] In step S3, the second bidirectional LSTM layer receives the output of the first bidirectional LSTM layer and performs further processing to obtain richer dynamic features. The calculation process is as follows:

[0251] i 2，t =σ(W i2 *h 1，t +U i2 *h 2，t-1 +b i2 )

[0252] f 2，t =σ(W f2 *h 1，t +U f2 *h 2，t-1 +b f2 )

[0253] o 2，t =σ(W o2 *h 1，t +U o2 *h 2，t-1 +b o2 )

[0254] g 2，t =tanh(W g2 *h 1，t +U g2 *h 2，t-1 +b g2 )

[0255] c 2，t =f 2，t *c 2，t-1 +i 2，t *g2，t

[0256] h 2，t =o 2,t *tanh(c 2,t )

[0257] In the formula, i, f, o, g represent the candidate states of the input gate, forget gate, output gate, and memory unit, respectively; h and c represent the hidden state and memory state, respectively; W represents the weight between the input and the hidden state, U represents the weight between the input and the candidate memory unit, b represents the bias vector; σ and tanh represent the sigmoid function and the hyperbolic tangent function, respectively; the subscript 2 indicates the second layer, and t represents the time step;

[0258] Its output dimension is 2*hidden_size. Since this layer also needs to consider forward and backward information, an attention mechanism is introduced to further improve the model's performance. The attention mechanism allows the model to assign different attention weights to different input elements when processing sequence data, thereby better capturing important information in the sequence. Scaled Dot-Product Attention is used.

[0259]

[0260] In the formula, Q, K, and V represent the query, key, and value, respectively, and d k The dimension is the key; during the calculation, the dot product of the query and the key is first calculated, and then divided by... The values ​​are scaled, then converted into a probability distribution using the softmax function, and finally weighted and summed using this distribution.

[0261] Furthermore, a normalization operation is applied to the output of the second bidirectional LSTM layer, ensuring that each dimension of the output has the same scale, thereby improving the model's stability and accelerating training.

[0262]

[0263] In the formula, x i It is the input, μ B and These are the mean and variance of the batch, respectively. is the normalized input, γ and β are learnable scaling factors and offset terms, and ∈ is a very small number to prevent division by zero.

[0264] In step S4, the attention layer assigns weights to each element in the input, and the assigned weights reflect the importance of that element. The attention layer includes three sub-attention layers. The attention mechanism is used to weight the output of the second bidirectional LSTM layer, allowing the model to focus more on key gestures. The attention weight calculation formula is as follows:

[0265]

[0266]

[0267]

[0268] In the formula, e ij It is the unnormalized attention weight, α ij It is the normalized attention weight, c i It is a context vector, h j The hidden state is represented by W, U, and V, which are the learned weights of the three sub-attention layers S. tanh and exp are the activation functions.

[0269] The calculation process for the three sub-attention layers is completed by three linear layers, W, U, and V, respectively, and the calculation process is as follows:

[0270] S41. Calculate attention weights

[0271] The attention weights are calculated by passing the output of the bidirectional LSTM through two linear layers, W and U. The calculation process is as follows:

[0272] Process the output of W using the tanh activation function:

[0273]

[0274] Process the output of U using the σ activation function:

[0275]

[0276] Multiply the results of the above two steps element-wise to obtain the attention weights:

[0277]

[0278] S42. Calculate attention scores

[0279] The attention scores are obtained by passing the attention weights calculated in the previous step through layer V, as expressed by:

[0280] s ij =V T ·e ij

[0281] S43. Calculate attention probability

[0282] Applying the softmax function to the attention scores normalizes all scores to the range of 0 to 1, and sums them to 1, thus obtaining the attention probability at each position, as shown in the following formula:

[0283]

[0284] S44. Introducing an attention mechanism

[0285] The attention probabilities obtained in the previous step are applied to the LSTM output to calculate the attention-weighted output, resulting in the final attention output:

[0286]

[0287] S45, Combine forward and backward attention outputs

[0288] The forward and backward attention outputs are concatenated to obtain the final output:

[0289]

[0290]

[0291] In step S5, the fully connected layer integrates the local feature information extracted by each of the aforementioned convolutional or pooling layers and fuses them into a global feature description. This global information not only contains the details of each individual feature but also embeds the relationships between features, injecting richer information into subsequent classification tasks.

[0292] By introducing a multilayer perceptual feedforward mechanism in the hidden layer, the number of neurons can be dynamically adjusted according to the complexity of the task and the characteristics of the dataset.

[0293] Let the input be Where n (0) It is the number of neurons in the input layer;

[0294] The weight matrix and bias vector of hidden layer l are W, respectively. (l) and b (l) Then the output h of the l-th hidden layer (l) for:

[0295] h (l) =f(W (l)T h (l-1) +b (l) )

[0296] In the formula, when l = 0, h (0) That is, the initial input x, and f(·) is the activation function;

[0297] Let W be the weight matrix of the output layer. (L+1)The bias matrix is ​​b (L+1) Then the output y of the output layer is:

[0298] y = g(W (L+1)T h (L) +b (L+1) )

[0299] In the formula, g(·) is the activation function of the output layer;

[0300] By finding an ideal balance point, the model can effectively reveal complex patterns in the data, while avoiding the introduction of too many parameters to prevent overfitting.

[0301] Residual connections are used to optimize model output:

[0302] z (l) =W (l) α (l-1) +b (l)

[0303] a (l) =φ(z) (l) )+a( l-1)

[0304] In the formula, z (l) It is the weighted input of the l-th layer, a (l-1) It's the activation of the previous layer, W (l) and b (l) These are the weights and bias parameters, and φ is the activation function;

[0305] The advantage of using residual connections is that it effectively avoids two common problems encountered when deepening neural networks: vanishing gradients and representation bottlenecks. Vanishing gradients occur when the magnitude of the gradient decreases layer by layer during backpropagation, making it difficult for shallow layers in the neural network to be updated. Representation bottlenecks, on the other hand, refer to the limitation of the model's overall learning ability by the complexity of a particular layer.

[0306] When implementing residual connections, the input of a layer in a neural network is directly added to its output, and then this is used as the input to the next layer. This allows the model to better adapt to identity mappings when necessary, and even if a part of the model does not learn a useful transformation, it can still pass information through the identity path.

[0307] To enhance the fully connected layers, a batch normalization layer is introduced. This layer normalizes each batch of data, making the output mean close to 0 and the variance close to 1, thereby improving network stability during training. This method reduces the network's sensitivity to weight initialization choices and helps prevent the vanishing gradient problem.

[0308] Calculate the mean and variance of each batch of data:

[0309]

[0310] Standardize the input by subtracting the mean and dividing by the standard deviation:

[0311]

[0312] Introduce scaling factor γ and offset factor β:

[0313]

[0314] Embedding layers are introduced to process data with high-dimensional discrete features. An embedding layer maps each discrete feature to a low-dimensional real-valued vector space, ensuring that similar inputs are mapped to similar embedding vectors, thereby effectively capturing and utilizing the relationships between input features.

[0315] Suppose each hand keypoint sequence has M keypoints, and each keypoint is any one of k possible positions. Initialize the embedding matrix E for each possible keypoint position. keypoint Its size is K×D, where D is the dimension of the embedding vector.

[0316] For each hand keypoint sequence, find the corresponding row in the embedding matrix to obtain its embedding vector, and then concatenate the embedding vectors of all keypoints to obtain the embedding vector representing the entire keypoint sequence:

[0317] e i，m =E ktypoint [k i，nv :]

[0318]

[0319] In the formula, e i，m It is the embedding vector of the m-th keypoint in the i-th hand keypoint sequence, k i，m It is the location index of the key point, E keypoint It is the keypoint embedding matrix, e i It is the embedding vector of the entire i-th hand keypoint sequence.

[0320] Based on this method, the embedding of each keypoint position can be learned, and an embedding vector representing the overall keypoint sequence can be obtained, which can better capture the relationship between keypoints in the hand keypoint sequence.

[0321] A recurrent layer is added to process the sequential data. This recurrent layer preserves information from previous time steps, allowing the model to capture the time-series nature of the input data. The hidden state h... t It is based on the hidden state h of the previous moment. t-1and the current input x t The calculation yielded the following results:

[0322] h t =φ(W hh h t-1 +W xh x t +b h )

[0323] In the formula, W hh and W xh It is the weight matrix, b h φ is the bias term, and φ is the non-linear activation function. This structure allows the network to remember past information and use that information to influence the current output when necessary.

[0324] Experimental setup:

[0325] During training, 15 frames per second were extracted from independent sign language video samples (the experiments used representative CSL and PHOENIX14 datasets for analysis). The specific frame rate could be dynamically adjusted. Each frame underwent noise reduction and normalization, and was normalized to 224x224 pixels. The 2D coordinates of 21 hand keypoints were selected as features to capture key dynamic information of the gestures. The range of epochs was set, with a lower limit no less than one-fifth of the data volume and an upper limit no more than one-half of the data volume. The batch size was set to 32, the learning rate to 0.001, and the Adam optimizer was used. A weight decay strategy was implemented with a value of 0.0005 to prevent overfitting and improve the model's generalization ability. A learning rate decay strategy was also implemented, with an initial learning rate of 0.001, which decayed at a rate of 0.95 after each epoch. This strategy allows the model to learn quickly in the early stages of training, and then slows down the learning speed as it approaches the optimal solution, thus approximating the optimal solution more precisely.

[0326] To avoid the gradient explosion problem, gradient clipping is implemented with an upper limit of 1.0. When the gradient value exceeds this upper limit, it will be clipped to this value. This strategy ensures the stability of model training. An early stopping strategy is also introduced, stopping training if the loss on the validation set does not decrease for five consecutive epochs. This strategy prevents the model from overfitting the training data during training, thereby improving its performance on unseen data.

[0327] To increase the robustness of the model, data augmentation is performed on the training data, including random flipping, rotation, and scaling. This strategy increases the model's ability to recognize various gestures and improves the model's generalization performance.

[0328] 1. Comparative Experimental Analysis

[0329]

[0330] In the table above: CSL dataset: Chinese Sign Language dataset; PHOENIX14 dataset: RWTH-PHOENIX-Weather2014 (continuous sign language recognition dataset); WER is the Word Error Rate, used to quantify the degree of difference between the output result and the reference text during the recognition process. The lower the WER, the better the model's recognition performance. Detailed comparative experimental analysis shows that the two-layer bidirectional LSTM network model in this application demonstrates superior accuracy and recognition performance.

[0331] 2. Ablation test

[0332] BiLSTM 68.70％ BiLSTM_Prior 70.62％ BiLSTM_Prior_Attention 74.61％ BiLSTM_Prior_Attention_TTA 76.44％ Double_BiLSTM_Prior_Attention_TTA 87.82％

[0333] Based on the ablation experiment results, the effectiveness of different network model variables and their impact on accuracy can be compared:

[0334] BiLSTM is a basic bidirectional long short-term memory network; BiLSTM_Prior introduces priority replay sampling; BiLSTM_Prior_Attention introduces an attention mechanism on top of BiLSTM_Prior; BiLSTM_Prior_Attention_TTA introduces test-time augmentation on top of BiLSTM_Prior_Attention; Double_BiLSTM_Prior_Attention_TTA uses a two-layer BiLSTM structure on top of BiLSTM_Prior_Attention_TTA. The first BiLSTM layer accepts the input sequence and learns its hidden state representation, while the second BiLSTM layer takes the output of the first BiLSTM layer as input and further learns higher-level feature representations. This hierarchical structure allows the model to extract features at a higher level of abstraction to better capture information from the input sequence. Experiments show that Double_BiLSTM_Prior_Attention_TTA improves accuracy by 87.82% compared to BiLSTM_Prior_Attention_TTA.

[0335] In summary, the ablation experiments show that Prior Sampling, Attention mechanisms, Test Time Augmentation (TTA), and the Double-layer BiLSTM structure all have a positive impact on model performance. Prior Sampling and Attention mechanisms have a certain positive effect on model performance and can further improve it. Test Time Augmentation (TTA) can increase the model's robustness and generalization ability, thereby further improving accuracy. The Double-layer BiLSTM structure performs well in this task, significantly improving the model's accuracy.

[0336] 3. Cross-experiment

[0337]

[0338]

[0339] In the table above, accuracy is the proportion of samples correctly classified by the model in its predictions, used to evaluate the overall classification accuracy of the model; precision is the proportion of samples predicted as positive that are actually positive, used to measure the model's accuracy in predicting positive classes; recall is the proportion of samples that are actually positive that the model successfully predicts as positive, used to measure the model's recall rate for positive samples; and the F1 score is the harmonic mean of precision and recall, taking into account both the model's prediction accuracy and recall rate to comprehensively evaluate the model's performance.

[0340] Cross-tabulation analysis shows that the model exhibits relatively consistent performance across all tradeoffs: accuracy ranges from 82.85% to 85.50%, with an average of approximately 83.88%; precision ranges from 82.28% to 85.47%, with an average of approximately 83.95%; recall ranges from 82.49% to 84.96%, with an average of approximately 83.65%; and F1 score ranges from 0.8034 to 0.8364, with an average of approximately 0.8246. This indicates that the model has good robustness and is relatively insensitive to random changes in the training set.

[0341] The above specific embodiments should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, any alternative improvements or modifications made to the embodiments of the present invention shall fall within the scope of protection of the present invention.

[0342] Any aspects of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A dynamic gesture recognition method based on hand key points and a two-layer bidirectional LSTM network, characterized in that, Includes the following steps: S1. Obtain the hand key point dataset. The process involves acquiring sign language video data from a sign language video dataset and then sequentially initializing the Mediapipe human pose model, sampling the matrix with given axes, initializing the Holistic model, reading each frame of the video in a loop, performing keypoint detection and saving the detected keypoint information data to a list, drawing hand pose keypoints and connecting lines, converting the keypoint list into a NumPy array, and sampling the keypoint list to obtain the input sequence. S2, Input Layer of Generative Network Model The input sequence obtained in the previous step is subjected to time-flipped data augmentation to generate a flipped sequence. The input sequence and the flipped sequence are used as the input layer of the network model. S3. Construct a two-layer bidirectional LSTM network model for dynamic sign language recognition. The two-layer bidirectional LSTM network model consists of two sequentially designed bidirectional LSTM layers. Each bidirectional LSTM layer includes a forward LSTM layer and a backward LSTM layer. The input sequence passes through the forward LSTM layer to obtain the forward hidden state and cell state. The forward hidden state and cell state are passed and updated within the forward LSTM layer. The reversed sequence passes through the backward LSTM layer to obtain the reversed hidden state and cell state. The reversed hidden state and cell state are passed and updated within the backward LSTM layer. The input layer is processed by the two-layer bidirectional LSTM network model to extract the temporal features of the data and obtain the reversed output sequence. S4. Use an attention layer to process the flipped and reversed output sequence, calculate the attention-weighted outputs for the forward and reverse directions respectively, and concatenate the attention outputs for the forward and reverse directions. S5. Use a fully connected layer to map the attention-weighted output to the output dimension.

2. The dynamic gesture recognition method based on hand key points and a two-layer bidirectional LSTM network according to claim 1, characterized in that, In step S1, during the hand keypoint data collection phase, a capture object is instantiated to access and decode consecutive video frames. Hand tracking is implemented using the MediaPipe hand model, identifying and locating keypoints of the hand from the video frames. Changes in keypoint positions between consecutive frames are compared, and a NumPy set of keypoint coordinates from the dynamic gesture video stream is collected. Priority empirical playback sampling is performed based on the sparsity of the temporal data. The specific steps of priority empirical playback are as follows: S11, Calculation Error, calculated for each empirical case error: In the formula, In time step of error, In time step The actual reward received It is a discount factor used to calculate the current estimate of future rewards. It is in state Select action The probability of the strategy, It is a state-action value function that calculates the expected future reward given a state and action. Represents the network weights; S12, Assign priority, use Error is used to assign priority to each sample, while giving Add positive numbers to the error This ensures that every experience has the potential to be sampled. In the formula, It is a time step priority, Indicates time step of The absolute value of the error, a positive number This is to ensure that every experience has the potential to be sampled; S13, Calculate the sampling probability Based on experience, priorities and hyperparameters To calculate the probability that each experience is sampled: In the formula, In time step The probability that the experience is sampled. The priority of this experience. It is a hyperparameter between 0 and 1, used to control the degree of influence of priority; S14, Weight Update After sampling and learning using experience, importance sampling weights are calculated to adjust the experience in updates. The effect of the value: In the formula, It is the size of the experience pool. It is the probability that experience is sampled. It is a hyperparameter between 0 and 1: In the formula, It is a recessionary factor. It's the learning rate. It is the moving average of the squared gradient over the past period. It is about The gradient of the loss function.

3. The dynamic gesture recognition method based on hand key points and a two-layer bidirectional LSTM network according to claim 2, characterized in that, In step S1, during the keypoint detection process, pixel-level masks and deformation fluctuation transformations are introduced to guide the network's attention and track fine-grained changes in gestures. The continuous form expression of the pixel-level mask is: The discrete form expression of the pixel-level mask is: In the formula, For deformation vector field Energy; The definition of a region is a two-dimensional or three-dimensional space; and for The components, in two-dimensional space, represent exist and Change in direction; These four items indicate The gradient; A small element for integration represents a small region in space; This is the regularization parameter, which controls... The magnitude of the energy contribution; and These are the coordinates in an image or a discrete grid. , , , These four terms represent the discrete grid. Differences in various directions; and To represent the size of an image or discrete grid; The deformation wave transformation expression is: In the formula, For pixels Attention weights For the input image in The pixel values ​​are summed, and the summation operation in the denominator is performed on the entire image; , and Representing pixels The intensity of the red, green, and blue channels, and , and It represents the weight of the corresponding channel.

4. The dynamic gesture recognition method based on hand key points and a two-layer bidirectional LSTM network according to claim 1 or 3, characterized in that, In step S2, for the RGB channel time series containing high-dimensional features at each time step, time flip enhancement is used by combining time translation and time noise injection. Each RGB channel is treated as an independent sequence, and the sequence of each channel is flipped. The time shift refers to moving the entire time series forward or backward by a certain number of time units, reducing the model's dependence on the absolute time of events and focusing more on the relative order or pattern of events. Specific steps include: Suppose the obtained input data is a tensor , ,in It is the number of time steps. and It is the image height and width at each time step. Each element of the tensor represents one of the three color channels: RGB. Indicates time ,high ,width Color channels The pixel value; select a translation step. , tensor Move along the time axis Step, to obtain a new tensor For the new tensor Allocate the corresponding space, as shown below: Define a new tensor Each element , represented as: In the formula, Indicates on the timeline Perform translation. Ensuring that the index remains within a valid range can be represented by the following equation: For each and The above system of equations all hold true; In the formula, This indicates a modulo operation, ensuring the index remains within a valid range. The time noise injection involves adding random noise to the time series to simulate measurement errors or other forms of uncertainty that may occur in the real world, making the model more robust and capable of handling noisy data in practical applications. Specific steps include: Suppose the obtained input data is a tensor Noise injection is performed, for tensors. Add noise to each element to obtain a new tensor. : In the formula, This is the Gaussian noise generation function; Suppose there is a four-dimensional tensor ,in It is the number of time steps. and These are the height and width of the image, respectively. This involves the number of color channels, and a noise generation function also needs to be defined. The noise generated by this function follows a Gaussian distribution. Define a new tensor. Its elements Add the generated noise to the corresponding elements of the original tensor: In the formula, This indicates that the mean is 0 and the standard deviation is 0. Gaussian noise, this process is represented by the following system of equations: For each , and All of the above equations hold true.

5. The dynamic gesture recognition method based on hand key points and a two-layer bidirectional LSTM network according to claim 4, characterized in that, In step S3, the forward LSTM layer and the backward LSTM layer each include multiple LSTM units. Each LSTM unit includes an input gate, a forget gate, an output gate, a cell state, and a hidden state. The input gate is used to control the weights of the input data and the flow of information into the cell state. The forget gate is used to control the information that needs to be forgotten in the cell state. The output gate is used to control the flow of information from the cell state to the hidden state. The cell state is the memory part of the LSTM unit, responsible for storing and transmitting information. The hidden state is the output of the LSTM unit and is the input of the next time step.

6. The dynamic gesture recognition method based on hand key points and a two-layer bidirectional LSTM network according to claim 5, characterized in that, In step S3, the first bidirectional LSTM layer receives and processes the sign language key point sequence input. By simultaneously considering both positive and negative information in the data, LSTM captures the temporal characteristics of sign language by characterizing the dynamic changes in the keypoint sequence. In the formula, It is in a hidden state. It is the input key point. and It refers to the weights and biases of the learning process. It is an activation function; For each time step The first layer of the forward LSTM performs the following calculations: In the formula, , , , These represent the candidate states of the input gate, forget gate, output gate, and memory unit, respectively. and These represent the hidden state and the remembered state, respectively. This represents the weights between the input and the hidden state. This represents the weights between the input and the candidate memory units. Represents the bias vector; and These represent the sigmoid function and the hyperbolic tangent function, respectively; subscripts Indicates the first layer. Indicates a time step; LSTM adds a reverse information flow on top of this. The reverse LSTM calculation is the same as above, but the time step is reversed. Let the dimension of the hidden layer be `hidden_size`, then the calculation of the forward and reverse information is represented as follows: In the formula, and These represent forward and backward propagation, respectively. This is the weight matrix. , For bias terms; Since LSTM is bidirectional, its output dimension is 2. To enhance the model's expressive power, hidden_size is introduced, along with residual blocks and skip connections, for the output of each layer. Add a skip connection so that it propagates directly to the next layer, i.e.: In the formula, Presentation layer Function mapping, The weights of this layer are used to increase the receptive field of the network without increasing model parameters and computational complexity, enabling the model to capture long-distance dependencies in the input data. In choosing the activation function, parameterized ReLU is used, that is: In the formula, It is a learnable parameter that, by introducing more nonlinearity, enables the model to learn more complex mapping relationships. PReLU is more flexible than ReLU because it introduces a learnable parameter, allowing the network to adaptively adjust the shape of the activation function during training. To improve the training stability and speed of the model, batch normalization is used: In the formula, and They are respectively The mean and variance, It is a very small number to prevent division by zero. Batch standardization makes the input of each layer approximately zero mean and unit variance, which can accelerate the convergence of the model and also help prevent overfitting and improve the generalization ability of the model.

7. The dynamic gesture recognition method based on hand key points and a two-layer bidirectional LSTM network according to claim 6, characterized in that, In step S3, the second bidirectional LSTM layer receives the output of the first bidirectional LSTM layer and performs further processing to obtain richer dynamic features. The calculation process is as follows: In the formula, , , , These represent the candidate states of the input gate, forget gate, output gate, and memory unit, respectively. and These represent the hidden state and the remembered state, respectively. This represents the weights between the input and the hidden state. This represents the weights between the input and the candidate memory units. Represents the bias vector; and These represent the sigmoid function and the hyperbolic tangent function, respectively; subscripts Indicates the second layer. Indicates a time step; Its output dimension is 2. `hidden_size` is used because this layer also needs to consider both forward and backward information. To further improve model performance, an attention mechanism is introduced. This mechanism assigns different attention weights to different input elements when processing sequence data, thereby better capturing important information in the sequence. Scaled Dot-Product Attention is used. In the formula, These are query, key, and value, respectively. The dimension is the key; during the calculation, the dot product of the query and the key is first calculated, and then divided by... Scaling is performed, then via The function transforms the result into a probability distribution, and finally uses this distribution to perform a weighted summation of the values; Furthermore, a normalization operation is applied to the output of the second bidirectional LSTM layer, ensuring that each dimension of the output has the same scale, thereby improving the model's stability and accelerating training. In the formula, It is input. and These are the mean and variance of the batch, respectively. It is the normalized input. and These are learnable scaling factors and offset terms. It is a very small number to prevent division by zero.

8. The dynamic gesture recognition method based on hand key points and a two-layer bidirectional LSTM network according to claim 7, characterized in that, In step S4, the attention layer assigns weights to each element in the input, and the assigned weights reflect the importance of that element. The attention layer includes three sub-attention layers. The attention mechanism is used to weight the output of the second bidirectional LSTM layer, allowing the model to focus more on key gestures. The attention weight calculation formula is as follows: In the formula, These are unnormalized attention weights. These are normalized attention weights. It is a context vector. It is in a hidden state. , and These are the learning weights of the three sub-attention layers. and It is an activation function; The computation process of the three sub-attention layers is respectively by , and The calculation formula and data transfer process are as follows: Three linear layers are completed. S41. Calculate attention weights The output of the bidirectional LSTM is passed through and The attention weights are calculated from these two linear layers. The calculation process includes the following steps: use Activation function processing Output: use Activation function processing Output: Multiply the results of the above two steps element-wise to obtain the attention weights: S42. Calculate attention scores The attention weights calculated in the previous step are then passed through... The layer obtains the attention score, expressed as: S43. Calculate attention probability Application of attention scores A function is used to normalize all scores to the range of 0 to 1, with a sum of 1, thus yielding the attention probability at each position, as shown in the following formula: S44. Introducing an attention mechanism The attention probabilities obtained in the previous step are applied to the output of the LSTM to calculate the attention-weighted output. This process can be viewed as multiplying the output at each position by its corresponding attention probability, and then summing the results at all positions to obtain the final attention output. S45, Combine forward and backward attention outputs The forward and backward attention outputs are concatenated to obtain the final output: 。 9. The dynamic gesture recognition method based on hand key points and a two-layer bidirectional LSTM network according to claim 8, characterized in that, In step S5, the fully connected layer integrates the local feature information extracted by each of the aforementioned convolutional or pooling layers and fuses them into a global feature description. This global information not only contains the details of each individual feature but also embeds the relationships between features, injecting richer information into subsequent classification tasks. By introducing a multilayer perceptual feedforward mechanism in the hidden layer, the number of neurons can be dynamically adjusted according to the complexity of the task and the characteristics of the dataset. Let the input be ,in It is the number of neurons in the input layer; Hidden layer The weight matrix and bias vector are respectively and Then the first The output of each hidden layer for: In the formula, when hour, That is, the initial input , It is an activation function; Let the weight matrix of the output layer be... The bias matrix is Then the output of the output layer for: In the formula, It is the activation function of the output layer; By finding an ideal balance point, the model can effectively reveal complex patterns in the data, while avoiding the introduction of too many parameters to prevent overfitting. Residual connections are used to optimize model output: In the formula, It is the first Weighted input of the layer, It is the activation of the previous layer. and These are the weights and bias parameters. It is an activation function; The fully connected layers are enhanced by introducing a batch normalization layer. This layer normalizes each batch of data, making the average value of the output close to 0 and the variance close to 1, thereby enhancing the stability of the network during training. This method reduces the network's sensitivity to the choice of weight initialization and helps prevent the gradient vanishing problem. Calculate the mean and variance of each batch of data: Standardize the input by subtracting the mean and dividing by the standard deviation: Introducing scaling factor and offset factor : By introducing an embedding layer, data with high-dimensional discrete features can be processed. The embedding layer can map each discrete feature to a low-dimensional real vector space, so that similar inputs are mapped to close embedding vectors, thereby effectively capturing and utilizing the relationship between input features. Suppose that each hand key point sequence has There are 1 key points, each of which is: For any one of the possible keypoint locations, initialize the embedding matrix for each possible keypoint location. Its size is ,in It is the dimension of the embedded vector; For each hand keypoint sequence, find the corresponding row in the embedding matrix to obtain its embedding vector, and then concatenate the embedding vectors of all keypoints to obtain the embedding vector representing the entire keypoint sequence: In the formula, It is the first In the sequence of key hand points, the first... Embedding vectors of key points It is the location index of the key point. It is a keypoint embedding matrix. It is the entire first Embedded vectors of a sequence of key hand points; Adding a recurrent layer to process sequential data preserves information from previous time steps, allowing the model to capture the time-series nature of the input data and the hidden states. It is based on the hidden state of the previous moment. and current input The calculation yielded the following results: In the formula, and It is a weight matrix. It is a bias term. It is a non-linear activation function. This structure allows the network to remember past information and use that information to influence the current output.

Images