A dynamic gesture recognition method, apparatus, and device

Abstract

The application discloses a dynamic gesture recognition method and device, equipment and storage medium, which comprises the following steps: acquiring a depth image sequence and a three-dimensional hand skeleton sequence in a to-be-recognized gesture video; preprocessing the depth image sequence and the three-dimensional hand skeleton sequence to obtain a hand region set and a first skeleton sequence; sampling the hand region set and the first skeleton sequence through a preset index array to obtain a hand region sampling sequence and a skeleton sampling sequence; inputting the hand region sampling sequence and the skeleton sampling sequence into a pre-trained double-flow network model to perform dynamic gesture recognition and obtain a gesture recognition result; wherein the double-flow network model comprises a depth recurrent network branch and a convolution recurrent network branch. The application can save computing resources and effectively improve the precision of gesture recognition.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a dynamic gesture recognition method, apparatus, and device. Background Technology

[0002] Most computer vision-based gesture recognition tasks employ deep learning methods. Convolutional neural networks (CNNs) excel in image classification, and existing research largely uses images or skeletons of the human hand as input. Most recognition algorithms use color images captured by RGB cameras as raw data. The robustness of gesture recognition is significantly affected by changes in ambient lighting or complete darkness, and spatial information is inevitably ignored. Although deep learning methods have made great strides in gesture recognition, they suffer from excessive consumption of computer resources, redundant model parameters, and low recognition accuracy in specific scenarios. Most studies use only a single feature as input, ignoring spatiotemporal information. Furthermore, due to the inherent characteristics of the hand, such as self-occlusion and complex structure, the application prospects and recognition of dynamic gestures in real life remain highly challenging. Summary of the Invention

[0003] In view of this, the purpose of this invention is to propose a dynamic gesture recognition method, apparatus, and device, which aims to solve the problems of low recognition accuracy and limited application scenarios of existing dynamic gesture recognition methods.

[0004] To achieve the above objectives, the present invention provides a dynamic gesture recognition method, the method comprising:

[0005] Obtain depth image sequences and 3D hand skeleton sequences from the video of the gesture to be recognized;

[0006] The depth image sequence and the three-dimensional hand skeleton sequence are preprocessed to obtain a set of hand regions and a first skeleton sequence;

[0007] The hand region set and the first skeleton sequence are sampled using a preset index array to obtain a hand region sampling sequence and a skeleton sampling sequence.

[0008] The hand region sampling sequence and the skeleton sampling sequence are input into a pre-trained two-stream network model for dynamic gesture recognition to obtain gesture recognition results; wherein, the two-stream network model includes a deep recurrent network branch and a convolutional recurrent network branch.

[0009] Preferably, the step of acquiring the depth image sequence and the three-dimensional hand skeleton sequence from the video of the gesture to be recognized includes:

[0010] The depth image sequence and the three-dimensional hand skeleton sequence are obtained using the depth sensor of a structured light depth camera and a gesture tracking algorithm.

[0011] Preferably, the preprocessing of the depth image sequence and the three-dimensional hand skeleton sequence to obtain a hand region set and a first skeleton sequence includes:

[0012] The hand region of each depth image in the depth image sequence is extracted and quantized to obtain the set of hand regions;

[0013] The three-dimensional hand skeleton sequence is normalized to obtain the first skeleton sequence. The normalization process includes subtracting the coordinates of the second joint point of the first frame sequence from the coordinate system of each frame sequence of the three-dimensional hand skeleton sequence.

[0014] Preferably, the step of inputting the hand region sampling sequence and the skeleton sampling sequence into a pre-trained two-stream network model for dynamic gesture recognition to obtain gesture recognition results includes:

[0015] The spatial and temporal features of the skeleton sampling sequence are extracted through the deep recurrent network branches to obtain the joint feature map;

[0016] Multi-scale features of the hand region sampling sequence are extracted through the branches of the convolutional recurrent network to obtain a multi-scale feature map;

[0017] The hand gesture recognition result is obtained by fusing the joint feature map and the multi-scale feature map.

[0018] Preferably, the network structure of the deep recurrent network branch includes multiple stacked first networks, a spatiotemporal multi-head attention mechanism module, and a bidirectional long short-term memory network, wherein the first network includes a fully connected layer, a ReLU activation function, a batch normalization module, and a dropout module.

[0019] Preferably, the network structure of the convolutional recurrent network branch includes multiple stacked multi-scale feature extraction modules and a bidirectional long short-term memory network, wherein the multi-scale feature extraction module includes a depthwise separable convolution module, an activation function, a batch normalization module, and an Inception module.

[0020] Preferably, the two-stream network model is optimized during training using the Adam adaptive moment estimator and a preset loss function; wherein, the preset loss function includes the cross-entropy loss function L. ce and regularization loss function L reg .

[0021] To achieve the above objectives, the present invention also provides a dynamic gesture recognition device, the device comprising:

[0022] The acquisition unit is used to acquire the depth image sequence and the three-dimensional hand skeleton sequence from the video of the gesture to be recognized.

[0023] A preprocessing unit is used to preprocess the depth image sequence and the three-dimensional hand skeleton sequence to obtain a set of hand regions and a first skeleton sequence;

[0024] A sampling unit is used to sample the hand region set and the first skeleton sequence using a preset index array to obtain a hand region sampling sequence and a skeleton sampling sequence.

[0025] The recognition unit is used to input the hand region sampling sequence and the skeleton sampling sequence into a pre-trained two-stream network model for dynamic gesture recognition and to obtain the gesture recognition result; wherein, the two-stream network model includes a deep recurrent network branch and a convolutional recurrent network branch.

[0026] To achieve the above objectives, the present invention also proposes a dynamic gesture recognition device, including a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a dynamic gesture recognition method as described in the above embodiments.

[0027] To achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a computer program that is executed by a processor to implement the steps of a dynamic gesture recognition method as described in the above embodiments.

[0028] Beneficial effects:

[0029] The above scheme achieves dynamic gesture recognition from depth image sequences and 3D hand skeleton sequences through the proposed dual-stream network model, including deep recurrent network branches and convolutional recurrent network branches. It can save computing resources, is highly robust, practical, and has good anti-interference ability, while effectively improving the accuracy of gesture recognition. Its application range is wide.

[0030] The above solution utilizes the depth sensor of a structured light depth camera and a gesture tracking algorithm to acquire depth image sequences and 3D hand skeleton sequences. The required equipment is simple and low-cost, and its application scenarios are wider. The structured light depth camera can quickly and accurately acquire depth images and 3D hand skeleton sequence information of the target from all directions, providing high depth measurement accuracy. It is not easily affected by changes in ambient light and can work normally in dark environments, which can improve the accuracy and robustness of gesture recognition.

[0031] The above scheme improves the performance and effectiveness of extracting spatial and temporal features of the skeleton sampling sequence through deep recurrent network branches, thereby enhancing the branch's expressive power, information interaction ability, and generalization ability in processing sequence data. It also improves the performance and effectiveness by extracting multi-scale features of the hand region sampling sequence through convolutional recurrent network branches, which can capture image details and semantic information at different scales. Through the advantages of multi-scale feature extraction, parameter efficiency, feature concatenation and sparse connection, and network scalability, the network performance can be effectively improved. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart illustrating a dynamic gesture recognition method according to an embodiment of the present invention.

[0034] Figure 2 This is a schematic diagram of the overall network structure of a two-stream network model provided in an embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of the network structure of a bidirectional long short-term memory network provided in an embodiment of the present invention.

[0036] Figure 4 This is a schematic diagram of the network structure of the Inception module provided in an embodiment of the present invention.

[0037] Figure 5 This is a schematic diagram of the structure of a dynamic gesture recognition device provided in an embodiment of the present invention.

[0038] The realization of the invention's objective, its functional characteristics, and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0041] The present invention will be described in detail below with reference to the embodiments.

[0042] Reference Figure 1 The diagram shown is a flowchart of a dynamic gesture recognition method provided in an embodiment of the present invention.

[0043] In this embodiment, the method includes:

[0044] S11, acquire the depth image sequence and the three-dimensional hand skeleton sequence from the video of the gesture to be recognized.

[0045] Step S11 further includes:

[0046] The depth image sequence and the three-dimensional hand skeleton sequence are obtained using the depth sensor of a structured light depth camera and a gesture tracking algorithm.

[0047] In this embodiment, depth image sequences and 3D hand skeleton sequences are acquired using the depth sensor of an Intel RealSense Depth camera and a gesture tracking algorithm. Structured light stereo vision typically projects specific coded light onto the surface of the object being measured, captures the modulation signal of the coded light on the object's surface using a camera, demodulates the signal to obtain the modulation signal related to depth information, and finally calibrates it to obtain the 3D shape of the object's surface. The structured light depth camera can acquire depth images in real time, providing high depth measurement accuracy, is not easily affected by changes in ambient light, and can operate normally in dark environments. The hand skeleton can clearly describe the hand's movement state; introducing it into gesture recognition can improve the accuracy and robustness of complex gesture recognition. The acquired depth images include the scene's 2D XY coordinate information and pixel depth values ​​reflecting the distance from the camera in the scene; the 3D hand skeleton sequence includes the 3D coordinates of 22 hand joints. The frame rate for capturing images using the structured light depth camera is set to 30 FPS, and the image size acquired by the structured light depth camera is 640x480 pixels. The Intel RealSense SDK is used to acquire depth images and hand joint coordinate information.

[0048] S12, preprocess the depth image sequence and the three-dimensional hand skeleton sequence to obtain a set of hand regions and a first skeleton sequence.

[0049] Step S12 further includes:

[0050] S12-1, Extract the hand region of each depth image in the depth image sequence and perform quantization processing to obtain the hand region set;

[0051] S12-2, normalize the three-dimensional hand skeleton sequence to obtain the first skeleton sequence, wherein the normalization process includes subtracting the coordinates of the second joint point of the first frame sequence from the coordinate system of each frame sequence of the three-dimensional hand skeleton sequence.

[0052] In this embodiment, a hand region is extracted by setting a depth threshold. Morphological operations are then performed on the extracted hand region to remove noise, and the depth value of the region of interest is obtained by combining the depth image with the morphological operation. Specifically, firstly, pixels with depth values ​​within the preset depth threshold are selected as candidate hand regions. Then, for the extracted candidate hand regions, small noise is removed by continuous erosion and dilation operations, while retaining the main structure of the hand region. Finally, the denoised hand region is subjected to a pixel-by-pixel bitwise AND operation with the depth image to obtain the depth value of the region of interest. The depth value is then quantized (grayscale transformation) to convert it into a discrete grayscale level with strong intermittent contrast, thereby enhancing the contrast of the hand region of interest and obtaining a hand region set. Each hand image in the hand region set is 96x96 pixels in size. Pixels outside the specified depth range are filtered out based on the input depth threshold and parameters, and the depth values ​​of the pixels that meet the conditions are mapped to the specified grayscale range.

[0053] By subtracting the coordinates of the second joint point (palm center) of the first frame sequence from the coordinate system of each frame sequence of the 3D hand skeleton sequence, palm center normalization is achieved. Through translation and rotation transformations with the palm center as the origin, differences in hand position and rotation are eliminated, ensuring that the 3D hand skeleton sequences in different hand positions and rotation states have a consistent reference coordinate system, thus enabling better differentiation of different hand gestures.

[0054] S13, the hand region set and the first skeleton sequence are sampled using a preset index array to obtain a hand region sampling sequence and a skeleton sampling sequence.

[0055] In this embodiment, sampling the hand region set and the first skeleton sequence is to discretize the continuous gesture signal into a series of static frames. Using sampled frames can avoid inputting too much invalid information, making it easier for the neural network to process and classify. That is, a sampled frame refers to the data obtained by sampling the hand region set and the first skeleton sequence obtained after the above preprocessing, wherein the number of sampled frames is 32 frames.

[0056] The step size between each hand region set and the starting frame of the first skeleton sequence is calculated, and then an index array of sampling samples (a sampled sample refers to a randomly selected portion of samples from the dataset used for subsequent training, validation, or testing of the model; the dataset refers to the data obtained after preprocessing the depth image sequence and the 3D hand skeleton sequence) is randomly generated according to the step size and rounded down. If the number of generated sample indices (a sample index is a unique identifier for each sample in the dataset, used to locate and access a specific sample in the dataset) is insufficient, additional sample indices are generated using random numbers to ensure that the final number of samples meets the requirements; the hand region set and the first skeleton sequence corresponding to the sampling frame are read through the index array.

[0057] Define a function `get_samples(start, end, T)->S`, where `start` represents the starting index, `end` represents the ending index, `T` represents the number of samples to be sampled, and `S` represents the index array. First, calculate the total index range, i.e., `N = end - start + 1`, representing the number of elements between the starting and ending values. Then, the value of `S` can be expressed as follows:

[0058]

[0059] In this context, [] represents a sequence, with elements separated by commas. [start,start+1,...,end] generates an arithmetic sequence of length N, from the starting value to the ending value (inclusive). [start,start+(end-start) / (T-1),...,end] generates an equally spaced sequence of length T, from the starting value to the ending value. Finally, S represents the index array returned by the function.

[0060] S14, the hand region sampling sequence and the skeleton sampling sequence are input into the pre-trained two-stream network model for dynamic gesture recognition to obtain the gesture recognition result; wherein, the two-stream network model includes a deep recurrent network branch and a convolutional recurrent network branch.

[0061] Step S14 further includes:

[0062] S14-1, The spatial and temporal features of the skeleton sampling sequence are extracted through the deep recurrent network branches to obtain the joint feature map;

[0063] S14-2, Extract multi-scale features of the hand region sampling sequence through the convolutional recurrent network branch to obtain a multi-scale feature map;

[0064] S14-3, The joint feature map and the multi-scale feature map are fused to obtain the gesture recognition result.

[0065] In this embodiment, the two-stream network model includes a deep recurrent network branch and a convolutional recurrent network branch. See also Figure 2 As shown, the left side represents the deep recurrent network branch. Its network structure consists of a first network composed of fully connected layers, ReLU activation function, batch normalization, and dropout, stacked three times in sequence. A spatiotemporal multi-head attention mechanism and a bidirectional long short-term memory network are then embedded on top of this. The right side represents the convolutional recurrent network branch. Its network structure includes multiple stacked multi-scale feature extraction modules for feature extraction and a bidirectional long short-term memory network for learning temporal information. The multi-scale feature extraction modules consist of deep separable convolutions, SiLU activation function, batch normalization, and an Inception module.

[0066] For deep recurrent network branches, introducing a spatiotemporal multi-head attention mechanism into a bidirectional long short-term memory (LSTM) network allows the network to focus on the most relevant information at each temporal and spatial location, thereby better capturing important patterns and features in sequence data. By weighted combining features from different temporal and spatial locations, the model can better adapt to different input data (skeleton sampling sequences) and better handle unseen samples. The introduced bidirectional LSTM network captures spatiotemporal relationships; it accepts variable-length sequence inputs and consists of forward and backward LSTM network layers. Figure 3 As shown, the input sequences are fed into the Long Short-Term Memory network in both forward and reverse order for feature extraction. The two output vectors (i.e., the extracted feature vectors) are concatenated to form the final feature representation, thereby improving the expressive power, information interaction ability, and generalization ability of the deep recurrent network branches in processing sequence data, and improving the performance and effectiveness of the model.

[0067] However, for convolutional recurrent network branches, depthwise separable convolutions can reduce the computational cost of the network, while the Inception module extracts and fuses feature maps of different scales from the input image data. Furthermore, the hand region sampling sequence first undergoes preliminary feature extraction through depthwise separable convolutional layers. Linear functions have limited ability to learn complex function mappings from features; therefore, the SiLU activation function introduces non-linearity, enabling this convolutional recurrent network branch to learn and fit complex data. Then, a batch normalization module stabilizes the feature distribution, accelerating model learning and simplifying parameter tuning. Based on this, the Inception module is introduced, such as... Figure 4The Inception module shown passes features in parallel to convolutional filters of different sizes, including 1x1, 3x3 and 5x5 convolutions and 1x1 max pooling layers, fusing feature maps of different scales to represent the complex structure of dynamic gestures, and introducing a bidirectional long short-term memory network to capture temporal information, thereby obtaining the global structure and local information of the input data.

[0068] Specifically, the skeleton sampling sequence is used to construct a fully connected skeleton graph. Nodes in the fully connected skeleton graph represent hand joints, and edges include spatial edges connecting different joints and temporal edges connecting the same joint. Node features are input into a deep recurrent network branch based on a spatiotemporal attention mechanism for encoding and information updating, thus completing the node feature update. The node feature is a three-dimensional tensor (time_len, joint_num, 3), where the first dimension represents the number of sample frames in the time series, the second dimension represents the number of hand joints, and the third dimension represents the coordinate dimension of each joint. This method uses spatial and temporal multi-head attention mechanisms to extract spatial and temporal features from the hand skeleton graph (skeleton sampling sequence), respectively.

[0069] The principle behind spatial multi-head attention mechanisms is to model the dependencies between different spatial locations in a sequence. Specifically, the sequence is split according to its spatial dimension, with each sub-tensor corresponding to a spatial location. Then, in each attention head, the correlation between different spatial locations is captured by calculating the attention weights between the query, key, and value corresponding to each spatial location. Similarly, the principle behind temporal multi-head attention is to model the dependencies between different time steps in a sequence. Mathematically, the calculation formulas for spatial and temporal multi-head attention mechanisms can be expressed as:

[0070]

[0071] in, d represents the linear transformation parameters for query, key, and value used in multi-head attention, respectively; when i = s, it represents spatial multi-head attention, and when i = t, it represents temporal multi-head attention. k This indicates the dimensions of query and key in attention.

[0072] Based on the set time step and number of joints, a temporal domain mask and a structural domain mask are created. The weights in the learned spatial and temporal edges are used to analyze the influence of each node on its neighboring nodes. This limits the scope of the attention mechanism, allowing the model to focus on the spatial or temporal domain. This helps the model better understand the connection methods and related information between nodes, and avoids the model relying on future information during prediction, which could lead to inaccurate results.

[0073] By inputting the hand region sampling sequence into a convolutional recurrent network branch and using depthwise separable convolution kernels instead of traditional convolutions, the number of parameters is reduced, resource utilization is improved, and computational cost is reduced. The Inception module can capture image details and semantic information at different scales. Through advantages such as multi-scale feature extraction, parameter efficiency, feature concatenation and sparse connections, and network scalability, the network performance can be effectively improved. A bidirectional long short-term memory network can learn the temporal characteristics of the hand region sampling sequence.

[0074] Depthwise separable convolution consists of two parts: depthwise convolution and pointwise convolution. The overall operation can be understood as first extracting spatial features from the input using depthwise convolution, and then transforming the channels using pointwise convolution. Depthwise convolution applies a convolution kernel to each channel of the input, generating the same number of channels, and then stacks these channels according to their channel dimensions. It only focuses on the spatial dimension of the input and does not consider information between channels. For an input feature map X, the depthwise convolution of the output feature map Y can be represented as:

[0075]

[0076] Where i,j are the spatial locations of the output feature map, k is the channel index, m,n are the spatial locations of the convolution kernel, and K m,n These are the weights of the convolution kernel.

[0077] Pointwise convolution applies a 1x1 kernel to convolve across different channels, mixing information from different channels. It does not change the spatial dimensions of the feature map, only the number of channels. For an input feature map X, the pointwise convolution of the output feature map Y can be represented as:

[0078]

[0079] Where i,j are the spatial locations of the output feature map, k' is the channel index after pointwise convolution, k is the channel index before pointwise convolution, and K... k,k′ These are the weights of the convolution kernel.

[0080] After obtaining the feature representations of the depth image and the 3D hand skeleton sequence, the two-stream network model connects them together to form a multimodal feature vector, and then maps the multimodal feature vector to the target variable space through a multilayer perceptron to achieve dynamic gesture recognition.

[0081] During the training of the two-stream network model, PyTorch was used as the training framework, with the training and validation sets split in a 7:3 ratio. The model was trained for 15 epochs using the Adam Adaptive Moment Estimation (Adam) optimizer and cross-entropy loss, with a learning rate of 0.0003, a batch size of 128, and L2 regularization to prevent overfitting. The Adam optimizer's task is to calculate the gradient of the loss function in each iteration and then update the parameters. Its formula is as follows:

[0082]

[0083] In the formula, the gradient of the first term g t Represents the loss function For θ t Find the partial derivative; the second term m t For time t, the first moment estimate of the gradient in momentum form; the third term v t This is the second moment estimate of the gradient in momentum form; the fourth term This is the first-order moment estimate after bias correction, where, It is beta 1 raised to the power of t, and the same applies below; the fifth term The term is the second-order moment estimate after bias correction; the last term is the update formula.

[0084] Cross-entropy loss is a loss function that measures the difference between the predicted distribution and the true distribution. Minimizing cross-entropy loss optimizes the model, enabling it to perform classification tasks more accurately. When the model's predictions perfectly match the true results, the cross-entropy loss is zero. As the difference between the two increases, the cross-entropy loss increases, indicating that the model's predictions are inaccurate.

[0085] The calculation formula is as follows:

[0086]

[0087] Among them, y i p represents the i-th element of the real label. i This represents the probability of the i-th category predicted by the model.

[0088] L2 regularization is a commonly used regularization method to control model complexity and prevent overfitting. Its principle is based on introducing the sum of squares of the model parameters as a regularization term into the loss function. Its calculation formula is as follows:

[0089]

[0090] Among them, w i Let represent the i-th parameter of the model, and λ represent the regularization parameter, used to control the strength of regularization. This formula is a weighted sum of the squares of all parameters; larger parameters are penalized more, thus reducing model complexity.

[0091] Adding the regularization term to the loss function yields the following total loss function:

[0092] L total =L ce +L reg

[0093] Among them, L total L represents the data loss, which is the difference between the model's predictions on the training data and the true labels. reg This represents the regularization term.

[0094] Reference Figure 5 The diagram shown is a structural schematic of a dynamic gesture recognition device provided in an embodiment of the present invention.

[0095] In this embodiment, the device 20 includes:

[0096] Acquisition unit 21 is used to acquire depth image sequence and three-dimensional hand skeleton sequence in the video of the gesture to be recognized;

[0097] Preprocessing unit 22 is used to preprocess the depth image sequence and the three-dimensional hand skeleton sequence to obtain a set of hand regions and a first skeleton sequence;

[0098] The sampling unit 23 is used to sample the hand region set and the first skeleton sequence through a preset index array to obtain a hand region sampling sequence and a skeleton sampling sequence.

[0099] The recognition unit 24 is used to input the hand region sampling sequence and the skeleton sampling sequence into a pre-trained two-stream network model for dynamic gesture recognition and to obtain the gesture recognition result; wherein, the two-stream network model includes a deep recurrent network branch and a convolutional recurrent network branch.

[0100] Each unit module of the device 20 can execute the corresponding steps in the above method embodiment, so the details of each unit module will not be elaborated here. Please refer to the description of the corresponding steps above for details.

[0101] This invention also provides a dynamic gesture recognition device, which includes the dynamic gesture recognition apparatus as described above, wherein the dynamic gesture recognition apparatus can employ... Figure 5 The structure of the embodiment, correspondingly, can be executed Figure 1The technical solutions of the method embodiments shown are similar in implementation principle and technical effect. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0102] The device includes: a mobile phone, digital camera, or tablet computer, or other device with a camera function; or a device with an image processing function; or a device with an image display function. The device may include components such as a memory, processor, input unit, display unit, and power supply.

[0103] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function (such as image playback function), etc.; the data storage area can store data created according to the use of the device. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide access to the memory for the processor and input units.

[0104] The input unit can be used to receive input numerical, character, or image information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in addition to a camera, the input unit of this embodiment may also include a touch-sensitive surface (e.g., a touch screen) and other input devices.

[0105] The display unit can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The display unit may include a display panel, optionally configured as an LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar display panel. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to the processor to determine the type of touch event. Subsequently, the processor provides corresponding visual output on the display panel based on the type of touch event.

[0106] This invention also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement... Figure 1 The dynamic gesture recognition method shown is illustrated. The computer-readable storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0107] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the device embodiments, equipment embodiments, and storage medium embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions in the method embodiments.

[0108] Furthermore, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0109] The foregoing description illustrates and describes preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept by means of the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A dynamic gesture recognition method, characterized in that, The method includes: Obtain depth image sequences and 3D hand skeleton sequences from the video of the gesture to be recognized; The depth image sequence and the three-dimensional hand skeleton sequence are preprocessed to obtain a set of hand regions and a first skeleton sequence; The hand region set and the first skeleton sequence are sampled using a preset index array to obtain a hand region sampling sequence and a skeleton sampling sequence. The hand region sampling sequence and the skeleton sampling sequence are input into a pre-trained two-stream network model for dynamic gesture recognition to obtain gesture recognition results. The two-stream network model includes a deep recurrent network branch and a convolutional recurrent network branch. The deep recurrent network branch's network structure includes multiple stacked first networks, a spatiotemporal multi-head attention mechanism module, and a bidirectional long short-term memory network. The first network includes fully connected layers, a ReLU activation function, a batch normalization module, and a dropout module. The convolutional recurrent network branch's network structure includes multiple stacked multi-scale feature extraction modules and a bidirectional long short-term memory network. The multi-scale feature extraction module includes a depthwise separable convolution module, an activation function, a batch normalization module, and an Inception module. The step of inputting the hand region sampling sequence and the skeleton sampling sequence into a pre-trained two-stream network model for dynamic gesture recognition to obtain gesture recognition results includes: The spatial and temporal features of the skeleton sampling sequence are extracted through the deep recurrent network branches to obtain the joint feature map; Multi-scale features of the hand region sampling sequence are extracted through the branches of the convolutional recurrent network to obtain a multi-scale feature map; The hand gesture recognition result is obtained by fusing the joint feature map and the multi-scale feature map. The skeleton sampling sequence is used to construct a fully connected skeleton graph. The nodes of the fully connected skeleton graph represent hand joints, and the edges of the fully connected skeleton graph include spatial edges connecting different joints and temporal edges connecting the same joint. A temporal domain mask and a structural domain mask are created based on the set time step and the number of joints. The weights in the learned spatial and temporal edges are used to analyze the influence of each node on its neighboring nodes, so as to limit the scope of action in the spatiotemporal multi-head attention mechanism module.

2. The dynamic gesture recognition method according to claim 1, characterized in that, The acquisition of the depth image sequence and the three-dimensional hand skeleton sequence from the video of the gesture to be recognized includes: The depth image sequence and the three-dimensional hand skeleton sequence are obtained using the depth sensor of a structured light depth camera and a gesture tracking algorithm.

3. The dynamic gesture recognition method according to claim 1, characterized in that, The preprocessing of the depth image sequence and the three-dimensional hand skeleton sequence to obtain a set of hand regions and a first skeleton sequence includes: The hand region of each depth image in the depth image sequence is extracted and quantized to obtain the set of hand regions; The three-dimensional hand skeleton sequence is normalized to obtain the first skeleton sequence. The normalization process includes subtracting the coordinates of the second joint point of the first frame sequence from the coordinate system of each frame sequence of the three-dimensional hand skeleton sequence.

4. The dynamic gesture recognition method according to claim 1, characterized in that, During training, the dual-stream network model is optimized using the Adam adaptive moment estimator and a preset loss function; wherein the preset loss function includes the cross-entropy loss function. and regularization loss function .

5. A dynamic gesture recognition device, characterized in that, Using a dynamic gesture recognition method as described in any one of claims 1-4, the apparatus comprises: The acquisition unit is used to acquire the depth image sequence and the three-dimensional hand skeleton sequence from the video of the gesture to be recognized. A preprocessing unit is used to preprocess the depth image sequence and the three-dimensional hand skeleton sequence to obtain a set of hand regions and a first skeleton sequence; A sampling unit is used to sample the hand region set and the first skeleton sequence using a preset index array to obtain a hand region sampling sequence and a skeleton sampling sequence. The recognition unit is used to input the hand region sampling sequence and the skeleton sampling sequence into a pre-trained two-stream network model for dynamic gesture recognition and to obtain the gesture recognition result; wherein, the two-stream network model includes a deep recurrent network branch and a convolutional recurrent network branch.

6. A dynamic gesture recognition device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory, the computer program being executed by the processor to implement the steps of a dynamic gesture recognition method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the steps of a dynamic gesture recognition method as described in any one of claims 1 to 4.

Images