A dynamic gesture recognition method, device, equipment and storage medium

Abstract

The application discloses a dynamic gesture recognition method and device, equipment and storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a to-be-recognized video containing a dynamic gesture, performing key frame extraction on the to-be-recognized video to obtain a target key frame; inputting the target key frame into a target double-path network to obtain first feature information output by a slow path in the target double-path network and second feature information output by a fast path in the target double-path network; wherein a convolution layer of the slow path is constructed based on cascaded small convolution kernels, and a convolution layer of the fast path is constructed based on a 3D deep separable convolution; and performing feature weight distribution and feature fusion on the first feature information and the second feature information by using an attention module to obtain fused features, so that a classifier performs gesture type recognition according to the fused features. The method can reduce the calculation cost and improve the recognition speed and accuracy.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a dynamic gesture recognition method, apparatus, device, and storage medium. Background Technology

[0002] Currently, human-computer interaction (HCI) technology has evolved into a key factor driving the development of many fields. Dynamic gesture recognition, as a highly promising research direction, is gradually changing the way people interact with electronic devices, and its importance is increasingly prominent. Traditional HCI methods, such as keyboards, mice, and touchscreens, can no longer meet users' demands for immersive experiences and ease of operation in certain scenarios. Dynamic gesture recognition technology has emerged to address this need, allowing users to communicate with devices through natural hand movements, greatly improving the intuitiveness and fluency of interaction. Dynamic gesture recognition has a wide range of applications, including VR / AR, smart home systems, autonomous driving, and the medical field.

[0003] Traditional dynamic gesture recognition relies on conventional computer vision techniques, such as skin color-based segmentation, contour extraction, and feature point tracking. While the algorithms are relatively simple, they suffer from poor robustness in complex environments, leading to false positives and false negatives, and the expressive power of gestures is limited. To address these issues, existing technologies propose deep learning-based dynamic gesture recognition, such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and their variants. By learning from large amounts of labeled gesture data, these technologies can automatically extract high-level features of hand movements, achieving more accurate and robust gesture recognition. However, the large number of parameters in these models makes them unsuitable for deployment on resource-constrained edge devices (such as mobile and embedded devices), hindering real-time performance and low power consumption. Furthermore, when processing rapidly changing gestures, information loss or delays may occur, affecting the accuracy and real-time performance of the recognition. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a dynamic gesture recognition method, apparatus, device, and storage medium, which can reduce computing costs and improve recognition speed and accuracy. The specific solution is as follows:

[0005] In a first aspect, this application discloses a dynamic gesture recognition method, including:

[0006] A video containing dynamic gestures is acquired to be recognized, and keyframes are extracted from the video to be recognized to obtain target keyframes.

[0007] The target keyframe is input into the target dual-path network to obtain the first feature information output by the slow channel in the target dual-path network and the second feature information output by the fast channel in the target dual-path network; wherein the convolutional layer of the slow channel is constructed based on cascaded small convolutional kernels and the convolutional layer of the fast channel is constructed based on 3D depth-separable convolution.

[0008] The attention module is used to assign feature weights and fuse features to the first feature information and the second feature information to obtain fused features, so that the classifier can use the fused features to recognize the gesture type.

[0009] Optionally, the step of extracting target keyframes from the video to be identified includes:

[0010] Calculate the image entropy of each video frame in the video to be identified, and determine local extreme points from the video to be identified based on all the image entropies; the local extreme points include local maxima and local minima;

[0011] Calculate the local density of each local extremum point, and determine the minimum distance between each local extremum point and the other local extremum points based on the local density;

[0012] Cluster centers are determined based on the minimum distance and the first preset target number, and a target keyframe is determined for each cluster center.

[0013] Optionally, the step of inputting the target keyframe into the target dual-path network to obtain the first feature information output by the slow channel in the target dual-path network and the second feature information output by the fast channel in the target dual-path network includes:

[0014] The target keyframe is input into the slow channel; the slow channel is constructed in the order of data layer, convolutional layer, pooling layer and residual layer;

[0015] Using the data layer of the slow channel, a second preset number of target keyframes are selected from the target keyframes as the input of the convolutional layer of the slow channel, and the first feature information of the target keyframes is obtained based on the output of the residual layer of the slow channel.

[0016] The target keyframe is input into the fast channel, and the second feature information of the target keyframe is obtained based on the output of the fast channel; the fast channel is constructed in the order of convolutional layer, pooling layer and residual layer.

[0017] Optionally, the convolutional layer of the slow channel is constructed based on multiple cascaded 3×3 convolutional kernels.

[0018] Optionally, the 3D depthwise separable convolution includes a 3D depthwise convolutional layer and a 3D pointwise convolutional layer;

[0019] The 3D deep convolutional layer is used to perform independent convolution on each channel of the input feature map to obtain an intermediate feature map with the same number of channels.

[0020] The 3D pointwise convolutional layer is used to aggregate channel information of the intermediate feature map.

[0021] Optionally, the attention module is constructed by cascading a channel attention submodule and a spatial depth attention submodule.

[0022] Optionally, the step of using the attention module to assign feature weights and fuse features between the first feature information and the second feature information includes:

[0023] The channel attention submodule is used to assign weights to the spatial features of the first feature information and the second feature information;

[0024] The spatial depth attention submodule is used to assign weights to the temporal features of the first feature information and the second feature information.

[0025] Secondly, this application discloses a dynamic gesture recognition device, comprising:

[0026] The video acquisition module is used to acquire a video to be recognized containing dynamic gestures, and to extract keyframes from the video to be recognized to obtain target keyframes.

[0027] The feature extraction module is used to input the target keyframe into the target dual-path network to obtain the first feature information output by the slow channel in the target dual-path network and the second feature information output by the fast channel in the target dual-path network; wherein the convolutional layer of the slow channel is constructed based on cascaded small convolutional kernels and the convolutional layer of the fast channel is constructed based on 3D depth-separable convolution.

[0028] The feature fusion module is used to perform feature weight allocation and feature fusion on the first feature information and the second feature information using the attention module to obtain fused features, so that the classifier can use the fused features to perform gesture type recognition.

[0029] Thirdly, this application discloses an electronic device, including:

[0030] Memory, used to store computer programs;

[0031] A processor is used to execute the computer program to implement the aforementioned dynamic gesture recognition method.

[0032] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned dynamic gesture recognition method.

[0033] In this application, a video containing dynamic gestures is acquired, and keyframes are extracted from the video to obtain target keyframes. The target keyframes are then input into a target dual-path network to obtain first feature information output by the slow channel and second feature information output by the fast channel. The convolutional layer of the slow channel is constructed based on cascaded small convolutional kernels, and the convolutional layer of the fast channel is constructed based on 3D depth-separable convolution. An attention module is used to assign feature weights and fuse the first and second feature information to obtain fused features, which are then used by a classifier to recognize gesture types. It is evident that extracting keyframes and performing recognition based on them can reduce useless and redundant data during model training, thereby improving the model's generalization ability and recognition accuracy. Video-based recognition can avoid information loss or delay, efficiently process long sequences of gesture data, and improve recognition accuracy. Using cascaded small convolutional kernels in the slow channel can increase the network's receptive field while reducing the number of model parameters, capturing data details and high-level features. Using 3D depthwise separable convolutions in the fast channel can reduce computational costs and the number of model parameters, thereby improving recognition speed. Utilizing attention modules for feature weight allocation and feature fusion enhances feature effectiveness, improves model adaptability, and increases the accuracy of inference and recognition. This addresses the problems of existing algorithms, such as large model parameter counts, difficulty in deploying on edge devices, and poor real-time recognition performance. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0035] Figure 1 A flowchart of a dynamic gesture recognition method provided in this application;

[0036] Figure 2 A specific keyframe extraction diagram is provided for this application;

[0037] Figure 3 A flowchart of a specific 3D depthwise separable convolution method is provided in this application;

[0038] Figure 4A schematic diagram illustrating a specific feature fusion method provided in this application;

[0039] Figure 5 This application provides a schematic diagram of the structure of a dynamic gesture recognition system;

[0040] Figure 6 This application provides a schematic diagram of the structure of a dynamic gesture recognition device;

[0041] Figure 7 This application provides a structural diagram of an electronic device. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] In existing technologies, dynamic gesture recognition based on deep learning techniques, such as convolutional neural networks, recurrent neural networks, and their variants, can automatically extract high-level features of hand movements by learning from a large amount of labeled gesture data, thereby achieving more accurate and robust gesture recognition. However, these models have a large number of parameters, making them unsuitable for deployment on resource-constrained edge devices and hindering the achievement of real-time and low-power requirements. Furthermore, when processing rapidly changing gestures, information loss or delays may occur, affecting the accuracy and real-time performance of the recognition. To overcome these technical problems, this application proposes a dynamic gesture recognition method that can reduce computational costs and improve recognition speed and accuracy.

[0044] This application discloses a dynamic gesture recognition method. See also Figure 1 As shown, the method may include the following steps:

[0045] Step S11: Obtain the video to be recognized containing dynamic gestures, and extract the target keyframes from the video to be recognized.

[0046] In this embodiment, the video to be identified containing dynamic gestures is the original video, which cannot be directly used as input to the neural network. Each video segment needs to be decomposed into consecutive video frames for processing. However, in general, the target action in the video only accounts for a small part of the original video. At the same time, the moving target may also be obscured and interfered with by a lot of background information, generating a lot of invalid or redundant information, which has a negative effect on the training of the network model. Therefore, redundant information is removed by extracting key frames.

[0047] In some embodiments, the step of extracting target keyframes from the video to be identified may include: calculating the image entropy of each video frame in the video to be identified, determining local extrema points from the video to be identified based on all the image entropies; the local extrema points include local maxima points and local minima points; calculating the local density of each local extrema point, determining the minimum distance between each local extrema point and the remaining local extrema points based on the local density; determining cluster centers based on the minimum distance and a first preset target number, and determining a target keyframe for each cluster center.

[0048] Specifically, this involves extracting target keyframes from a video using image entropy and density clustering methods. For example, in a video V containing n consecutive video frames Fframes, the target keyframe Skeyframes can be represented as: Skeyframes = Fframes(V); the keyframe extraction steps are as follows:

[0049] S101: Calculate the image entropy of each frame in the video: Where f i This represents a video frame, where i is the index of the video frame, and p... fi (j) represents image frame f i The probability density function is obtained by analyzing the image frame f. i E(f) is obtained by normalizing the grayscale pixel histogram, where j represents the index of the grayscale value, i.e., the different grayscale levels in the image, with a grayscale value range of 0-255. i ) represents image frame f i The entropy value.

[0050] S102: E(f) i Mapping to a two-dimensional coordinate space, using the formula:

[0051] Calculate the local maxima and local minima in the two-dimensional coordinate space. The set of local maxima and local minima of all video frames in the two-dimensional coordinate space is called Pextreme; Pextreme = Pmax ∪ Pmin.

[0052] S103: Calculate the local density ρ for each extremum point in Pextreme, and based on ρ, calculate the minimum distance δ between that extremum point and other points. Assume there are N extremum points, where P... k For an extreme point, P i (i = 1, 2, ..., N-1) is the number of elements except p. k Other extreme points besides dp k p i Let be the distance between two extreme points, and dc be the threshold distance. The formulas for calculating local density and minimum distance are as follows:

[0053]

[0054] That is, for each extreme point P i Calculate its relationship with the extreme point P. k distance dp k p i By examining all P i The extreme point P is obtained by calculating and summing the local densities. k density value Filter out The point set, and then calculate the relationship between these points and P. k The distance is taken as the minimum value. To some extent, it reflects the extreme point P k The degree of proximity to denser areas is represented by the minimum distance.

[0055] S104: Select N maximum values ​​δ as cluster centers and calculate keyframes. For example, N=16, i.e., frame 06, the target keyframe. Figure 2 The image shown is a video segment to be identified, with the selected frame representing the target keyframe.

[0056] Step S12: Input the target keyframe into the target dual-path network to obtain the first feature information output by the slow channel in the target dual-path network and the second feature information output by the fast channel in the target dual-path network; wherein, the convolutional layer of the slow channel is constructed based on cascaded small convolutional kernels, and the convolutional layer of the fast channel is constructed based on 3D depth-separable convolution.

[0057] In this embodiment, a dual-path network architecture is used as the main framework, and targeted improvements are made to better adapt to dynamic gesture recognition tasks. The slow channel captures spatial semantic information in the video at a lower frame rate, focusing primarily on overall scene information and slowly changing information. It uses a larger number of channels to learn high-level, long-term spatiotemporal information. The fast channel samples the video at a higher frame rate, capturing details and rapidly changing information. It uses fewer channels and focuses on local details of fast-moving objects and short-term spatiotemporal information. Existing dual-path network architectures use large convolutional kernels in the slow channel, increasing the number of model parameters. This application replaces large convolutional kernels with cascaded small convolutional kernels, reducing the number of model parameters while increasing the network's receptive field to capture data details and high-level features. Existing dual-path network architectures use C3D convolutions in the fast channel, resulting in a slow inference process that cannot meet real-time recognition requirements and is unsuitable for edge devices. This application improves inference speed by using 3D depthwise separable convolutions.

[0058] In this embodiment, the step of inputting the target keyframes into the target dual-path network to obtain the first feature information output by the slow channel within the target dual-path network includes: inputting the target keyframes into the slow channel; the slow channel is constructed in the order of data layer, convolutional layer, pooling layer, and residual layer; using the data layer of the slow channel, selecting a second preset number of target keyframes from the target keyframes as the input to the convolutional layer of the slow channel, and obtaining the first feature information of the target keyframes based on the output of the residual layer of the slow channel; wherein, the convolutional layer of the slow channel is constructed based on multiple cascaded 3×3 convolutional kernels. For example, if the stride of the data layer of the slow channel (SlowPathway Network) is 4×1×1, and the first dimension is set to 4, then if 16 keyframes are input, 4 frames are selected after passing through the data layer as the actual input of the slow channel, and the last two dimensions of 1×1 are used to ensure that the size of each input frame image remains unchanged. The network receives four video frames as input, selected after grayscale conversion and keyframe extraction. These frames are preprocessed to provide the network with a relatively concise and representative image sequence. Convolutional layers are constructed using cascaded small convolutional kernels. Compared to traditional large convolutional kernels, cascading small kernels not only reduces the number of model parameters but also increases the network's receptive field, more effectively capturing the spatial semantic information of gestures and some slowly changing details. Specifically, multiple 3×3 small convolutional kernels can be cascaded to replace larger 5×5 convolutional kernels, reducing parameters while enabling more detailed learning of local features in the image. The slow channel uses a larger number of channels to fully learn high-level, long-term spatiotemporal information. More channels also allow the network to extract gesture features from different dimensions, such as different color channels and texture channels, thus better grasping the overall shape and changing trends of the gesture.

[0059] In this embodiment, inputting the target keyframe into the target dual-path network to obtain the second feature information output by the fast channel within the target dual-path network includes: inputting the target keyframe into the fast channel, and obtaining the second feature information of the target keyframe based on the output of the fast channel; the fast channel is constructed in the order of convolutional layers, pooling layers, and residual layers. The 3D depthwise separable convolution includes 3D depthwise convolutional layers and 3D pointwise convolutional layers; the 3D depthwise convolutional layers are used to independently convolve each channel on the input feature map to obtain an intermediate feature map with an equal number of channels; the 3D pointwise convolutional layers are used to aggregate channel information from the intermediate feature maps. Using all extracted target keyframes as input to the fast channel allows the advantage of a high frame rate to be utilized to capture rapid changes in gestures and more subtle motion details in the video.

[0060] Applying 3D depthwise separable convolution in convolutional layers, which is divided into 3D depthwise convolution and 3D pointwise convolution, significantly reduces computational cost and the number of model parameters, while effectively extracting spatiotemporal features. 3D depthwise convolution independently convolves each channel of the 3D feature map to obtain channel-independent intermediate feature maps, improving the model's real-time inference performance with minimal loss of accuracy. The formula for 3D depthwise convolution is as follows:

[0061]

[0062] Where W1 represents the weights of the 3D depthwise convolution, V represents the input 3D feature map, i, j, u represent position indices, and K, L, M represent the kernel size. This represents the dot product of elements. 3D pointwise convolution will be applied to the channel-independent feature maps obtained in the previous step. Further aggregation and channel information. The formula is defined as follows:

[0063]

[0064] Where W2 represents the weights of the 3D pointwise convolution, and n represents the size of the convolution kernel. 3D depthwise convolution and 3D pointwise convolution are performed sequentially to form a complete 3D depthwise separable convolution process, for example... Figure 3 As shown, the formula is as follows:

[0065] Conv sepConv (V)=Conv Point (Conv Depth (V));

[0066] Table 1 below compares the number of parameters between 3D separable convolution and traditional 3D convolution:

[0067] Table 1. Comparison of parameter counts between 3D separable convolution and traditional 3D convolution.

[0068] Convolution name Parameters 3D Convolution <![CDATA[M*K 3 *C*C^]]> 3D depthwise convolution <![CDATA[M*K 3 *C]]> 3D pointwise convolution M*C*C^ 3D separable convolution <![CDATA[M*K 3 *C+M*C*C^]]>

[0069] Where M = H * W * D, for example, H and W are set to 112 and D is 16. As can be seen from the table above, when K = 3 and C^ = 32, the number of parameters of 3D separable convolution is only 1 / 14 of the number of parameters of 3D convolution, which significantly reduces the complexity of the model.

[0070] The Fast Pathway maintains a smaller number of channels because it primarily focuses on the local details of fast-moving motions and short-term spatiotemporal information. Fewer channels reduce computational load and increase network speed while ensuring the capture of key information, matching the high frame rate input and enabling effective capture of fast gestures.

[0071] The improved network structure and the feature map size output after each layer are shown in Table 2. Where T represents the temporal depth, i.e., the number of input video frames, and S... 2 This indicates the square size of the feature map.

[0072] Table 2. Improved slowfast network structure and its output

[0073]

[0074] Violation of convolutional layers in slow channels, 1×7 2 This indicates a convolution kernel size of 1×7×7, resulting in 64 output channels and a stride of 1×2×2. The maximum number of channels in the slow channel is 512, and the maximum number of channels in the fast channel is 256. The fast channel's Data Layer uses fewer video frames than the Input. The fast channel's Data Layer uses the same number of video frames and image size as the Input. The output data shapes [BS, T, W, H, C] of each layer after passing through the improved slowfast network are shown in the table below.

[0075] Table 3. Shape of output feature maps for each layer of the improved slowfast network

[0076] Stage Slow Pathway Network Fast Pathway Network Input.shape [32,16,112,112,1] [32,16,112，112,1] Data layer.shape [32，4，112，112.32] [32,16,112,112,1] Conv1.shape [32,4,56,56,64] [32,16,56,56,16] Pool1.shape [32,4,56,56,64] [32,16,56,56,16] Res2.shape [32，4，28,28,128] [32,16,28,28,64] Res3.shape [32,4,14,14,256] [32,16，14,14,128] Res4.shape [32,4，7，7,512] [32,16,7,7,256]

[0077] Where BS stands for BatchSize, which can be manually set according to the hardware resources of the training network. Taking BS=32 as an example, T is the number of input video frames, W and H represent the width and height of the input video frame / feature map, respectively, and C represents the number of channels of the feature map. The number of channels of Input is 1, which means that the input is a grayscale image.

[0078] Step S13: Use the attention module to assign feature weights and fuse features to the first feature information and the second feature information to obtain fused features, so that the classifier can use the fused features to recognize the gesture type.

[0079] Finally, feature weights are assigned and features are fused based on the first and second feature information to obtain the fused features. In this embodiment, feature fusion is not simply a matter of stitching together; it also incorporates an attention module, specifically a 3D spatiotemporal attention module. This module can adaptively allocate weights according to the importance of features from different paths, fully aggregating spatial and temporal features to enhance feature effectiveness. To fully utilize and fuse the temporal and spatial features between consecutive video frames, a 3D spatiotemporal attention module is proposed to improve the model's recognition accuracy.

[0080] In some embodiments, the attention module is constructed by cascading a channel attention submodule and a spatial depth attention submodule. The aforementioned use of the attention module to assign feature weights and fuse features of the first and second feature information includes: using the channel attention submodule to assign weights to the spatial features of the first and second feature information; and using the spatial depth attention submodule to assign weights to the temporal features of the first and second feature information. That is, the channel attention submodule focuses on spatial features, such as background features, while the spatial depth attention submodule focuses more on the correlation between different frames, i.e., features in the temporal dimension. For example, for gesture parts that have obvious spatial features but do not change significantly in time, the 3D spatiotemporal attention module will give higher weights to the corresponding features in the slow channel; while for rapidly changing gesture details, it will increase the weights of features in the fast channel, thereby achieving more accurate feature fusion.

[0081] Specifically, the structure of the 3D spatiotemporal attention module is as follows: Figure 4 As shown, the channel attention submodule and the spatial depth attention submodule are cascaded to obtain the 3D spatiotemporal attention module. The channel attention weights are calculated by a multilayer perceptron (MLP), and the spatial depth attention weights are calculated by applying different convolutional kernels to multidimensional (spatial and depth) feature maps. Finally, the two modules are cascaded to form the complete 3D spatiotemporal attention module. The formula for calculating the weights of the channel attention submodule is shown below:

[0082]

[0083] Where V represents the input feature map, Avg Pool () indicates average pooling, Max... Pool () represents max pooling, and σ() represents the activation function, which is used to introduce nonlinear characteristics into the model and enhance its expressive power. The activation function is the ReLU() function. The weights are calculated after channel attention. This yields the intermediate feature map V′. Furthermore, the weight calculation formula for the spatial depth attention submodule is as follows:

[0084]

[0085] Among them, f 1×7×7 This represents a convolution operation, where 1×7×7 is the convolution kernel, indicating convolution in the horizontal direction of the feature map; f 7×1×1 f represents a convolution operation with a 7×1×1 kernel, and f represents a convolution along the depth direction of the feature map (i.e., the time dimension); 7×7×7 This represents the overall convolution operation. The final feature map output by the 3D spatiotemporal attention module is as follows: By integrating feature information extracted from two paths through a target dual-path network, various dynamic gestures can be classified more comprehensively and accurately. Finally, a representative feature vector is obtained after the improved target dual-path network and a 3D spatiotemporal attention module. This vector is used as the input to the fully connected classification layer and fed into the classifier for gesture classification.

[0086] For example Figure 5 The diagram shown is a schematic of a specific dynamic gesture recognition system.

[0087] The system comprises four parts: a data acquisition module, a keyframe extraction module, a network main framework, and a classifier. The specific recognition process is as follows: Gesture data is acquired via a camera, and the acquired continuous video frames are converted to grayscale; keyframes are extracted using keyframe technology; a subset of keyframes are input into the slow channel of the target dual-path network to capture spatial semantic information in the video at a lower frame rate; simultaneously, through parallel processing, all extracted keyframes are input into the fast channel to sample the video at a higher frame rate, capturing details and rapidly changing information in the video; the feature information from the two paths is fused using a 3D spatiotemporal attention module to achieve a comprehensive understanding of the video, considering both the overall information and locally rapidly changing information; finally, the fused information is input into the classifier (Softmax) for gesture classification, with classification results including but not limited to: no gesture, shaking, and thumbs-up. Significant results have been achieved both on PCs and on system-on-a-chip (SoC), meeting the needs of various scenarios in terms of recognition accuracy and real-time inference speed.

[0088] As can be seen from the above, in this embodiment, a video containing dynamic gestures is acquired, and keyframes are extracted from the video to obtain target keyframes. The target keyframes are input into a target dual-path network to obtain the first feature information output by the slow channel and the second feature information output by the fast channel. The convolutional layer of the slow channel is constructed based on cascaded small convolutional kernels, and the convolutional layer of the fast channel is constructed based on 3D depth-separable convolution. The attention module is used to perform feature weight allocation and feature fusion on the first and second feature information to obtain fused features, so that the classifier can use the fused features to recognize the gesture type. It is evident that extracting keyframes and performing recognition based on them can reduce useless and redundant data during model training, thereby improving the model's generalization ability and recognition accuracy. Video-based recognition can avoid information loss or delay and efficiently process long sequences of gesture data. Using cascaded small convolutional kernels in the slow channel can increase the network's receptive field while reducing the number of model parameters, capturing data details and high-level features. Using 3D depthwise separable convolutions in the fast channel can reduce computational costs and the number of model parameters. Utilizing attention modules for feature weight allocation and feature fusion enhances feature effectiveness and improves the model's adaptability and inference recognition accuracy. This addresses the problems of existing algorithms, such as large model parameters, difficulty in deploying on edge devices, and poor real-time recognition performance.

[0089] Accordingly, this application also discloses a dynamic gesture recognition device, see [link to relevant documentation]. Figure 6 As shown, the device includes:

[0090] The video acquisition module 11 is used to acquire a video to be recognized containing dynamic gestures, and to extract key frames from the video to be recognized to obtain target key frames.

[0091] The feature extraction module 12 is used to input the target keyframe into the target dual-path network to obtain the first feature information output by the slow channel in the target dual-path network and the second feature information output by the fast channel in the target dual-path network; wherein the convolutional layer of the slow channel is constructed based on cascaded small convolutional kernels and the convolutional layer of the fast channel is constructed based on 3D depth-separable convolution.

[0092] The feature fusion module 13 is used to perform feature weight allocation and feature fusion on the first feature information and the second feature information using the attention module to obtain fused features, so that the classifier can use the fused features to perform gesture type recognition.

[0093] As can be seen from the above, in this embodiment, a video containing dynamic gestures is acquired, and keyframes are extracted from the video to obtain target keyframes. The target keyframes are input into a target dual-path network to obtain the first feature information output by the slow channel and the second feature information output by the fast channel. The convolutional layer of the slow channel is constructed based on cascaded small convolutional kernels, and the convolutional layer of the fast channel is constructed based on 3D depth-separable convolution. The attention module is used to perform feature weight allocation and feature fusion on the first and second feature information to obtain fused features, so that the classifier can use the fused features to recognize the gesture type. It is evident that extracting keyframes and performing recognition based on them can reduce useless and redundant data during model training, thereby improving the model's generalization ability and recognition accuracy. Video-based recognition can avoid information loss or delay and efficiently process long sequences of gesture data. Using cascaded small convolutional kernels in the slow channel can increase the network's receptive field while reducing the number of model parameters, capturing data details and high-level features. Using 3D depthwise separable convolutions in the fast channel can reduce computational costs and the number of model parameters. Utilizing attention modules for feature weight allocation and feature fusion enhances feature effectiveness and improves the model's adaptability and inference recognition accuracy. This addresses the problems of existing algorithms, such as large model parameters, difficulty in deploying on edge devices, and poor real-time recognition performance.

[0094] In some specific embodiments, the video acquisition module 11 may specifically include:

[0095] The image entropy calculation unit is used to calculate the image entropy of each video frame in the video to be identified, and to determine local extreme points from the video to be identified based on all the image entropies; the local extreme points include local maxima and local minima.

[0096] A minimum distance determination unit is used to calculate the local density of each local extremum point and determine the minimum distance between each local extremum point and the other local extremum points based on the local density.

[0097] The target keyframe extraction unit is used to determine cluster centers based on the minimum distance and the first preset number of targets, and to determine a target keyframe for each cluster center.

[0098] In some specific embodiments, the feature extraction module 12 may specifically include:

[0099] An input unit is used to input the target keyframe into the slow channel; the slow channel is constructed in the order of data layer, convolutional layer, pooling layer and residual layer;

[0100] The first feature information acquisition unit is used to select a second preset number of target key frames from the target key frames using the data layer of the slow channel as the input of the convolutional layer of the slow channel, and obtain the first feature information of the target key frames according to the output of the residual layer of the slow channel.

[0101] The second feature information acquisition unit is used to input the target keyframe into the fast channel and obtain the second feature information of the target keyframe according to the output of the fast channel; the fast channel is constructed in the order of convolutional layer, pooling layer and residual layer.

[0102] In some specific embodiments, the convolutional layer of the slow channel can be constructed based on multiple cascaded 3×3 convolutional kernels.

[0103] In some specific embodiments, the 3D depthwise separable convolution may specifically include a 3D depthwise convolutional layer and a 3D pointwise convolutional layer; the 3D depthwise convolutional layer is used to perform independent convolution on each channel of the input feature map to obtain an intermediate feature map with the same number of channels; the 3D pointwise convolutional layer is used to aggregate channel information of the intermediate feature map.

[0104] In some specific embodiments, the attention module can be constructed by cascading a channel attention submodule and a spatial depth attention submodule.

[0105] In some specific embodiments, the feature fusion module 13 may specifically include:

[0106] The first weight allocation unit is used to allocate weights to the spatial features of the first feature information and the second feature information using the channel attention submodule.

[0107] The second weight allocation unit is used to allocate weights to the temporal features of the first feature information and the second feature information using the spatial depth attention submodule.

[0108] Furthermore, this application also discloses an electronic device, see [link to relevant documentation]. Figure 7 As shown, the content in the figure should not be considered as any limitation on the scope of use of this application.

[0109] Figure 7 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the dynamic gesture recognition method disclosed in any of the foregoing embodiments.

[0110] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0111] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include operating system 221, computer program 222 and data 223 including video to be identified, etc. The storage method can be temporary storage or permanent storage.

[0112] The operating system 221 manages and controls the various hardware devices on the electronic device 20 and the computer program 222 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the dynamic gesture recognition method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0113] Furthermore, this application also discloses a computer storage medium storing computer-executable instructions. When the computer-executable instructions are loaded and executed by a processor, they implement the dynamic gesture recognition method steps disclosed in any of the foregoing embodiments.

[0114] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0115] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0116] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0117] The present invention provides a detailed description of a dynamic gesture recognition method, apparatus, device, and storage medium. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A dynamic gesture recognition method, characterized in that, Applied to edge devices, including: A video containing dynamic gestures is acquired to be recognized, and keyframes are extracted from the video to be recognized to obtain target keyframes. The target keyframe is input into a target dual-path network to obtain the first feature information output by the slow channel and the second feature information output by the fast channel. The convolutional layer of the slow channel is constructed based on cascaded small convolutional kernels, and the convolutional layer of the fast channel is constructed based on 3D depthwise separable convolution. The 3D depthwise separable convolution includes a 3D depthwise convolutional layer and a 3D pointwise convolutional layer. The 3D depthwise convolutional layer is used to independently convolve each channel on the input feature map to obtain an intermediate feature map with the same number of channels. The 3D pointwise convolutional layer is used to aggregate channel information from the intermediate feature maps. The attention module is used to assign feature weights and fuse the first feature information and the second feature information to obtain fused features, so that the classifier can use the fused features to recognize the gesture type. The step of extracting target keyframes from the video to be identified includes: Calculate the image entropy of each video frame in the video to be identified, and determine local extreme points from the video to be identified based on all the image entropies; the local extreme points include local maxima and local minima; Calculate the local density of each local extremum point, and determine the minimum distance between each local extremum point and the other local extremum points based on the local density; wherein, the formula for calculating the local density is: ;in, For an extreme point, To remove Other extreme points outside, i =1,2,...,N-1; is the distance between two extreme points, and dc is the threshold distance; Cluster centers are determined based on the minimum distance and the first preset number of targets, and a target keyframe is determined for each cluster center; The step of inputting the target keyframe into the target dual-path network to obtain the first feature information output by the slow channel within the target dual-path network and the second feature information output by the fast channel within the target dual-path network includes: The target keyframe is input into the slow channel; the slow channel is constructed in the order of data layer, convolutional layer, pooling layer and residual layer; Using the data layer of the slow channel, a second preset number of target keyframes are selected from the target keyframes as the input of the convolutional layer of the slow channel, and the first feature information of the target keyframes is obtained based on the output of the residual layer of the slow channel. The target keyframe is input into the fast channel, and the second feature information of the target keyframe is obtained based on the output of the fast channel; the fast channel is constructed in the order of convolutional layer, pooling layer and residual layer; The attention module is constructed by cascading a channel attention submodule and a spatial depth attention submodule; the process of using the attention module to assign feature weights and fuse features of the first feature information and the second feature information includes: The channel attention submodule is used to assign weights to the spatial features of the first feature information and the second feature information; Using the spatial depth attention submodule, weights are assigned to the temporal features of the first feature information and the second feature information; The weight calculation formula for the spatial depth attention submodule is as follows: ； in, This represents a convolution operation in the horizontal direction of the feature map with a 1×7×7 kernel. This represents a convolution operation with a 7×1×1 kernel on the temporal dimension of the feature map. This represents the overall convolution operation. This represents the intermediate feature map after passing through the channel attention submodule. Indicates average pooling. This indicates max pooling.

2. The dynamic gesture recognition method according to claim 1, characterized in that, The slow channel's convolutional layer is constructed based on multiple cascaded 3×3 convolutional kernels.

3. A dynamic gesture recognition device, characterized in that, Applied to edge devices, including: The video acquisition module is used to acquire a video to be recognized containing dynamic gestures, and to extract keyframes from the video to be recognized to obtain target keyframes. A feature extraction module is used to input the target keyframe into a target dual-path network to obtain first feature information output by the slow channel and second feature information output by the fast channel within the target dual-path network. The convolutional layer of the slow channel is constructed based on cascaded small convolutional kernels, and the convolutional layer of the fast channel is constructed based on 3D depthwise separable convolution. The 3D depthwise separable convolution includes a 3D depthwise convolutional layer and a 3D pointwise convolutional layer. The 3D depthwise convolutional layer is used to independently convolve each channel on the input feature map to obtain an intermediate feature map with the same number of channels. The 3D pointwise convolutional layer is used to aggregate channel information from the intermediate feature maps. The feature fusion module is used to perform feature weight allocation and feature fusion on the first feature information and the second feature information using the attention module to obtain fused features, so that the classifier can use the fused features to perform gesture type recognition. The video acquisition module is configured to calculate the image entropy of each video frame in the video to be identified, and determine local extrema points from the video to be identified based on all the image entropies; the local extrema points include local maxima points and local minima points; calculate the local density of each local extrema point, and determine the minimum distance between each local extrema point and the other local extrema points based on the local density; wherein the formula for calculating the local density is: ;in, For an extreme point, To remove Other extreme points outside, i =1,2,...,N-1; The distance between the two extreme points is denoted by dc, which is the threshold distance. Cluster centers are determined based on the minimum distance and the first preset target quantity, and a target keyframe is determined for each cluster center. The feature extraction module is used to input the target keyframes into the slow channel; the slow channel is constructed in the order of data layer, convolutional layer, pooling layer, and residual layer; using the data layer of the slow channel, a second preset number of target keyframes are selected from the target keyframes as the input of the convolutional layer of the slow channel, and the first feature information of the target keyframes is obtained based on the output of the residual layer of the slow channel; the target keyframes are then input into the fast channel, and the second feature information of the target keyframes is obtained based on the output of the fast channel; the fast channel is constructed in the order of convolutional layer, pooling layer, and residual layer. The attention module is constructed by cascading a channel attention submodule and a spatial depth attention submodule. The feature fusion module is used to assign weights to the spatial features of the first and second feature information using the channel attention submodule, and to assign weights to the temporal features of the first and second feature information using the spatial depth attention submodule. The weight calculation formula for the spatial depth attention submodule is as follows: ;in, This represents a convolution operation in the horizontal direction of the feature map with a 1×7×7 kernel. This represents a convolution operation with a 7×1×1 kernel on the temporal dimension of the feature map. This represents the overall convolution operation. This represents the intermediate feature map after passing through the channel attention submodule. Indicates average pooling. This indicates max pooling.

4. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the dynamic gesture recognition method as described in claim 1 or 2.

5. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein the computer programs, when executed by a processor, implement the dynamic gesture recognition method as described in claim 1 or 2.

Images