Dynamic gesture recognition method and device, equipment and storage medium
Through adaptive window intercepting video frames and combining spatiotemporal feature fusion and CBAM feature enhancement, the problem of sliding window overlap and model structure in existing gesture recognition methods is solved, and the accuracy and real-timeness of dynamic gesture recognition are improved.
Patent Information
- Application Number
- CN202411235967.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-04
AI Technical Summary
When existing gesture recognition methods handle dynamic gestures, they can easily cause sliding windows to overlap and cannot accurately intercept video frame groups, resulting in hand detection failure and misrecognition. The gesture recognition model structure is simple, and it is impossible to effectively express the space-time characteristics of dynamic gestures, resulting in Low recognition accuracy.
The video frame is intercepted by an adaptively adjustable window, and the continuous gesture video frame group is extracted, and the 3D hand key point extraction and feature calculation are performed. Combined with spatiotemporal feature fusion, multi-scale feature extraction and CBAM feature enhancement, it is mapped to the recognition results.
It improves the accuracy and response speed of dynamic gesture recognition, reduces the error recognition rate, can effectively deal with short-term occlusion and excessive hand speed, and improves real-time and recognition accuracy.
Smart Images

Figure CN119091509A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the technical field of gesture recognition, and in particular to a dynamic gesture recognition method, device, equipment and computer-readable storage medium. Background Art
[0002] In existing gesture recognition methods, a sliding window with a fixed length and a fixed step size is usually used to capture a video frame group for data processing. However, in practice, the time intervals between multiple dynamic gestures are not equal. The use of existing data processing methods is prone to sliding window overlap, and it cannot be guaranteed that the captured video frame group belongs to the trajectory of the same dynamic gesture, and redundant video frames are captured, etc., which leads to blurred definitions of the starting and ending positions of dynamic gestures, and it is difficult to eliminate the influence of hand detection failures caused by occlusion, fast movements, etc., which leads to misrecognition problems, and is time-consuming and low in accuracy. In addition, the existing gesture recognition model has a simple structure and cannot express features of different scales, such as the spatiotemporal evolution of dynamic gestures and the diverse changes of different dynamic gestures, resulting in low gesture recognition accuracy. Summary of the invention
[0003] In view of the above problems, embodiments of the present invention provide a dynamic gesture recognition method, apparatus, device, and computer-readable storage medium to improve the accuracy of dynamic gesture recognition.
[0004] According to one aspect of an embodiment of the present invention, a dynamic gesture recognition method is provided, the method comprising:
[0005] Extracting gesture video frames from the video data to be recognized, and taking continuous gesture video frames of no less than a preset number of frames as gesture video frame groups, to obtain a plurality of gesture video frame groups;
[0006] Hand key points are extracted for each gesture video frame group to obtain a corresponding 3D hand key point sequence, and each of the 3D hand key point sequences is input into a recognition model in turn, so that the recognition model performs feature calculation on each of the 3D key point sequences to obtain spatial features and temporal features; feature transformation is performed on the spatial features and the temporal features respectively, and spatiotemporal feature fusion, multi-scale feature extraction and CBAM feature enhancement processing are performed in turn to obtain gesture spatiotemporal features; feature mapping is performed on the gesture spatiotemporal features to obtain recognition results corresponding to each gesture video frame group.
[0007] In an optional manner, the gesture video frames are extracted from the video data to be identified, and continuous gesture video frames of not less than a preset number of frames are used as gesture video frame groups to obtain a plurality of gesture video frame groups, including:
[0008] Detecting gesture video frames of video data; the gesture video frames include: a first gesture frame and a second gesture frame; wherein the first gesture frame is a video frame in which a hand is detected, and the second gesture frame is a video frame in which detection fails due to a preset reason;
[0009] Using a window with an adaptively adjustable length to intercept continuous gesture video frames in the video data to be recognized, a plurality of gesture video frame groups are obtained;
[0010] Among them, the number of frames of the first gesture frame in each of the gesture video frame groups is a first preset value, the number of frames of the second gesture frame can be adaptively adjusted, and the sum of the number of frames of the first gesture frame and the number of frames of the second gesture frame is the length of the window; the length of the window is not less than a second preset value.
[0011] In an optional manner, hand key points are extracted for each of the gesture video frame groups to obtain a corresponding 3D hand key point sequence, and each of the 3D hand key point sequences is sequentially input into a recognition model so that the recognition model performs feature calculation on each of the 3D key point sequences to obtain spatial features and temporal features, including:
[0012] Extracting 3D hand key points in each gesture video frame in each gesture video frame group;
[0013] Arrange the 3D hand key points of each frame in time sequence to obtain a 3D hand key point sequence of each gesture video frame group;
[0014] Set key point pairs of 3D hand key points, and calculate distance information of the key point pairs;
[0015] Sorting the distance information of the key point pairs in spatial order to obtain spatial features;
[0016] The 3D hand key point sequence is used as trajectory information, and the trajectory information of the first preset scale is extracted to obtain the time feature.
[0017] In an optional manner, feature transformation is performed on the spatial feature and the temporal feature respectively, and spatiotemporal feature fusion, multi-scale feature extraction and CBAM feature enhancement processing are performed in sequence to obtain the spatiotemporal feature of the gesture, including:
[0018] Performing feature transformation on the spatial feature and the temporal feature respectively, converting the spatial feature and the temporal feature to the same dimension, and obtaining transformed features;
[0019] Performing spatiotemporal feature fusion on the transformed features to obtain fused feature data;
[0020] Performing feature extraction on the fused feature data according to a second preset scale to obtain extracted data of a corresponding scale;
[0021] The extracted data is subjected to CBAM feature enhancement to obtain gesture spatiotemporal features.
[0022] In an optional manner, performing feature transformation on the spatial feature and the temporal feature respectively, converting the spatial feature and the temporal feature to the same dimension, and obtaining the transformed feature, includes:
[0023] The spatial features and the temporal features are sequentially input into the same convolution kernel and filter, and the sizes of the spatial features and the temporal features are modified to preset sizes by using a pooling operation to obtain transformed features.
[0024] In an optional manner, feature extraction is performed on the fused feature data according to a second preset scale to obtain extracted data of a corresponding scale, including:
[0025] According to the second preset scale, a convolution kernel of a corresponding scale is used to perform feature extraction on the fused feature data to obtain extracted data of the corresponding scale.
[0026] In an optional manner, the extracted data is subjected to CBAM feature enhancement to obtain gesture spatiotemporal features, including:
[0027] The extracted data is input into the CBAM module so that the CBAM module calculates the attention weight of the extracted data on the preset dimension, and the attention weight is input into the feature map for adaptive feature weighting to obtain enhanced extracted data; the enhanced extracted data is used as the spatiotemporal feature of the gesture.
[0028] This embodiment uses a dynamic window to capture the video screen, which reduces the time consumption for processing invalid video frame groups and repeated frames, and has a higher response speed; it avoids the interference of invalid or even irrelevant gestures that may exist in certain video frame groups in the retrieval case, and reduces the misrecognition rate; this embodiment first transforms the calculated spatiotemporal features to the same dimension through the convolution module and then fuses them together, and then uses a multi-scale convolution module to extract features of different scales to capture the diverse changes and spatiotemporal evolution laws of dynamic gestures, and uses a CBAM module for feature enhancement, focusing on the important features of the gesture and suppressing invalid features, and then mapping the features to the recognition results, thereby improving the accuracy of gesture recognition. At the same time, this embodiment does not increase too many parameters and calculations, and has a high real-time performance. This embodiment can deal with the problems of short-term occlusion and excessive hand speed in individual hand video frames, and improve the accuracy of dynamic gesture recognition.
[0029] According to another aspect of an embodiment of the present invention, there is provided a dynamic gesture recognition device, comprising: a dynamic extraction module and a recognition module;
[0030] The dynamic extraction module is used to extract gesture video frames from the video data to be identified, and use continuous gesture video frames of no less than a preset number of frames as gesture video frame groups to obtain a plurality of gesture video frame groups;
[0031] The recognition module is used to extract hand key points from each gesture video frame group to obtain a corresponding 3D hand key point sequence, and input each 3D hand key point sequence into the recognition model in turn, so that the recognition model performs feature calculation on each 3D key point sequence to obtain spatial features and temporal features; perform feature transformation on the spatial features and the temporal features respectively, and perform spatiotemporal feature fusion, multi-scale feature extraction and CBAM feature enhancement processing in turn to obtain gesture spatiotemporal features; perform feature mapping on the gesture spatiotemporal features to obtain a recognition result corresponding to each gesture video frame group.
[0032] According to another aspect of an embodiment of the present invention, a dynamic gesture recognition device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation of any one of the dynamic gesture recognition methods described above.
[0033] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores at least one executable instruction, wherein the executable instruction enables a dynamic gesture recognition device / apparatus to perform the operation of any one of the dynamic gesture recognition methods described above.
[0034] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:
[0036] Figure 1 A schematic diagram of the process of the dynamic gesture recognition method provided by the present invention is shown;
[0037] Figure 2A schematic diagram of the workflow of the recognition model of the dynamic gesture recognition method provided by the present invention is shown;
[0038] Figure 3 A schematic diagram showing the structure of a dynamic gesture recognition device provided by the present invention is shown;
[0039] Figure 4 A structural schematic diagram of an embodiment of a dynamic gesture recognition device provided by the present invention is shown. DETAILED DESCRIPTION
[0040] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.
[0041] Embodiment 1, Figure 1 FIG. 1 is a flow chart showing a first embodiment of a dynamic gesture recognition method according to the present invention, wherein the method is executed by a dynamic gesture recognition device. Figure 1 As shown, the method comprises the following steps:
[0042] Step 110: extracting gesture video frames from the video data to be identified, and taking continuous gesture video frames of no less than a preset number of frames as gesture video frame groups, to obtain a plurality of gesture video frame groups.
[0043] In some embodiments of the present invention, satisfying the preset condition includes: satisfying a continuous gesture video frame number of not less than a preset number of frames.
[0044] In some embodiments of the present invention, the step of extracting gesture video frames from the video data to be identified and taking continuous gesture video frames of not less than a preset number of frames as gesture video frame groups to obtain a plurality of gesture video frame groups includes:
[0045] Detecting gesture video frames of video data; the gesture video frames include: a first gesture frame and a second gesture frame; wherein the first gesture frame is a video frame in which a hand is detected, and the second gesture frame is a video frame in which detection fails due to a preset reason;
[0046] Using a window with an adaptively adjustable length to intercept continuous gesture video frames in the video data to be recognized, a plurality of gesture video frame groups are obtained;
[0047] Among them, the number of frames of the first gesture frame in each of the gesture video frame groups is a first preset value, the number of frames of the second gesture frame can be adaptively adjusted, and the sum of the number of frames of the first gesture frame and the number of frames of the second gesture frame is the length of the window; the length of the window is not less than a second preset value.
[0048] Specifically, the number of video frames that need to be processed for each continuous hand video frame collected using the adaptively adjustable window is not a fixed value, but is determined by the number of frames in which the hand appears continuously in the video frame. Each gesture video frame includes a first gesture frame and a second gesture frame. The second gesture frame is a number of video frames in which hand detection fails due to short-term occlusion or too fast hand speed in the continuous gesture. The number of frames of the first gesture frame is a fixed value, and the number of frames of the second gesture frame is a variable value, so the window length can be adaptively adjusted according to the actual situation.
[0049] The selection basis of the gesture video frame group is: the continuous gesture video frame is not less than the preset number of frames, so the length of the gesture video frame group will change dynamically, and the change is adaptive rather than manually adjusted. Preferably, in order to deal with problems such as short-term occlusion and excessive hand speed, in one embodiment, the continuous hand video frames mentioned in this article are not the 1st, 2nd, 3rd, 4th, 5th... frames in the video picture frame, but the 1st, 3rd, 4th, 6th... frames, etc., and short interruptions of several frames are acceptable, so the length of the gesture video frame group is adaptively adjusted.
[0050] For example, a dynamic window with an adaptively adjustable length is used to capture the video screen, where the number of frames in which the hand appears exceeds 16 frames for dynamic gesture recognition, and it is necessary to ensure that the number of frames in which the hand appears continuously in the video screen reaches 16 frames. In this process, problems such as short-term occlusion and too fast hand speed will cause individual hand frames to fail to detect the hand, and the actual video screen may be 16, 17 or 18 frames, etc., so the corresponding video frames are captured using the dynamic window.
[0051] The method of using a dynamic window to capture hand video frame groups reduces the time spent on processing invalid video frame groups and repeated frames, has a higher response speed, and avoids interference from invalid or even irrelevant gestures that may exist in certain video frame groups.
[0052] Step 120: extract hand key points for each of the gesture video frame groups to obtain a corresponding 3D hand key point sequence, input each of the 3D hand key point sequences into the recognition model in turn, so that the recognition model performs feature calculation on each of the 3D key point sequences to obtain spatial features and temporal features; perform feature transformation on the spatial features and the temporal features respectively, perform spatiotemporal feature fusion, multi-scale feature extraction and CBAM feature enhancement processing in turn to obtain gesture spatiotemporal features; perform feature mapping on the gesture spatiotemporal features to obtain a recognition result corresponding to each of the gesture video frame groups.
[0053] In some embodiments of the present invention, hand key points are extracted from each of the gesture video frame groups to obtain a corresponding 3D hand key point sequence, and each of the 3D hand key point sequences is sequentially input into a recognition model so that the recognition model performs feature calculation on each of the 3D key point sequences to obtain spatial features and temporal features, including:
[0054] Extracting 3D hand key points in each gesture video frame in each gesture video frame group;
[0055] Arrange the 3D hand key points of each frame in time sequence to obtain a 3D hand key point sequence of each gesture video frame group;
[0056] Set key point pairs of 3D hand key points, and calculate distance information of the key point pairs;
[0057] Sorting the distance information of the key point pairs in spatial order to obtain spatial features;
[0058] The 3D hand key point sequence is used as trajectory information, and the trajectory information of the first preset scale is extracted to obtain the time feature.
[0059] Preferably, after obtaining the 3D hand key point sequence of each of the gesture video frame groups, the method further includes: performing median filtering and data normalization on the extracted 3D hand key point sequence to improve data quality and recognition accuracy.
[0060] Specifically, the 3D hand key point sequence is subjected to median filtering, including: arranging the extracted key point sequence by frame, wherein the coordinate value of each point is replaced by the median of three points in the neighborhood of the point, so as to make the dynamic gesture trajectory smoother and eliminate the interference of outliers.
[0061] The 3D hand key point sequence is subjected to data normalization, including: subtracting the mean of each dimension (x, y, z) from the key point coordinates of that dimension and dividing by the variance of that dimension, eliminating the influence of large differences between key point coordinates to facilitate subsequent feature calculation.
[0062] In some embodiments of the present invention, feature transformation is performed on the spatial features and the temporal features respectively, and spatiotemporal feature fusion, multi-scale feature extraction and CBAM feature enhancement processing are performed in sequence to obtain the spatiotemporal features of the gesture, including:
[0063] Performing feature transformation on the spatial feature and the temporal feature respectively, converting the spatial feature and the temporal feature to the same dimension, and obtaining transformed features;
[0064] Performing spatiotemporal feature fusion on the transformed features to obtain fused feature data;
[0065] Performing feature extraction on the fused feature data according to a second preset scale to obtain extracted data of a corresponding scale;
[0066] The extracted data is subjected to CBAM feature enhancement to obtain gesture spatiotemporal features.
[0067] Preferably, among all the 3D hand key points, the 3D hand key points are grouped into pairs, set as multiple key point pairs, and the key point pairs in the k-th frame of the gesture video frame are calculated. and The Euclidean distance between is: Key point pair in the kth frame of the gesture video The Euclidean distance from the origin o(0,0,0) is:
[0068] Construct the initial distance matrix of the k-th gesture video frame:
[0069]
[0070] Initial distance matrix KD k It represents the Euclidean distance between any two key points in the kth frame and the Euclidean distance between all key points and the origin.
[0071] Furthermore, in order to avoid duplication of information, this embodiment only uses the upper triangular elements of the complete matrix to construct the distance matrix, and the distance matrix is:
[0072]
[0073] The KD of each gesture video frame k The feature is flattened into a one-dimensional vector, and the distance feature vector of each gesture video frame is arranged in time order to obtain a two-dimensional vector of the gesture video frame group, and the two-dimensional vector of the gesture video frame group is used as the spatial feature. For example, assuming that the number of gesture video frames is 12, the number of key points is 21, and the coordinate dimension is 3, the size of the 3D hand key point sequence is (12, 21, 3), and the size of the spatial feature is (12, 231).
[0074] Preferably, the 3D hand key point sequence is recorded as a S low trajectory, and the 3D hand key point sequence is sliced by a slicing method with a step size of 2 to obtain a sequence recorded as a Fast trajectory, thereby obtaining trajectory information of two scales. For example, if the size of the 3D hand key point sequence is (12, 21, 3), then the size of the S low trajectory is (12, 21, 3), and the size of the Fast trajectory is (6, 21, 3).
[0075] Calculate the coordinate difference between the key points of two adjacent frames of the Slow trajectory and the Fast trajectory to obtain the Slow coordinate difference and the Fast trajectory. Where x = p[1:,...]-p[:-1,...]; x represents the coordinate difference between the key points of two adjacent frames, p represents the key point sequence, p[1:,...] represents all key point information except the first frame, and p[:-1,...] represents all key point information except the last frame.
[0076] The dimension of the Slow coordinate difference is (11, 21, 3), and the dimension of the Fast coordinate difference is (5, 21, 3). The bilinear interpolation method is used to modify the Slow coordinate difference dimension to the same as the input and transform the dimension to (12, 63). For the Fast trajectory information, the final dimension after the same processing is (6, 63), and the trajectory information of different scales or dimensions is used as the time feature.
[0077] In some embodiments of the present invention, feature transformation is performed on the spatial feature and the temporal feature respectively, and the spatial feature and the temporal feature are transformed to the same dimension to obtain the transformed feature, including:
[0078] The spatial features and the temporal features are sequentially input into the same convolution kernel and filter, and the sizes of the spatial features and the temporal features are modified to preset sizes by using a pooling operation to obtain transformed features.
[0079] Exemplarily, the size of the spatial feature is (12, 231). The temporal feature includes a Slow trajectory and a Fast trajectory, the size of the Slow trajectory is (12, 63), and the size of the Fast trajectory is (6, 63). The spatial feature and the temporal feature are sequentially input into the same convolution kernel and filter to transform the feature dimensions of the spatial feature, the Slow trajectory, and the Fast trajectory to (12, 64), (12, 64), and (6, 64), respectively. The spatial feature and the Slow trajectory are subjected to a pooling operation to transform the feature dimension to (6, 64) so that the feature dimensions of the spatial feature, the Slow trajectory, and the Fast trajectory are the same.
[0080] Preferably, the spatial features and time features after feature transformation are stacked to obtain comprehensive spatiotemporal trajectory information, and the feature dimension of the spatiotemporal trajectory information is (6, 192).
[0081] In some embodiments of the present invention, feature extraction is performed on the fused feature data according to the second preset scale to obtain extracted data of the corresponding scale, including:
[0082] According to the second preset scale, a convolution kernel of a corresponding scale is used to perform feature extraction on the fused feature data to obtain extracted data of the corresponding scale.
[0083] Preferably, this embodiment uses 1*1, 3*1 and 5*1 multi-scale convolution kernels to extract multi-scale feature information from the spatiotemporal trajectory information to capture the diverse changes and spatiotemporal evolution laws of dynamic gestures, and keeps the feature dimension unchanged through feature filling, which facilitates the integration of multi-scale features.
[0084] In some embodiments of the present invention, feature enhancement is performed on the extracted data to obtain gesture features, including:
[0085] The extracted data is input into a convolutional attention module so that the convolutional attention module calculates the attention weight of the extracted data on a preset dimension, and the attention weight is input into a feature map for adaptive feature weighting to obtain enhanced extracted data; the enhanced extracted data is used as a gesture feature.
[0086] Preferably, CBAM is a lightweight attention module. Given an input feature map, the module sequentially calculates attention weights along two independent dimensions (channel and space), and multiplies the attention weights by the input feature map for adaptive feature weighting. The CBAM module can be embedded in a one-dimensional convolutional layer for end-to-end training to enhance feature representation.
[0087] Please refer to Figure 2 , which is the workflow of the recognition model, calculates the 3D hand key point sequence to obtain the key point pair distance, S l ow trajectory and Fast trajectory information. The recognition model inputs the key point pair distance, S l ow trajectory and Fast trajectory into three one-dimensional convolution modules respectively, transforms the three features to the same dimension and fuses them together to obtain the fused feature data. The recognition model inputs the fused feature data into 1*1, 3*1 and 5*1 multi-scale convolution kernels respectively for feature extraction to obtain three sets of extracted data. The recognition model inputs the three sets of extracted data into the CBAM model for feature enhancement, obtains the enhanced features as gesture features, performs feature mapping on the gesture features, and obtains the recognition results corresponding to each of the gesture video frame groups.
[0088] Implementing this embodiment has the following beneficial effects:
[0089] This embodiment uses a dynamic window to capture the video screen, which reduces the time consumption for processing invalid video frame groups and repeated frames, and has a higher response speed; it avoids the interference of invalid or even irrelevant gestures that may exist in certain video frame groups, and reduces the misrecognition rate; this embodiment first transforms the calculated spatiotemporal features to the same dimension through a convolution module and then fuses them together, and then uses a multi-scale convolution module to extract features of different scales to capture the diverse changes and spatiotemporal evolution laws of dynamic gestures, and uses a CBAM module for feature enhancement, focusing on the important features of gestures and suppressing invalid features, and then mapping the features to the recognition results, thereby improving the accuracy of gesture recognition. At the same time, this embodiment does not increase too many parameters and calculations, and has high real-time performance. This embodiment can deal with problems such as short-term occlusion and excessive hand speed in individual hand video frames, and improve the accuracy of dynamic gesture recognition.
[0090] Embodiment 2, Figure 3 FIG. 4 is a schematic diagram showing the structure of a first embodiment of a dynamic gesture recognition device according to the present invention; Figure 3 As shown, the device 300 includes: a dynamic extraction module 310 and an identification module 320;
[0091] The dynamic extraction module 310 is used to extract gesture video frames from the video data to be identified, and use continuous gesture video frames of no less than a preset number of frames as gesture video frame groups to obtain a plurality of gesture video frame groups;
[0092] The recognition module 320 is used to extract the hand key points of each gesture video frame group to obtain the corresponding 3D hand key point sequence, and input each 3D hand key point sequence into the recognition model in turn, so that the recognition model performs feature calculation on each 3D key point sequence to obtain spatial features and temporal features; perform feature transformation on the spatial features and the temporal features respectively, and perform spatiotemporal feature fusion, multi-scale feature extraction and CBAM feature enhancement processing in turn to obtain gesture spatiotemporal features; perform feature mapping on the gesture spatiotemporal features to obtain the recognition results corresponding to each gesture video frame group. In an optional manner, the gesture video frames are extracted from the video data to be recognized, and continuous gesture video frames of not less than a preset number of frames are used as gesture video frame groups to obtain a plurality of gesture video frame groups, including:
[0093] Detecting gesture video frames of video data; the gesture video frames include: a first gesture frame and a second gesture frame; wherein the first gesture frame is a video frame in which a hand is detected, and the second gesture frame is a video frame in which detection fails due to a preset reason;
[0094] Using a window with an adaptively adjustable length to intercept continuous gesture video frames in the video data to be recognized, a plurality of gesture video frame groups are obtained;
[0095] Among them, the number of frames of the first gesture frame in each of the gesture video frame groups is a first preset value, the number of frames of the second gesture frame can be adaptively adjusted, and the sum of the number of frames of the first gesture frame and the number of frames of the second gesture frame is the length of the window; the length of the window is not less than a second preset value.
[0096] In an optional manner, hand key points are extracted for each of the gesture video frame groups to obtain a corresponding 3D hand key point sequence, and each of the 3D hand key point sequences is sequentially input into a recognition model so that the recognition model performs feature calculation on each of the 3D key point sequences to obtain spatial features and temporal features, including:
[0097] Extracting 3D hand key points in each gesture video frame in each gesture video frame group;
[0098] Arrange the 3D hand key points of each frame in time sequence to obtain a 3D hand key point sequence of each gesture video frame group;
[0099] Set key point pairs of 3D hand key points, and calculate distance information of the key point pairs;
[0100] Sorting the distance information of the key point pairs in spatial order to obtain spatial features;
[0101] The 3D hand key point sequence is used as trajectory information, and the trajectory information of the first preset scale is extracted to obtain the time feature.
[0102] In an optional manner, feature transformation is performed on the spatial feature and the temporal feature respectively, and spatiotemporal feature fusion, multi-scale feature extraction and CBAM feature enhancement processing are performed in sequence to obtain the spatiotemporal feature of the gesture, including:
[0103] Performing feature transformation on the spatial feature and the temporal feature respectively, converting the spatial feature and the temporal feature to the same dimension, and obtaining transformed features;
[0104] Performing spatiotemporal feature fusion on the transformed features to obtain fused feature data;
[0105] Performing feature extraction on the fused feature data according to a second preset scale to obtain extracted data of a corresponding scale;
[0106] The extracted data is subjected to CBAM feature enhancement to obtain gesture spatiotemporal features.
[0107] In an optional manner, performing feature transformation on the spatial feature and the temporal feature respectively, converting the spatial feature and the temporal feature to the same dimension, and obtaining the transformed feature, includes:
[0108] The spatial features and the temporal features are sequentially input into the same convolution kernel and filter, and the sizes of the spatial features and the temporal features are modified to preset sizes by using a pooling operation to obtain transformed features.
[0109] In an optional manner, feature extraction is performed on the fused feature data according to a second preset scale to obtain extracted data of a corresponding scale, including:
[0110] According to the second preset scale, a convolution kernel of a corresponding scale is used to perform feature extraction on the fused feature data to obtain extracted data of the corresponding scale.
[0111] In an optional manner, the extracted data is subjected to CBAM feature enhancement to obtain gesture spatiotemporal features, including:
[0112] The extracted data is input into the CBAM module so that the CBAM module calculates the attention weight of the extracted data on the preset dimension, and the attention weight is input into the feature map for adaptive feature weighting to obtain enhanced extracted data; the enhanced extracted data is used as the spatiotemporal feature of the gesture.
[0113] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0114] Embodiment 3, Figure 4 The schematic diagram of the structure of an embodiment of the dynamic gesture recognition device of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the dynamic gesture recognition device.
[0115] like Figure 4 As shown, the dynamic gesture recognition device may include: a processor 402, a communication interface 404, a memory 406, and a communication bus 408.
[0116] The processor 402, the communication interface 404, and the memory 406 communicate with each other via the communication bus 408. The communication interface 404 is used to communicate with other devices such as a client or other server network elements. The processor 402 is used to execute the program 410, which can specifically execute the relevant steps in the above-mentioned embodiment of the method for dynamic gesture recognition.
[0117] Specifically, the program 410 may include program code including computer executable instructions.
[0118] The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention. The one or more processors included in the dynamic gesture recognition device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.
[0119] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.
[0120] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system or other device. In addition, the embodiments of the present invention are not directed to any particular programming language.
[0121] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. Similarly, in order to simplify the present invention and help understand one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. Wherein, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself is a separate embodiment of the present invention.
[0122] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive.
[0123] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.
Claims
1. A dynamic gesture recognition method, characterized in that: The method comprises: Extracting gesture video frames from the video data to be recognized, and taking continuous gesture video frames of no less than a preset number of frames as gesture video frame groups, to obtain a plurality of gesture video frame groups; Hand key points are extracted for each gesture video frame group to obtain a corresponding 3D hand key point sequence, and each of the 3D hand key point sequences is input into a recognition model in turn, so that the recognition model performs feature calculation on each of the 3D key point sequences to obtain spatial features and temporal features; feature transformation is performed on the spatial features and the temporal features respectively, and spatiotemporal feature fusion, multi-scale feature extraction and CBAM feature enhancement processing are performed in turn to obtain gesture spatiotemporal features; feature mapping is performed on the gesture spatiotemporal features to obtain recognition results corresponding to each gesture video frame group.
2. The dynamic gesture recognition method according to claim 1, characterized in that: The step of extracting gesture video frames from the video data to be identified and taking continuous gesture video frames of no less than a preset number of frames as gesture video frame groups, thereby obtaining a plurality of gesture video frame groups, including: Detecting gesture video frames of video data; the gesture video frames include: a first gesture frame and a second gesture frame; wherein the first gesture frame is a video frame in which a hand is detected, and the second gesture frame is a video frame in which detection fails due to a preset reason; Using a window with an adaptively adjustable length to intercept continuous gesture video frames in the video data to be recognized, a plurality of gesture video frame groups are obtained; Among them, the number of frames of the first gesture frame in each of the gesture video frame groups is a first preset value, the number of frames of the second gesture frame can be adaptively adjusted, and the sum of the number of frames of the first gesture frame and the number of frames of the second gesture frame is the length of the window; the length of the window is not less than a second preset value.
3. The dynamic gesture recognition method according to claim 1, characterized in that: Extracting hand key points from each of the gesture video frame groups to obtain a corresponding 3D hand key point sequence, and sequentially inputting each of the 3D hand key point sequences into a recognition model so that the recognition model performs feature calculation on each of the 3D key point sequences to obtain spatial features and temporal features, including: Extracting 3D hand key points in each gesture video frame in each gesture video frame group; Arrange the 3D hand key points of each frame in time sequence to obtain a 3D hand key point sequence of each gesture video frame group; Set key point pairs of 3D hand key points, and calculate distance information of the key point pairs; Sorting the distance information of the key point pairs in spatial order to obtain spatial features; The 3D hand key point sequence is used as trajectory information, and the trajectory information of the first preset scale is extracted to obtain the time feature.
4. The dynamic gesture recognition method according to claim 1, characterized in that: Performing feature transformation on the spatial feature and the temporal feature respectively, The spatiotemporal features are fused, multi-scale features are extracted, and CBAM feature enhancement is performed in sequence to obtain the spatiotemporal features of gestures, including: Performing feature transformation on the spatial feature and the temporal feature respectively, converting the spatial feature and the temporal feature to the same dimension, and obtaining transformed features; Performing spatiotemporal feature fusion on the transformed features to obtain fused feature data; Performing feature extraction on the fused feature data according to a second preset scale to obtain extracted data of a corresponding scale; The extracted data is subjected to CBAM feature enhancement to obtain gesture spatiotemporal features.
5. The dynamic gesture recognition method according to claim 4, characterized in that: Performing feature transformation on the spatial feature and the temporal feature respectively, converting the spatial feature and the temporal feature to the same dimension, and obtaining transformed features, including: The spatial features and the temporal features are sequentially input into the same convolution kernel and filter, and the sizes of the spatial features and the temporal features are modified to preset sizes by using a pooling operation to obtain transformed features.
6. The dynamic gesture recognition method according to claim 4, characterized in that: The fused feature data is subjected to feature extraction according to the second preset scale to obtain extracted data of the corresponding scale, including: According to the second preset scale, a convolution kernel of a corresponding scale is used to perform feature extraction on the fused feature data to obtain extracted data of the corresponding scale.
7. The dynamic gesture recognition method according to claim 4, characterized in that: The extracted data is subjected to CBAM feature enhancement to obtain gesture spatiotemporal features, including: The extracted data is input into the CBAM module so that the CBAM module calculates the attention weight of the extracted data on the preset dimension, and the attention weight is input into the feature map for adaptive feature weighting to obtain enhanced extracted data; the enhanced extracted data is used as the spatiotemporal feature of the gesture.
8. A dynamic gesture recognition device, characterized in that: The device comprises: a dynamic extraction module and a recognition module; The dynamic extraction module is used to extract gesture video frames from the video data to be identified, and use continuous gesture video frames of no less than a preset number of frames as gesture video frame groups to obtain a plurality of gesture video frame groups; The recognition module is used to extract hand key points from each gesture video frame group to obtain a corresponding 3D hand key point sequence, and input each 3D hand key point sequence into the recognition model in turn, so that the recognition model performs feature calculation on each 3D key point sequence to obtain spatial features and temporal features; perform feature transformation on the spatial features and the temporal features respectively, and perform spatiotemporal feature fusion, multi-scale feature extraction and CBAM feature enhancement processing in turn to obtain gesture spatiotemporal features; perform feature mapping on the gesture spatiotemporal features to obtain a recognition result corresponding to each gesture video frame group.
9. A dynamic gesture recognition device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation of the dynamic gesture recognition method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction. When the executable instruction is executed on the dynamic gesture recognition device / apparatus, the dynamic gesture recognition device / apparatus performs the operation of the dynamic gesture recognition method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Multi-modal pedestrian detection method based on improved YOLO model
CN111767882A
Dynamic gesture real-time recognition method and device, electronic equipment and storage medium
CN115565253A
Gesture interaction method in intelligent cabin environment and related equipment
CN117373121A
Dynamic gesture recognition method combining multi-modal inter-frame movement and shared attention weight
CN118072395A
Dynamic gesture recognition method and device and storage medium
CN118470793A
