Attention mechanism-based dynamic gesture recognition method and apparatus

Abstract

The present application discloses an attention mechanism-based dynamic gesture recognition method and apparatus. In the present application, the method comprises: by means of a video acquisition device, acquiring, on the basis of a time sequence, a preset number of first video frames containing hand key points so as to obtain a first video frame sequence; traversing the first video frame sequence, and calculating the Euclidean distances between hand key points in each first video frame, so as to obtain a two-dimensional distance matrix; and inputting the two-dimensional distance matrix into a preset gesture recognition model to obtain a gesture recognition result, wherein the gesture recognition model is constructed by connecting a lightweight convolutional neural network and a bidirectional long short-term memory network via an attention mechanism module. In the present application, spatial-temporal feature information of a gesture can be considered in a dynamic gesture recognition process, thereby improving the accuracy of gesture recognition. In addition, the real-time performance of dynamic gesture recognition is improved by means of lightweight convolution.

Description

A dynamic gesture recognition method and device based on an attention mechanism TECHNICAL FIELD

[0001] The present application relates to the field of gesture recognition, in particular to a dynamic gesture recognition method and device based on an attention mechanism. BACKGROUND

[0002] With the development of related industries such as automatic driving and consumer electronics that require human-computer interaction operations, users have increasingly high requirements for operation convenience and speed, and gesture recognition technology effectively meets the needs of users.

[0003] The existing public technology uses overlap rate, distance or joint information entropy as the basis for dynamic gesture recognition, which can simply identify gesture features with small action amplitude, but the recognition accuracy is not high because the time-space variation characteristics of gestures are not considered. In order to improve the accuracy of gesture recognition, the existing technology uses three-dimensional convolution and GRU (Gated Recurrent Unit) network for dynamic gesture recognition, but this method has high technical cost and long calculation time, and is not suitable for implementation in low-config hardware facilities.

[0004] Therefore, in the face of the challenge of recognizing various complex gestures of users, how to ensure the accuracy of dynamic gesture recognition and the recognition efficiency under the condition of low hardware cost has become a technical problem to be solved. SUMMARY

[0005] The present application provides a dynamic gesture recognition method and device based on an attention mechanism to solve the technical problems of low accuracy, high cost and poor real-time performance of dynamic gesture recognition.

[0006] To solve the above technical problems, in a first aspect, the present application provides a dynamic gesture recognition method based on an attention mechanism, comprising:

[0007] acquiring a first video frame sequence by collecting a preset number of first video frames containing hand key points in time sequence through a video acquisition device;

[0008] calculating the Euclidean distance between each hand key point in each first video frame to obtain a two-dimensional distance matrix by traversing the first video frame sequence;

[0009] inputting the two-dimensional distance matrix into a preset gesture recognition model to obtain a gesture recognition result; the gesture recognition model is built by connecting a lightweight convolutional neural network and a bidirectional long short-term memory network through an attention mechanism module.

[0010] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by generating a video frame sequence composed of video frames containing hand key points in chronological order, the time information features in the dynamic hand gesture change process are retained, and then a two-dimensional distance matrix containing hand spatial information features is extracted based on the Euclidean distance between the hand key points. At this time, the two-dimensional distance matrix contains the feature information of the dynamic hand gesture change in space-time, and then the spatial information features in the two-dimensional distance matrix are extracted by a lightweight convolutional neural network, and further the feature data of the part that needs to be focused on in the spatial information features is strengthened by an attention mechanism. Since the bidirectional long short-term memory network can process sequence data and maintain long-term memory, and the two-dimensional distance matrix is obtained from the video frame sequence sorted in chronological order, therefore, the strengthened spatial information features are input into the bidirectional long short-term memory network, the space-time feature information containing time information can be extracted from the spatial information features, thereby improving the accuracy of subsequent dynamic hand gesture recognition.

[0011] In some embodiments of the first aspect of the present application, the inputting the two-dimensional distance matrix into a preset gesture recognition model to obtain a gesture recognition result comprises:

[0012] extracting features of the two-dimensional distance matrix by the lightweight convolutional neural network to obtain first feature information;

[0013] enhancing the first feature information by the attention mechanism module to obtain second feature information;

[0014] extracting features of the second feature information by a bidirectional long short-term memory network model to obtain third feature information;

[0015] mapping the third feature information to the gesture recognition result by a fully connected layer.

[0016] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by extracting features through a lightweight convolutional neural network, the efficiency of the spatial information feature extraction process can be improved, so that the hand recognition model can be built on a lightweight hardware facility, and the real-time performance of dynamic hand gesture recognition is improved; by the attention mechanism module, the attention degree of important features is enhanced, and the enhanced features are input into the bidirectional long short-term memory network model in the order of corresponding video frames, the feature information of the dynamic hand gesture in the time dimension is extracted, thereby obtaining feature information containing time and space dimension information, and the accuracy of the subsequent gesture recognition result is improved.

[0017] In some embodiments of the first aspect of the present application, the extracting features of the two-dimensional distance matrix by the lightweight convolutional neural network to obtain first feature information comprises:

[0018] The lightweight convolutional neural network comprises a plurality of convolution kernels of different scales.

[0019] The spatial information features of the two-dimensional distance matrix under different scale convolution operations are extracted through the convolution kernels of different scales.

[0020] All the spatial information features are fused to obtain the first feature information.

[0021] Compared with the prior art, the embodiments of the present application have the following beneficial effects: different levels of feature information are captured through the convolution kernels of different scales, and the different levels of feature information are fused, so that more generalized features are obtained, and the risk of overfitting of the subsequent dynamic gesture recognition result is reduced.

[0022] In some embodiments of the first aspect of the present application, the traversal of the first video frame sequence, the calculation of the Euclidean distances between the hand key points in each of the first video frames, and the acquisition of the two-dimensional distance matrix comprise:

[0023] According to the spatial order of the hand key points, all the hand key points in each of the first video frames are sorted;

[0024] The first Euclidean distances between any two hand key points in each of the first video frames are calculated, and the second Euclidean distances between each of the hand key points and a preset origin point in each of the first video frames are calculated;

[0025] According to all the first Euclidean distances and all the second Euclidean distances corresponding to each first video frame, the two-dimensional distance matrix is obtained.

[0026] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the spatial information features of the gesture can be clearly represented by calculating the Euclidean distances between the hand key points, which is beneficial to the subsequent convolutional neural network to extract the effective spatial information feature vector of the dynamic gesture, and the generated two-dimensional distance matrix is obtained by traversing the first video frame sequence obtained in time sequence and calculating the Euclidean distances, so the two-dimensional distance matrix actually also contains the features of the dynamic gesture in the time dimension, and therefore the efficiency of subsequent data processing can be improved.

[0027] In some embodiments of the first aspect of the present application, the first video frame sequence is obtained by sequentially judging whether the video frame collected by the video collection device contains the hand key points until the length of the first video frame sequence meets a first preset threshold.

[0028] The video frame collected by the video collection device is sequentially judged whether it contains the hand key points until the length of the first video frame sequence meets a first preset threshold.

[0029] if the first video frame sequence is not empty, deleting the first video frame added earliest in the first video frame sequence.

[0030] if the first video frame sequence is not empty, deleting the first video frame added earliest in the first video frame sequence.

[0031] Compared with the prior art, the embodiments of the application have the following beneficial effects: generating a first video frame sequence according to the order of video frames collected by a video device can effectively retain the time information features of dynamic gestures; at the same time, if the current video frame does not exist a hand key point, by deleting the earliest added video frame in the current first video frame sequence, it can avoid the existence of a large number of video frames with large time span in the same first video frame sequence, effectively guaranteeing the continuity of dynamic gesture changes in time and space, and improving the accuracy of dynamic gesture recognition.

[0032] In a second aspect, the embodiments of the application also provide a dynamic gesture recognition device based on an attention mechanism, comprising: a first video frame sequence acquisition module, a two-dimensional distance matrix acquisition module, and a gesture recognition result acquisition module.

[0033] The first video frame sequence acquisition module is configured to collect a preset number of first video frames containing hand key points in time sequence through a video acquisition device, and acquire a first video frame sequence.

[0034] The two-dimensional distance matrix acquisition module is configured to traverse the first video frame sequence, calculate the Euclidean distance between each hand key point in each first video frame, and acquire a two-dimensional distance matrix.

[0035] The gesture recognition result acquisition module is configured to input the two-dimensional distance matrix into a preset gesture recognition model to acquire a gesture recognition result; the gesture recognition model is built by connecting a lightweight convolutional neural network and a bidirectional long short-term memory network through an attention mechanism module.

[0036] In some embodiments of the second aspect of the application, the gesture recognition result acquisition module is configured to input the two-dimensional distance matrix into a preset gesture recognition model to acquire a gesture recognition result, comprising:

[0037] The lightweight convolutional neural network is configured to perform feature extraction on the two-dimensional distance matrix to acquire first feature information.

[0038] The attention mechanism module is configured to perform feature enhancement on the first feature information to acquire second feature information.

[0039] The bidirectional long short-term memory network model is configured to perform feature extraction on the second feature information to acquire third feature information.

[0040] The third feature information is mapped to the gesture recognition result by a full connection layer.

[0041] In some embodiments of the second aspect of the application, the feature extraction on the two-dimensional distance matrix by the lightweight convolutional neural network to obtain first feature information comprises:

[0042] The lightweight convolutional neural network comprises a plurality of convolution kernels of different scales.

[0043] The spatial information features of the two-dimensional distance matrix under different scale convolution operations are extracted by the convolution kernels of different scales.

[0044] All the spatial information features are fused to obtain the first feature information.

[0045] In some embodiments of the second aspect of the application, the two-dimensional distance matrix acquisition module is configured to traverse the sequence of first video frames, calculate the Euclidean distances between the hand key points in each of the first video frames, and obtain a two-dimensional distance matrix, comprising:

[0046] All the hand key points in each of the first video frames are sorted according to the spatial order of the hand key points.

[0047] The first Euclidean distances between any two of the hand key points in each of the first video frames and the second Euclidean distances between each of the hand key points and a preset origin point in each of the first video frames are calculated.

[0048] The two-dimensional distance matrix is obtained according to all the first Euclidean distances and all the second Euclidean distances corresponding to each of the first video frames.

[0049] In some embodiments of the second aspect of the application, the sequence of first video frames acquisition module is configured to acquire a preset number of first video frames containing hand key points in time sequence by a video acquisition device, and obtain a sequence of first video frames, comprising:

[0050] It is sequentially determined whether the video frames currently acquired by the video acquisition device contain the hand key points until the length of the sequence of first video frames meets a first preset threshold.

[0051] If yes, the video frame currently acquired is determined as a first video frame, and the first video frame is added to the sequence of first video frames.

[0052] If no and the sequence of first video frames is not empty, the first video frame added earliest in the sequence of first video frames is deleted. BRIEF DESCRIPTION OF DRAWINGS

[0053] FIG. 1 is a flowchart of a dynamic gesture recognition method based on an attention mechanism according to some embodiments of the present application;

[0054] FIG. 2 is a flowchart of a first video sequence acquisition method according to some embodiments of the present application;

[0055] FIG. 3 is a data processing flowchart of a gesture recognition model according to some embodiments of the present application;

[0056] FIG. 4 is a structural diagram of a dynamic gesture recognition device based on an attention mechanism according to some embodiments of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0058] Embodiment One

[0059] Referring to FIG. 1, a dynamic gesture recognition method based on an attention mechanism according to some embodiments of the present application includes S10 to S30, specifically:

[0060] S10: acquiring a first video frame sequence by acquiring a preset number of first video frames containing hand key points in time sequence through a video acquisition device.

[0061] In some embodiments of the present application, the first video frame sequence is acquired by acquiring a preset number of first video frames containing hand key points in time sequence through a video acquisition device, including:

[0062] sequentially determining whether the video frame currently acquired by the video acquisition device contains the hand key point until the length of the first video frame sequence meets a first preset threshold;

[0063] If yes, the video frame currently acquired is determined as the first video frame, and the first video frame is added to the first video frame sequence.

[0064] If no and the first video frame sequence is not empty, the first video frame added earliest in the first video frame sequence is deleted.

[0065] According to the order in which the video device collects video frames, the first video frame sequence is generated, which can effectively retain the time information characteristics of dynamic gestures; at the same time, if the current video frame does not exist hand key points, by deleting the earliest added video frame in the current first video frame sequence, it can avoid the existence of a large number of video frames with large time span in the same first video frame sequence, which can effectively guarantee the continuity of dynamic gesture changes in time and space, and improve the accuracy of dynamic gesture recognition.

[0066] Preferably, referring to FIG. 2, in some embodiments of the present application, step S10 can be executed by the following preferred embodiments:

[0067] S11: Turn on any type of camera device, and continuously collect real-time video frames in time sequence; the video frames can be continuous video frames extracted from the video shot by the camera device, or continuous images obtained by quickly shooting the camera device at a predetermined time interval, such as 30 images per second for a 30-frame video. The present application does not limit the acquisition of video frames, but requires that the video frames remain continuous;

[0068] S12: Process the video frames obtained in S11 by the key point extraction module, and determine whether there are hand key points in the current video frame; the hand key points include: fingertips, knuckles, wrists, etc.

[0069] S13: If there are, add the current video frame to the first video frame sequence; otherwise, delete the old frame from the first video frame sequence (if the current video frame obtained in S11 is the first video frame, no processing is required for this video frame, and jump to S12), until the video frames of the first video sequence reach 50 frames. The present application does not limit the number of frames, which can be reduced or increased according to the actual gesture recognition requirements;

[0070] S14: Determine whether the number of key points meets the predetermined requirements; if so, input the first video frame sequence into the gesture recognition model, and clear the first video frame sequence to obtain the final gesture recognition result; otherwise, jump to S12.

[0071] Preferably, the definition of the old frame in step S13 can be: in the process of reading video frames in time sequence, the video frame being detected is a new frame, and the video frame that detects hand key points before the new frame is an old frame.

[0072] Further, based on the above definition of the old frame, when performing the old frame deletion operation in step S13, only the earliest obtained old frame can be deleted.

[0073] S20: traversing the first video frame sequence, calculating the Euclidean distance between each hand key point in each first video frame, and obtaining a two-dimensional distance matrix.

[0074] In some embodiments of the present application, the step of traversing the first video frame sequence, calculating the Euclidean distance between each hand key point in each first video frame, and obtaining a two-dimensional distance matrix comprises:

[0075] sequencing all hand key points in each first video frame according to the spatial order of the hand key points;

[0076] calculating the first Euclidean distance between any two hand key points in each first video frame, and the second Euclidean distance between each hand key point in each first video frame and a preset origin;

[0077] obtaining the two-dimensional distance matrix according to all the first Euclidean distances and all the second Euclidean distances corresponding to each first video frame.

[0078] By calculating the Euclidean distance between hand key points, the spatial information features of the gesture can be clearly represented, which is beneficial for the subsequent convolutional neural network to extract effective spatial information feature vectors of dynamic gestures, and the generated two-dimensional distance matrix is obtained by traversing the first video frame sequence obtained in time sequence and calculating the Euclidean distance, so the two-dimensional distance matrix actually also contains the features of dynamic gestures in the time dimension, thus the efficiency of subsequent data processing can be improved.

[0079] Preferably, step S20 can be executed by the following preferred embodiments, specifically:

[0080] Arranging the hand key point coordinates of each video frame in the first video sequence in spatial order, calculating the Euclidean distance between any two hand key point coordinates of each video frame, and the Euclidean distance between all hand key points and the origin, a first two-dimensional distance matrix containing all key point pairs can be obtained; wherein the spatial order can be the order of each hand key point prepared in advance, so as to avoid the existence of Euclidean distance data corresponding to different hand key points in each row of the flattened one-dimensional matrix when splicing the first two-dimensional distance matrix. In order to avoid repeated information, the upper triangular elements in the first two-dimensional distance matrix are used to construct the feature vector.

[0081] wherein the Euclidean distance calculation formula of any two points is:

[0082] ,

[0083] ,

[0084] In the formula, representing the first hand key points in the video frame and distance; representing the first hand key points in the video frame between the origin and the Euclidean distance.

[0085] Further, according to the Euclidean distance calculation formula, the Euclidean distance is obtained, and a first two-dimensional distance matrix corresponding to the current first video frame is generated:

[0086]

[0087] wherein, representing the first video frame corresponding to the first two-dimensional distance matrix, in order to avoid information duplication, only the upper triangular elements are retained to construct a second two-dimensional distance matrix.

[0088] Finally, the first two-dimensional distance matrix corresponding to each video frame is flattened into a one-dimensional vector, and then the corresponding one-dimensional vectors are sequentially spliced in the order of the video frames obtained, to obtain a second two-dimensional distance matrix.

[0089] S30: inputting the two-dimensional distance matrix into a preset gesture recognition model to obtain a gesture recognition result; the gesture recognition model is built by connecting a lightweight convolutional neural network and a bidirectional long short-term memory network through an attention mechanism module.

[0090] In some embodiments of the present application, the inputting the two-dimensional distance matrix into the preset gesture recognition model to obtain the gesture recognition result comprises:

[0091] extracting features of the two-dimensional distance matrix through the lightweight convolutional neural network to obtain first feature information;

[0092] enhancing the first feature information through the attention mechanism module to obtain second feature information;

[0093] extracting features of the second feature information through the bidirectional long short-term memory network model to obtain third feature information;

[0094] mapping the third feature information to the gesture recognition result through a fully connected layer.

[0095] The feature extraction is performed through a lightweight convolutional neural network, which can improve the efficiency of the spatial information feature extraction process, so that the hand recognition model can be built on a lightweight hardware facility, and the real-time performance of the dynamic gesture recognition is improved; through the attention mechanism module, the attention degree of important features is enhanced, and the enhanced features are input into the bidirectional long short-term memory network model according to the order in which the corresponding video frames are obtained, the feature information of the dynamic gesture in the time dimension is extracted, so that the feature information containing the time and space dimension information is obtained, and the accuracy of the subsequent gesture recognition result is improved.

[0096] In some embodiments of the present application, the feature extraction of the two-dimensional distance matrix through the lightweight convolutional neural network to obtain the first feature information comprises:

[0097] The lightweight convolutional neural network comprises a plurality of convolution kernels of different scales.

[0098] The spatial information features of the two-dimensional distance matrix under different scale convolution operations are extracted through the convolution kernels of different scales.

[0099] All the spatial information features are fused to obtain the first feature information.

[0100] Different levels of feature information are captured through the convolution kernels of different scales, and the feature information of different levels is fused to obtain more generalized features, thereby reducing the risk of overfitting of the subsequent dynamic gesture recognition result.

[0101] Preferably, referring to FIG. 3, step S30 can be performed through the following preferred implementation, specifically:

[0102] S31: inputting the two-dimensional distance matrix obtained in S20 into a lightweight multi-scale convolution module, specifically, inputting into a 1*1 convolution kernel, a 3*1 convolution kernel and a 5*1 convolution kernel respectively, and extracting spatial information features of different levels; here, any means of lightweight convolution can be used, and the present application does not limit this technical feature;

[0103] S32: through a Concat feature fusion operation, the spatial information features of different levels extracted in S31 are spliced in the channel to obtain first feature information;

[0104] S33: the first feature information obtained in S32 is processed by a CBAM (Convolutional Block Attention Module) attention mechanism to enhance the attention degree of important features, and second feature information after feature enhancement is obtained;

[0105] S34: The second feature information obtained in S33 is input into a BiLSTM (Bi-directional Recurrent Neural Network) model in the order of the acquisition time of the first two-dimensional distance matrix corresponding to the video frame in the preferred embodiment of S20, to extract third feature information containing spatial and temporal changes;

[0106] S35: The third feature information obtained in S34 is mapped to the final gesture recognition result through a plurality of fully connected modules.

[0107] In summary, the dynamic gesture recognition method based on the attention mechanism provided in the embodiments of the application has the following beneficial effects: by generating a video frame sequence composed of video frames containing hand key points in time sequence, the temporal information features in the dynamic gesture change process are retained, and then based on the Euclidean distance between the hand key points, a two-dimensional distance matrix containing hand spatial information features is extracted. At this time, the two-dimensional distance matrix contains the feature information of the dynamic gesture change in space-time, and then the spatial information features in the two-dimensional distance matrix are extracted through a lightweight convolutional neural network, and further through an attention mechanism, the feature data of the part that needs to be focused on in the spatial information features is strengthened. Since the bidirectional long short-term memory network can process sequence data and maintain long-term memory, and the two-dimensional distance matrix is obtained from the video frame sequence sorted in time sequence, therefore, the strengthened spatial information features are input into the bidirectional long short-term memory network, the space-time feature information containing the temporal information can be extracted from the spatial information features, thereby improving the accuracy of subsequent dynamic gesture recognition.

[0108] Embodiment Two

[0109] Referring to FIG. 4, the dynamic gesture recognition device based on the attention mechanism provided in the embodiments of the application includes a first video frame sequence acquisition module 11, a two-dimensional distance matrix acquisition module 12, and a gesture recognition result acquisition module 13.

[0110] In some embodiments of the application, the first video frame sequence acquisition module 11 is configured to acquire a first video frame sequence by acquiring a preset number of first video frames containing hand key points in time sequence through a video acquisition device; the two-dimensional distance matrix acquisition module 12 is configured to traverse the first video frame sequence, calculate the Euclidean distance between each hand key point in each first video frame, and obtain a two-dimensional distance matrix; and the gesture recognition result acquisition module 13 is configured to input the two-dimensional distance matrix into a preset gesture recognition model to obtain a gesture recognition result; and the gesture recognition model is built by connecting a lightweight convolutional neural network and a bidirectional long short-term memory network through an attention mechanism module.

[0111] In some embodiments of the present application, the gesture recognition result acquisition module 13 is configured to input the two-dimensional distance matrix into a preset gesture recognition model to obtain a gesture recognition result, including: extracting features of the two-dimensional distance matrix through the lightweight convolutional neural network to obtain first feature information; enhancing the features of the first feature information through the attention mechanism module to obtain second feature information; extracting features of the second feature information through a bidirectional long short-term memory network model to obtain third feature information; and mapping the third feature information to the gesture recognition result through a fully connected layer.

[0112] In some embodiments of the present application, the extracting features of the two-dimensional distance matrix through the lightweight convolutional neural network to obtain the first feature information includes: the lightweight convolutional neural network includes a plurality of convolution kernels of different scales; the spatial information features of the two-dimensional distance matrix under different scale convolution operations are extracted through the convolution kernels of different scales; and all the spatial information features are fused to obtain the first feature information.

[0113] In some embodiments of the present application, the two-dimensional distance matrix acquisition module 12 is configured to traverse the first video frame sequence, calculate the Euclidean distances between the hand key points in each of the first video frames, and obtain a two-dimensional distance matrix, including: sorting all the hand key points in each of the first video frames according to the spatial order of the hand key points; calculating the first Euclidean distances between any two of the hand key points in each of the first video frames, and the second Euclidean distances between each of the hand key points and a preset origin point; and obtaining the two-dimensional distance matrix according to all the first Euclidean distances and all the second Euclidean distances corresponding to each of the first video frames.

[0114] In some embodiments of the present application, the first video frame sequence acquisition module 11 is configured to acquire a preset number of first video frames containing hand key points in time sequence through a video acquisition device to obtain a first video frame sequence, including: sequentially judging whether the video frame currently acquired by the video acquisition device contains the hand key points until the length of the first video frame sequence meets a first preset threshold; if yes, determining that the currently acquired video frame is a first video frame and adding the first video frame to the first video frame sequence; and if no and the first video frame sequence is not empty, deleting the earliest added first video frame in the first video frame sequence.

[0115] In summary, the embodiment of the application provides a dynamic gesture recognition device based on an attention mechanism, which has the following beneficial effects: by generating a video frame sequence composed of video frames containing hand key points in chronological order, time information features in the dynamic gesture change process are retained, and then a two-dimensional distance matrix containing hand spatial information features is extracted based on the Euclidean distance between the hand key points. At this time, the two-dimensional distance matrix contains the feature information of the dynamic gesture change in space-time, and then the spatial information features in the two-dimensional distance matrix are extracted through a lightweight convolutional neural network, and further through the attention mechanism, the feature data of the part that needs to be focused on in the spatial information features is strengthened. Since the bidirectional long short-term memory network can process sequence data and maintain long-term memory, and the two-dimensional distance matrix is obtained from the video frame sequence sorted in chronological order, therefore, the strengthened spatial information features are input into the bidirectional long short-term memory network, the space-time feature information containing time information can be extracted from the spatial information features, thereby improving the accuracy of subsequent dynamic gesture recognition.

[0116] Embodiment three

[0117] On the basis of the above-mentioned embodiment of the dynamic gesture recognition method based on the attention mechanism, another embodiment of the application provides a dynamic gesture recognition terminal device based on the attention mechanism. The dynamic gesture recognition terminal device based on the attention mechanism includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the dynamic gesture recognition method based on the attention mechanism of any embodiment of the application is realized.

[0118] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the application. The one or more modules can be a series of computer program instructions for describing the execution process of the computer program in the dynamic gesture recognition device based on the attention mechanism.

[0119] The dynamic gesture recognition device based on the attention mechanism can be a desktop computer, a notebook, a palm computer, and a cloud server, etc. The dynamic gesture recognition terminal device based on the attention mechanism can include, but is not limited to, a processor and a memory.

[0120] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor is a control center of the dynamic gesture recognition device based on the attention mechanism, and connects various parts of the dynamic gesture recognition device based on the attention mechanism through various interfaces and lines. The memory can be used to store the computer programs and / or modules, and the processor realizes various functions of the dynamic gesture recognition device based on the attention mechanism by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required by a function, etc. The data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a nonvolatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0121] Embodiment Four

[0122] On the basis of the above-mentioned embodiments of the dynamic gesture recognition method based on the attention mechanism, another embodiment of the present application provides a storage medium including a stored computer program, wherein when the computer program runs, the device where the storage medium is located executes the dynamic gesture recognition method based on the attention mechanism of any one of the embodiments of the present application.

[0123] In this embodiment, the storage medium described above is a computer readable storage medium, the computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0124] The specific embodiments described above further illustrate the purposes, technical solutions and beneficial effects of the present application. It should be understood that the above description is only for specific embodiments of the present application and is not intended to limit the protection scope of the present application. It is particularly pointed out that any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A dynamic gesture recognition method based on an attention mechanism, characterized in that, The application comprises the following steps: acquiring a first video frame sequence by collecting a preset number of first video frames containing hand key points in time sequence through a video acquisition device; traversing the first video frame sequence to calculate the Euclidean distance between each hand key point in each first video frame and obtain a two-dimensional distance matrix; inputting the two-dimensional distance matrix into a preset gesture recognition model to obtain a gesture recognition result; the gesture recognition model is built by connecting a lightweight convolutional neural network and a bidirectional long short-term memory network through an attention mechanism module. 2.The dynamic gesture recognition method based on attention mechanism of claim 1, wherein, The gesture recognition result is obtained by inputting the two-dimensional distance matrix into the preset gesture recognition model, which comprises the following steps: extracting features of the two-dimensional distance matrix through the lightweight convolutional neural network to obtain first feature information; enhancing the first feature information through the attention mechanism module to obtain second feature information; extracting features of the second feature information through a bidirectional long short-term memory network model to obtain third feature information; mapping the third feature information to the gesture recognition result through a fully connected layer. 3.The dynamic gesture recognition method based on attention mechanism of claim 2, wherein, The first feature information is obtained by extracting features of the two-dimensional distance matrix through the lightweight convolutional neural network, which comprises the following steps: the lightweight convolutional neural network comprises a plurality of convolution kernels of different scales; extracting spatial information features of the two-dimensional distance matrix under different scale convolution operations through convolution kernels of different scales; fusing all the spatial information features to obtain the first feature information. 4.The dynamic gesture recognition method based on attention mechanism of claim 1, wherein, The two-dimensional distance matrix is obtained by traversing the first video frame sequence to calculate the Euclidean distance between each hand key point in each first video frame, which comprises the following steps: sorting all the hand key points in each first video frame according to the spatial order of the hand key points; calculating the first Euclidean distance between any two hand key points in each first video frame and the second Euclidean distance between each hand key point and a preset origin point in each first video frame; obtaining the two-dimensional distance matrix according to all the first Euclidean distances and all the second Euclidean distances corresponding to each first video frame.

5. The dynamic gesture recognition method based on attention mechanism according to claim 1, wherein, The first video frame sequence is obtained by collecting a preset number of first video frames containing hand key points in time sequence through a video acquisition device, which comprises the following steps: judging whether the video frames collected by the video acquisition device currently exist hand key points in sequence until the length of the first video frame sequence meets a first preset threshold; if yes, the video frame currently collected is determined as a first video frame, and the first video frame is added to the first video frame sequence; if no and the first video frame sequence is not empty, the first video frame added earliest in the first video frame sequence is deleted.

6. An attention mechanism based dynamic gesture recognition apparatus, characterized in that, The application comprises the following steps: a first video frame sequence acquisition module, a two-dimensional distance matrix acquisition module and a gesture recognition result acquisition module; the first video frame sequence acquisition module is used to acquire a first video frame sequence by collecting a preset number of first video frames containing hand key points in time sequence through a video acquisition device. The two-dimensional distance matrix acquisition module is used to traverse the first video frame sequence, calculate the Euclidean distance between each hand key point in each first video frame, and obtain a two-dimensional distance matrix. The gesture recognition result acquisition module is used to input the two-dimensional distance matrix into a preset gesture recognition model to obtain the gesture recognition result; the gesture recognition model is constructed by connecting a lightweight convolutional neural network and a bidirectional long short-term memory network through an attention mechanism module.

7. The dynamic gesture recognition device based on attention mechanism according to claim 6, wherein, The gesture recognition result acquisition module is used to input the two-dimensional distance matrix into a preset gesture recognition model to obtain gesture recognition results, including: The lightweight convolutional neural network is used to extract features from the two-dimensional distance matrix to obtain first feature information; The attention mechanism module is used to enhance the first feature information to obtain the second feature information. The second feature information is extracted using a bidirectional long short-term memory network model to obtain the third feature information; The third feature information is mapped to the gesture recognition result through a fully connected layer.

8. The dynamic gesture recognition device based on attention mechanism according to claim 7, wherein, The step of extracting features from the two-dimensional distance matrix using the lightweight convolutional neural network to obtain first feature information includes: The lightweight convolutional neural network includes: several convolutional kernels of different scales; By using convolution kernels of different scales, spatial information features of the two-dimensional distance matrix under convolution operations of different scales are extracted; By fusing all the aforementioned spatial information features, the first feature information is obtained.

9. The dynamic gesture recognition device based on attention mechanism according to claim 6, wherein, The two-dimensional distance matrix acquisition module is used to traverse the first video frame sequence, calculate the Euclidean distance between each of the hand keypoints in each of the first video frames, and acquire a two-dimensional distance matrix, including: Sort all the hand key points in each of the first video frames according to the spatial order of the hand key points; Calculate the first Euclidean distance between any two hand key points in each of the first video frames, and the second Euclidean distance between each hand key point in each of the first video frames and a preset origin. The two-dimensional distance matrix is ​​obtained based on all the first Euclidean distances and all the second Euclidean distances corresponding to each first video frame.

10. The dynamic gesture recognition device based on attention mechanism according to claim 6, wherein, The first video frame sequence acquisition module is used to acquire a preset number of first video frames containing key hand points in chronological order using a video acquisition device, thereby acquiring a first video frame sequence, including: The system sequentially determines whether the hand key point exists in the video frame captured by the video capture device until the length of the first video frame sequence meets the first preset threshold. If it exists, the currently acquired video frame is determined to be the first video frame, and the first video frame is added to the first video frame sequence; If it does not exist and the first video frame sequence is not empty, then delete the earliest first video frame added in the first video frame sequence.

Images