A dynamic gesture recognition method, system, device and medium

Abstract

The application discloses a dynamic gesture recognition method, system, device and medium, and relates to the technical field of gesture recognition. The method comprises the following steps: acquiring a to-be-recognized gesture video; inputting the to-be-recognized gesture video into a dynamic gesture recognition model for recognition to obtain a predicted category of the to-be-recognized gesture video; the dynamic gesture recognition model comprises an image encoder, a text encoder and a similarity calculator; the image encoder comprises a feature extraction module and a feature fusion module; the feature extraction module is obtained through knowledge distillation training by taking an I3D network as a teacher model and taking a ResNet network as a student model; and the feature fusion module is obtained based on a coordinate attention mechanism. The application can solve the problems of complex background interference, model dependence on labeled data, large calculation amount and poor real-time performance in dynamic gesture recognition, and improve the recognition performance of dynamic gesture recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of gesture recognition, in particular to a dynamic gesture recognition method, system, device and medium. BACKGROUND

[0002] Gesture is the most common way of communication for human, which conveys information by using fingers, arms and other parts of the body to make actions with specific meanings, and to a certain extent, it helps to understand language. In the past, the recognition of gesture actions generally only occurred between people, and with the development of technology and the increasing application demand of people on intelligent devices, at present, recognizing gesture actions by machines is a popular research direction. The main task of gesture recognition based on deep learning is to extract features from images or videos, and then classify or recognize each sample as a specific label to understand human motion. Through gesture input, the communication and interaction between people and computers no longer need other media, and people can use some pre-defined and appropriate gestures to control the machine, thereby reducing the complexity of interaction and effectively improving the interaction experience of people.

[0003] Gesture recognition can be divided into static and dynamic according to whether the recognized object is a picture or a video. Static gesture recognition is to recognize a single image, only considering the spatial information at a certain time, ignoring the information on the time sequence, so the information contained in static gesture is limited. Dynamic gesture recognition refers to recognizing the motion trajectory of a gesture on a time sequence, which is to recognize a video sequence, increasing the time information and action features. Compared with static gesture recognition, dynamic gesture recognition has more diversified information, is more in line with application and practicality, has strong practical significance, and has great research value. Since dynamic gesture recognition technology can naturally realize human-computer interaction, dynamic gesture recognition has a wide range of applications in smart home, automatic driving, virtual reality, aseptic medical surgery and other fields.

[0004] In dynamic gesture recognition, the current method often targets public data sets, but there are inevitably influences of light, skin color, clothing and the like in the shooting process of public data sets, which causes background interference problems. Dynamic gesture categories are various, not all categories have a large number of labeled instance data available for use, and the labeling of a large amount of data is very time-consuming and labor-intensive, which causes the problem of dependence of model performance on labeled data. The training process uses a complex model, a large amount of computing resources, and is convenient for extracting information from very large and highly redundant data sets, but the large model is not convenient to deploy to the service, not only the inference speed is slow, but also the deployment resource requirement is high (memory, video memory, etc.), therefore, the problems of large amount of calculation and poor real-time performance occur. Background interference, dependence on labeled data and large amount of calculation and poor real-time performance are important problems to be solved in the field of gesture recognition.

[0005] Background interference refers to that in the dynamic gesture video, the effective information of the key region such as hand movement and motion trajectory accounts for a small proportion, and other interference factors such as background, light, skin color and dressing account for a large proportion, resulting in poor final gesture recognition accuracy. Relying on labeled data refers to that the model based on deep learning relies on supervised learning, and its performance depends largely on the labeled training data. In addition, the model is limited to recognizing the categories seen during training, and in actual situations, not all categories have a large number of labeled instance data available for use, so these models are not necessarily practical. Large amount of calculation and poor real-time performance refer to that the deep neural network needs to store a large number of parameters, and in order to improve the training efficiency in a large-scale training set, a large memory storage is needed to meet the training data sent in large batches each time for training. In addition, a large-scale training set not only needs a large memory, but also needs a powerful computing device to speed up the training and testing speed.

[0006] In view of the problems of gesture background interference, model relying on labeled data and large amount of network calculation and poor real-time performance, the current main solving strategies include traditional artificial feature-based method and deep learning-based method.

[0007] The traditional dynamic gesture recognition method usually needs to accurately detect the gesture region in the scene from the image and segment and extract the gesture key point or hand region, and then manually design the spatiotemporal feature or model the time sequence information, and the processing process is relatively complex. Hand detection is an important preprocessing link in gesture recognition, which can mine important gesture information and remove background interference. Early hand detection methods mainly use manually extracted features to obtain the hand region in the picture, such as skin color-based method, shape-based method, motion information-based method, etc. These methods are easily affected by light changes, skin color differences, posture changes, finger self-occlusion, etc., and the detection effect is not ideal and unstable, and the algorithm complexity is high and the calculation speed is slow, which is difficult to meet the detection requirements in actual scenes, and the improvement of gesture recognition performance is very limited.

[0008] With the continuous development of convolutional neural network, using deep learning to realize dynamic gesture recognition based on video has become the mainstream. Convolutional neural network has strong feature extraction ability and high robustness, and can learn the spatiotemporal features of gestures from the data set autonomously without human intervention. In view of the above problems, people have studied many solutions, and from different angles, gestures are analyzed and processed to improve the accuracy of gesture recognition.

[0009] A common method is to fuse the time domain information and spatial domain information of dynamic gesture video. Researchers have successively proposed video feature extraction architectures based on dual-stream method, 3D convolution structure and long short-term memory (LSTM) network. The dual-stream method is to process two input streams of spatial stream of RGB image and time domain motion stream using dense optical flow map respectively, and then fuse the results to realize behavior recognition task. In view of the problem that the dual-stream network has greater time sequence dependence, some people propose a human behavior recognition algorithm based on improved dual-stream space-time network, which enhances the feature expression ability of the network to improve the recognition ability of time sequence dominant behavior. Some people combine 2D convolution and LSTM structure, first use 2D convolution to extract features from images to obtain a sequence of visual features, and then input the feature sequence directly into LSTM to further mine context information. Some people use 3D convolution in the field of video analysis, but only use 3D convolution in the shallow layer, and use manual method to extract image gray scale, gradient and other information, which cannot realize real-time processing of video. Some people introduce 3D convolution into the dual-stream framework and propose I3D architecture, which matches all points in the image one by one using dense optical flow to form an optical flow field. This method has good video target detection accuracy, but the calculation amount is too large. Some researches fuse channel and spatial attention mechanism CBAM into I3D network, which effectively improves the accuracy of dynamic gesture recognition. Although large-scale convolution kernel is introduced in CBAM to extract spatial features, the long-range dependence problem is ignored. Convolutional neural network applied to dynamic gesture recognition has obtained good recognition effect, but it is time-consuming and easy to overfit, so a large amount of data is needed to train the model, which requires high performance of storage space and hardware equipment. In addition, the annotation of a large amount of data used for training is very time-consuming and labor-intensive, and not all categories have a large number of annotated instance data available for use.

[0010] Since the current technology is mainly based on traditional artificial features and deep learning-based dynamic gesture recognition methods, the following will briefly describe several existing dynamic gesture recognition methods.

[0011] (1) The classic method based on artificial features includes regular dynamic time (DTW) algorithm and hidden Markov model (HMM).

[0012] DTW is a method to measure the similarity of two different length time series, which was first used in speech recognition, and then used in dynamic gesture recognition. When using DTW to recognize dynamic gestures, first, each gesture in the training set needs to be preprocessed, then the features are extracted and normalized into a sequence template, and then the test gesture is processed in the same way as the training gesture. Finally, the generated result is matched with each template in the training set, and the class of the template with the smallest distance is taken as the gesture recognition result. DTW method does not use statistical model framework for training, and it is also difficult to use various knowledge of context in image recognition algorithm, so it has disadvantages in solving large data, complex gestures and other problems.

[0013] In order to solve this problem, many people have improved DTW. Someone proposed an improved dynamic time warping (iDTW) dynamic gesture recognition algorithm, which uses a combination of points and lines to constrain the search path to prevent unreasonable search based on DTW, and uses the distance variance of nodes in the motion sequence to dynamically assign weights to each node. Compared with DTW, iDTW improves the accuracy on public data sets as a whole, and significantly reduces the amount of calculation.

[0014] HMM is a classic model, which is mainly used to process problems based on time series or state sequence; there are two types of data in HMM-based algorithm, one is the data that can be observed, that is, the observation sequence; the other is hidden data, that is, the state sequence. In dynamic gesture recognition, a series of actions made by the hand is the observation sequence, while the action of the hand is the hidden sequence, and the task of HMM is to determine the content of the hidden data from a series of observable data. In dynamic gesture recognition, each type of gesture corresponds to a hidden Markov model; during training, first, each gesture sample is divided by category, and then the forward and backward algorithms are used to train the corresponding HMM model for each type of gesture. When testing, the test sample uses the forward algorithm to traverse all HMM models, and calculates the probability value of each HMM model producing the gesture sequence, and the class corresponding to the HMM model with the largest value is the recognition result of the test sample.

[0015] Although the dynamic gesture recognition algorithm based on artificial features has achieved some good results, it is difficult for researchers to design a feature extraction method suitable for all samples for different background environments and different motion gestures. There are still great challenges in dynamic gesture recognition based on artificial features.

[0016] (2) In recent years, with the rise of deep learning, dynamic gesture recognition based on deep learning has been widely concerned. The core idea is to first build a neural network, initialize the model, and then use the forward propagation, loss calculation, and backward feedback of the neural network to learn the network parameters from the data. Unlike artificial features, the processing method of the input is fixed, the parameters of the neural model are completely determined by the characteristics of the data and the structure of the neural network, which directly affects the performance of gesture recognition. Because the input data is fixed, building a robust neural network is the key to the entire deep learning algorithm. According to the encoding method of spatial information and temporal information, the current commonly used neural network can be divided into two categories: one is two stream network based on 2DCNNs, and the other is 3DCNNs.

[0017] The network based on two stream algorithm consists of two sub-networks of spatial network and temporal network, which are responsible for mining the spatial information of the hand from the RGB picture and the motion information of the hand from the stacked optical flow, respectively. After fusing the two kinds of information, the spatio-temporal information is used for video analysis tasks. The spatial network and the temporal network have the same structure, and classical 2DCNNs such as AlexNet, VGG, ResNet, and Inception can be used to construct them. With the successful application of two stream algorithm in behavior recognition, a large number of researchers at home and abroad have proposed dynamic gesture recognition based on two stream. 2DCNNs can only process video data in time domain and spatial domain, and it is difficult to extract features separately because time and space are intertwined. Therefore, researchers have proposed 3DCNNs to extract time domain and spatial domain information simultaneously in one network.

[0018] 3DCNNs can extract spatial and temporal information from sequences simultaneously, providing a good method for gesture recognition. However, inputting raw images alone is not enough to obtain enough features. Many researchers have begun to use convolutional neural networks and multi-stream input to obtain more information, such as optical flow information, depth information, and posterior information, which helps to improve the accuracy of network recognition. Drawing on the advantages of dual-flow networks and 3DCNNs, the Deep mind team proposed the I3D network, which extended the 2D convolution Inception v1 to a 3D structure, while the optical flow was separately formed as a branch and the raw image formed a dual-flow network. RGB images and optical flow were trained separately. Although 3D convolution can directly learn the temporal features of the video, it only performs pure forward propagation, while the optical flow algorithm provides an iterative idea to some extent. Adding optical flow can improve the recognition accuracy of the network. This method has good video target detection accuracy, but the computational load is too large. Some research has integrated the channel and spatial attention mechanism CBAM into the I3D network, effectively improving the accuracy of dynamic gesture recognition. Although large-scale convolution kernels are introduced in CBAM to extract spatial features, the long-range dependency problem is ignored. 3DCNNs applied to dynamic gesture recognition have achieved good recognition results, but they have poor processing capability for long sequence data and large parameters, and have high requirements for storage space and hardware device performance. In addition, the annotation of a large amount of data used for training is very time-consuming and labor-intensive, but not all classes have a large number of annotated instance data available for use, resulting in a model performance that is too dependent on labeled data.

[0019] (3) Dynamic gesture recognition using knowledge distillation

[0020] Knowledge distillation is one of the three main methods of compressing neural networks, making them suitable for less powerful hardware. Unlike the other two powerful compression methods, weight pruning and quantization, knowledge distillation does not directly reduce the network. Instead, it uses the original model to train a smaller model, called the "student model." The main idea of knowledge distillation is to train an efficient student model under the guidance of a teacher model to achieve comparable accuracy. Since the teacher model can even provide predictions for unlabeled data, the student model can learn how to generalize like a teacher. As a representative type of model compression and acceleration, knowledge distillation effectively learns a small student model from a large teacher model, which has quickly gained attention in the industry.

[0021] Knowledge distillation for model compression is similar to the way humans learn. Inspired by this, recent knowledge distillation methods have been extended to teacher-student learning, mutual learning, assisted teaching, lifelong learning, and self-learning. Most extensions of knowledge distillation focus on compressing deep neural networks. The resulting lightweight student networks can be easily deployed in applications such as visual recognition, speech recognition, and natural language processing.

[0022] With the emergence of knowledge distillation technology, a method for realizing gesture recognition estimation by using knowledge distillation also emerges. However, the existing gesture recognition network based on knowledge distillation only migrates the gesture recognition knowledge of the teacher network to the student network, and the student network is often not powerful enough to cause too much decline in recognition efficiency and low recognition accuracy. Therefore, how to efficiently apply the knowledge distillation technology to dynamic gesture recognition still needs further research.

[0023] In summary, the current gesture recognition often faces the problems of complex background interference, model dependence on labeled data, large amount of calculation and poor real-time performance, which will seriously reduce the accuracy of gesture recognition. SUMMARY

[0024] The purpose of the present application is to provide a dynamic gesture recognition method, system, device and medium to solve the problems of complex background interference, model dependence on labeled data, large amount of calculation and poor real-time performance in dynamic gesture recognition, and improve the recognition performance of dynamic gesture recognition.

[0025] To achieve the above purpose, the present application provides the following scheme:

[0026] A dynamic gesture recognition method, comprising:

[0027] Obtaining a to-be-recognized gesture video; the to-be-recognized gesture video comprises to-be-recognized true color data, to-be-recognized depth data and to-be-recognized optical flow data; the to-be-recognized optical flow data is determined based on the to-be-recognized true color data;

[0028] Inputting the to-be-recognized gesture video into a dynamic gesture recognition model for recognition to obtain a predicted category of the to-be-recognized gesture video; the dynamic gesture recognition model comprises an image encoder, a text encoder and a similarity calculator; the image encoder comprises a feature extraction module and a feature fusion module; the feature extraction module is obtained by knowledge distillation training with an I3D network as a teacher model and a ResNet network as a student model; and the feature fusion module is constructed based on a coordinate attention mechanism;

[0029] The feature extraction module is used for feature extraction on the to-be-recognized gesture video to obtain to-be-recognized true color features, to-be-recognized depth features and to-be-recognized optical flow features; the feature fusion module is used for feature fusion on the to-be-recognized true color features, the to-be-recognized depth features and the to-be-recognized optical flow features to obtain to-be-recognized image features; the text encoder is used for feature extraction on all category labels in a gesture recognition task data set to obtain corresponding text features; and the similarity calculator is used for calculating the similarity between the to-be-recognized image features and all text features, and taking the category label corresponding to the text feature with the highest similarity as the predicted category of the to-be-recognized gesture video.

[0030] Optionally, the training process of the image encoder comprises:

[0031] constructing a gesture recognition task dataset; the gesture recognition task dataset comprises a plurality of sample gesture videos and corresponding true class labels; the sample gesture video comprises sample true color data, sample depth data and sample optical flow data with uniform frame numbers;

[0032] constructing a teacher model based on an I3D network, and inputting the sample gesture video into the teacher model for feature extraction to obtain first sample true color features, first sample depth features and first sample optical flow features;

[0033] constructing a student model based on a ResNet network, and inputting the sample gesture video into the student model for feature extraction to obtain second sample true color features, second sample depth features and second sample optical flow features;

[0034] constructing a feature fusion network based on a coordinate attention mechanism;

[0035] inputting the first sample true color features, the first sample depth features and the first sample optical flow features into the feature fusion network for feature fusion to obtain first fused image features;

[0036] inputting the second sample true color features, the second sample depth features and the second sample optical flow features into the feature fusion network for feature fusion to obtain second fused image features;

[0037] determining a sample soft label according to the first fused image features;

[0038] determining a soft prediction label and a hard prediction label according to the second fused image features;

[0039] determining a total loss according to the sample soft label, the soft prediction label, the hard prediction label and a sample hard label; the sample hard label is a true class label of the sample gesture video;

[0040] performing distillation training with the minimum total loss as the target to obtain an image encoder.

[0041] Optionally, the total loss is determined according to the sample soft label, the soft prediction label and a sample hard label, and specifically comprises:

[0042] determining a distillation loss according to the sample soft label and the soft prediction label;

[0043] determining a student loss according to the hard prediction label and the sample hard label;

[0044] A total loss is determined according to the distillation loss and the student loss.

[0045] Optionally, a calculation formula of the total loss is:

[0046]

[0047] wherein, is a total loss, N is a training batch size, y i is a true class label of the i-th sample, is a student loss, is a distillation loss, is a hard prediction label output by the student model, is a soft prediction label output by the student model, is a sample soft label output by the teacher model, is a class probability distribution output by the student model, is a class probability distribution output by the teacher model, T is a temperature parameter of knowledge distillation, and sigma() represents a Softmax activation function, L CE () represents cross-entropy, I() represents an indicator function, the indicator function is 1 when the teacher model can correctly predict the class of the input sample, and the indicator function is 0 when the teacher model cannot correctly predict the class of the input sample.

[0048] Optionally, the I3D network comprises a first convolutional layer, a second convolutional layer, a third convolutional layer, two first 3D CA-Inceptionv1 structures, a first maximum pooling layer, five second 3D CA-Inceptionv1 structures, a second maximum pooling layer, two third 3D CA-Inceptionv1 structures, and a fourth convolutional layer connected in sequence; the 3D CA-Inceptionv1 structure comprises a 3D Inceptionv1 structure and a coordinate attention layer connected in sequence.

[0049] A dynamic gesture recognition system comprises:

[0050] A data acquisition module is configured to acquire a to-be-recognized gesture video; the to-be-recognized gesture video comprises to-be-recognized true color data, to-be-recognized depth data, and to-be-recognized optical flow data; the to-be-recognized optical flow data is determined based on the to-be-recognized true color data.

[0051] The gesture recognition module is configured to input the to-be-recognized gesture video into a dynamic gesture recognition model to perform recognition, and obtain a predicted category of the to-be-recognized gesture video; the dynamic gesture recognition model comprises an image encoder, a text encoder and a similarity calculator; the image encoder comprises a feature extraction module and a feature fusion module; the feature extraction module is obtained through knowledge distillation training by taking an I3D network as a teacher model and taking a ResNet network as a student model; and the feature fusion module is constructed based on a coordinate attention mechanism.

[0052] The feature extraction module is configured to perform feature extraction on the to-be-recognized gesture video to obtain to-be-recognized true color features, to-be-recognized depth features and to-be-recognized optical flow features; the feature fusion module is configured to perform feature fusion on the to-be-recognized true color features, the to-be-recognized depth features and the to-be-recognized optical flow features to obtain to-be-recognized image features; the text encoder is configured to perform feature extraction on all category labels in a gesture recognition task data set to obtain corresponding text features; and the similarity calculator is configured to calculate the similarity between the to-be-recognized image features and all text features, and take the category label corresponding to the text feature with the highest similarity as the predicted category of the to-be-recognized gesture video.

[0053] An electronic device comprises a memory and a processor, the memory is configured to store a computer program, and the processor is configured to run the computer program to enable the electronic device to perform the dynamic gesture recognition method.

[0054] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the dynamic gesture recognition method.

[0055] According to the embodiments of the present application, the following technical effects are achieved:

[0056] The application provides a dynamic gesture recognition method, which adopts a dynamic gesture recognition model to recognize the category of a to-be-recognized gesture video, and the dynamic gesture recognition model comprises an image encoder, a text encoder and a similarity calculator; the image encoder comprises a feature extraction module and a feature fusion module; the feature extraction module is obtained through knowledge distillation training by taking an I3D network as a teacher model and a ResNet network as a student model; and the feature fusion module is obtained based on a coordinate attention mechanism. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0058] Figure 1 The flowchart of the dynamic gesture recognition method provided by the present application;

[0059] Figure 2 The framework diagram of the dynamic gesture recognition method provided by the present application;

[0060] Figure 3 The schematic diagram of the knowledge distillation training provided by the present application;

[0061] Figure 4 The 3D Inception v1 structure schematic diagram provided by the present application;

[0062] Figure 5 The I3D network schematic diagram based on the 3D Inception v1 structure provided by the present application;

[0063] Figure 6A schematic diagram of the 3D CA-Inceptionv1 structure provided by this invention;

[0064] Figure 7 A schematic diagram of the I3D network based on the 3D CA-Inception v1 structure provided by this invention;

[0065] Figure 8 This is a schematic diagram of the three sets of cross-modal distillation training provided by the present invention;

[0066] Figure 9 A block diagram of the dynamic gesture recognition system provided by the present invention.

[0067] Symbol explanation:

[0068] Data acquisition module—1, gesture recognition module—2. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] The purpose of this invention is to provide a dynamic gesture recognition method, system, device, and medium to solve the problems of complex background interference, model dependence on labeled data, large computational load, and poor real-time performance in dynamic gesture recognition, thereby improving the recognition performance of dynamic gesture recognition.

[0071] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0072] Example 1

[0073] This embodiment provides a dynamic gesture recognition method, such as Figure 1 As shown, the dynamic gesture recognition method includes:

[0074] Step S1: Acquire the gesture video to be recognized; the gesture video to be recognized includes: true color data to be recognized, depth data to be recognized, and optical flow data to be recognized; the optical flow data to be recognized is determined based on the true color data to be recognized, for example: optical flow data is calculated based on the true color data using the energy equation proposed by Brox et al. based on the assumptions of brightness constancy, gradient constancy, and spatiotemporal smoothness constraints.

[0075] Step S2: inputting the to-be-identified gesture video into a dynamic gesture recognition model for identification to obtain a predicted category of the to-be-identified gesture video; the dynamic gesture recognition model comprises an image encoder, a text encoder and a similarity calculator; the image encoder comprises a feature extraction module and a feature fusion module; the feature extraction module is obtained through knowledge distillation training by taking an I3D network as a teacher model and a ResNet network as a student model; and the feature fusion module is obtained based on a coordinate attention mechanism.

[0076] The feature extraction module is configured to extract features of the to-be-identified gesture video to obtain to-be-identified true color features, to-be-identified depth features and to-be-identified optical flow features; the feature fusion module is configured to fuse the to-be-identified true color features, the to-be-identified depth features and the to-be-identified optical flow features to obtain to-be-identified image features; the text encoder is configured to extract features of all category labels in a gesture recognition task data set to obtain corresponding text features; and the similarity calculator is configured to calculate similarities between the to-be-identified image features and all text features, and take a category label corresponding to a text feature with the highest similarity as a predicted category of the to-be-identified gesture video.

[0077] Further, the training process of the image encoder comprises:

[0078] 1) constructing a gesture recognition task data set; the gesture recognition task data set comprises a plurality of sample gesture videos and corresponding real category labels; the sample gesture videos comprise sample true color data, sample depth data and sample optical flow data with uniform frame numbers.

[0079] 2) constructing a teacher model based on an I3D network, and inputting the sample gesture videos into the teacher model for feature extraction to obtain first sample true color features, first sample depth features and first sample optical flow features.

[0080] 3) constructing a student model based on a ResNet network, and inputting the sample gesture videos into the student model for feature extraction to obtain second sample true color features, second sample depth features and second sample optical flow features.

[0081] 4) constructing a feature fusion network based on a coordinate attention mechanism.

[0082] 5) inputting the first sample true color features, the first sample depth features and the first sample optical flow features into the feature fusion network for feature fusion to obtain first fused image features.

[0083] 6) inputting the second sample true color feature, the second sample depth feature and the second sample optical flow feature into the feature fusion network for feature fusion to obtain a second fusion image feature.

[0084] 7) determining a sample soft label according to the first fusion image feature.

[0085] 8) determining a soft prediction label and a hard prediction label according to the second fusion image feature.

[0086] 9) determining a total loss according to the sample soft label, the soft prediction label, the hard prediction label and a sample hard label; the sample hard label is a true class label of the sample gesture video.

[0087] Step 9) specifically comprises: determining a distillation loss according to the sample soft label and the soft prediction label; determining a student loss according to the hard prediction label and the sample hard label; and determining the total loss according to the distillation loss and the student loss.

[0088] 10) performing distillation training with the total loss being minimum to obtain an image encoder.

[0089] The calculation formula of the total loss is as follows:

[0090]

[0091] wherein, is a total loss, N is a training batch size, y i is a true class label of the i-th sample, is a student loss, is a distillation loss, is a hard prediction label output by the student model, is a soft prediction label output by the student model, is a sample soft label output by the teacher model, is a class probability distribution output by the student model, is a class probability distribution output by the teacher model, T is a temperature parameter of knowledge distillation, and σ() represents a Softmax activation function, L CE () represents cross-entropy, I() represents an indicator function, the indicator function is 1 when the teacher model can correctly predict the class of the input sample, and the indicator function is 0 when the teacher model cannot correctly predict the class of the input sample.

[0092] The dynamic gesture recognition method provided by the application mainly comprises three tasks, namely image feature extraction of a teacher model, image feature extraction of a student model and text feature extraction, as shown in the framework. Figure 2 The framework describes the implementation process and the relationship of the above three tasks.Figure 2 The upper part is an image feature extraction schematic diagram of the I3D network, the lower part is an image feature extraction schematic diagram of the ResNet network, and the middle part is a CLIP text feature extraction schematic diagram. The upper and lower parts are respectively used as a teacher network (i.e., a teacher model) and a student network (i.e., a student model) for knowledge distillation training, and are respectively combined with the middle part for image-text contrast learning. RGB data (i.e., true color data), depth data (i.e., depth data), and flow data (i.e., optical flow data) are used as input, and after image feature extraction, CA attention features are selected for fusion.

[0093] As a specific implementation, the specific process of the dynamic gesture recognition provided by the application includes:

[0094] Step 1: Obtain true color RGB data, depth Depth data, and optical flow Flow data according to the RGB data.

[0095] Step 2: Process the three kinds of data into a unified 32-frame video.

[0096] Step 3: Input the 32-frame RGB, Depth, and Flow data into the teacher model I3D and the student model ResNet structure for training, respectively to obtain RGB features, Depth features, and Flow features.

[0097] Step 4: Fuse the three kinds of features obtained by using a coordinate attention CA module, and obtain image features Image feature1 of the teacher model I3D and image features Image feature2 of the student model ResNet after fusion.

[0098] Step 5: Use the Image feature1 of the teacher model to assist in distilling the student model with real labels, respectively calculate the distillation loss and the student loss of the student model, and finally weight-sum the two losses as the final loss to update the network parameters. The trained teacher model and student model are used as image encoders of the CLIP visual language model, i.e., I3D image encoders and ResNet image encoders.

[0099] Step 6: According to all the class labels of the gesture data set, corresponding sentence text templates are made, which are input into the text encoder of the CLIP pre-trained to obtain all the text features Text feature.

[0100] Step 7: In the prediction recognition, given a gesture video, it is simultaneously input into the I3D image encoder and the ResNet image encoder, and the obtained image features Image feature1 and Image feature2 are respectively compared and learned with all text features, that is, the similarity between the image-text feature pairs is calculated, and the probability with the highest similarity is the classification category of the gesture video. After knowledge distillation training, the ResNet small model has the same prediction recognition performance as the I3D large model, but the number of parameters is greatly reduced, thereby realizing model compression and acceleration.

[0101] Figure 3 Further show the knowledge distillation training process of the teacher network and the student network, as shown in Figure 3 , the process is as follows:

[0102] 1) The left teacher network is a complex large model I3D, and the class probability value soft label (i.e. sample soft label) output by the Softmax activation function layer with temperature parameter T of the teacher network is used as the soft target for the student network to learn, which assists the student training.

[0103] 2) The right is the ResNet student network, which also predicts the probability distribution Softprediction (i.e. soft prediction label) through the Softmax activation function layer with temperature parameter T during learning, and calculates the distillation loss with the class probability value soft label output by the teacher network.

[0104] 3) At the same time, the student network also obtains the predicted sample category Hard prediction (i.e. hard prediction label) through the normal training process, and calculates the student loss with the real sample category Hard label (i.e. sample hard label).

[0105] Among them, the result of the network using the Softmax activation function is easy to be extreme, that is, the confidence of a certain class is very high, and the confidence of other classes is very low, at this time the positive class information that the student model pays attention to may still belong to a certain class. In addition, because the negative class information of different categories also has relative importance, it is not good to have similar negative class scores, which cannot achieve the purpose of knowledge distillation. In order to solve this problem, the concept of temperature (Temperature) is introduced, and high temperature is used to distill the information carried by small probability values. Specifically, the output value Logits (corresponding to Y) is divided by the temperature T before passing through the Softmax activation function. Neural networks usually use the Softmax output layer to produce classification probabilities, which converts the calculated Logits of each class into classification probabilities.

[0106] During distillation training, first, the knowledge learned by the teacher model is distilled to the student model. Specifically, for a sample x, the second-to-last layer of the teacher model is divided by a temperature T first, and then a class probability Softlabel is predicted through a Softmax activation function layer. The student model is the same, the second-to-last layer is divided by the same temperature T, and then a result Softprediction is predicted through a Softmax activation function layer. Then, the cross-entropy of the result and the Soft label is taken as the training distillation loss. Next, the cross-entropy of the normal output Hardprsdiction of the student model and the real label Hard label is taken as the training student loss. Then, the two losses are weighted and summed to serve as the final loss, i.e., the total loss, for training the student model.

[0107] The present application designs a whole-process dynamic gesture recognition strategy, solves the problems of complex background interference, model dependence on labeled data, large amount of calculation and low real-time network recognition performance in dynamic gesture recognition, and mainly improves as follows:

[0108] 1) Improve the I3D network structure and parameters: first, remove the first two maximum pooling layers to prevent the loss of low-level features of the image caused by the pooling operation; second, remove the last average pooling operation and only keep the 1x1x1 convolution. This is to preserve the global information of the image while reducing a large number of parameters, thereby increasing the robustness of the network. The network weights are randomly initialized with a standard normal distribution (mean 0, variance 1).

[0109] 2) Select the coordinate attention mechanism Coordinate Attention (CA) for feature fusion. CA can consider both the relationship between channels and the long-distance position information. This operation can improve the feature expression ability of the network model and enhance the network's attention to the effective information of the local area of the gesture, further reducing the background interference unrelated to the gesture.

[0110] 3) Improve the traditional knowledge distillation loss and add an indicator function to enable it to learn the correct soft target output by the teacher network, reduce the computational cost, and improve the real-time performance.

[0111] 4) A multi-granularity context prompt template integration method is proposed, and an improved CLIP visual language model (MG-CLIP) is constructed. Through image-text paired contrastive learning for Zero Shot inference, the generalization performance is improved, and the dependence on labeled data is reduced.

[0112] The four strategies above jointly build the entire dynamic gesture recognition process, ensuring the efficiency of the recognition result, and having the ability to solve the problems of complex background interference, model dependence on labeled data, and low network recognition performance in the case of large calculation amount and poor real-time performance. The following will describe the four improvement strategies in detail.

[0113] First, the I3D network structure is improved.

[0114] A sequence of different modalities carries different information. RGB pictures contain rich color and texture information, depth pictures contain object contour and depth information, and optical flow pictures contain object motion information. These data are complementary in information description, and their fusion can effectively make up for the limitations of single modal data. Therefore, the idea of three streams is considered to be added to the 3D convolution, in which one 3D convolution network receives RGB information, the second 3D convolution network receives optimized smooth optical flow information, and the other 3D convolution network receives depth information. The 3D convolution structure in the I3D network is extended from the 2D convolution Inception v1, as shown in Figure 4 The specific method is to first use 1x1x1 convolution of different branches to reduce the dimension of the input feature matrix, and then use 3x3x3 and 5x5x5 convolution kernels on different branches to extract multi-scale information. The receptive field of different size convolution kernels sees different features, which increases the sparsity of the network while being consistent with the behavior of multi-scale visual information extraction, so that the information extraction capability of the network is enhanced.

[0115] As a specific embodiment, the I3D network used by the application is improved on the basis of the traditional I3D network structure. The specific improvements are as follows:

[0116] 1) Remove the first two max pooling layers to prevent loss of low-level features of the image caused by the pooling operation.

[0117] 2) Remove the last average pooling operation and only keep the convolution. This is to preserve the global information of the image while reducing the number of parameters, thereby increasing the robustness of the network.

[0118] The specific connection mode of the network is shown in Figure 5 The network weights are randomly initialized with standard normal distribution. In this embodiment, the receptive field of different size convolution kernels sees different features, so the data is extracted through different convolution kernels to extract multi-scale image feature information. The max pooling layer is used to increase the receptive field and extract features, and finally a 1x1x1 3D convolution is used for dimension reduction to reduce the amount of calculation.

[0119] Second, feature fusion based on CA (Coordinate Attention) coordinate attention mechanism.

[0120] Introducing attention mechanism in neural network can improve the feature expression ability of network model. Attention can not only tell the network model what to pay attention to, but also enhance the representation ability of specific region of image. CA is an attention module that encodes channel relationship and long-term dependency through precise location information. Compared with SENet, CA is more comprehensive in attention, and compared with CBAM, CA considers not only the relationship between space and channel, but also the long-term dependency problem, which can not only achieve precision improvement, but also has less parameter quantity and calculation amount. Unlike channel attention that converts input into a single feature vector through 2D global pooling, CA decomposes channel attention into two 1D feature encoding processes that aggregate features along different directions. The advantage is that long-term dependency can be captured along one spatial direction, and precise location information is preserved along the other spatial direction. Then, the generated feature maps are respectively encoded to form a pair of direction-aware and location-sensitive feature maps, which can be applied to the input feature map complementarily to enhance the representation of the target of interest, thereby solving the problems of complex background and multiple interference factors of dynamic gesture.

[0121] CA is added after the channel splicing layer of the three-dimensional Inceptionv1 module of I3D, which can realize lossless transmission of original input information, and automatically learn the importance of image space position and channel, and then enhance useful features and suppress useless features according to the importance, so as to realize adaptive calibration of space and channel. The addition of CA attention mechanism has little effect on the network structure, but can make the network learn more important channel features and spatial positions in the image. The three-dimensional Inceptionv1 structure after adding CA is shown in Figure 6 The connection mode of I3D network after adding CA is shown in Figure 7

[0122] As a specific implementation, the I3D network used in the application comprises a first convolutional layer, a second convolutional layer, a third convolutional layer, two first 3D CA-Inceptionv1 structures, a first maximum pooling layer, five second 3D CA-Inceptionv1 structures, a second maximum pooling layer, two third 3D CA-Inceptionv1 structures and a fourth convolutional layer connected in sequence; the 3D CA-Inceptionv1 structure comprises a 3D Inceptionv1 structure and a coordinate attention layer connected in sequence.

[0123] Thirdly, the traditional knowledge distillation loss is improved.

[0124] The process of knowledge distillation training is divided into two steps: ​

[0125] 1) Train a teacher model using training data, save the parameters of the model.

[0126] 2) The basic idea of knowledge distillation is to make the student network approximate the teacher network by minimizing the difference between the predictive distributions of the teacher network and the student network. Neural networks usually produce classification probabilities by using a Softmax output layer, which converts the calculated Logits output value of each class into a classification probability, as shown in the following formula.

[0127]

[0128] where z i is the i-th component of Logits, T is the temperature parameter, and a higher temperature will produce a softer inter-class classification probability, q i is the classification probability of the i-th class, which is the object of student model learning (Soft label), z i is the i-th class logits output value, which is the output before the Softmax layer of the neural network, exp() is the exponential function with the natural constant e as the base. When T = 1, the formula is the standard Softmax formula. When T approaches 0, the maximum classification probability value will be closer to 1, and the other classification probability values will approach 0. If T is larger, the distribution of the output classification probability will be smoother, which is equivalent to a smoothing effect that preserves similar information. If T is equal to infinity, it is a uniform distribution.

[0129] The total loss of knowledge distillation training consists of two parts: the cross-entropy between classification probabilities, and the cross-entropy loss between the student network's classification prediction and the true label, with a temperature of 1, as shown in the following formula.

[0130]

[0131] where N is the size of the training batch, L CE represents the cross-entropy. σ() represents the Softmax function, T is the temperature parameter of knowledge distillation, y i is the true label of sample i, z S ∈ RC and z T ∈ RC are the Logits output by the student network and the teacher network for C-class classification tasks, respectively, and L KD is the total loss of traditional knowledge distillation, which is the weighted sum of the distillation loss and the student loss.

[0132] Although the teacher network is more accurate than the student network at the beginning of training, the teacher will still make some prediction errors. When the teacher network makes a prediction error, knowledge is also transferred to the student network, which will affect the performance of the student network. Therefore, the present application improves the method of traditional knowledge distillation, ignores the prediction distribution of the teacher network error, and only transmits the correct prediction distribution to the student network. Finally, the target function used to train the student network is improved, and the total loss after improvement is as follows.

[0133]

[0134]

[0135] wherein, is an indicator function, is the label predicted by the teacher network. When the teacher network can correctly predict the classification of the input sample, the indicator function is 1, and the student network learns the sample label and the soft target output by the teacher network at the same time; when the teacher network cannot correctly classify, the indicator function is 0, and only the cross entropy between the classification of the student network and the real label is calculated, is the total loss of the improved knowledge distillation.

[0136] Fourthly, a multi-granularity context prompt template integration method MG-CLIP is constructed.

[0137] The picture-text "contrast learning" is to make the picture and the text correspond better, and the implementation method is to give the text feature or the visual feature, and then give several other features, and select the one corresponding to them, and the "positive example" is the picture-text pair in the data set. The similarity between the two features is calculated by vector dot product.

[0138] The training method, the CLIP model architecture is divided into two parts, image encoder and text encoder, in the present application, the image encoder adopts I3D and ResNet (i.e. the teacher model and student model trained by knowledge distillation), and the text encoder adopts the TextEncoder of CLIP, i.e. transformer.

[0139] The training data is 400 million image-text data pairs collected on network social media. With these data, the task that CLIP needs to complete is: given an image, find the one that can match among the 249 randomly sampled text segments of IsoGDP data set.

[0140] To accomplish this task, the CLIP model learns to recognize various visual concepts in images and relate the concepts to the pictures. Thus, the CLIP model can be applied to almost any visual classification task. For example, if the task of a dataset is to classify photos of dogs and cats, the CLIP model predicts which of the two textual descriptions "a photo of a dog" and "a photo of a cat" matches the picture better.

[0141] In the training phase, for a batch of data, first, the text and image features are obtained through the text encoder and image encoder, and then the inner products of all text and image features are calculated to obtain an n x n similarity matrix. From the perspective of images, the row direction is a classifier, and from the perspective of text, the column direction is also a classifier. Since the matching relationship between text and image in a batch is known, the objective function is to maximize the inner product of the same pair of image and text features, that is, the elements on the diagonal of the matrix, and to minimize the inner product of unrelated features. By doing cross-entropy loss on each row image-text and each column text-image, the goal is to optimize the sum of the two losses, so as to achieve the cross-modal alignment of visual information and text information.

[0142] While standard image models jointly train an image feature extractor and a linear classifier to predict certain labels, CLIP jointly trains an image encoder and a text encoder to predict the correct pairing of a batch of (image, text) training examples. In testing, the learned text encoder synthesizes a zero-shot linear classifier by embedding the descriptions of the target dataset classes.

[0143] In the testing phase, the trained CLIP is directly used for the dynamic gesture IsoGD dataset without the need for fine-tuning. Similar to the training phase, first, the image to be classified is encoded to obtain the feature, and then for the gesture recognition task dataset, a multi-granularity context prompt template integration method is proposed to construct an improved CLIP visual language model (MG-CLIP). For each label in the IsoGD dataset, a corresponding text is constructed, which contains multi-granularity context information of the corresponding image information. According to the different backgrounds of the gestures in the IsoGD gesture dataset, a template fusion method is used to fuse the multi-granularity context prompt templates, such as: A photo of a {OK}, a type of hand gesture, one hand.

[0144] Then the text and image features are obtained through the encoder, then the text features and the image features are calculated by inner product to obtain the similarity, and the label corresponding to the maximum similarity is the classification result of the gesture image. This completes the zero-shot learning classification of gesture recognition task, so that the model can recognize image categories that have not been seen in the training stage, thereby improving the generalization performance of the MG-CLIP model.

[0145] Fifth, cross-modal distillation.

[0146] Knowledge distillation can not only be used for model compression, but also can improve the performance of a complex model through the features of cross-modal data, and has a significant effect on model enhancement. Cross-modal distillation is an important application direction in model enhancement, and the synchronous alignment of the modal information of the teacher can be used to make up for the information flow that the student network originally does not have, and the performance of the student network is continuously enhanced through knowledge distillation training.

[0147] The present application also uses the information of different modal data (RGB, Depth and optical flow) to provide complementary clues for cross-modal distillation for gesture recognition tasks, thereby improving the performance of the student network. As shown in Figure 8 The teacher and student networks are both I3D networks, the first group of cross-modal distillation teacher networks are distilled and trained with RGB and Flow data as input to the student network with Depth data as input; the second group of cross-modal distillation teacher networks are distilled and trained with RGB data as input to the student network with Depth data as input; the third group of cross-modal distillation teacher networks are distilled and trained with Depth data as input to the student network with RGB data as input.

[0148] In summary, the dynamic gesture recognition method based on double-flow three-dimensional convolution and knowledge distillation provided by the present application integrates knowledge distillation technology and attention mechanism into the I3D network for research on dynamic gesture recognition, which has the following advantages:

[0149] 1) The related parameters and structure of the I3D network model are improved, the first two max pooling layers are removed to prevent the loss of low-level features of the image caused by the pooling operation; in order to retain the global information of the image while reducing a large number of parameters, the last average pooling operation is removed, which increases the robustness of the network; the network weight is randomly initialized with standard normal distribution.

[0150] 2) The coordinate attention mechanism CA is selected for feature fusion, which considers the relationship between channels and long-distance position information to improve the feature expression ability of the network model, enhance the attention of the network to effective information, and thus enhance the representation ability of specific regions of the image. Reduce the interference information existing in the dynamic gesture video data set.

[0151] 3) Improve the traditional knowledge distillation loss, add an indicator function, which can learn the correct soft target of the teacher network output, reduce the operation cost, and improve the real-time performance. In addition, cross-modal distillation can be used to implicitly embed the features of different modal data in the student network of single modal data, improving the performance of the student network using single modal data as input when predicting.

[0152] 4) A multi-granularity context prompt template integration method is proposed, and an improved CLIP visual language model (MG-CLIP) is constructed. Through image-text paired contrastive learning, zero-shot inference is performed to improve the generalization performance and reduce the dependence of the model on labeled data.

[0153] In order to verify the advantages of the technical solutions provided by the present application, the present application is compared with a variety of double-flow 3D convolution methods on the Chinese sign language data set (CSL) and the open source data set ChaLearn LAP IsoGD (IsoGD). For the technical solutions of the present application, the experimental environment, data set and parameter settings are as follows.

[0154] (1) Experimental environment

[0155] The hardware configuration of the project experiment is Intel Xeon E5-2660 v4 CPU, 64GB memory, and GPU card is NVIDIA GeForce RTX 2080Ti. The software environment is 64-bit Ubuntu 16.04 operating system, CUDA 8.0.61, cuDNN 5.1.10, deep learning framework is PyTorch, version is 1.0.1, and Python version is 3.5.3.

[0156] (2) Data set

[0157] For the dynamic gesture recognition task, the public dataset ChaLearn LAP IsoGD, referred to as IsoGD dataset, is adopted. The dataset is collected by Kinect sensor, containing dynamic gesture video data of RGB and depth modalities. Both modalities contain 47933 labeled video sequences, and there are 249 gesture actions. In the dataset, there is only single human gesture action in the video sequence, and there is no interaction between gesture and human or object, nor is there interaction between people. The IsoGD dataset is composed of three parts: training set, validation set and test set, each of which contains 249 gestures. In order to avoid the influence of the action habits of the performer and the human features on the experimental results, different performers are used in the training set, the validation set and the test set. The training set includes 35878 gesture video sequences, which are completed by 17 performers; the validation set contains 5784 video sequences, which are completed by 2 performers; and the test set contains 6271 videos, which are completed by 2 performers. Each video contains only one gesture action, and the performer starts from the naturally drooping hands to perform the corresponding gesture action, and after completing the gesture action, the hands naturally droop. All samples are collected in a natural scene, and the background, lighting, environment, clothing, etc. are changed. The dataset is a large RGB-D dynamic gesture recognition dataset with high complexity, with complex background, various gestures and close-to-life scene, and is one of the commonly used datasets in the field of dynamic gesture recognition.

[0158] (3) Parameter setting

[0159] Since the number of frames in each video in the IsoGD dataset is inconsistent, it is difficult for the network to process, so the frame number is processed into a uniform frame number video sequence, and the RankPooling method is used for sampling during each training. In the experiment, 32 frames and 16 frames are used as the input of the network, and the size of each frame image is changed to 224x224. In the experiment, the random gradient descent is used as the optimizer to optimize the parameters of the neural network, the BatchSize batch size is set to 4, the initial learning rate is set to 0.03, and the learning rate is decayed according to the cosine annealing method.

[0160] The results show that the method can well solve the problems of background interference, model dependence on labeled data, large calculation and poor real-time performance in the dataset, can well improve the performance of the network, and can improve the generalization ability of the model to adapt to gesture recognition in real scenes. The recognition rate is higher than that of other dynamic gesture recognition methods.

[0161] Embodiment two

[0162] In order to perform the method corresponding to the above-mentioned embodiment one to realize the corresponding functions and technical effects, a dynamic gesture recognition system is provided below. As shown inFigure 9 As shown in the figure, the dynamic gesture recognition system comprises:

[0163] A data acquisition module 1 is configured to acquire a to-be-recognized gesture video; the to-be-recognized gesture video comprises to-be-recognized true color data, to-be-recognized depth data and to-be-recognized optical flow data; the to-be-recognized optical flow data is determined based on the to-be-recognized true color data.

[0164] A gesture recognition module 2 is configured to input the to-be-recognized gesture video into a dynamic gesture recognition model for recognition, to obtain a predicted category of the to-be-recognized gesture video; the dynamic gesture recognition model comprises an image encoder, a text encoder and a similarity calculator; the image encoder comprises a feature extraction module and a feature fusion module; the feature extraction module is obtained by knowledge distillation training with an I3D network as a teacher model and a ResNet network as a student model; and the feature fusion module is constructed based on a coordinate attention mechanism.

[0165] The feature extraction module is configured to perform feature extraction on the to-be-recognized gesture video, to obtain to-be-recognized true color features, to-be-recognized depth features and to-be-recognized optical flow features; the feature fusion module is configured to perform feature fusion on the to-be-recognized true color features, the to-be-recognized depth features and the to-be-recognized optical flow features, to obtain to-be-recognized image features; the text encoder is configured to perform feature extraction on all category labels in a gesture recognition task data set, to obtain corresponding text features; and the similarity calculator is configured to calculate the similarity between the to-be-recognized image features and all text features, and take the category label corresponding to the text feature with the highest similarity as the predicted category of the to-be-recognized gesture video.

[0166] Embodiment three

[0167] The embodiment of the present application also provides an electronic device comprising a memory and a processor, the memory being configured to store a computer program, and the processor being configured to run the computer program to enable the electronic device to perform the dynamic gesture recognition method in embodiment one. The electronic device can be a server.

[0168] In addition, the present application also provides a computer readable storage medium storing a computer program, which, when executed by a processor, implements the dynamic gesture recognition method in embodiment one.

[0169] In summary, the application provides a dynamic gesture recognition method based on three-flow coordinate attention network and knowledge distillation, which mainly studies the use of three-flow input based on I3D network, the use of attention feature fusion and knowledge distillation technology, and the use of image-text paired contrast learning technology to improve network performance. First, the optical flow video is generated from the RGB video to further extract the motion information. Through the analysis of the data distribution characteristics, all videos with different frame numbers are preprocessed to obtain videos with a unified frame number of 32 frames. Then, the RGB, Depth and optical flow data are input into the I3D network for feature extraction. Second, the CA attention module is used to fuse the three-flow features extracted, which helps the network pay more attention to the gesture area in the input image, avoids the interference of the performer's clothes, skin color and other factors, and further extracts the image features of the video frames. Third, in order to run the model on resource-limited devices, the I3D network is used as the teacher model, and the ResNet network is used as the student model for knowledge distillation training to compress the model and shorten the network training time. In addition, the teacher model and the student model can also use the same model structure I3D, and the cross-data distillation training experiment of the teacher model and the student model is carried out on different data to achieve the effect of model enhancement. At the same time, the text feature extraction is carried out by using the text encoder of CLIP, and the classification result is obtained by image-text pairing. Finally, combined with the text label, the transformer text encoder of CLIP is used to extract the text features, and the contrast learning of the text features and the image features of the teacher model and the student model is carried out, the similarity between the image features and all the text features is calculated, and the maximum similarity is obtained. The text label corresponding to the maximum similarity is the label category of the given picture, which reduces the dependence of the model on the labeled data and improves the generalization performance of the model.

[0170] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0171] The principles and implementation modes of the application are described by using specific examples in this paper. The above embodiment description is only used to help understand the method and core idea of the application. For those skilled in the art, according to the idea of the application, the specific implementation mode and application range can be changed. In summary, the content of the specification should not be understood as a limitation of the application.

Claims

1. A dynamic gesture recognition method, characterized in that, The method comprises the following steps: acquiring a to-be-recognized gesture video; the to-be-recognized gesture video comprises to-be-recognized true color data, to-be-recognized depth data and to-be-recognized optical flow data; the to-be-recognized optical flow data is determined based on the to-be-recognized true color data; inputting the to-be-recognized gesture video into a dynamic gesture recognition model for recognition to obtain a predicted category of the to-be-recognized gesture video; the dynamic gesture recognition model comprises an image encoder, a text encoder and a similarity calculator; the image encoder comprises a feature extraction module and a feature fusion module; the feature extraction module is obtained by knowledge distillation training with an I3D network as a teacher model and a ResNet network as a student model; the feature fusion module is constructed based on a coordinate attention mechanism; the feature extraction module is used for feature extraction on the to-be-recognized gesture video to obtain to-be-recognized true color features, to-be-recognized depth features and to-be-recognized optical flow features; the feature fusion module is used for feature fusion on the to-be-recognized true color features, the to-be-recognized depth features and the to-be-recognized optical flow features to obtain to-be-recognized image features; the text encoder is used for feature extraction on all category labels in a gesture recognition task data set to obtain corresponding text features; the similarity calculator is used for calculating the similarity between the to-be-recognized image features and all text features, and taking the category label corresponding to the text feature with the highest similarity as the predicted category of the to-be-recognized gesture video; wherein the training process of the image encoder comprises: constructing a gesture recognition task data set; the gesture recognition task data set comprises a plurality of sample gesture videos and corresponding real category labels; the sample gesture videos comprise sample true color data, sample depth data and sample optical flow data with uniform frame numbers; constructing a teacher model based on an I3D network, and inputting the sample gesture videos into the teacher model for feature extraction to obtain first sample true color features, first sample depth features and first sample optical flow features; constructing a student model based on a ResNet network, and inputting the sample gesture videos into the student model for feature extraction to obtain second sample true color features, second sample depth features and second sample optical flow features; constructing a feature fusion network based on a coordinate attention mechanism; inputting the first sample true color features, the first sample depth features and the first sample optical flow features into the feature fusion network for feature fusion to obtain first fused image features; inputting the second sample true color features, the second sample depth features and the second sample optical flow features into the feature fusion network for feature fusion to obtain second fused image features; determining sample soft labels according to the first fused image features; determining soft prediction labels and hard prediction labels according to the second fused image features; determining a total loss according to the sample soft labels, the soft prediction labels, the hard prediction labels and sample hard labels; the sample hard labels are real category labels of the sample gesture videos; performing distillation training with the minimum total loss as the target to obtain an image encoder.

2. The dynamic gesture recognition method of claim 1, wherein, Determine a total loss according to the sample soft label, the soft prediction label and a sample hard label, specifically comprising: Determine a distillation loss according to the sample soft label and the soft prediction label; Determine a student loss according to the hard prediction label and the sample hard label; Determine the total loss according to the distillation loss and the student loss.

3. The dynamic gesture recognition method of claim 1, wherein, The formula for calculating the total loss is: ； wherein, is the total loss, N is the training batch size, is the true class label of the i-th sample, is the student loss, is the distillation loss, is the hard predicted label output by the student model, is the soft predicted label output by the student model, is the sample soft label output by the teacher model, is the class probability distribution output by the student model, is the class probability distribution output by the teacher model, T is the temperature parameter of knowledge distillation, and σ() represents the Softmax activation function, () represents the cross-entropy, I() represents the indicator function, which is 1 when the teacher model can correctly predict the class of the input sample, and 0 when the teacher model cannot correctly predict the class of the input sample.

4. The dynamic gesture recognition method of claim 1, wherein, The I3D network comprises a first convolutional layer, a second convolutional layer, a third convolutional layer, two first 3D CA-Inception v1 structures, a first maximum pooling layer, five second 3D CA-Inception v1 structures, a second maximum pooling layer, two third 3D CA-Inception v1 structures and a fourth convolutional layer connected in sequence; the 3D CA-Inception v1 structure comprises a 3D Inception v1 structure and a coordinate attention layer connected in sequence.

5. A dynamic gesture recognition system, characterized in that, Comprise: A data acquisition module for acquiring a to-be-identified gesture video; The to-be-identified gesture video comprises to-be-identified true color data, to-be-identified depth data and to-be-identified optical flow data; the to-be-identified optical flow data is determined based on the to-be-identified true color data; A gesture recognition module for inputting the to-be-identified gesture video into a dynamic gesture recognition model for recognition to obtain a predicted category of the to-be-identified gesture video; the dynamic gesture recognition model comprises an image encoder, a text encoder and a similarity calculator; the image encoder comprises a feature extraction module and a feature fusion module; the feature extraction module is obtained by knowledge distillation training with an I3D network as a teacher model and a ResNet network as a student model; the feature fusion module is constructed based on a coordinate attention mechanism; The feature extraction module is configured to extract features from the to-be-identified gesture video to obtain to-be-identified true color features, to-be-identified depth features and to-be-identified optical flow features; the feature fusion module is configured to fuse the to-be-identified true color features, the to-be-identified depth features and the to-be-identified optical flow features to obtain to-be-identified image features; the text encoder is configured to extract features from all category labels in a gesture recognition task dataset to obtain corresponding text features; the similarity calculator is configured to calculate the similarity between the to-be-identified image features and all text features, and take the category label corresponding to the text feature with the highest similarity as the predicted category of the to-be-identified gesture video; The training process of the image encoder comprises: Construct a gesture recognition task dataset; the gesture recognition task dataset comprises a plurality of sample gesture videos and corresponding real category labels; the sample gesture videos comprise sample true color data, sample depth data and sample optical flow data with uniform frame numbers; Construct a teacher model based on an I3D network, and input the sample gesture videos into the teacher model for feature extraction to obtain first sample true color features, first sample depth features and first sample optical flow features; The student model is constructed based on a ResNet network, and the sample gesture video is input into the student model for feature extraction, to obtain a second sample true color feature, a second sample depth feature and a second sample optical flow feature; The feature fusion network is constructed based on a coordinate attention mechanism; The first sample true color feature, the first sample depth feature and the first sample optical flow feature are input into the feature fusion network for feature fusion, to obtain a first fused image feature; The second sample true color feature, the second sample depth feature and the second sample optical flow feature are input into the feature fusion network for feature fusion, to obtain a second fused image feature; A sample soft label is determined according to the first fused image feature; A soft prediction label and a hard prediction label are determined according to the second fused image feature; A total loss is determined according to the sample soft label, the soft prediction label, the hard prediction label and a sample hard label; the sample hard label is a true class label of the sample gesture video; Distillation training is performed with the minimum total loss as a target, to obtain an image encoder.

6. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory is used for storing a computer program, and the processor runs the computer program to enable the electronic device to perform the dynamic gesture recognition method in any one of claims 1 to 4.

7. A computer readable storage medium characterized in that, The computer program is stored in the memory and is executed by the processor to implement the dynamic gesture recognition method in any one of claims 1 to 4.

Images