A dynamic gesture recognition method and device

Abstract

The present invention discloses a dynamic gesture recognition method and device in the field of artificial intelligence, and relates to the field of artificial intelligence (AI). The dynamic gesture recognition method comprises: obtaining multiple images containing gesture actions; recognizing the multiple images through a gesture recognition model, and obtaining the type and attribute of the gesture actions in the multiple images, wherein the attribute includes an outbound path and a return path. The above method can improve the recognition accuracy.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI), and in particular to a dynamic gesture recognition method and device. Background Art

[0002] Gestures are expressive body movements that can convey a variety of meaningful information. Dynamic gesture recognition has always been one of the research hotspots in the field of deep learning. As an emerging human-computer interaction (HCI) method, dynamic gesture recognition has broad application prospects in many fields such as virtual reality, smart home, early childhood education, and medical robots. In dynamic gesture recognition, when a user waves his hand continuously in one direction, he must return to the starting point in the other direction. This is the gesture return. The gesture return can easily cause the terminal device to misjudge the direction of the gesture. For example, if a user needs to wave his hand to the right twice in a row, he needs to return his hand to the starting point of the first wave and wave it again for the second time. During the return process, the terminal device can easily judge it as a left gesture.

[0003] Therefore, how to accurately recognize gestures is a problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present application provide a dynamic gesture recognition method and device, which can improve recognition accuracy.

[0005] In a first aspect, an embodiment of the present application provides a dynamic gesture recognition method, comprising: acquiring multiple images containing gesture actions; recognizing the multiple images through a gesture recognition model to obtain the types and attributes of the gesture actions in the multiple images, wherein the attributes include outbound and return paths.

[0006] In the solution provided in the present application, a gesture recognition model is used to identify multiple images containing gesture actions, so that the types and attributes of the gesture actions in the multiple images can be obtained, and then corresponding operations can be performed according to the types and attributes of the gesture actions in the multiple images obtained, thereby avoiding the terminal's misjudgment of gestures due to backhaul problems.

[0007] In a possible implementation, a plurality of sample images carrying annotation information are obtained, wherein the sample images are multiple images containing gesture actions, and the annotation information includes the type and attribute of the gesture actions in the sample images; and an initial gesture recognition model is trained according to the plurality of sample images carrying annotation information to obtain the gesture recognition model.

[0008] In the solution provided in the present application, the types and attributes of gesture actions recorded in sample images can be obtained in advance, and then the initial gesture recognition model can be trained using multiple sample images carrying types and attributes, so that the trained gesture recognition model has the ability to recognize the types and attributes of gesture actions recorded in multiple images. In this way, multiple images input into the gesture recognition model can be recognized, so that the types and attributes of gesture actions recorded in multiple images can be output.

[0009] In a possible implementation, the gesture recognition model includes a spatial feature module, a temporal feature module and a classification module; the identifying the multiple images through the gesture recognition model to obtain the types and attributes of the gesture actions in the multiple images includes: inputting the multiple images into the spatial feature module to obtain first feature data, the first feature data including the spatial features of the gesture actions in the multiple images; inputting the first feature data into the temporal feature module to obtain second feature data, the second feature data including the temporal features of the first feature data in the time dimension; inputting the second feature data into the classification module to obtain the types and attributes of the gesture actions in the multiple images.

[0010] In the solution provided in the present application, after the spatial features of the input multiple images are extracted by the spatial feature module of the gesture recognition model, the first feature data is obtained, and the time domain feature module of the gesture recognition model extracts the time domain features of the multiple images for the first feature data in the time dimension, and finally inputs the classification module of the gesture recognition model to obtain the types and attributes of the gesture actions in the multiple images.

[0011] In a possible implementation, the time domain feature module includes a dimension transformation layer, a convolution layer, a batch normalization (BN) layer, a rectified linear unit (ReLu) layer, a maximum pooling layer and a feature union layer; the first feature data is input into the time domain feature module to obtain second feature data, and the second feature data includes the time domain features of the first feature data in the time dimension, including: determining the first time domain feature data corresponding to the first feature data in the time dimension through the dimension transformation layer according to the time information of the multiple images; performing convolution processing on the first time domain feature data through the convolution layer to obtain the second time domain feature data; and the second time domain feature data is sequentially passed through the BN layer, the ReLu layer, the maximum pooling layer and the feature union layer to obtain the second feature data.

[0012] In the solution provided in the present application, the dimensionality transformation layer of the time domain feature module determines the first time domain feature data corresponding to the pixel points with the same position in the first feature data corresponding to the multiple images in the time dimension according to the time information of each image in the multiple images, and the convolution layer of the time domain feature module performs convolution processing on the first feature data to obtain the corresponding second time domain feature data, and the second time domain feature data is sequentially passed through the BN layer, the ReLu layer, the maximum pooling layer and the feature union layer to obtain the second feature data.

[0013] In a possible implementation, the convolution processing of the first time domain feature data through a convolution layer to obtain the second time domain feature data includes: using a first preset number of one-dimensional convolution layers with different convolution kernel sizes to perform convolution processing on the first time domain feature data to obtain the second time domain feature data, the second time domain feature data including a first preset number of feature data of different scales; the passing the second time domain feature data through the BN layer, the ReLu layer, the maximum pooling layer and the feature union layer in sequence to obtain the second feature data includes: passing the first preset number of feature data of different scales through the BN layer, the ReLu layer, the maximum pooling layer and the feature union layer in sequence to obtain the second feature data.

[0014] In the solution provided in the present application, for the first time domain feature data, the convolution layer of the time domain feature module uses a first preset number of one-dimensional convolution layers with different convolution kernel sizes to perform convolution processing on the first time domain feature data to obtain a first preset number of feature data of different scales, and the feature data passes through the BN layer, the ReLu layer, the maximum pooling layer and the feature union layer, and the feature union layer fuses the first preset number of feature data of different scales corresponding to the first time domain feature data to obtain the second feature data corresponding to the first feature data. The use of a one-dimensional convolution layer can effectively reduce the amount of calculation and improve the processing efficiency of the convolution layer of the time domain feature module.

[0015] In a possible implementation, the gesture recognition model also includes a first classifier and a second classifier, and the inputting of the second feature data into the classification module to obtain the types and attributes of the gesture actions in the multiple images includes: inputting the second feature data into the first classifier to obtain the first probability that the gesture actions in the multiple images belong to each type; classifying the first gesture action into the type with the highest first probability corresponding to the first gesture action, and the first gesture action is any gesture action among the gesture actions in the multiple images; inputting the second feature data into the second classifier to obtain the second probability that the gesture actions in the multiple images belong to each attribute; classifying the first gesture action into the attribute with the highest second probability corresponding to the first gesture action.

[0016] In the solution provided in the present application, the second feature data is input into the trained first classifier and the second classifier to obtain the first probability that the gesture action belongs to each type and the second probability that it belongs to each attribute, and the first gesture action is classified into the type with the highest first probability and the attribute with the highest second probability corresponding to the first gesture action.

[0017] In a possible implementation, the acquiring of multiple images including gesture actions includes: acquiring a video to be identified; and extracting an image from the video to be identified every second preset number of images to obtain multiple images including gesture actions.

[0018] In the solution provided in the present application, a video to be identified is obtained, and according to the time sequence of the images in the video to be identified, one image is extracted from the video to be identified every second preset number of images, and when the number of extracted images reaches a third preset number, the third preset number of extracted images are determined as the multiple images.

[0019] In a possible implementation manner, the method further includes: when the attribute of the gesture action in the multiple images is outgoing, executing a function corresponding to the type of the gesture action.

[0020] In the solution provided in the present application, after the gesture recognition model identifies the type and attribute of the gesture action in the multiple images, when the attribute is an outbound trip, the terminal device executes the function corresponding to the type of the identified gesture action, and when the attribute is a return trip, the terminal device does not process it.

[0021] In a second aspect, an embodiment of the present application provides a dynamic gesture recognition device, comprising: a first acquisition unit, used to acquire multiple images containing gesture actions; a recognition unit, used to recognize the multiple images through a gesture recognition model, and obtain the types and attributes of the gesture actions in the multiple images, wherein the attributes include outbound and return trips.

[0022] In a possible implementation, the device also includes: a second acquisition unit, used to acquire multiple sample images carrying annotation information, the sample images are multiple images containing gesture actions, and the annotation information includes the type and attributes of the gesture actions in the sample images; a training unit, used to train an initial gesture recognition model according to the multiple sample images carrying annotation information to obtain the gesture recognition model.

[0023] In a possible implementation, the gesture recognition model includes a spatial feature module, a temporal feature module and a classification module; the recognition unit is specifically used to: input the multiple images into the spatial feature module to obtain first feature data, wherein the first feature data includes the spatial features of the gesture actions in the multiple images; input the first feature data into the temporal feature module to obtain second feature data, wherein the second feature data includes the temporal features of the first feature data in the time dimension; input the second feature data into the classification module to obtain the type and attribute of the gesture actions in the multiple images.

[0024] In a possible implementation, the time domain feature module includes a dimension transformation layer, a convolution layer, a BN layer, a ReLu layer, a maximum pooling layer and a feature union layer; the recognition unit is used to input the first feature data into the time domain feature module to obtain second feature data, and when the second feature data includes the time domain features of the first feature data in the time dimension, it is specifically used to: determine the first time domain feature data corresponding to the first feature data in the time dimension through the dimension transformation layer according to the time information of the multiple images; perform convolution processing on the first time domain feature data through the convolution layer to obtain second time domain feature data; and pass the second time domain feature data through the BN layer, the ReLu layer, the maximum pooling layer and the feature union layer in sequence to obtain the second feature data.

[0025] In a possible implementation, the identification unit is used to perform convolution processing on the first time domain feature data through the convolution layer to obtain the second time domain feature data, and is specifically used to: use a first preset number of one-dimensional convolution layers with different convolution kernel sizes to perform convolution processing on the first time domain feature data to obtain the second time domain feature data, and the second time domain feature data includes a first preset number of feature data of different scales; the second time domain feature data is sequentially passed through the BN layer, the ReLu layer, the maximum pooling layer and the feature union layer to obtain the second feature data, including: passing the first preset number of feature data of different scales through the BN layer, the ReLu layer, the maximum pooling layer and the feature union layer in sequence to obtain the second feature data.

[0026] In a possible implementation, the gesture recognition model also includes a first classifier and a second classifier, and the recognition unit is used to input the second feature data into the classification module to obtain the type and attribute of the gesture actions in the multiple images, and is specifically used to: input the second feature data into the first classifier to obtain the first probability that the gesture actions in the multiple images belong to each type; classify the first gesture action into the type with the highest first probability corresponding to the first gesture action, and the first gesture action is any gesture action among the gesture actions in the multiple images; input the second feature data into the second classifier to obtain the second probability that the gesture actions in the multiple images belong to each attribute; classify the first gesture action into the attribute with the highest second probability corresponding to the first gesture action.

[0027] In a possible implementation, the first acquisition unit is specifically configured to: acquire a video to be identified; and extract an image from the video to be identified every second preset number of images to obtain a plurality of images including gesture actions.

[0028] In a possible implementation manner, the device further includes: an execution unit, configured to execute a function corresponding to the type of the gesture action when the attribute of the gesture action in the multiple images is outgoing.

[0029] In a third aspect, an embodiment of the present application provides a computing device, comprising a processor and a memory, wherein the memory is used to store programs, and the processor executes the programs stored in the memory. When the programs stored in the memory are executed, the computing device implements the dynamic gesture recognition method provided in the first aspect and in combination with any one of the implementation methods of the first aspect.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable medium is used to store computer-executable instructions, and when the computer-executable instructions are called by the computer, the computer is used to enable the computer to implement the dynamic gesture recognition method provided in the above-mentioned first aspect and in combination with any one of the implementation methods of the above-mentioned first aspect.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the computer program product is executed by a computer, the computer can execute the process of the dynamic gesture recognition method provided in the above-mentioned first aspect and in combination with any one of the implementation methods in the above-mentioned first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0033] Figure 1 This is a scenario of dynamic gesture interaction in an embodiment of the present application;

[0034] Figure 2 This is a schematic diagram of a dynamic gesture recognition system architecture in an embodiment of the present application;

[0035] Figure 3 is a schematic diagram of a gesture recognition model in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of a CNN in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of a chip hardware structure in an embodiment of the present application;

[0038] Figure 6 is a schematic flow chart of a dynamic gesture recognition method in an embodiment of the present invention;

[0039] Figure 7 A schematic diagram of a spatial feature module in an embodiment of the present application;

[0040] Figure 8 A schematic diagram of first feature data extraction in an embodiment of the present application;

[0041] Fig. 9 A schematic diagram of a time domain feature module in an embodiment of the present application;

[0042] Fig.10 A schematic diagram of a Reshape operation performed on a dimensional transformation layer in an embodiment of the present application;

[0043] Fig.11 is a flow chart of another dynamic gesture recognition method in an embodiment of the present invention;

[0044] Fig.12 is a flow chart of another dynamic gesture recognition method in an embodiment of the present invention;

[0045] Fig.13 A flowchart of a gesture recognition model training method according to an embodiment of the present invention;

[0046] Fig.14A schematic diagram of feature extraction in gesture action recognition in an embodiment of the present invention;

[0047] Fig.15 A schematic diagram of time domain feature extraction in an embodiment of the present invention;

[0048] Fig.16 is a structural schematic diagram of a dynamic gesture recognition device in an embodiment of the present invention;

[0049] Fig.17 A schematic diagram of the structure of a computing device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application are described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0051] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0052] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0053] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0054] (1) Artificial Intelligence

[0055] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0056] (2) Neural Network

[0057] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit with the intercept 1 as input can be:

[0058]

[0059] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0060] (3) Deep Neural Networks

[0061] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, b is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the number of coefficients W and offset vectors b is also large. The definitions of these parameters in DNN are as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0062] (4) Convolutional Neural Network

[0063] Convolutional neural network (CNN) is a deep neural network with convolutional structure. Convolutional neural network contains a feature extractor consisting of convolution layer and subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (feature map). Convolution layer refers to the neuron layer in convolutional neural network that performs convolution processing on the input signal. In the convolution layer of convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature planes, and each feature plane can be composed of some rectangular arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in a part can also be used in another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0064] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0065] (5) Loss Function

[0066] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually a process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0067] (6) Gesture return

[0068] In dynamic gesture recognition, when the user waves the gesture continuously in one direction, he or she will definitely return to the starting point in the other direction. This is the gesture return. The gesture return may easily cause the terminal device to misjudge the direction of the gesture.

[0069] With the rapid development of computer technology, dynamic gesture recognition has become one of the human-computer interaction methods. Figure 1 This is a scenario of dynamic gesture interaction in an embodiment of the present application, such as Figure 1As shown, in the field of interactive design, users can use one or two hands to perform contactless operations on terminal devices, and the terminal devices respond to user gestures and execute related commands through dynamic gesture recognition. At present, the mainstream dynamic gesture recognition methods can be divided into two types: the first is to implement dynamic gesture recognition methods by combining neural networks and video input. This method is based on multi-image input, uses convolutional neural networks (CNN) to extract spatial features (image features), uses one-dimensional convolution (1DCONV) or fully connected networks (multilayer perceptron, MLP) to extract time domain features, and finally obtains dynamic gesture recognition results in the video. This method can complete the self-recognition of dynamic gesture recognition end-to-end (learning the characteristics of actions during training), but there is a serious backhaul problem. The second is to perform static image recognition (detection tracking, classification or key point recognition) through neural networks, and infer dynamic actions through the combination of classification of continuous frames, such as position and hand shape classification. This method can adjust the accuracy of dynamic gesture recognition through some classification results and classification thresholds, but the backhaul problem is also difficult to solve.

[0070] Based on the above problems, the present application provides a method for dynamic gesture recognition, which can obtain multiple images containing gesture actions, recognize multiple images through a gesture recognition model, obtain the type and attribute of the gesture actions in the multiple images, and then execute the corresponding operation command according to the obtained type and attribute of the gesture action. Through this method, the recognition of the return and the inbound in dynamic gesture recognition can be solved, and the recognition accuracy of gesture actions can be improved.

[0071] The following introduces the system architecture provided by the embodiments of the present application.

[0072] See also Figure 2 , Figure 2 FIG. 1 is a schematic diagram of a dynamic gesture recognition system architecture in an embodiment of the present application. Figure 2 As shown, the dynamic gesture recognition system architecture 200 may include an execution device 210 , a training device 220 , a database 230 , a user device 240 , a data storage system 250 , and a data collection device 260 .

[0073] The data acquisition device 260 is used to collect multiple image data containing gesture actions and store the multiple image data in the database 230. The training device 220 trains the gesture recognition model 201 based on the multiple image data maintained in the database 230. The training process may include: the training device 220 inputs the multiple image data into the initial gesture recognition model 221 to obtain the gesture recognition model 201, and the initial gesture recognition model 221 is a deep neural network. The work of each layer in the deep neural network can be expressed by the mathematical expression To describe: From a physical level, the work of each layer in a deep neural network can be understood as completing the transformation from input space to output space (i.e., from row space to column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by Completed, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the classified object is not a single thing, but a class of things, and space refers to the set of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in this layer of the neural network. The vector W determines the spatial transformation from the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training a deep neural network is to eventually obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially to learn how to control spatial transformation, and more specifically to learn the weight matrix.

[0074] Because we want the output of the deep neural network to be as close as possible to the value we really want to predict, we can compare the current network's predicted value with the target value we really want, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the network's predicted value is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the neural network can predict the target value we really want. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0075] The gesture recognition model 201 obtained by training the training device 220 can be applied to different systems or devices, such as Figure 2 The execution device 210 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an AR / VR, a vehicle terminal, etc., or a server or a cloud. The execution device 210 may execute the dynamic gesture recognition method in the embodiment of the present application. Figure 2 In the embodiment, the execution device 210 is configured with an I / O interface 212 for data interaction with an external device. The user can input data to the I / O interface 212 through the user device 240. The input data in the embodiment of the application can be multiple image data containing gesture actions, or it can be a request to the execution device 210 to recognize dynamic gestures.

[0076] During the process of the calculation module 211 of the execution device 210 performing calculations and other related processing, the execution device 210 can call the data, code, etc. in the data storage system 250 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 250.

[0077] The calculation module 211 can use the gesture recognition model 201 to process the input multiple image data containing gesture actions. Specifically, multiple images containing gesture actions are first obtained, and first feature data is obtained through the spatial feature module in the gesture recognition model 201. The first feature data is input into the time domain feature module in the gesture recognition model 201 to obtain second feature data, and the second feature data is input into the classification module in the gesture recognition model 201 to obtain the types and attributes of the gesture actions in the multiple images.

[0078] Finally, the I / O interface 212 returns the processing results, such as the types and attributes of gesture actions in multiple images obtained by the recognition method of the gesture recognition model 201, to the user device 240. The user device 240 can be a terminal, such as a mobile phone terminal, a laptop computer, an AR / VR terminal or a vehicle terminal, etc., to respond to the corresponding needs of the terminal user.

[0079] exist Figure 2In the case shown in , the user can manually give input data (such as multiple images containing gesture actions in the embodiment of the present application), and the manual giving can be operated through the interface provided by the I / O interface 212. In another case, the user device 240 can automatically send input data to the I / O interface 212. If the user's authorization is required to require the user device 240 to automatically send input data, the user can set the corresponding permissions in the user device 240. The user can view the recognition results output by the execution device 210 on the user device 240, and the recognition results include the types and attributes of gesture actions in multiple images. After receiving the recognition results, the user device 240 can convert the recognition results into corresponding instructions to respond to the user's dynamic gestures.

[0080] It is worth noting that Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, Figure 2 In the embodiment, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may be placed in the execution device 210. The data acquisition device 260 may be a separate external device relative to the user device 240, or an internal device placed in the user device 240.

[0081] like Figure 2 As shown, the gesture recognition model 201 in this embodiment may be obtained through training with the training device 220. Specifically, the gesture recognition model 201 provided in the embodiment of the present application may be a neural network model for dynamic gesture recognition.

[0082] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a gesture recognition model in an embodiment of the present application. Figure 3 As shown, the gesture recognition model 300 may include a spatial feature module 301 , a temporal feature module 302 , and a classification module 303 . The temporal feature module 302 may be arranged after the spatial feature module 301 . Figure 3The spatial feature module 301 in the image extracts first feature data layer by layer from the input multiple images containing gestures, and the first feature data contains spatial features that characterize the gestures in the multiple images. The input data of the time domain feature module 302 is the first feature data output by the spatial feature module 301 located at the previous level. The time domain feature module 302 processes the first feature data to obtain second feature data. The input data of the classification module 303 is the second feature data output by the time domain feature module 302 located at the previous level. The classification module 303 classifies the second feature data to determine the type and attributes of the gestures in the multiple images. The output value of the classification module 303 can be passed to two outputs, one output can be classified using softmax logistic regression to characterize the type of gesture, and the other output can be classified using a sigmoid function to characterize the attributes of the gesture.

[0083] In a specific implementation, the gesture recognition model 300 may include multiple spatial feature modules and multiple temporal feature modules, and the structures of the multiple spatial feature modules may be the same or different. A single spatial feature module may include only one neural network layer, for example, a single spatial feature module includes only one convolution layer; a single spatial feature module may also include multiple identical or different neural network layers, for example, a single spatial feature module includes a convolution layer and a pooling layer, or a single spatial feature module includes multiple different convolution layers. Figure 3 The gesture recognition model 300 is only an example. In actual applications, the number, structure, and position of the spatial feature modules and the number, structure, and position of the temporal feature modules included in the gesture recognition model 300 can be set according to actual needs, and the embodiments of the present application are not limited thereto.

[0084] In this embodiment, the spatial feature module 301 may be a CNN architecture.

[0085] See also Figure 4 , Figure 4 This is a schematic diagram of a CNN in an embodiment of the present application. As described in the previous basic concept introduction, CNN is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different abstract levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, and each neuron in the feed-forward artificial neural network can respond to multiple images containing gestures input therein.

[0086] like Figure 4As shown, the convolutional neural network (CNN) 400 may include an input layer 410 , a convolutional layer / pooling layer 420 (wherein the pooling layer is optional), and a neural network layer 430 .

[0087] Convolutional layer / pooling layer 420:

[0088] Convolutional Layer:

[0089] like Figure 4 The convolution layer / pooling layer 420 shown may include layers 421-426, for example: in one implementation, layer 421 is a convolution layer, layer 422 is a pooling layer, layer 423 is a convolution layer, layer 424 is a pooling layer, layer 425 is a convolution layer, and layer 426 is a pooling layer; in another implementation, layers 421 and 422 are convolution layers, layer 423 is a pooling layer, layers 424 and 425 are convolution layers, and layer 426 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0090] The following will take convolutional layer 421 as an example to introduce the internal working principle of a convolutional layer.

[0091] The convolution layer 421 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator may be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in an image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of an image, and another weight matrix is ​​used to blur unwanted noise in an image, etc. The multiple weight matrices have the same size (rows × columns), and the size of the feature maps extracted by the multiple weight matrices of the same size is also the same. The extracted feature maps of the same size are then merged to form the output of the convolution operation. The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications, and the weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 400 can make correct predictions.

[0092] When the convolutional neural network 400 has multiple convolutional layers, the initial convolutional layer (for example, 421) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 400 increases, the features extracted by the later convolutional layers (for example, 426) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0093] Pooling layer:

[0094] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 4The layers 421-426 illustrated in 420 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0095] Neural network layer 430:

[0096] After being processed by the convolution layer / pooling layer 420, the convolution neural network 400 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 520 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 400 needs to use the neural network layer 430 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 430 may include multiple hidden layers (such as Figure 4 431, 432 to 43n) and the output layer 440 shown, the parameters contained in the multi-layer hidden layers can be pre-trained according to relevant training data of a specific task type.

[0097] After the multiple hidden layers in the neural network layer 430, that is, the last layer of the entire convolutional neural network 400 is the output layer 440, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 400 (such as Figure 4 The propagation from 410 to 440 is forward propagation), and the reverse propagation (such as Figure 4 The propagation from 440 to 410 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 400 and the error between the result output by the convolutional neural network 400 through the output layer and the ideal result.

[0098] It should be noted that if Figure 4The convolutional neural network 400 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

[0099] The following introduces a chip hardware structure provided by an embodiment of the present application.

[0100] See also Figure 5 , Figure 5 FIG. 1 is a schematic diagram of a chip hardware structure in an embodiment of the present application. Figure 5 As shown, the chip includes a neural network processor 50. The chip can be arranged as follows Figure 2 The execution device 210 shown in FIG. 1 is used to complete the calculation work of the calculation module 211. The chip can also be set in Figure 2 The training device 220 shown in FIG. 2 is used to complete the training work of the training device 220 and output the target model 201. Figure 3 The algorithms of each module in the gesture recognition model shown can be found in Figure 5 is implemented in the chip shown.

[0101] The neural network processor 50 can be any processor suitable for large-scale XOR operation processing, such as NPU, TPU, or GPU. Take NPU as an example: NPU can be mounted on the host CPU as a coprocessor, and the host CPU assigns tasks to it. The core part of NPU is the operation circuit 503, which is controlled by the controller 504 to extract the matrix data in the memory (501 and 502) and perform multiplication and addition operations.

[0102] In some implementations, the operation circuit 503 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 503 is a two-dimensional systolic array. The operation circuit 503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 503 is a general-purpose matrix processor.

[0103] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 503 takes the weight data of the matrix B from the weight memory 502 and caches it on each PE in the operation circuit 503. The operation circuit 503 takes the input data of the matrix A from the input memory 501, performs matrix operations based on the input data of the matrix A and the weight data of the matrix B, and stores the partial results or the final results of the matrix in the accumulator 508.

[0104] The unified memory 506 is used to store input data and output data. The weight data is directly transferred to the weight memory 502 through the direct memory access controller (DMAC) 505. The input data is also transferred to the unified memory 506 through the DMAC.

[0105] The bus interface unit (BIU) 510 is used for the interaction between the DMAC and the instruction fetch buffer 509. The bus interface unit 501 is also used for the instruction fetch buffer 509 to obtain instructions from the external memory. The bus interface unit 501 is also used for the storage unit access controller 505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0106] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 506, or to transfer weight data to the weight memory 502, or to transfer input data to the input memory 501.

[0107] The vector calculation unit 507 includes multiple operation processing units, and when necessary, further processes the output of the operation circuit 503, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. The vector calculation unit 507 is mainly used for the calculation of non-convolutional layers or fully connected layers (FC) in the neural network, and can specifically process: pooling, normalization, etc. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both.

[0108] In some implementations, the vector calculation unit 507 stores the processed vectors in the unified memory 506. In some implementations, the vectors processed by the vector calculation unit 507 can be used as activation inputs of the operation circuit 503, for example, for use in subsequent layers in a neural network, such as Figure 4 As shown, if the current processing layer is hidden layer 1 (431), the vector processed by the vector calculation unit 507 can also be used in the calculation of hidden layer 2 (432).

[0109] An instruction fetch buffer 509 connected to the controller 504, for storing instructions used by the controller 504;

[0110] The unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories. The external memory is independent of the NPU hardware architecture.

[0111] in, Figure 3 The operations of each module in the gesture recognition model shown may be performed by the operation circuit 503 or the vector calculation unit 507 .

[0112] The following provides a Figure 2 A dynamic gesture recognition method for a system architecture. Figure 6 , Figure 6 FIG. 1 is a flow chart of a dynamic gesture recognition method according to an embodiment of the present invention. Figure 6 As shown, the dynamic gesture recognition method may include the following steps.

[0113] 601. Acquire multiple images containing gesture actions.

[0114] In an embodiment, when a hand gesture needs to be recognized, multiple images containing the hand gesture may be acquired. The multiple images containing the hand gesture may include only the hand object or the background of the hand object.

[0115] In a specific implementation, the video to be identified can be obtained first, and an image can be extracted from the video to be identified every second preset number of images to obtain multiple images including gestures. In one implementation, when the number of extracted images reaches a third preset number, the extracted third preset number of images are determined as multiple images containing gestures. For example, if the second preset number is 14 and the third preset number is 8, then starting from the first frame in the video to be identified, an image can be extracted every 14 frames, and finally the first 8 images consisting of the 1st, 15th, 29th, 43rd, 57th, 71st, 85th, and 99th frames are obtained. One image can be extracted every 14 frames to obtain the second 8 images.

[0116] 602. Recognize the multiple images through the gesture recognition model to obtain types and attributes of gesture actions in the multiple images.

[0117] After acquiring multiple images containing gestures, the multiple images can be identified by a gesture recognition model to obtain the types and attributes of the gestures in the multiple images. The gesture recognition model may include a spatial feature module, a temporal feature module, a classification module, a first classifier, and a second classifier.

[0118] In specific implementation, the multiple images input to the gesture recognition model can be a continuous image, or multiple images obtained by arranging multiple discontinuous images captured from a video to be recognized in time sequence. The multiple images are essentially a four-dimensional matrix (B×T, C, H, W), where B is the batch size, that is, the number of multiple images that the gesture recognition model can process at one time, T is the image length, that is, the number of images contained in the multiple images, C is the number of channels of the image, H is the height of the image, and W is the width of the image. The image referred to at this time is a video frame. Take the input information of batch size B=2, image length T=8, RGB channel number C=3, height H=224, and width W=224 as an example, that is, the multiple images input to the gesture recognition model are a four-dimensional matrix (2×8,3,224,224). If the gesture recognition model only processes one group of images at the same time, B can be set to 1, that is, the gesture recognition model can process T images in a group of images at one time.

[0119] In a specific implementation, multiple images containing gestures may be input into a spatial feature module to obtain first feature data, where the first feature data includes spatial features of the gestures in the multiple images. Figure 7 , Figure 7 FIG. 1 is a schematic diagram of a spatial feature module in an embodiment of the present application. Figure 7 As shown, the spatial feature module 700 may include an input layer 710, a convolution layer / pooling layer 720 (where the pooling layer is optional), an attention mechanism 730, and a neural network layer 740. In the network structure design of the spatial feature module 700, the backbone network may adopt a CNN architecture. The spatial feature module 700 is a lightweight network based on the CNN architecture with an attention mechanism added. Batch normalization BatchNorm is used instead of L2Norm to obtain a better effect. Among them, the attention mechanism 730 may be added after the convolution layer / pooling layer 720 (where the pooling layer is optional).

[0120] See also Figure 8 , Figure 8 FIG. 1 is a schematic diagram of extracting a first feature data in an embodiment of the present application. Figure 8As shown in (1), this embodiment adopts a solution of multiple image inputs, and a method of sharing parameters is used for each image. Blocks 0 to 3 use the same parameters to extract spatial features. Among them, a block is a feature block formed after the convolution layer in the spatial feature module extracts spatial features from a local area of ​​the image. An attention mechanism can be added after the block to enhance the spatial feature extraction of the local area. In this embodiment, an attention mechanism is added after each block. Specifically, Figure 8 As shown in (2), for example, when performing spatial feature extraction on image 0, features of multiple local regions can be obtained, and the features of multiple local regions can be added together to obtain spatial feature 0. Similarly, performing spatial feature extraction on image 1 can obtain spatial feature 1, ..., and performing spatial feature extraction on image n can obtain spatial feature n.

[0121] In an embodiment of the present application, the first feature data corresponding to each image includes multiple two-dimensional pictures (i.e., two-dimensional matrices (H, W)), each of which is a feature map, and the number of feature maps contained in the first feature data is equal to the corresponding number of channels. For example, if the dimension of the data output by the spatial feature module is (16, 64, 112, 112), then the number of feature maps contained in the first feature data corresponding to an image is 64, and the size of each feature map is 112×112. It should be noted that the dimension and size of the first feature data corresponding to each image output by the same module are the same. Similarly, the second feature data corresponding to each image also includes multiple feature maps.

[0122] After obtaining the first feature data, the first feature data can be input into a time domain feature module to obtain second feature data, where the second feature data includes the time domain features of the first feature data in the time dimension. In one implementation, the time domain feature module can be a CNN architecture. Fig. 9 , Fig. 9 FIG. 1 is a schematic diagram of a time domain feature module in an embodiment of the present application. Fig. 9 As shown, the time domain feature module 900 may include a dimension transformation layer 901, a convolution layer 902, a batch normalization layer 903, an activation function layer 904, a maximum pooling layer 905 and a feature combination layer 906. Specifically, the first time domain feature data corresponding to the first feature data in the time dimension is determined by the dimension transformation layer 901 according to the time information of the multiple images. Fig.10 , Fig.10 Schematic diagram of a dimensional transformation layer in an embodiment of the present application performing a Reshape operation. Reshape is a function that can re-adjust the number of rows, columns, and dimensions of a matrix. Fig.10As shown, the dimension transformation layer 901 can realize the dimensional transformation of the first feature data (B×T, C, H, W) output by the previous spatial feature module, that is, the spatial dimension (H, W) in the first feature data (B×T, C, H, W) is merged into the Batch batch dimension, and the time dimension T is separated separately to obtain a three-dimensional matrix (B×H×W, C, T). The first time domain feature data is composed of the first feature data (C, H, W) corresponding to multiple images. The pixels with the same H, W, and C are arranged in time order. Each first time domain feature data contains T data, and the first time domain feature data is a one-dimensional vector composed of these T data. For example, when B=1, T=8, C=64, H=56, and W=56, 1×56×56×64 first time domain feature data can be obtained after the Reshape operation, and each first time domain feature data contains 8 data.

[0123] After obtaining the first time domain feature data, the first time domain feature data is convolved by the convolution layer 902 to obtain the second time domain feature data. Specifically, the first time domain feature data output by the dimension transformation layer 901 is convolved by the convolution layer 902. The convolution layer 902 may include a first preset number of one-dimensional convolution layers with different convolution kernel sizes. For the first time domain feature data output by the dimension transformation layer 901, the first time domain feature data is convolved by the first preset number of one-dimensional convolution layers to obtain the first preset number of second time domain feature data of different scales corresponding to the first time domain feature data. The second time domain feature data is sequentially passed through the BN layer 903, the ReLu layer 904, and the maximum pooling layer 905. Further, the feature joint layer 906 fuses the first preset number of second time domain feature data of different scales to obtain the corresponding second feature data. Specifically, the feature joint layer 906 can add the first preset number of second time domain feature data of different scales to obtain the corresponding second feature data. By setting multiple one-dimensional convolution layers with different convolution kernel sizes through the convolution layer 902, time series features of different scales can be extracted from the same first time domain feature data. The feature combination layer 906 fuses these multiple time series features of different scales to obtain the second feature data, which better retains the time series characteristics of the gesture action.

[0124] Among them, the BN layer 903 is a network layer like the convolution layer 902, which is used to speed up the training speed and improve the generalization ability of the network. The BN layer is essentially a normalized network layer, which can replace the local response normalization layer (LRN layer). The ReLu layer 904 is used to increase the nonlinear relationship between the layers of the neural network and alleviate the gradient disappearance problem. The maximum pooling operator in the maximum pooling layer 905 can take the pixel with the largest value in a specific range as the result of the maximum pooling, and the first time domain feature data output by the convolution layer 902 can be sampled to obtain an image of a smaller size.

[0125] In one implementation, the convolution layer 902 may include four one-dimensional convolution layers with different convolution kernel sizes, and the convolution kernels are k=3, k=4, k=5, and k=6, respectively. These four one-dimensional convolution layers are used to perform convolution processing on the first time domain feature data to obtain four feature data of different scales corresponding to the first time domain feature data. After passing through the BN layer 903, the ReLu layer 904 and the maximum pooling layer 905, the feature union layer 906 adds the feature data of four different scales to obtain the second feature data corresponding to the first time domain feature data.

[0126] In one implementation, see Fig.11 , Fig.11 is a schematic diagram of a data assembly method in an embodiment of the present application. The first feature data and the second feature data can be as follows: Fig.11 After being assembled in a manner, it is input into the feature union layer 906 for data fusion.

[0127] After obtaining the second feature data, the second feature data can be input into the first classifier to obtain the first probability that the gesture action in the multiple images belongs to each type, and the first gesture action is classified into the type with the largest first probability corresponding to the first gesture action, and the first gesture action is any gesture action among the gesture actions in the multiple images. The second feature data is input into the second classifier to obtain the second probability that the gesture action in the multiple images belongs to each attribute, and the first gesture action is classified into the attribute with the largest second probability corresponding to the first gesture action. The output values ​​of the first classifier and the second classifier can be passed to two outputs, one output can be classified using softmax logistic regression (softmax regression) for characterization type classification, and the other output can be classified using sigmoid function for characterization attribute classification.

[0128] exist Figure 6 In the dynamic gesture recognition method described, multiple images containing gesture actions are obtained, and Figure 3 The spatial feature module 301, the temporal feature module 302 and the classification module 303 in the gesture recognition model 300 shown in the figure identify multiple images and obtain the types and attributes of gestures in the multiple images. Among them, the first feature data including the spatial features of gestures in the multiple images is obtained by the spatial feature module 301, and the second feature data including the temporal features of the first feature data in the time dimension is obtained by the temporal feature module 302. The types and attributes of gestures in the multiple images are obtained by the classification module 303. Therefore, based on the above-mentioned gesture recognition model 300 and the dynamic gesture recognition method, more comprehensive feature information of gestures can be obtained from multiple images, thereby improving the recognition accuracy of hand movements.

[0129] The following provides a Figure 2 Another dynamic gesture recognition method based on the system architecture of . Fig.12 , Fig.12 FIG. 1 is a flow chart of another dynamic gesture recognition method in an embodiment of the present invention. Fig.12 As shown, the dynamic gesture recognition method may include the following steps.

[0130] 1201. Acquire multiple images containing gesture actions.

[0131] Among them, step 1201 is the same as step 601. Please refer to step 601 for detailed description, which will not be repeated here.

[0132] 1202. Recognize multiple images through a gesture recognition model to obtain types and attributes of gesture actions in the multiple images.

[0133] Among them, step 1202 is the same as step 602. Please refer to step 602 for detailed description, which will not be repeated here.

[0134] 1203. When the attribute of the gesture action in the multiple images is outgoing, execute a function corresponding to the type of the gesture action.

[0135] After obtaining the types and attributes of the gesture actions in the multiple images, when the attributes of the gesture actions in the multiple images are outward, the function corresponding to the type of the gesture action can be executed. For example, if the types and attributes of the gesture actions are respectively left and outward, a left command is executed in response to the gesture action; if the types and attributes of the gesture actions are respectively left and return, no command is executed.

[0136] In the embodiment of the present application, the gesture recognition model may be an AI model. Before using the gesture recognition model for recognition, the initial gesture recognition model needs to be trained so that the trained gesture recognition model has the ability to recognize the types and attributes of gestures in multiple images. The gesture recognition model in the present application may have the ability to determine the types and attributes of gestures (outbound and return). The following provides a method based on Figure 2 A gesture recognition model training method based on the system architecture. Fig.13 , Fig.13 FIG. 1 is a flow chart of a method for training a gesture recognition model according to an embodiment of the present invention. Fig.13 As shown, the gesture recognition model training method may include the following steps.

[0137] 1301. Obtain multiple sample images carrying annotation information.

[0138] In the process of training the initial gesture recognition model, special training data is required for training. Based on the model capability requirements, analysis is performed and sample images with annotation information are required for training. The sample images record the gesture actions. The annotation information may include the type and attribute of the gesture actions in the sample images. The type information of the gesture action is used to indicate the type of the gesture action, for example: "continuous left flip", "continuous right flip", "continuous up flip", "continuous down flip", etc. The attribute information is used to indicate the attribute of the gesture action in multiple images. The attribute information may be the outbound or return trip. For example, the gesture action is the outbound trip in images 0-7, and the gesture action is the return trip in images 8-15. It should be noted that the annotation information can be saved in files such as extensible markup language (XML) or JavaScript object notation (JSON).

[0139] The annotation information including the type and attribute of the gesture action can be obtained by detecting the sample image using a gesture action detection algorithm, or by obtaining the type information and attribute information of the gesture action recorded in the sample image through manual annotation.

[0140] 1302. Train an initial gesture recognition model according to a plurality of sample images carrying annotation information to obtain a gesture recognition model.

[0141] After obtaining multiple sample images carrying annotation information, the multiple sample images carrying annotation information constitute a training set, and the model is trained using the training samples in the training set. First, the initial gesture recognition model is determined. In an embodiment of the present application, the initial gesture recognition model can be an AI model, specifically a deep neural network model, which can identify the category of gesture actions and the attributes of gesture actions.

[0142] The initial gesture recognition model may include a spatial feature module, a temporal feature module, a loss function calculation module, and a classification module.

[0143] First, initialize the parameters of the initial gesture recognition model, then input the sample image into the spatial module of the initial gesture recognition model, extract the spatial features of the input sample image, and obtain the abstract features. The hand features and hand key point features in the gesture can be detected by the spatial feature module. Then, the time domain features corresponding to the time information of the hand key point features can be obtained by the time domain feature model. Fig.14 , Fig.14 FIG. 1 is a schematic diagram of feature extraction in gesture recognition in an embodiment of the present invention. Fig.14As shown in the figure, for the sample image, the spatial feature module can detect the position information of the hand in the image, mark the position information of the hand in the image with a rectangular frame, and then perform key point detection on the hand in the rectangular frame. The temporal feature module can obtain the temporal features corresponding to the time information of the key points in multiple images, and the differential method can be used to obtain the temporal features of the key points in the previous and next images. For details, please refer to Fig.15 , Fig.15 FIG. 1 is a schematic diagram of a time domain feature extraction in an embodiment of the present invention. Fig.15 As shown in FIG. 1 , the key points of the hand may include fingertip points and phalange key points, and the displacement information and velocity information of the key points of the hand may be extracted. The displacement of the key points of the hand relative to the image may be expressed by S x and S y To indicate that S x represents the displacement in the x direction, S y Indicates the displacement in the y direction. When the frame rates of the previous and next images are the same, S x and S y It can also represent the moving speed of the key points of the hand. According to a large amount of data experience, the gesture action has different moving speeds for different attributes of the outbound and return journeys. Therefore, when the frame rates of the previous and next images are different, the moving speed of the key points of the hand can also be extracted. For different key points, multiple displacement information and speed information can be extracted. The displacement information and speed information of multiple different hand key points are synthesized into a vector to obtain the vector F, F = [f1, f2, f3, ..., f n ].

[0144] The features extracted by the spatial feature module and the temporal feature module are detected and recognized, and the type and attribute of the gesture action in the input sample image are predicted and output to the loss function calculation module. Then, the annotation information corresponding to the sample image is also input to the loss function calculation module. The loss function calculation module compares the predicted result with the annotation information corresponding to the sample image, and calculates the loss function. The weight parameters in the initial gesture recognition model and the classifier are updated and adjusted using optimization algorithms such as backpropagation (BP), gradient descent (GD) or stochastic gradient descent (SGD) with the loss function as the objective function. Sample images with annotation information are input in a loop in sequence, and the above training process is continuously iterated until the preset probability corresponding to the sample image obtained based on the initial gesture recognition model and the classifier is consistent with the probability of the annotation information corresponding to the sample image reaching the expected value, which means that a gesture recognition model that meets the requirements has been obtained, and the training can be terminated to obtain the gesture recognition model, that is, the gesture recognition model has the function of identifying the type and attribute of gesture actions in multiple images, and can be used for dynamic gesture recognition.

[0145] See also Fig.16 , Fig.16 FIG. 1 is a schematic diagram of a dynamic gesture recognition device disclosed in an embodiment of the present invention. Fig.16 As shown, the dynamic gesture recognition device 1600 may include:

[0146] The first acquisition unit 1601 is used to acquire multiple images containing gesture actions;

[0147] The recognition unit 1602 is used to recognize multiple images through a gesture recognition model to obtain types and attributes of gesture actions in the multiple images, where the attributes include outbound and return paths.

[0148] In an optional implementation, the apparatus 1600 may further include:

[0149] The second acquisition unit 1603 is used to acquire multiple sample images carrying annotation information, where the sample images are multiple images containing gestures, and the annotation information includes the type and attribute of the gestures in the sample images;

[0150] The training unit 1604 is used to train the initial gesture recognition model according to a plurality of sample images carrying annotation information to obtain a gesture recognition model.

[0151] In an optional implementation, the gesture recognition model includes a spatial feature module, a temporal feature module, and a classification module;

[0152] The identification unit 1602 is specifically configured to:

[0153] Inputting the plurality of images into a spatial feature module to obtain first feature data, the first feature data including spatial features of gesture actions in the plurality of images;

[0154] Inputting the first feature data into a time domain feature module to obtain second feature data, where the second feature data includes the time domain features of the first feature data in the time dimension;

[0155] The second feature data is input into the classification module to obtain the types and attributes of the gesture actions in the multiple images.

[0156] In an optional implementation, the time domain feature module includes a dimension transformation layer, a convolution layer, a BN layer, a ReLu layer, a maximum pooling layer and a feature combination layer;

[0157] The identification unit 1602 is used to input the first feature data into the time domain feature module to obtain the second feature data, and when the second feature data includes the time domain feature of the first feature data in the time dimension, it is specifically used to:

[0158] Determine first time-domain feature data corresponding to the first feature data in the time dimension through a dimensionality transformation layer according to time information of the plurality of images;

[0159] Performing convolution processing on the first time domain feature data through a convolution layer to obtain second time domain feature data;

[0160] The second time domain feature data is sequentially passed through a BN layer, a ReLu layer, a maximum pooling layer, and a feature combination layer to obtain the second feature data.

[0161] In an optional implementation, the identification unit 1602 is used to perform convolution processing on the first time domain feature data through a convolution layer to obtain the second time domain feature data, specifically for:

[0162] Using a first preset number of one-dimensional convolution layers with different convolution kernel sizes to perform convolution processing on the first time domain feature data to obtain second time domain feature data, where the second time domain feature data includes a first preset number of feature data of different scales;

[0163] The second time domain feature data is sequentially passed through the BN layer, the ReLu layer, the maximum pooling layer and the feature joint layer to obtain the second feature data including:

[0164] The first preset number of feature data of different scales are sequentially passed through a BN layer, a ReLu layer, a maximum pooling layer and a feature combination layer to obtain second feature data.

[0165] In an optional implementation, the gesture recognition model further includes a first classifier and a second classifier, and the recognition unit 1602 is used to input the second feature data into the classification module to obtain the types and attributes of gesture actions in multiple images, specifically for:

[0166] Inputting the second feature data into the first classifier to obtain a first probability that the gesture actions in the plurality of images belong to each type;

[0167] Classifying the first gesture action into a first most likely type corresponding to the first gesture action, where the first gesture action is any gesture action among the gesture actions in the multiple images;

[0168] Inputting the second feature data into a second classifier to obtain a second probability that the gesture actions in the plurality of images belong to each attribute;

[0169] The first gesture action is classified into the attribute with the second highest probability corresponding to the first gesture action.

[0170] In an optional implementation, the first acquiring unit 1601 is specifically configured to:

[0171] Get the video to be identified;

[0172] An image is extracted from the video to be identified every second preset number of images to obtain multiple images including gesture actions.

[0173] In an optional implementation, the device 1600 may further include: an execution unit 1605, configured to execute a function corresponding to the type of the gesture action when the attribute of the gesture action in the multiple images is outgoing.

[0174] See also Fig.17 , Fig.17 FIG. 1 is a schematic diagram of a computing device disclosed in an embodiment of the present invention. Fig.17 As shown, the computing device 1700 may include: a memory 1701, a processor 1702, a communication interface 1703 and a bus 1704. The memory 1701, the processor 1702 and the communication interface 1703 are connected to each other through the bus 1704.

[0175] The memory 1701 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1701 may store a program. When the program stored in the memory 1701 is executed by the processor 1702, the processor 1702 and the communication interface 1703 are used to execute the various steps of the dynamic gesture recognition method of the embodiment of the present application.

[0176] Processor 1702 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the dynamic gesture recognition device of the embodiment of the present application, or to execute the dynamic gesture recognition method of the method embodiment of the present application.

[0177] The processor 1702 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the dynamic gesture recognition method of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1702. The above-mentioned processor 1702 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1701, and the processor 1702 reads the information in the memory 1701, and combines its hardware to complete the functions required to be performed by the units included in the dynamic gesture recognition device of the embodiment of the present application, or executes the dynamic gesture recognition method of the method embodiment of the present application.

[0178] The communication interface 1703 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 1700 and other devices or communication networks. The bus 1704 may include a path for transmitting information between the various components of the device 1700 (e.g., the memory 1701, the processor 1702, and the communication interface 1703). The specific implementation of the above-mentioned various functional devices can refer to the relevant description of the dynamic gesture recognition method in the above-mentioned embodiment, and the embodiments of this application will not be repeated.

[0179] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0180] Those skilled in the art will appreciate that the functions described in conjunction with the various illustrative logic blocks, modules, and algorithm steps disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions described in the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media, such as data storage media, or includes any communication media that facilitates the transfer of computer programs from one place to another (e.g., according to a communication protocol). In this manner, computer-readable media may generally correspond to (1) non-temporary tangible computer-readable storage media, or (2) communication media, such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in this application. A computer program product may include computer-readable media.

[0181] As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer. Also, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave are included in the definition of media. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals or other temporary media, but are actually directed to non-temporary tangible storage media. As used herein, disks and optical disks include compact disks (CDs), laser optical disks, optical optical disks, digital versatile disks (DVDs), and Blu-ray disks, where disks typically reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0182] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor" as used herein may refer to any of the aforementioned structures or any other structures suitable for implementing the techniques described herein. Additionally, in some aspects, the functions described by the various illustrative logic blocks, modules, and steps described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated in a combined codec. Moreover, the techniques may be fully implemented in one or more circuits or logic elements.

[0183] The techniques of the present application may be implemented in a variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this application to emphasize functional aspects of devices for performing the disclosed techniques, but do not necessarily need to be implemented by different hardware units. In fact, as described above, the various units may be combined in a codec hardware unit in conjunction with appropriate software and / or firmware, or provided by interoperating hardware units (including one or more processors as described above).

[0184] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A dynamic gesture recognition method, characterized in that: include: Get multiple images containing gesture actions; Recognize the multiple images through a gesture recognition model to obtain types and attributes of gesture actions in the multiple images, wherein the attributes include outbound and return paths; When the attribute of the recognized gesture action is outgoing, executing the function corresponding to the recognized gesture action; The gesture recognition model includes a spatial feature module, a temporal feature module and a classification module; The step of recognizing the plurality of images by using a gesture recognition model to obtain types and attributes of gesture actions in the plurality of images includes: Inputting the plurality of images into the spatial feature module to obtain first feature data, wherein the first feature data includes spatial features of gesture actions in the plurality of images; Inputting the first feature data into the time domain feature module to obtain second feature data, wherein the second feature data includes the time domain features of the first feature data in the time dimension; The second feature data is input into the classification module to obtain the types and attributes of the gesture actions in the multiple images.

2. The method according to claim 1, characterized in that The method further comprises: Acquire a plurality of sample images carrying annotation information, wherein the sample images are a plurality of images containing gesture actions, and the annotation information includes types and attributes of the gesture actions in the sample images; The gesture recognition model is obtained by training an initial gesture recognition model according to the multiple sample images carrying the annotation information.

3. The method according to claim 1, characterized in that The time domain feature module includes a dimension transformation layer, a convolution layer, a batch normalization BN layer, a rectified linear unit ReLu layer, a maximum pooling layer and a feature combination layer; The step of inputting the first feature data into the time domain feature module to obtain second feature data, wherein the second feature data includes the time domain features of the first feature data in the time dimension, including: Determine, through the dimensionality transformation layer, first time-domain feature data corresponding to the first feature data in the time dimension according to the time information of the multiple images; Performing convolution processing on the first time domain feature data through the convolution layer to obtain second time domain feature data; The second time domain feature data is sequentially passed through the BN layer, the ReLu layer, the maximum pooling layer and the feature combination layer to obtain the second feature data.

4. The method according to claim 3, characterized in that The step of performing convolution processing on the first time domain feature data through a convolution layer to obtain second time domain feature data includes: Using a first preset number of one-dimensional convolution layers with different convolution kernel sizes to perform convolution processing on the first time domain feature data to obtain second time domain feature data, wherein the second time domain feature data includes a first preset number of feature data of different scales; The step of sequentially passing the second time domain feature data through the BN layer, the ReLu layer, the maximum pooling layer, and the feature combination layer to obtain the second feature data includes: The first preset number of feature data of different scales are sequentially passed through the BN layer, the ReLu layer, the maximum pooling layer and the feature combination layer to obtain the second feature data.

5. The method according to any one of claims 1 to 4, characterized in that: The gesture recognition model further includes a first classifier and a second classifier, and the inputting of the second feature data into the classification module to obtain the types and attributes of the gesture actions in the multiple images includes: Inputting the second feature data into the first classifier to obtain a first probability that the gesture actions in the plurality of images belong to each type; Classifying a first gesture action into a first most likely type corresponding to the first gesture action, wherein the first gesture action is any gesture action among the gesture actions in the multiple images; Inputting the second feature data into the second classifier to obtain a second probability that the gesture actions in the plurality of images belong to each attribute; The first gesture action is classified into the attribute with the second highest probability corresponding to the first gesture action.

6. The method according to any one of claims 1 to 4, characterized in that: The acquiring of multiple images containing gesture actions comprises: Get the video to be identified; An image is extracted from the video to be identified every second preset number of images to obtain multiple images containing gesture actions.

7. A dynamic gesture recognition device, characterized in that: include: A first acquisition unit, used to acquire multiple images containing gesture actions; A recognition unit, configured to recognize the plurality of images through a gesture recognition model, obtain types and attributes of gesture actions in the plurality of images, wherein the attributes include outbound and return paths; and when the attribute of the recognized gesture action is outbound, execute a function corresponding to the recognized gesture action; The gesture recognition model includes a spatial feature module, a temporal feature module and a classification module; The identification unit is specifically used for: Inputting the plurality of images into the spatial feature module to obtain first feature data, wherein the first feature data includes spatial features of gesture actions in the plurality of images; Inputting the first feature data into the time domain feature module to obtain second feature data, wherein the second feature data includes the time domain features of the first feature data in the time dimension; The second feature data is input into the classification module to obtain the types and attributes of the gesture actions in the multiple images.

8. The device according to claim 7, characterized in that The device also includes: A second acquisition unit is used to acquire a plurality of sample images carrying annotation information, wherein the sample images are a plurality of images containing gesture actions, and the annotation information includes the type and attribute of the gesture actions in the sample images; A training unit is used to train an initial gesture recognition model according to the multiple sample images carrying annotation information to obtain the gesture recognition model.

9. The device according to claim 7, characterized in that The time domain feature module includes a dimension transformation layer, a convolution layer, a batch normalization BN layer, a rectified linear unit ReLu layer, a maximum pooling layer and a feature combination layer; The recognition unit is used to input the first feature data into the time domain feature module to obtain second feature data, and when the second feature data includes the time domain feature of the first feature data in the time dimension, it is specifically used to: Determine, through the dimensionality transformation layer, first time-domain feature data corresponding to the first feature data in the time dimension according to the time information of the multiple images; Performing convolution processing on the first time domain feature data through the convolution layer to obtain second time domain feature data; The second time domain feature data is sequentially passed through the BN layer, the ReLu layer, the maximum pooling layer and the feature combination layer to obtain the second feature data.

10. The device according to claim 9, characterized in that The recognition unit is used to perform convolution processing on the first time domain feature data through the convolution layer to obtain the second time domain feature data, specifically for: Using a first preset number of one-dimensional convolution layers with different convolution kernel sizes to perform convolution processing on the first time domain feature data to obtain second time domain feature data, wherein the second time domain feature data includes a first preset number of feature data of different scales; The step of sequentially passing the second time domain feature data through the BN layer, the ReLu layer, the maximum pooling layer, and the feature combination layer to obtain the second feature data includes: The first preset number of feature data of different scales are sequentially passed through the BN layer, the ReLu layer, the maximum pooling layer and the feature combination layer to obtain the second feature data.

11. The device according to any one of claims 7 to 10, characterized in that: The gesture recognition model further includes a first classifier and a second classifier, and the recognition unit is used to input the second feature data into the classification module to obtain the types and attributes of the gesture actions in the multiple images, specifically for: Inputting the second feature data into the first classifier to obtain a first probability that the gesture actions in the plurality of images belong to each type; Classifying a first gesture action into a first most likely type corresponding to the first gesture action, wherein the first gesture action is any gesture action among the gesture actions in the multiple images; Inputting the second feature data into the second classifier to obtain a second probability that the gesture actions in the plurality of images belong to each attribute; The first gesture action is classified into the attribute with the second highest probability corresponding to the first gesture action.

12. The device according to any one of claims 7 to 10, characterized in that: The first acquisition unit is specifically used to: Get the video to be identified; An image is extracted from the video to be identified every second preset number of images to obtain multiple images containing gesture actions.

13. A computing device, characterized in that: The computing device comprises a processor and a memory, wherein the memory is used to store a program, and the processor executes the program stored in the memory. When the program stored in the memory is executed, the computing device implements the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are called by a computer, the computer is used to enable the computer to implement the method according to any one of claims 1 to 6.

Images