A method for gesture action recognition

Abstract

This invention discloses a method for gesture recognition. Based on the existing ResNet network, three network modules are proposed: a spatial attention module that calculates the RGB differences between frames separated by long distances and uses this difference to calculate a spatial attention map, thereby assigning different weights to different spatial positions; a long-short distance temporal displacement module that simultaneously performs long-distance temporal displacement and short-distance temporal displacement operations, simultaneously modeling long-term and short-term temporal information; and a label relationship module that models semantic information between different categories. This module first uses a semantic model to encode the labels of gesture categories and then uses the label relationship module to model the relationships between categories to assist in classification. This invention can improve the accuracy of gesture recognition with reduced computational cost and memory consumption.

Description

Technical Field

[0001] The present invention belongs to the technical field of pattern recognition, and in particular relates to a method for gesture recognition. Background Art

[0002] Gesture recognition is a fundamental problem in computer vision, which aims to identify gestures in video sequences. This technology has received increasing attention due to its huge application value.

[0003] In related technologies, gesture recognition models based on deep neural networks typically use convolutional neural networks to recognize gestures. Specifically, they include 2D convolutional neural networks and 3D convolutional neural networks. Although 3D convolutional networks can simultaneously model temporal and spatial features, 3D convolutions often require more parameters, which leads to large and time-consuming computations. When using 2D convolutional neural networks to recognize actions, only spatial features can be modeled, not temporal features. Previous methods typically used additional optical flow information as an additional stream to model temporal features. However, optical flow information is usually computationally intensive. Recent methods use additional modules to extract temporal information.

[0004] However, these methods based on additional modules and 2D convolutional neural networks also have some problems. First, existing methods cannot efficiently model both long-term and short-term temporal information. In addition, existing methods cannot pay close attention to where gestures mainly occur in the entire video. Finally, existing methods usually only use visual information without using the relationship information between categories. Summary of the Invention

[0005] To overcome the shortcomings of the existing technology, the present invention provides a method for gesture recognition. Based on the existing ResNet network, three network modules are proposed: a spatial attention module, which calculates the RGB differences between frames separated by long distances and calculates a spatial attention map based on this difference, thereby assigning different weights to different spatial positions; a long-short distance temporal displacement module, which performs long-distance temporal displacement and short-distance temporal displacement operations simultaneously, modeling both long-term and short-term temporal information; and a label relationship module, which models semantic information between different categories. This module first uses a semantic model to encode the labels of gesture categories and then uses the label relationship module to model the relationships between categories to assist in classification. The present invention can improve the accuracy of gesture recognition with lower computational cost and memory consumption.

[0006] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0007] Step 1: Obtain T video frames from the video to be processed, perform data preprocessing and use them as input for the gesture recognition model;

[0008] Step 2: Feed the processed video frames into the ResNet neural network with the added spatial attention module and long-short distance temporal displacement module. For each bottleneck in ResNet, the specific operations are as follows:

[0009] Step 2-1: For the feature map X={X(1),...X(T)}∈R sent to Bottleneck T×C×H×W , X(1),...X(T) represent the input images of each frame, C, H, and W represent the number of channels of the input image and the height and width of the image respectively; first, the feature map X is passed through the spatial attention module to calculate the RGB difference between long-distance frames, and the spatial attention map is calculated based on the difference, thereby assigning different weights to different spatial positions; the details are as follows:

[0010] First, calculate the RGB difference between frames separated by T / 2, that is, calculate the RGB difference between X(t) and X(t+T / 2):

[0011] D(t)=X(t+T / 2)-X(t),t∈{1,2,...T / 2}

[0012] Where D(t) is the RGB difference between frames separated by T / 2. Next, D(t) is concatenated along the time dimension to obtain the final motion pattern change matrix D:

[0013] D=[D(1),...,D(T / 2)∈R T / 2×C×H×W

[0014] The average pooling layer in the time dimension is used to pool the motion mode change matrix D to obtain the location where the mode change occurs in the entire video:

[0015] D avg =Pool(D),D avg ∈R 1×C×H×W

[0016] Where D avg represents the result of pooling the motion pattern change matrix D along the time dimension;

[0017] Next, a 3×3 convolutional layer is used to avg Perform convolution and then pass through a sigmoid layer to obtain the spatial attention map M:

[0018] M=σ(conv*D avg ),M∈R 1×C×H×W

[0019] Where, σ(.) represents the sigmoid function; conv represents a 3×3 convolutional layer;

[0020] Finally, the corresponding elements of the input X and the spatial attention map M matrices are multiplied to obtain the output of the final spatial attention module:

[0021] X o =X⊙M

[0022] Step 2-2: Long and short distance time displacement module;

[0023] First, the input of the long-short distance time displacement module is divided into three parts along the channel dimension, that is, 1 / 8 of the channel is selected as the short-term time displacement part X s , 1 / 8 of the channel is used as the long-term time displacement part X l , the remainder is X o ;

[0024] For the short-term displacement part X s , make a short time displacement, select X s Half of the channels move forward 1 time step, and the other half move backward 1 time step;

[0025] For the long-term time displacement part X l , make a long time displacement, select X l Half of the channels move forward by T / 2 time steps, and the other half move backward by T / 2 time steps;

[0026] For the remaining X o remains unchanged;

[0027] Finally, the weighted sum of the above three parts is calculated:

[0028] X'=X s ×α+X l ×β+X o

[0029] Where α and β represent the importance coefficients of long-distance time displacement and short-distance time displacement, respectively;

[0030] Step 3: After passing through multiple Bottlenecks in step 2, the visual information v for classification is obtained; in the category relationship module, the visual information v and the semantic information constructed from the semantic encoding model are integrated for classification.

[0031] Step 3-1: Construct semantic information; encode the category labels of different gestures through the phrase-Bert semantic encoding model to obtain semantic information W e;

[0032] Step 3-2: Model the relationship between category labels; Based on the semantic information obtained in step 3-1, the semantic information encoding is transformed into Q, K, and V through three linear layers; Q, K, and V represent the query matrix, keyword matrix, and the sum obtained by transforming the semantic information encoding through the linear layer respectively;

[0033] Q=f·W e ,K=g·W e ,V=h·W e

[0034] Where f, g, and h represent the three linear layers that transform semantic information; N represents the number of action categories, that is, the number of labels.

[0035] Calculate the self-attention matrix through Q and K, and then multiply the self-attention matrix with V to get Attn;

[0036]

[0037] Align Attn and W through a linear layer l e dimension, and align Attn with W e Add:

[0038] G e =l·Attn+W e

[0039] Step 3-3: Replace the original classification layer of ResNet with a classification layer Glinear that combines both semantic information and visual information;

[0040] Through the self-attention module in step 3-2, a new semantic space containing inter-class relationships is obtained. This semantic space is combined with the previous visual information v to obtain the classification result p:

[0041] Glinear:p=softmax(G e Pv+b)

[0042] in is a learnable parameter used to fuse visual and semantic information;

[0043] Step 4: Average the results of step 3 over time to obtain the final classification and recognition results.

[0044] Preferably, the gesture action recognition model is trained using the following steps:

[0045] (1) obtaining training samples, each training sample including a preset number of video frames to be tested and a real behavior category corresponding to the video frames to be tested;

[0046] (2) inputting the training sample into the neural network model to be trained, where the neural network model to be trained is a preset initial neural network model;

[0047] (3) Using the output of the neural network model to be trained, determine the predicted true behavior category of each training sample;

[0048] (4) Determine the loss value based on the predicted true behavior category of each training sample, the true behavior category and the preset loss function, and the loss function used is the cross entropy loss function;

[0049] (5) judging whether the neural network model to be trained has converged based on the loss value; if it has converged, the neural network model to be trained is the trained gesture action recognition model;

[0050] (6) If convergence has not occurred, adjust the network parameters of the neural network model to be trained and return to (2) until convergence occurs.

[0051] The beneficial effects of the present invention are as follows:

[0052] The spatial attention module in this invention obtains a spatial attention map based on the RGB differences between frames, assigning different weights to different spatial locations, thereby making the network pay more attention to the spatial locations where the action mainly occurs. The long-short distance temporal displacement module exchanges information between frames of different distances through long-term and short-term temporal displacement operations, thereby simultaneously modeling long-term and short-term temporal information, thereby paying more attention to the locations where gestures mainly occur. The label relationship module is inserted before the classification layer of the network. It fully models the relationship between categories based on the semantic information of the labels to assist in classification. This invention can improve the accuracy of gesture recognition with low computational cost and memory consumption.

[0053] On two commonly used datasets, EgoGesture and NVGesture, the method of the present invention can achieve the highest recognition accuracy among similar methods, which are 93.77% and 81.73% respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Flowchart of the method of the present invention.

[0055] Figure 2 Diagram of the spatial attention module of the method of the present invention.

[0056] Figure 3 Diagram of the long and short distance time displacement module of the method of the present invention.

[0057] Figure 4 The label relationship module diagram of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further described below with reference to the accompanying drawings and examples.

[0059] The method provided by the present invention is based on the existing ResNet network. The present invention mainly proposes three network modules. First, a spatial attention module is provided. In this module, the RGB difference between frames separated by a long distance is calculated, and the spatial attention map is calculated based on the difference, so that different spatial positions are given different weights. Second, based on the idea of time displacement, a long-short distance time displacement module is provided. In this module, long-distance time displacement and short-distance time displacement operations are performed simultaneously, so that long-term time information and short-term time information can be modeled at the same time. Third, a label relationship module is provided to model semantic information between different categories. This module first uses a semantic model to encode the label of the gesture category, and then uses the label relationship module to model the relationship between the categories to assist classification. The long-short distance time displacement module and the spatial attention module are both inserted into each Bottleneck of ResNet. The label relationship module is inserted before the last classification layer of ResNet.

[0060] like Figure 1 As shown, a method for gesture recognition includes the following steps:

[0061] Step 1: Obtain T video frames from the video to be processed, perform data preprocessing and use them as input for the gesture recognition model;

[0062] Step 2: Feed the processed video frames into the ResNet neural network with the added spatial attention module and long-short distance temporal displacement module. For each bottleneck in ResNet, the specific operations are as follows:

[0063] Step 2-1: For the feature map X={X(1),...X(T)}∈R sent to Bottleneck T×C×H×W First, the feature map X is passed through the spatial attention module. The spatial attention module calculates the RGB differences between different frames and uses this difference to calculate the spatial attention map, thereby assigning different weights to different spatial locations. When calculating RGB differences, unlike other methods that directly calculate the RGB differences between adjacent frames, we calculate the RGB differences between frames over a long distance. This allows us to better focus on the areas in the image where gestures are primarily occurring. The details are as follows:

[0064] First, calculate the RGB difference between frames separated by T / 2, that is, calculate the RGB difference between X(t) and X(t+T / 2):

[0065] D(t)=X(t+T / 2)-X(t),t∈{1,2,...T / 2}

[0066] Where D(t) is the RGB difference between frames separated by T / 2, which reflects the module in which the motion in the picture mainly occurs. Next, D(t) is spliced together along the time dimension to obtain the final motion pattern change matrix D:

[0067] D=[D(1),...,D(T / 2)∈R T / 2×C×H×W

[0068] The average pooling layer in the time dimension is used to pool the motion mode change matrix D to obtain the location where the mode change occurs in the entire video:

[0069] D avg =Pool(D),D avg ∈R 1×C×H×W

[0070] Next, a 3×3 convolutional layer is used to avg Perform convolution and then pass through a sigmoid layer to obtain the spatial attention map M:

[0071] M=σ(conv*D avg ),M∈R 1×C×H×W

[0072] Where, σ(.) represents the sigmoid function;

[0073] Finally, the corresponding elements of the input X and the spatial attention map M matrices are multiplied to obtain the output of the final spatial attention module:

[0074] X o =X⊙M

[0075] Through the above-mentioned spatial attention module, we can better focus on where the movement pattern mainly occurs;

[0076] Step 2-2: Long and short distance time displacement module;

[0077] The temporal displacement module effectively exchanges information between different frames, thereby enabling temporal information modeling. Previous temporal displacement modules exchanged information between adjacent frames, which only captured short-term temporal information but not long-term temporal information. Therefore, a long-short distance temporal displacement module was proposed. This module performs both long-distance and short-distance temporal displacement simultaneously, thus capturing both long-term and short-term temporal information.

[0078] First, the input of the long-short distance time displacement module is divided into three parts along the channel dimension, that is, 1 / 8 of the channel is selected as the short-term time displacement part X s , 1 / 8 of the channel is used as the long-term time displacement part X l , the remainder is X o ;

[0079] For the short-term displacement part X s , make a short time displacement, select X s Half of the channels move forward 1 time step, and the other half move backward 1 time step;

[0080] For the long-term time displacement part X l , make a long time displacement, select X l Half of the channels move forward by T / 2 time steps, and the other half move backward by T / 2 time steps;

[0081] For the remaining X o remains unchanged;

[0082] Finally, the weighted sum of the above three parts is calculated:

[0083] X'=X s ×α+X l ×β+X o

[0084] Where α and β represent the importance coefficients of long-distance time displacement and short-distance time displacement, respectively;

[0085] Step 3: After passing through multiple Bottlenecks in step 2, the visual information v for classification is obtained; in the category relationship module, the visual information v and the semantic information constructed from the semantic encoding model are integrated for classification.

[0086] Step 3-1: Construct semantic information; encode the category labels of different gestures through the phrase-Bert semantic encoding model to obtain semantic information W e ;

[0087] Step 3-2: Model the relationship between category labels; Based on the semantic information obtained in step 3-1, the semantic information is encoded and converted into Q, K, and V through three linear layers;

[0088] Q=f·W e ,K=g·W e ,V=h·W e

[0089] Calculate the self-attention matrix through Q and K, and then multiply the self-attention matrix with V to get Attn;

[0090]

[0091] Align Attn and W through a linear layer l e dimension, and align Attn with W e Add:

[0092] G e =l·Attn+W e

[0093] Step 3-3: Replace the original classification layer of ResNet with a classification layer Glinear that combines both semantic information and visual information;

[0094] Through the self-attention module in step 3-2, a new semantic space containing inter-class relationships is obtained. This semantic space is combined with the previous visual information v to obtain the classification result p:

[0095] Glinear:p=softmax(G e Pv+b)

[0096] in is a learnable parameter used to fuse visual and semantic information;

[0097] Step 4: Average the results of step 3 over time to obtain the final classification and recognition results. Specific embodiment:

[0099] Specifically, for an input video V,

[0100] 1: First, extract video frames, then use sparse sampling to sample T frames. Optionally, T represents a preset number of video frames selected from the video to be processed. In this embodiment, T = 8. The T video frames are scaled, randomly cropped, and mean-normalized to obtain video data with a dimension of 3*T*224*224. This video data serves as the input to the gesture recognition model.

[0101] 2: The processed video frames are fed into a ResNet with added long- and short-distance temporal displacement modules and spatial attention modules. For each Bottleneck in the ResNet, the specific operations are as follows:

[0102] 2-1: For the feature map X={X(1),...X(T)}∈R sent to Bottleneck T×C×H×W , first pass it through Figure 2 The spatial attention module shown.

[0103] The spatial attention module calculates the RGB differences between frames and creates a spatial attention map based on these differences, assigning different weights to different spatial locations. Unlike other methods that directly calculate the RGB differences between adjacent frames, we instead calculate the RGB differences between frames over long distances. This allows us to better focus on the areas of the image where gestures are primarily occurring.

[0104] The specific calculation steps are as follows:

[0105] First, calculate the RGB difference between frames separated by T / 2, that is, calculate the RGB difference between X(t) and X(t+T / 2):

[0106] D(t)=X(t+T / 2)-X(t),t∈{1,2,...T / 2}

[0107] Where D(t) is the RGB difference between frames separated by T / 2. This difference reflects the module where the motion in the picture mainly occurs. Next, D(t) is spliced along the time dimension to obtain the final motion pattern change matrix D:

[0108] D=[D(1),...,D(T / 2)∈R T / 2×C×H×W

[0109] Then, the average pooling layer in the time dimension is used to pool the motion pattern change matrix D to obtain the position where the main pattern changes occur in the entire video.

[0110] D avg =Pool(D),D avg ∈R 1×C×H×W

[0111] Next, a 3×3 convolutional layer is used to convolve Davg, and then a sigmoid layer is passed to obtain the spatial attention map:

[0112] M=σ(conv*D avg ),M∈R 1×C×H×W

[0113] Finally, the corresponding elements of the two matrices of input x and spatial attention map M are multiplied to obtain the output of the final spatial attention module:

[0114] X o =X⊙M

[0115] Through the above-mentioned spatial attention module, we can better focus on where the movement pattern mainly occurs.

[0116] 2-2: Through the long and short distance time displacement module;

[0117] After the spatiotemporal features pass through the above-mentioned spatial attention module, they are fed into Figure 3 The long and short distance time displacement modules are shown.

[0118] The time displacement module effectively exchanges information between different frames, thereby enabling temporal information modeling. Previous time displacement modules exchanged information between adjacent frames, which only captured short-term temporal information but not long-term temporal information. Therefore, the present invention proposes a long-short distance time displacement module, which simultaneously performs long-distance and short-distance time displacement, thus capturing both long-term and short-term temporal information.

[0119] The specific steps are as follows:

[0120] For the input X of this module, it is first divided into three parts along the channel dimension, that is, 1 / 8 of the channel is selected as the short-term time displacement part X s , the other 1 / 8 of the channel is used as the long-term time displacement part X l and the remaining part X o .

[0121] For the short-term displacement part X s , and do a short time displacement. Select X s Half of the channels are shifted forward by 1 time step, and the other half are shifted backward by 1 time step.

[0122] For the short-term displacement part X l , we will do a long time shift. We choose X l Half of the channels are shifted forward by T / 2 time steps, and the other half are shifted backward by T / 2 time steps.

[0123] For the remaining X o remains unchanged.

[0124] Finally, calculate the weighted sum of the above three parts:

[0125] X'=X s ×α+X l ×β+X o

[0126] Where α and β indicate the importance of long-distance and short-distance time displacements, respectively. In this embodiment, we set α = 1 and β = 1.5 to achieve the best effect.

[0127] 3: After several Bottleneck steps in step 2, the visual information v for classification is obtained.

[0128] Different from the original ResNet network which directly feeds the visual information into a linear layer for classification, the present invention uses Figure 4 The label relationship module shown in the figure combines visual information and semantic information for classification. The specific steps are as follows:

[0129] 3-1: First, we need to construct semantic information. We use the phrase-Bert semantic encoding model to encode the category labels of different gestures to obtain semantic information W e .

[0130] 3-2: Modeling the relationship between category labels. Based on the semantic information obtained in step 3-1, the semantic information is encoded and converted into Q, K, and V through three linear layers.

[0131] Q=f·W e ,K=g·W e ,V=h·W e

[0132] The self-attention matrix is calculated by Q and K, and then the self-attention matrix is multiplied by V to obtain Attn.

[0133]

[0134] Align Attn and W through a linear layer l e dimension, and align Attn with W e Add:

[0135] G e =l·Attn+W e

[0136] 3-3: Replace the original classification layer of ResNet with a classification layer Glinear that combines both semantic information and visual information.

[0137] Through the self-attention module in step 3-2, we can obtain a new semantic space containing inter-class relationships. Combining this semantic space with the previous visual information v, we can obtain the classification result p:

[0138] Glinear:p=softmax(G e Pv+b)

[0139] in is a learnable parameter used to fuse visual and semantic information.

[0140] 4: Average the results of step 3 over time to get the final prediction result.

Claims

1. A method for gesture recognition, characterized in that: The steps include: Step 1: Obtain T video frames from the video to be processed, perform data preprocessing and use them as input for the gesture recognition model; Step 2: Feed the processed video frames into the ResNet neural network with the added spatial attention module and long-short distance temporal displacement module. For each bottleneck in ResNet, the specific operations are as follows: Step 2-1: For the feature map X = {X(1),...X(T)}∈R sent to Bottleneck T×C×H×W , X(1),...X(T) represent the input images of each frame respectively, C, H, W represent the number of channels of the input image and the height and width of the image respectively; first, the feature map X is passed through the spatial attention module to calculate the RGB difference between long-distance frames, and the spatial attention map is calculated based on the difference, thereby assigning different weights to different spatial positions; the details are as follows: First, calculate the RGB difference between frames separated by T / 2, that is, calculate the RGB difference between X(t) and X(t+T / 2): D(t)=X(t+T / 2)-X(t), t∈{1, 2,...T / 2} Where D(t) is the RGB difference between frames separated by T / 2. Next, D(t) is concatenated along the time dimension to obtain the final motion pattern change matrix D: D=[D(1),...,D(T / 2)∈R T / 2×C×H×W The average pooling layer in the time dimension is used to pool the motion mode change matrix D to obtain the location where the mode change occurs in the entire video: D avg =Pool(D),D avg ∈R 1×C×H×W Where D avg represents the result of pooling the motion pattern change matrix D along the time dimension; Next, a 3×3 convolutional layer is used to avg Perform convolution and then pass through a sigmoid layer to obtain the spatial attention map M: M=σ(conv*D avg ),M∈R 1×C×H×W Where, σ(.) represents the sigmoid function; conv represents a 3×3 convolutional layer; Finally, the corresponding elements of the input X and the spatial attention map M matrices are multiplied to obtain the output of the final spatial attention module: X o =X⊙M Step 2-2: Long and short distance time displacement module; First, the input of the long-short distance time displacement module is divided into three parts along the channel dimension, that is, 1 / 8 of the channel is selected as the short-term time displacement part X s , 1 / 8 of the channel is used as the long-term time displacement part X l , the remainder is X o ; For the short-term displacement part X s , make a short time displacement, select X s Half of the channels move forward 1 time step, and the other half move backward 1 time step; For the long-term time displacement part X l , make a long time displacement, select X l Half of the channels move forward by T / 2 time steps, and the other half move backward by T / 2 time steps; For the remaining X o remains unchanged; Finally, the weighted sum of the above three parts is calculated: X‘=X s ×α+X l ×β+X o Where α and β represent the importance coefficients of long-distance time displacement and short-distance time displacement, respectively; Step 3: After passing through multiple bottlenecks in step 2, visual information v for classification is obtained; in the category relationship module, the visual information v is integrated with the semantic information constructed from the semantic encoding model for classification; Step 3-1: Construct semantic information; encode the category labels of different gestures through the phrase-Bert semantic encoding model to obtain semantic information W e ; Step 3-2: Model the relationship between category labels; Based on the semantic information obtained in step 3-1, the semantic information encoding is transformed into Q, K, and V through three linear layers; Q, K, and V represent the query matrix, keyword matrix, and the sum obtained by transforming the semantic information encoding through the linear layer respectively; f,g,h: Q=f·W e ,K=g·W e ,V=h·W e Where f, g, and h represent the three linear layers that transform semantic information; N represents the number of action categories, that is, the number of labels, Q, K, Calculate the self-attention matrix through Q and K, and then multiply the self-attention matrix with V to get Attn; Align Attn and W through a linear layer l e dimension, and align Attn with W e Add: G e =l·Attn+W e Step 3-3: Replace the original classification layer of ResNet with a classification layer Glinear that combines both semantic information and visual information; Through the self-attention module in step 3-2, a new semantic space containing inter-class relationships is obtained. This semantic space is combined with the previous visual information v to obtain the classification result p: Glinear:p=softmax(G e Pv+b) in is a learnable parameter used to fuse visual and semantic information; Step 4: Average the results of step 3 over time to obtain the final classification and recognition results.

2. The method for gesture recognition according to claim 1, wherein: The gesture recognition model is trained using the following steps: (1) obtaining training samples, each training sample including a preset number of video frames to be tested and a real behavior category corresponding to the video frames to be tested; (2) inputting the training sample into the neural network model to be trained, where the neural network model to be trained is a preset initial neural network model; (3) Using the output of the neural network model to be trained, determine the predicted true behavior category of each training sample; (4) Determine the loss value based on the predicted true behavior category of each training sample, the true behavior category and the preset loss function, and the loss function used is the cross entropy loss function; (5) judging whether the neural network model to be trained has converged based on the loss value; if it has converged, the neural network model to be trained is the trained gesture action recognition model; (6) If convergence has not occurred, adjust the network parameters of the neural network model to be trained and return to (2) until convergence occurs.

Images