A gesture category recognition method based on HGCN and gesture skeleton sequence

Abstract

The application discloses a gesture category recognition method based on HGCN and gesture skeleton sequences, which can recognize the corresponding gesture category of a user according to the gesture action of the user, and two improvements are made on the basis of an early-stage basic model to more effectively utilize the structure information of hand joint space and time. Firstly, the application proposes hand feature enhancement based on graph convolution, constructs a hand joint dependency graph, and proposes to use a full connection network to strengthen the information flow between nodes far away from each other. In addition, the application proposes to use space-time position coding and a self-attention mechanism to learn the dependency relationship of hand joints in space and time. Based on these improvements, the application achieves better effects on sequence gesture skeleton generation, and proves the effectiveness of the improved model.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and human pose estimation, and particularly relates to a gesture category recognition method based on HGCN and gesture skeleton sequence. BACKGROUND

[0002] Gesture is an important non-verbal medium in communication and interaction, and often appears in daily conversation interaction. It not only assists language communication, but also has independent expression and plays an important role in communication. Representative and typical gestures and imitative gestures that are easy to express intuitively can better complete the expression and understanding between communication participants. When language is difficult to convey communication information, gestures can well assist in expressing communication intentions. Whether gestures express the surface meaning or the implied meaning of communication, gestures have reference for communication.

[0003] For example, when expressing "heart" or "love", the hands can be combined to form a heart shape, or the thumb and index finger of a single hand can be overlapped at the first knuckle, and the other fingers can be curled to represent it. In recent years, gestures have attracted attention from academia and industry due to their wide range of human-computer interaction, game, control of household appliances and other non-verbal communication applications, and as a natural way of interaction in augmented reality and virtual reality applications, and have gradually become an active research field and are considered a new trend in the future. Gesture generation, as an important task of gesture expression behavior modeling, has a large number of practical applications such as video synthesis, gesture classification, gesture prediction and gesture recognition.

[0004] The complexity, high dimensionality and overlapping with other parts of the body of natural scenes make it challenging to directly generate image-level gestures. A natural solution is to first generate more easily processed skeleton-based gesture sequences, and then render natural gesture sequences according to these sequences. Based on the above considerations, the generation of skeleton-based gesture sequences is studied, with the aim of exploring the generation of multiple categories of static gestures and continuous sequence of gesture actions. In fact, most of the motion information is stored in the skeleton sequence, so generating the corresponding gesture skeleton plays an important role in the gesture generation task.

[0005] In the study of human gestures, McNeill et al. proposed four main gesture types according to semiotics in the literature "Hand and Mind: What Gestures Reveal About Thought", namely iconic, metaphorical, deictic and beat gestures, and then Toshinori et al. summarized the gesture classification in the literature "A speech-driven hand gesture generation method and evaluation in android robots", added symbolic and concomitant gestures; among them, iconic, metaphorical and symbolic gestures are highly related to speech vocabulary, deictic gestures are indicative actions that point to specific targets or spaces, and are related to speech vocabulary and the spatial context in which the gesture is formed; beat gestures are rhythmic actions closely related to speech audio, and concomitant gestures often involve self-touching actions, and research has found that concomitant gestures help to indicate specific personality types, providing guidance on how to create specific roles.

[0006] However, current gesture generation work is mostly used to generate collaborative gestures from speech or text and to explore image or video generation of gestures; collaborative gestures as non-verbal signals assist communication, making communication more natural, but it is difficult to express semantics alone, and due to the highly uncertain mapping of speech to gestures, the generated gestures will be specific to individual gesture style. In addition, most of the gesture generation work treats the human body's skeletal joints as a pixel of an image and a frame of action as an image for processing, without considering the rich human body structure information between the skeletal joints, which may result in distorted and unnatural generated results. SUMMARY

[0007] In view of the above, the present application provides a gesture category recognition method based on HGCN and gesture skeleton sequence, which proposes corresponding improvement measures for the particularity of gesture skeleton problems, explores gesture generation schemes with independent expression of meaning, and achieves better results.

[0008] A gesture category recognition method based on HGCN and gesture skeleton sequence, comprising the following steps:

[0009] (1) Obtain a large number of video frame sequences about gesture actions and corresponding gesture category labels, and enhance the hand features in the video frames;

[0010] (2) Construct a model architecture based on HGCN (Hand-Gesture Graph Convolution Neural Networks), which is composed of a generator and two discriminators DF and DV. The generator is used to encode the input video frame sequence to generate a hand gesture skeleton graph sequence. The discriminator DF takes part of the frames in the hand gesture skeleton graph sequence as input to discriminate the corresponding gesture category result. The discriminator DF takes the entire hand gesture skeleton graph sequence as input to discriminate the corresponding gesture category result.

[0011] (3) Use the large number of video frame sequences and their gesture category labels obtained in step (1) to train the above model architecture.

[0012] (4) Input the gesture action video frame sequence to be identified into the trained model, and the corresponding gesture category can be identified.

[0013] Further, each frame in the video frame sequence is marked with a hand joint, including the palm root and the tip, first knuckle, second knuckle, and root of each finger. The tip, first knuckle, second knuckle, and root of the same finger are connected by edges in turn.

[0014] Further, the specific implementation of enhancing hand features in step (1) is to add edge connections between adjacent finger tips, adjacent first knuckles, adjacent second knuckles, and adjacent finger roots in the same frame. For the entire video frame sequence, add mapping connections between the same joints in different frames.

[0015] Further, the encoder is connected in turn from input to output by convolutional network D, self-attention module based on spatiotemporal position encoding, convolutional network D1, convolutional network D2, self-attention mechanism module, and graph convolutional network module. The outputs of convolutional networks D1 and D2 are processed by ReLU activation function.

[0016] Further, the self-attention mechanism module is used to learn the sparse action graph of the video frame sequence to reduce the computational burden of the graph convolutional network and provide the graph convolutional network with an adjacency matrix Specifically:

[0017] First, an initial adjacency matrix A0 is constructed as follows: As follows:

[0018]

[0019] Where: I is the identity matrix, A is an N×N adjacency matrix used to represent the connection relationship of key points in a video frame. If two key points are connected, the corresponding element value is set to 1, otherwise it is set to 0. N is the number of key points in the video frame.

[0020] Then, based on the input information of the module, the self-attention score is calculated for the entire video frame sequence to obtain the mask attention matrix S. mask ;

[0021]

[0022] Where: s i,j The self-attention score represents the influence of the i-th frame on the j-th frame in the video frame sequence, where i and j represent the frame numbers and T represents the total number of frames in the video frame sequence.

[0023] Furthermore, for matrix S mask For any row in the array, retain the top K elements with the highest self-attention scores, and set the remaining elements to 0. K is a preset positive integer.

[0024] Finally, S mask and Multiplication yields a matrix Furthermore, on After normalization, the final adjacency matrix is ​​obtained.

[0025] Furthermore, the graph convolutional network module is a multi-layer graph convolutional network structure. Each layer consists of a fully connected network FC1, a two-dimensional convolutional network Conv2d, and a fully connected network FC2 connected sequentially from input to output. The output of Conv2d is connected to the adjacency matrix. The sum is used as the input of FC2. The input of the first layer is the input information Hin of the module. Hin passes through a two-dimensional convolutional network and is added to the output of the first layer to serve as the input of the second layer. From the third layer onwards, the input of each layer is the output of the previous layer. The input of the second layer passes through a two-dimensional convolutional network and is added to the output of the last layer, and then passes through another two-dimensional convolutional network to serve as the output of the graph convolutional network module.

[0026] Furthermore, the self-attention module based on spatiotemporal position coding first encodes the key point positions in the video frame sequence in both time and space to obtain temporal position coding PET and spatial position coding PEs. Then, the input information of the module is added to PEs and passed through a multi-head self-attention layer and normalization processing to obtain an intermediate result. Finally, the intermediate result is added to PEt and passed through a multi-head self-attention layer and normalization processing to obtain the output result of the module.

[0027] Further, the time position encoding PEt and the space position encoding PEs are both expressed in the form of a position encoding matrix, and the expression of each element value in the matrix is as follows:

[0028]

[0029]

[0030] wherein PE pos,2r represents the element value of the pos-th row and the 2r-th column in the matrix, PE pos,2r+1 represents the element value of the pos-th row and the 2r+1-th column in the matrix, d model is the output dimension of the self-attention module, r is a natural number, pos represents the absolute position of the joint node in the current frame in the video frame sequence, pos [0, N) in the space position encoding PEs, and pos [0, N*T) in the time position encoding PEt, N is the number of joint nodes in the video frame, and T represents the total number of video frames.

[0031] The gesture category recognition model based on the HGCN can recognize the corresponding gesture category according to the gesture action of the user, two improvements are made on the basis of the early-stage basic model to more effectively utilize the structure information of the hand joint space and time. Firstly, the hand feature enhancement based on the graph convolution is proposed, the hand joint dependency graph is constructed, and the information flow between nodes with relatively long distance is enhanced by using the full connection network; in addition, the spatial and temporal position encoding and the self-attention mechanism are used to learn the dependency relationship of the hand joints in space and time. Based on these improvements, the sequence gesture skeleton generation is better, and the effectiveness of the improved model is proved. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is the dependency graph between hand joints.

[0033] Figure 2 is the overall framework diagram of the gesture category recognition model based on the HGCN.

[0034] Figure 3 is a structural schematic diagram of the generator.

[0035] Figure 4 is a structural schematic diagram of the frame-based discriminator DF.

[0036] Figure 5 is a structural schematic diagram of the video-based discriminator Dv.

[0037] Figure 6 is a structural schematic diagram of the graph convolution network module.

[0038] Figure 7 Structure diagram of a self-attention module based on space-time position coding.

[0039] Figure 8 Generate a graph sequence for the skeleton of the index finger click.

[0040] Figure 9 Generate a graph sequence for the skeleton of the thumb and the remaining four fingers. DETAILED DESCRIPTION

[0041] In order to describe the present application more specifically, the technical solutions of the present application are described in detail below in combination with the drawings and specific embodiments.

[0042] The sequence gesture skeleton generation method based on HGCN of the present application comprises the following steps:

[0043] (1) Input the sequence gesture skeleton data, such as Human3.6M, which is a sequence type body skeleton dataset, and the sequence gesture skeleton dataset here is self-made.

[0044] (2) Enhance the hand features.

[0045] In the static gesture generation work, the relationship graph between the hand nodes is obtained by directly observing the hand shape structure, and a natural connection mode is adopted, and the graph adjacency matrix is taken as the graph adjacency matrix; then, the graph convolutional neural network aggregates the information of the neighbor nodes of each joint node according to the adjacency matrix, and the purpose of utilizing the hand structure information is achieved.

[0046] On the basis of the present application, it is hoped that the structure between the hands can be explored more, and the hand features can be further enhanced. First, the natural connection mode between the hand joints realizes the linkage of the joints of the same finger, but the adjacent fingers are only connected through the finger root, and the correlation between different fingers is less considered, and the information of different fingers is relatively independent. Therefore, considering the more dependency between the fingers, the present application adds the symmetry dependency between the fingers in the generation task, that is, the fingertips, the first knuckles, the second knuckles and the finger roots of the adjacent fingers are also connected, and the final association dependency of the hand joints is as shown in Figure 1 , which can not only ensure the information flow between the same fingers, but also ensure the information transmission between different fingers. Compared with the full connection between the same fingers or the full connection between the corresponding joints of different fingers and the full connection of the entire hand joints, Figure 1 , the mode can better distinguish the strong association between the joints and the weak association between the joints, and also reduce the information redundancy, and can better reflect the connection between the joints of the hand.

[0047] In the edge set E of the gesture sequence undirected graph, in addition to the above-mentioned space edge, there are also self-connection edges between nodes and connection edges between time steps, that is, there are three types of edges, as follows:

[0048] Space edge v (t,i) →v (t,j) (i≠j) represents the connection between different joint nodes in the same time step.

[0049] Time edge v (t,i) →v (k,i) (t≠k) represents the connection of the same joint node between different time steps.

[0050] Self-connection edge v (t,i) →v (t,i) represent the connection of the joint node with itself.

[0051] (3) A GCN is used to build a sequence gesture skeleton generation model based on a GAN network, which includes a generator, a video discriminator and a frame discriminator.

[0052] As Figure 2 shown is the overall model framework, z is random noise subject to standard normal distribution (i.e. video image about gesture action), label is the label of gesture category, and x is a real gesture sample. Part of the frames and the corresponding labels are randomly selected from the sequence sample as the input of the frame discriminator DF, the generator G generates a sequence gesture sample from the random noise and the label, and then the generated sequence sample or the real sequence sample and the label are input into the video discriminator DV.

[0053] As Figure 3 shown, the generator takes Gaussian random noise and gesture category label as input, followed by a convolutional network, and then a self-attention module based on spatiotemporal position coding is used to learn the features of the joint nodes and the connection edges between the joint nodes in the spatial and temporal domains. Subsequently, two network blocks composed of one-dimensional convolutional networks and ReLU activation functions are used, and then the self-attention mechanism is used to capture the dependency in the sequence and construct a sparse graph of the gesture sequence, and finally the graph convolutional network is used to further learn to obtain the coordinate output of each joint node of the gesture sequence.

[0054] Since the action graph of the continuous gesture sequence is large, directly using the graph convolutional network to process it is challenging, therefore, the self-attention mechanism is used before the graph convolutional network to learn the sparse action graph of each gesture sequence to reduce the computational burden, and the specific construction process is as follows:

[0055] First, a directed graph G=(V,E) is constructed on the whole gesture sequence of T frames, the node set V={v ti{t=0,1,...,T-1, i=1,2,...,N} contains all nodes of the skeleton sequence, and N is the number of gesture joints in each frame. Then, the connection relationship between the hand joints in each frame can be represented by an adjacency matrix A, A∈R N×N The final intra-frame adjacency matrix can be obtained by adding self-connections to each joint. Then, the initial adjacency matrix of the entire sequence is defined as:

[0056]

[0057] wherein the unit matrix I represents connecting each node to all corresponding nodes in the time sequence, and the row and column sizes of are both N×T.

[0058] Next, the self-attention score of the entire sequence is calculated to obtain the mask attention matrix S mask Specifically, the input of the self-attention layer is represented as H={h0, h1,...,h T-1}, wherein h t ∈R N represents the hidden state vector of the N nodes at time t; following the self-attention mechanism, the calculation of the query vector, the key vector and the value vector is represented as:

[0059] Q=W q H, K=W k H, V=W v H

[0060] wherein W q , W k , W v are projection weights, the attention score S∈R T×T and the calculation result of the output H in of the attention layer are represented as follows:

[0061]

[0062] H in =SV

[0063] To prevent the influence of the current frame on the subsequent frames, the values in the upper triangular of the attention score S are all changed to 0 to obtain the mask attention matrix S mask , which can be represented as:

[0064]

[0065] wherein the element s m,n represents the influence of the nth frame on the mth frame.

[0066] In order to realize the sparse connection between frames, further in S mask select the K highest score values in each row of the matrix and set other elements to 0; when the number of non-zero elements in a row is less than K, all non-zero elements are reserved, and the adjacency matrix is represented as follows:

[0067]

[0068] Finally, the matrix is normalized to obtain the final adjacency matrix

[0069] The double discriminator is composed of a video-based discriminator DV and a frame-based discriminator DF, the discriminator DV takes the T-frame gesture sequence and the corresponding category label as input, the discriminator DF randomly selects part of the frames and the corresponding label from the T-frame gesture sequence as input, and finally outputs the gesture category result, the present application adopts the objective function of conditional GAN

[22] , which can be expressed as follows:

[0070]

[0071] Wherein: p(x) is the distribution of real data, p(z) is the standard Gaussian distribution, y is the one-hot category label, and the specific structures of the two discriminators DF and DV are shown in Figure 4 and Figure 5 .

[0072] As shown in Figure 6 is the structure of the graph convolution network module, after the first layer of graph convolution network, the updated joint node feature aggregates the information of the current node and its first-order neighbor node, and after the second layer of graph convolution network, the updated joint node feature aggregates the information of the current node and its first-order and second-order neighbor nodes, and so on. The multi-layer graph convolution network can learn the features of multi-order neighbor nodes. However, research shows that the operation of the graph convolution network overemphasizes the role of neighbor nodes in the gesture generation task, and easily ignores the nodes with less correlation, therefore, the present application adds a fully connected network before each layer of graph convolution network, first considers the correlation between nodes, and then uses the neighbor nodes to enhance the features of the current node by the graph convolution network. In this way, the information flow between nodes with weak correlation is ensured, and the importance of neighbor nodes is emphasized, so that the hand structure features can be more utilized.

[0073] As shown in Figure 7The structure diagram of the self-attention module based on space-time position coding is shown, and the model adopted by the time series task is usually a recurrent neural network, so that the network can remember the previous information and use the previous information to affect the subsequent output, and can describe the relationship between the current output of a sequence and the previous information. However, the sequential calculation of the recurrent neural network brings two problems: one is that the model cannot be accelerated by parallel calculation, and the other is the difficulty in long-time dependence; although the improved model such as LSTM and GRU introduces a gate mechanism, the ability to relieve the long-time dependence problem is limited. A new structure Transformer is designed in Advances in neural information processing systems, which better models the time series through the self-attention mechanism, but the self-attention does not record the position information, so the position coding is proposed to inject the position information into the input, so that the self-attention can remember the position information; combined with the gesture sequence, the original node features extracted from the input gesture joint node coordinates do not contain the spatial identification information corresponding to the current node and the time information that the current node is located at the corresponding time step. In order to integrate these information, the application proposes to use the self-attention method based on space-time position coding to learn the dependence between sequences, and the space-time position coding is composed of spatial position coding PEs (Spatial Positional Encoding) and temporal position coding PEt (Temporal Positional Encoding). Their values are still set using sine and cosine functions, which can be represented as:

[0074]

[0075] Wherein: pos represents the position of each joint node in the gesture sequence, i is the dimension, d model is the dimension of the output; The pos in the spatial position coding PEs is composed of a continuous N-dimensional vector, and each value represents a joint node; The pos in the temporal position coding PEt is composed of an (N x T) dimensional vector, corresponding to each joint node in the gesture sequence skeleton graph.

[0076] First, the spatial position coding PEs is added to the initial node feature vector, and after the multi-head self-attention layer and normalization processing, the time position coding PEt is added to the output, and then the updated node feature is processed through the multi-head self-attention layer and normalization processing, so as to learn the space and time features of the gesture joint node, which can be represented as:

[0077] H=ATT(PEs+ATT(Hin+PEt))

[0078] Wherein: H is the final output, ATT is the multi-head self-attention layer, and Hin is the initial feature input.

[0079] Spatiotemporal location encoding enables the model to encode the spatial and temporal information carried by the input sequence; self-attention mechanism can capture the importance of information between hand nodes, with different attention distributions on the connecting edges between nodes, and then aggregate the information of other nodes based on the learned edge weights; multi-head attention captures multiple associations between sequences, enhancing the model's expressive ability for the input vector. Therefore, the model can better learn the dependencies between sequences.

[0080] like Figure 8 and Figure 9 The images shown are the generated images of the index finger clicking and the thumb and the other four fingers zooming. The generated sequence of gesture skeletons is consistent with the meaning of the gestures, and the joints in the skeleton are natural, reasonable and not distorted. The skeleton images of each frame can also reflect the progressive nature of the sequence of gesture movements.

[0081] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A gesture category recognition method based on HGCN and gesture skeleton sequence, comprising the following steps: (1) Obtain a large number of video frame sequences about gesture actions and corresponding gesture category labels, and enhance the hand features in the video frames; Each frame in the video frame sequence is labeled with hand joint nodes, including the palm root and the tip, the first knuckle, the second knuckle and the root of each finger, and the tip, the first knuckle, the second knuckle and the root of the same finger are connected by edges in turn with the palm root; The specific implementation of enhancing the hand features is to add edge connections between adjacent tips, adjacent first knuckles, adjacent second knuckles and adjacent roots in the same frame; and for the entire video frame sequence, add mapping connections between the same joint nodes in different frames; (2) Construct a model architecture based on HGCN, which consists of a generator and two discriminators DF and DV, the generator is used to encode the input video frame sequence to generate a gesture skeleton graph sequence, the discriminator DF takes part of the frames in the gesture skeleton graph sequence as input to discriminate the corresponding gesture category result, and the discriminator DF takes the entire gesture skeleton graph sequence as input to discriminate the corresponding gesture category result; The generator is connected in turn from input to output by convolutional network D, self-attention module based on spatiotemporal position encoding, convolutional network D1, convolutional network D2, self-attention mechanism module and graph convolutional network module, and the outputs of convolutional networks D1 and D2 are processed by ReLU activation function; The self-attention mechanism module is used to learn a sparse motion graph of the video frame sequence, so as to reduce the calculation burden of the graph convolution network and provide the graph convolution network with an adjacency matrix In particular: First, an initial adjacency matrix is constructed as follows: wherein: I is an identity matrix, A is an N*N size adjacency matrix, used to represent the connection relationship of the junction nodes in the video frame, if two junction nodes exist connection, the corresponding element value is set to 1, otherwise 0, N is the number of junction nodes in the video frame; Then, according to the input information of the module, the self-attention scores of the entire video frame sequence are calculated to obtain a mask attention matrix S mask ; wherein: s i,j represents the influence of the i-th frame on the j-th frame in the video frame sequence, i.e., the self-attention score, i and j represent the frame number, and T represents the total number of frames of the video frame sequence; Further, for any row in the matrix S mask , the top K element values with the highest self-attention scores in the row are retained, and the rest of the element values are set to 0, K being a preset positive integer. Finally, S mask is multiplied by the matrix to obtain the matrix Further, the matrix is normalized to obtain the final adjacency matrix The graph convolution network module is a multi-layer graph convolution network structure, each layer is sequentially connected by a full connection network FC1, a two-dimensional convolution network Conv2d and a full connection network FC2 from input to output, the output of the Conv2d is added to an adjacency matrix The output of the FC2 is the input of the second layer, the input of the first layer is the input information Hin of the module, the Hin is added to the output of the first layer after passing through a two-dimensional convolution network to be the input of the second layer, the input of each layer from the third layer is the output of the previous layer, the input of the second layer is added to the output of the last layer after passing through a two-dimensional convolution network and then passing through a two-dimensional convolution network to be the output of the graph convolution network module. The self-attention module first encodes the position of the joint nodes in the video frame sequence in time and space respectively to obtain the time position encoding PEt and the space position encoding PEs; then adds the input information of the module to PEs, and obtains the intermediate result through multi-head self-attention layer and normalization processing, and then adds the intermediate result to PEt, and obtains the output result of the module through multi-head self-attention layer and normalization processing; (3) Use the large number of video frame sequences and their gesture category labels obtained in step (1) to train the above model architecture; (4) Input the gesture action video frame sequence to be recognized into the trained model, and the corresponding gesture category can be recognized. 2.The gesture class recognition method of claim 1, wherein: Both the time position encoding PEt and the space position encoding PEs are expressed in the form of a position encoding matrix, and the expression of each element value in the matrix is as follows: wherein: PE pos,2r denotes the element value of the pos-th row and the 2r-th column in the matrix, PE pos,2r+1 denotes the element value of the pos-th row and the 2r+1-th column in the matrix, d model is the output dimension of the self-attention module, r is a natural number, pos denotes the absolute position of the joint node in the video frame sequence in the current frame, pos ∈ [0, N) in the spatial position encoding PEs, pos ∈ [0, N × T) in the temporal position encoding PEt, N is the number of joint nodes in the video frame, and T represents the total number of video frames.

Images