Dynamic gesture recognition method and system based on pulse transformer

Abstract

The application discloses a dynamic gesture recognition method and system based on pulse Transformer, the recognition method comprising: acquiring dynamic gesture data to be recognized and converting the dynamic gesture data into vectors, then constructing position relations between sequence elements of the vectors; compressing the vectors with the constructed position relations to obtain compression information; inputting the compression information into a pulse linear Transformer block, a time domain compensation block and a pulse gate feedforward network block in sequence to extract time domain information and space domain information; repeating step S3 for a preset number of times, and inputting the time domain information and the space domain information extracted in the last execution of step S3 each time; and classifying the space domain information by using a classification head to obtain a recognition result of the dynamic gesture. The recognition system comprises a data acquisition module, an input embedding and position embedding module, a compression block, a pulse linear Transformer block, a time domain compensation block, a pulse gate feedforward network block, a judgment module and a classification head.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, in particular to a dynamic gesture recognition method and system based on pulse Transformer. BACKGROUND

[0002] Dynamic gesture recognition refers to recognizing gestures by capturing and analyzing dynamic features such as hand movement trajectory, speed and acceleration. It is widely used in human-computer interaction, virtual reality, security authentication, medical health, education and training, etc. In these application fields, there are usually a large number of device resource limited scenarios, such as battery powered human-computer interaction mobile devices, VR head-mounted devices that need to be worn, etc. For these devices, power consumption is a very important consideration. Therefore, when implementing gesture recognition technology, it is necessary to maintain high accuracy while meeting the requirement of low power consumption to ensure that the device can run stably for a long time. This requirement poses higher challenges to gesture recognition technology, and effective solutions need to be found.

[0003] Existing dynamic gesture recognition technology usually adopts machine learning based methods, including traditional machine learning algorithms and deep learning algorithms. These methods learn feature representation and classification decision of gestures by training models. However, with the rapid growth of data, traditional machine learning algorithms have the performance bottleneck of low accuracy when facing large-scale data, which cannot meet the needs of practical applications. Therefore, deep neural networks have become the mainstream technology of dynamic gesture recognition. In particular, the Transformer model has attracted widespread attention from researchers due to its excellent performance. The Transformer model realizes the modeling of global information through self-attention mechanism, which can better capture the temporal features in gesture sequence, thereby improving the accuracy and robustness of recognition.

[0004] Although the Transformer model has excellent performance, its model training and inference use process consumes a large amount of energy. For example, training the Transformer based GPT-3 model consumes 190,000 kWh of energy; such huge energy consumption cannot meet the requirements of device resource limitation. SUMMARY

[0005] In view of the above problems in the prior art, the dynamic gesture recognition method and system based on pulse Transformer provided by the present application solve the problem of long recognition time of existing gesture recognition methods based on neural network, which causes large power consumption of the device.

[0006] In order to achieve the above application purpose, the technical scheme adopted by the present application is:

[0007] In a first aspect, a dynamic gesture recognition method based on pulse Transformer is provided, which comprises the following steps:

[0008] S1, obtaining dynamic gesture data to be recognized and converting it into a vector, and then constructing the position relationship between the vector sequence elements;

[0009] S2, compressing the vector with the constructed position relationship to obtain compressed information;

[0010] S3, sequentially inputting the compressed information into a linear Transformer, a time domain compensation block and a pulse gate feedforward network block to extract time domain information and spatial domain information;

[0011] S4, repeating step S3 a predetermined number of times, and each time inputting the time domain information and spatial domain information extracted in the last execution of step S3;

[0012] S5, classifying the time domain information and spatial domain information using a classification head to obtain the recognition result of the dynamic gesture.

[0013] Further, the method for processing the compressed information by the linear Transformer comprises:

[0014] linearly transforming the input compressed information and converting it into a pulse sequence using a pulse encoding layer:

[0015] Converting Q and K into a pulse sequence using a pulse encoding layer and

[0016]

[0017] wherein X0 is the compressed information; W Q ,E K ,W V are linear transformation matrices; LIF(·) is a pulse encoding operation; and is a pulse sequence, is a floating-point number sequence; N is the image height multiplied by the width; D is the dimension of the vector embedding representation; represents a real number field with a dimension of 1×N×D.

[0018] using the pulse sequence and the floating-point number sequence to calculate the spatial domain feature information

[0019]

[0020] wherein is a multiplication symbol; d is a single embedding representation dimension.

[0021] The further beneficial effects of the above technical solution are: compared with the traditional Transformer quadratic calculation complexity, the linear Transformer does not need to calculate the attention weight for each position, which significantly reduces the calculation complexity and memory consumption; in addition, the pulse sequence operation has higher energy efficiency than the floating point sequence operation, which can further reduce the overall power consumption of the model.

[0022] Further, the time domain compensation block extracts the time domain information in the spatial domain feature information

[0023]

[0024] Among them, LIF1 and LIF2 are two threshold different pulse neuron layers; The spatial domain feature information extracted by the linear Transformer.

[0025] The further beneficial effects of the above technical solution are: since the compression of the time domain information to 1 time will cause performance loss, the time domain compensation block can calculate the pulse sequence at different thresholds, and through the rich and diverse pulse information, the performance loss can be minimized to improve the model performance.

[0026] Further, the pulse gate feedforward network block extracts the spatial domain information in the compensated information:

[0027]

[0028]

[0029]

[0030] Among them, K is a preset number of times; X k and X k-1 are the spatial domain information extracted for the kth and (k-1)th times respectively; is the time domain information extracted by the time domain compensation block; SGFFN(.) is the pulse feedforward network block; X′ k is the sum of the current time domain information and the (k-1)th spatial domain information; is the gate feedforward pulse sequence; is the gate feedforward floating point sequence; W O , W G and W P are gate feedforward linear transformation matrices respectively; W G , and and respectively represent a real number field with a dimension of Dx4D and a dimension of 4DxD.

[0031] The further beneficial effects of the above technical solution are: since the floating-point multiplication operation has the disadvantage of high power consumption, the scheme realizes the gating mechanism by binary pulse to extract sample space information, optimizes the floating-point multiplication operation into the AND operation of the gate circuit, and can significantly reduce the operation energy consumption.

[0032] Further, the method for converting the dynamic gesture data to be recognized into a vector and then constructing the position relationship between the elements of the vector sequence comprises:

[0033] The input embedding block is used to convert the dynamic gesture data to be recognized into a vector:

[0034] IE(I)=Pool(LIF(BN(Conv2d(I)))),I∈ T×C×H×W

[0035] wherein, IE(·) is an input embedding operation; I is the dynamic gesture data to be recognized; T, C, H and W are total time, image channel, image height and image width respectively; represents a real number set; Conv2d(·) is a two-dimensional convolution operation; BN(·) is a normalization operation; LIF(·) is a pulse coding operation; Pool(·) is a pooling operation; represents a real number field with a dimension of TxCxHxW.

[0036] The position information of the dynamic gesture data to be recognized is encoded by the position embedding module:

[0037] PE(I)=LIF(BN(Conv2d(I)))

[0038] wherein, PE(I) is a position embedding operation;

[0039] The position relationship between the elements of the vector sequence is constructed:

[0040]

[0041] wherein, x is a vector for constructing the position association; N is the image height multiplied by the width; D is the dimension of the vector embedding representation; represents a real number field with a dimension of TxNxD.

[0042] The further beneficial effects of the above technical solution are: by the input embedding module, the original image data is converted into a vector representation with more semantic information, so that higher-level features are learned at the pixel level; by the position embedding module, the model is provided with pixel position information, so that the spatial relationship in the image is maintained. The two parts jointly provide the model with more effective feature representation and image spatial information, so that the model has better migratability, interpretability and generalization ability.

[0043] Further, the time domain compression module is used to compress the vector x of the constructed position relationship:

[0044]

[0045] wherein x t is the dynamic gesture at the tth moment in x; X0 is the compressed information; T is the total moment included by the dynamic gesture.

[0046] The further beneficial effects of the above technical solution are that the T-dimensional time domain information of the dynamic gesture can be compressed to 1 dimension by the time domain compression module, so that it is not necessary to operate once at each moment, thereby solving the problem of repeated operation which is extremely time-consuming, and achieving the purpose of shortening the recognition time.

[0047] Further, the classification head includes one average pooling layer and one fully connected layer, and the classification formula is:

[0048] CH(X K )=FC(Avg_pool(X K ))

[0049] wherein X K is the Kth spatial domain information; K is a preset number; FC(·) is a fully connected layer operation; Avg_pool(·) is an average pooling operation; CH(·) is a predicted probability of the dynamic gesture belonging to each category;

[0050] The maximum predicted probability in all predicted probabilities is selected as the category of the dynamic gesture.

[0051] In a second aspect, a dynamic gesture recognition system based on pulse Transformer is provided, which includes:

[0052] A data acquisition module is configured to acquire dynamic gesture data to be recognized.

[0053] An input embedding and position embedding module is configured to convert the dynamic gesture data to be recognized into a vector, and then construct a position relationship between sequence elements of the vector.

[0054] A compression block is configured to compress the vector of the constructed position relationship to obtain compressed information.

[0055] A pulse linear Transformer block is configured to extract spatial domain feature information in the compressed information.

[0056] A time domain compensation block is configured to extract time domain information in the spatial domain feature information.

[0057] A pulse gate feedforward network block is configured to extract spatial domain information in the time domain information.

[0058] A judgment module is configured to judge whether the spatial domain information return pulse linear transformer block, the time domain compensation block pulse gate feedforward network block executes a preset number K of times.

[0059] A classification head is configured to classify the Kth spatial domain information to obtain a recognition result of the dynamic gesture.

[0060] Compared with the prior art, the present application has the following advantages:

[0061] The present application can compress the vector of the constructed position relationship to compress the dynamic gesture recognition information at multiple time points to one time point, thereby significantly reducing the high time and space overhead required for subsequent calculation. The pulse linear transformer block can reduce the quadratic calculation complexity in the traditional transformer structure to linear complexity, so that the present application can be more efficient (short recognition time and high recognition accuracy) in recognition.

[0062] The time domain compensation block of the present application uses a dynamic threshold technology to obtain pulse activation information at different time points, thereby improving the model performance through rich and diverse time domain information. The pulse gate feedforward network module uses a binary pulse signal to realize a network gating mechanism, thereby efficiently screening important information in the spatial domain, so that the model can focus on key features more accurately, thereby further improving the model performance.

[0063] The present application uses a pulse-based transformer block, which has the advantage of low power consumption compared with other deep neural network structures based on floating-point numbers. In addition, due to the time domain information compression and linear complexity calculation of the present application, the present application has a shorter training time and recognition time compared with other pulse-based network models, so that when it is applied to an embedded device, it has a low power consumption when it performs dynamic gesture recognition due to the short recognition time, thereby extending the use time of the embedded device. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 A flowchart of the pulse transformer-based dynamic gesture recognition method.

[0065] Figure 2 A principle block diagram of the pulse transformer-based dynamic gesture recognition system. DETAILED DESCRIPTION

[0066] The specific embodiments of the present application are described below to facilitate the understanding of the present application by those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

[0067] Reference Figure 1 , Figure 1 The flowchart of the dynamic gesture recognition method based on pulse Transformer is shown; as shown in Figure 1 , the method S includes steps S1 to S5.

[0068] In step S1, the dynamic gesture data to be recognized is obtained and converted into a vector, and then the position relationship between the vector sequence elements is constructed;

[0069] In implementation, the method of converting the dynamic gesture data to be recognized into a vector and then constructing the position relationship between the vector sequence elements preferably includes:

[0070] The input embedding block is used to convert the dynamic gesture data to be recognized into a vector:

[0071] IE(I)=Pool(LIF(BN(Conv2d(I)))),I∈ T×C×H×W

[0072] Wherein, IE(·) is an input embedding operation; I is the dynamic gesture data to be recognized; T, C, H and W are the total time, image channel, image height and image width respectively; is a real number set; Conv2d(·) is a two-dimensional convolution operation; BN(·) is a normalization operation; LIF(·) is a pulse coding operation; Pool(·) is a pooling operation; Indicates a real number field with dimensions TxCxHxW.

[0073] The position embedding module is used to encode the position information of the dynamic gesture data to be recognized:

[0074] PE(I)=LIF(BN(Conv2d(I)))

[0075] Wherein, PE(I) is a position embedding operation;

[0076] The position relationship between the vector sequence elements is constructed:

[0077]

[0078] Wherein, x is the vector for constructing the position association; N is the image height multiplied by the width; D is the dimension of the vector embedding representation. represents a real field with dimension T x N x D.

[0079] In step S2, the vector of constructing position relationship is compressed to obtain compressed information; the scheme preferably adopts a time domain compression module to compress the vector x of constructing position relationship:

[0080]

[0081] Wherein, x t is the dynamic gesture of the tth moment in x; X0 is the compressed information; T is the total moment included by the dynamic gesture.

[0082] In step S3, the compressed information is sequentially input into a linear Transformer, a time domain compensation block and a pulse gate feedforward network block, and time domain information and spatial domain information are extracted;

[0083] In an embodiment of the present application, the method for processing the compressed information by the linear Transformer comprises:

[0084] linearly transforming the input compressed information, and converting into a pulse sequence by using a pulse coding layer:

[0085] Converting Q and K into a pulse sequence by using a pulse coding layer and

[0086]

[0087] Wherein, X0 is the compressed information; W Q ,W K ,W V are all linear transformation matrices; LIF(·) is a pulse coding operation; and are pulse sequences, are floating point number sequences; N is the image height multiplied by the width; D is the dimension of the vector embedding representation. The pulse sequence and the floating point number sequence are used to calculate the spatial domain feature information

[0088]

[0089] Wherein, is a product symbol; d is a single embedding representation dimension.

[0090] In implementation, the time domain compensation block preferably comprises two LIF pulse coding layers with different thresholds, and the time domain compensation block is used to extract the time domain information in the spatial domain feature information

[0091]

[0092] wherein LIF1 and LIF2 are two threshold different pulse neuron layers, which are 0.5 and 1.0 respectively; is the spatial domain feature information extracted by the linear transformer.

[0093] The pulse gate feedforward network block in the scheme includes three linear transformations and two LIF pulse neuron layers. The spatial domain information in the compensated information is extracted by using the pulse gate feedforward network block:

[0094]

[0095]

[0096]

[0097] wherein K is a preset number of times; X k and X k-1 are the spatial domain information extracted in the kth and (k-1)th times respectively; is the time domain information extracted by the time domain compensation block; SGFFN(.) is the pulse feedforward network block; X' k is the sum of the current time domain information and the (k-1)th spatial domain information; is the gate feedforward pulse sequence; is the gate feedforward floating point number sequence; W O , W G and W P are gate feedforward linear transformation matrices respectively; W G , and and respectively represent a real number field with a dimension of D x 4D and a dimension of 4D x D.

[0098] The pulse gate feedforward neural network block of the scheme is a feedforward neural network block more suitable for pulse calculation. Inspired by the gate circuit, the scheme uses binary pulses to realize the gating mechanism to extract the spatial information in the dynamic gesture. The element product in SGFFN is optimized as a logical AND operation, which is more energy-efficient than the floating point multiplication operation.

[0099] In step S4, step S3 is repeated a preset number of times, and each time the input is the time domain information and the spatial domain information extracted in the last execution of step S3;

[0100] In step S5, the time domain information and the spatial domain information are classified by using a classification head to obtain the recognition result of the dynamic gesture. In the scheme, the classification head includes one average pooling layer and one fully connected layer, and the classification formula is:

[0101] CH(X K )=FC(Avg_pool(X K ))

[0102] wherein, X K is the Kth spatial domain information; K is a preset number of times; FC(·) is a full connection layer operation; Avg_pool(·) is an average pooling operation; CH(·) is a predicted probability of the dynamic gesture belonging to each category;

[0103] The maximum prediction probability in all prediction probabilities is selected as the category of the dynamic gesture.

[0104] In the scheme, the classification head is the last group of layers of the neural network model, responsible for converting the output of the model into a classification prediction. The network intermediate information X K is mapped to the output nodes equal to the number of task categories.

[0105] As shown in Figure 2 , the scheme also provides a dynamic gesture recognition system based on pulse Transformer, which comprises:

[0106] A data acquisition module is configured to acquire dynamic gesture data to be identified.

[0107] An input embedding and position embedding module is configured to convert the dynamic gesture data to be identified into a vector, and then build a position relationship between vector sequence elements.

[0108] A compression block is configured to compress the vector with the built position relationship to obtain compressed information.

[0109] A pulse linear Transformer block is configured to extract spatial domain feature information in the compressed information.

[0110] A time domain compensation block is configured to extract time domain information in the spatial domain feature information.

[0111] A pulse gate feedforward network block is configured to extract spatial domain information in the time domain information.

[0112] A judgment module is configured to judge whether the spatial domain information returns to the pulse linear Transformer block, the time domain compensation block, and the pulse gate feedforward network block to execute a preset number of times K.

[0113] A classification head is configured to classify the Kth obtained spatial domain information to obtain a recognition result of the dynamic gesture.

[0114] In order to verify the effectiveness of the dynamic gesture recognition method provided by the scheme, the following model training and evaluation work is carried out on the DVS128Gesture dataset widely used in the field of dynamic gesture recognition, and compared with the existing optimal dynamic gesture recognition method.

[0115] A, Dataset construction

[0116] DVS128Gesture is an event stream neuromorphic dataset, which contains 10000 gesture data samples of 29 people in 3 lighting conditions, 11 gesture categories, of which 9000 samples are used as the training set and 1000 samples are used as the test set. The main reasons for choosing this dataset are as follows: 1) The dataset is challenging in terms of sample size, classification categories, and noise; 2) The dataset has a wide range of applications in evaluating dynamic gesture recognition capabilities.

[0117] B, Task evaluation index

[0118] In terms of evaluating model performance, time step and accuracy are used as evaluation indexes, where time step is used to measure the training and inference delay (recognition time) of the model, in units of ms; accuracy is used to measure the correctness of the model in recognizing dynamic gestures, in units of %.

[0119] C, Algorithm parameter setting

[0120] In the training parameter setting, the input sample size is set to 128*128, and the batch size is set to 16. The initial learning rate is 0.001, the training iteration number is 200, and the initial time step is set to 16.

[0121] D, Result comparison and analysis

[0122] Table 1 Performance comparison results

[0123]

[0124] From Table 1, it can be seen that the recognition method of the present scheme has a time step reduction of 500 times compared to the DECOLLE method with the highest step, while the performance is improved by 3.1%, which shows that the recognition method has a lower training and inference delay while maintaining excellent recognition accuracy. In addition, compared with the optimal SEW-ResNet method, the time step of the recognition method is reduced by 16 times, while the performance is improved by nearly 1%, which shows that the recognition method has a great advantage in performance and power consumption compared with existing recognition methods.

Claims

1. A method for dynamic gesture recognition based on pulse Transformer, characterized in that, The method comprises the steps of: S1, obtaining dynamic gesture data to be recognized, converting the dynamic gesture data to be recognized into a vector, and then constructing a position relationship between elements of the vector sequence; S2, compressing the vector with the constructed position relationship to obtain compressed information; The method for processing the compressed information by the pulse linear Transformer comprises: performing linear transformation on the input compressed information, and converting the compressed information into a pulse sequence by using a pulse coding layer: ， 、 、 wherein, is a compression information; are linear transformation matrices; is a pulse encoding operation; and is a pulse sequence, is a sequence of floating point numbers; N is an image height times width; D is a dimension of a vector embedding representation; denotes a real number field of one dimension is ​ Adopting pulse sequence , and floating point number sequence Calculate spatial domain feature information : wherein, is a product symbol; d is a single embedding representation dimension; S3, the compressed information is sequentially input into a pulse linear transformer block, a time domain compensation block and a pulse gate feedforward network block to extract time domain information and spatial domain information; the time domain compensation block is used to extract time domain information in the spatial domain feature information : wherein, and are two threshold different pulse neuron layers; is spatial domain feature information extracted by the pulse linear Transformer; represents a real number field of one dimension is a real number field extracting spatial domain information in the compensated information by using a pulse gate feedforward network block: ， wherein K is a preset number of times; and are the spatial domain information extracted for the first k and k -1 times, respectively; is the time domain information extracted for the time domain compensation block; is the impulse feedforward network block; is the sum of the current time domain information and k -1 times of the spatial domain information; is the gate feedforward impulse sequence; is the gate feedforward float sequence; , and are the gate feedforward linear transformation matrices; 、 and ; and represent a real number field of and a real number field of dimensions, respectively; S4, repeating step S3 for a preset number of times, and each time inputting the time domain information and the spatial domain information extracted in the last execution of step S3; S5, classifying the spatial domain information by using a classification head to obtain a recognition result of the dynamic gesture.

2. The method of claim 1, wherein the pulse-based Transformer is a dynamic gesture recognition method. The method for converting the dynamic gesture data to be recognized into a vector and then constructing a position relationship between elements of the vector sequence comprises: converting the dynamic gesture data to be recognized into a vector by using an input embedding block: ， wherein, is an input embedding operation; I is dynamic gesture data to be recognized; T , C , H and W are total time, image channel, image height and image width, respectively; represents a set of real numbers; is a two-dimensional convolution operation; is a normalization operation; is a pulse coding operation; is a pooling operation; denotes a real field of dimension . encoding position information of the dynamic gesture data to be recognized by using a position embedding module: wherein, is a positional embedding operation; constructing a position relationship between elements of the vector sequence: wherein, x to construct position-dependent vectors; N is the image height times width; D is the dimension of the vector embedding representation; represents a real number field of dimension .

3. The method of claim 1, wherein the pulse-based Transformer is a dynamic gesture recognition method. Adopting time domain compression module to construct position relation vector x Performing compression: wherein, is x the t total time instants of the dynamic gesture; is the compressed information; T is the total time instants of the dynamic gesture.

4. The method of claim 1, wherein the pulse-based Transformer is a dynamic gesture recognition method. The classification head comprises one average pooling layer and one fully connected layer, and a classification formula thereof is: wherein, is the spatial domain information for the K th spatial domain; K is a preset number of times; is a fully connected layer operation; is an average pooling operation; is a predicted probability that the dynamic gesture belongs to each class. selecting the maximum prediction probability in all prediction probabilities as the category of the dynamic gesture.

5. A pulse Transformer-based dynamic gesture recognition system for implementing the pulse Transformer-based dynamic gesture recognition method of any one of claims 1-4. It comprises: a data acquisition module configured to acquire dynamic gesture data to be recognized; an input embedding and position embedding module configured to convert the dynamic gesture data to be recognized into a vector, and then construct a position relationship between elements of the vector sequence; a compression block configured to compress the vector with the constructed position relationship to obtain compressed information; a pulse linear Transformer block configured to extract spatial domain feature information in the compressed information; a time domain compensation block configured to extract time domain information in the spatial domain feature information; a pulse gate feedforward network block configured to extract spatial domain information in the time domain information; a judgment module configured to judge whether the spatial domain information is returned to the pulse linear Transformer block, the time domain compensation block, and the pulse gate feedforward network block to execute a preset number K of times; a classification head configured to classify the spatial domain information obtained for the Kth time to obtain a recognition result of the dynamic gesture.

Images