Gesture recognition system and method

Abstract

A gesture recognition system comprises an input module, a convolution feature extraction module and a control system module. The input module uses an event camera to capture a gesture image event stream in real time as input. The convolution feature extraction module is a convolutional neural network for extracting structural features from input image data. Using this information, the network can identify important feature information of the gesture in the image and only transmit this part of the image to the next module. The control system module is implemented by a neural circuit strategy network, which is composed of four layers of neurons and realizes the functions of accepting image features of the convolution network, constructing a recurrent connection structure and finally outputting the meaning of the gesture. The application of neural circuit strategy to gesture recognition in audio-visual auxiliary systems has high computational efficiency, is beneficial to time series modeling, has better effects with smaller parameter quantity, has robustness and strong interpretability, and is not easily disturbed by noise.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, neural networks and pattern recognition technology, and specifically relates to a gesture recognition system and method. Background Technology

[0002] Due to its naturalness and convenience, gesture interaction can be applied in numerous fields. For example, in intelligent transportation, gesture recognition enables rapid interaction between drivers and onboard computers. Furthermore, with the rapid development of autonomous driving technology in recent years, incorporating the recognition of traffic police gestures will further improve existing autonomous driving technology. In smart homes, gesture recognition can complement voice control, enabling more natural control of smart homes. In sign language recognition, mobile devices can detect and recognize the meaning of sign language used by deaf and mute individuals, which is revolutionary in addressing communication difficulties faced by this population.

[0003] Early implementations of gesture recognition, besides traditional shallow networks, primarily relied on deep learning networks based on restricted Boltzmann machines or LeNet-5 convolutional neural networks, or other methods improved from deep learning. While these deep learning methods outperformed traditional shallow networks, they often required massive amounts of computation. This approach not only incurred enormous computational costs but also resulted in long processing times and low accuracy, making it virtually impossible to guarantee accuracy and timeliness under conditions of energy efficiency and limited hardware computing power. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the present invention aims to provide a gesture recognition system and method based on neural circuit strategy, which can greatly reduce the demand for computing power, improve processing speed and intelligence, and improve recognition accuracy.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A gesture recognition system, characterized in that it includes:

[0007] The input module uses an event camera to capture gesture images in real time;

[0008] The convolutional feature extraction module is a convolutional neural network, including convolutional layers, pooling layers, and fully connected layers, used to extract digital structure features from the pixels of the gesture image to obtain a feature vector sequence;

[0009] The control system module is implemented by a neural circuit strategy network, which consists of four layers of neurons: a perception layer, a relay layer, a control layer, and a driver layer. The neurons in the perception layer are sensory neurons, the neurons in the relay layer are interneurons, the neurons in the control layer are command neurons, and the neurons in the driver layer are motor neurons. The interneurons establish synaptic connections with both the sensory neurons and the command neurons. The command neurons establish synaptic connections with both the interneurons and the motor neurons, and also establish self-feedback synaptic connections with other command neurons to form a cyclic connection structure.

[0010] The sensory neurons are used to receive the feature vector sequence, convert the feature vector sequence into pulse signals, and stimulate the interneurons, that is, to send inhibitory or excitation signals to the interneurons.

[0011] The interneuron is used to escape the feature vector sequence and send inhibition or activation signals to the instruction neuron;

[0012] The instruction neuron stores and makes decisions on temporal information, establishes a self-loop in the control layer, and stimulates the motor neuron, sending inhibition or excitation signals to the motor neuron and the other instruction neurons.

[0013] The motor neurons output a digital code sequence representing the type of gesture based on the intensity of their own pulse signals, thereby determining the result of gesture recognition.

[0014] The present invention also provides a gesture recognition method based on the gesture recognition system, comprising the following steps:

[0015] Step 1) Capture gesture images in real time using an event camera;

[0016] Step 2) Use the convolutional feature extraction module to perform convolution and pooling operations on the gesture image to extract the gesture features in the input image and encode them to obtain a feature vector sequence;

[0017] Step 3) After receiving the feature vector sequence, the perceptual layer transmits inhibition or excitation signals to the next layer through synapses of two positive and negative polarities. Based on the weight and polarity of the synapses, it controls the pulse delay length and dynamically updates the membrane potential of the target neuron. The signal is transmitted from the perceptual layer to the control layer through the intermediate layer, then cyclically transmitted and output to the driving layer. The driving layer updates the state of all neurons. Each motor neuron calculates its output membrane potential through a nonlinear activation function. The gesture corresponding to the bionic neuron with the highest membrane potential is the gesture recognized by the system, thus completing gesture recognition.

[0018] Compared with the prior art, the beneficial effects of the present invention are:

[0019] 1. This method constructs a four-layer neural circuit strategy network, which can efficiently and accurately complete gesture recognition. Compared with other technologies at present, this method has lower hardware computing power requirements, can have higher computational efficiency, and can achieve good results with only a small number of neurons.

[0020] 2. This method utilizes a multi-level neuron delay cascade structure and uses pulses to transmit information, resulting in a more flexible, faster, and more energy-efficient response. The use of biomimetic neurons with nonlinear time-varying characteristics is beneficial for time series modeling.

[0021] 3. This method is robust and highly interpretable, and is not easily affected by noise. That is, the model can reliably focus on the more critical information of the gesture, thereby improving the accuracy of recognition.

[0022] 4. This method is a specific implementation of third-generation neural networks in gesture recognition. Compared with other technologies at present, the working principle of this method is closer to the functional principle of nerve cells, has more advanced theoretical support, and has more development potential in the field of artificial intelligence. Attached Figure Description

[0023] Figure 1 This is a block diagram illustrating the principle of the present invention.

[0024] Figure 2 It is the basic model of neural circuit strategy networks.

[0025] Figure 3 , Figure 4 , Figure 5 These are the three rules followed in the design of neural circuit strategy network structures.

[0026] Figure 6 This is a diagram illustrating the output of the gesture type determination. Detailed Implementation

[0027] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples. This embodiment elaborates on an example of the specific implementation of a gesture recognition system built using a neural circuit-based strategy in the case of a gesture recognition training set DvsGesture.

[0028] DvsGesture is a dataset for pose recognition in real-world scenarios. It contains 11 gestures from 29 subjects under three lighting conditions. The class values ​​and gesture correspondences are as follows: 1: Clapping; 2: Right hand shaking; 3: Left hand shaking; 4: Right arm clockwise; 5: Right arm counterclockwise; 6: Left arm clockwise; 7: Left arm counterclockwise; 8: Arm rolling; 9: Air drum; 10: Air guitar; 11: Other gestures.

[0029] The system framework for gesture recognition in this invention is based on Neural Circuit Policy Construction (NCP), employing a convolutional neural network and a four-layer neural circuit policy network. Functionally, it includes an input module that captures the event stream of gesture images, a convolutional feature extraction module that extracts structural features from the pixels of the input image (implemented by the four-layer neural circuit policy network), and a control system module that finally outputs the meaning of the gesture.

[0030] For details on the architecture and functions of each module, please refer to [link / reference]. Figure 1 :

[0031] The input module uses an event camera to capture gesture images in real time. Specifically, by asynchronously monitoring the brightness changes of each pixel's sensor, changes in multiple pixels generate an event stream, thus capturing the gesture image event stream and outputting it as an AER format image. The events are then input as individual vector sequences into the convolutional feature extraction module; each vector sequence is an expanded representation of the event stream.

[0032] The convolutional feature extraction module is a compact convolutional neural network, including convolutional layers, pooling layers, and fully connected layers. It extracts digital structure features from the pixels of the input image. It performs convolution, pooling, and fully connected operations on the event flow gesture image to obtain a sequence of feature vectors corresponding to the image. Specifically, each vector sequence passes through two convolutional layers, one pooling layer, and one fully connected layer. The first convolutional layer identifies the more important features of the gesture in the image and only transmits this part of the image information to the next module.

[0033] The control system module is implemented using a neural circuit strategy network, which consists of four layers of neurons: a perception layer, a relay layer, a control layer, and a driver layer. The neurons in the perception layer are sensory neurons, the neurons in the relay layer are interneurons, the neurons in the control layer are command neurons, and the neurons in the driver layer are motor neurons. Interneurons establish synaptic connections with both sensory and command neurons. Command neurons establish synaptic connections with both interneurons and motor neurons, and also establish self-feedback synaptic connections with other command neurons to form a cyclic connection structure.

[0034] The neural circuit strategy network creates bionic neurons and establishes synapses between them that can transmit inhibitory and excitation signals. The synapses change the state of the target bionic neuron through asynchronous bipolar signal transmission between synapses. The state update of the bionic neuron corresponds to the processing of image feature vector pulse signals.

[0035] Specifically, sensory neurons receive feature vector sequences, convert them into pulse signals, and stimulate interneurons, sending inhibitory or excitation signals to the interneurons. Interneurons escape the obtained feature vector sequences and send inhibitory or excitation signals to instruction neurons. Instruction neurons store temporal information and make decisions, establishing self-loops in the control layer while simultaneously stimulating motor neurons, sending inhibitory or excitation signals to motor neurons and other instruction neurons. Motor neurons in the driver layer output a digital encoding sequence representing the gesture type based on the intensity of their own pulse signals, thereby determining the gesture recognition result. This process includes implementing image features to support convolutional networks, constructing an RNN-like recurrent connection structure, and ultimately outputting the gesture meaning.

[0036] In the control system module of this invention, each bionic neuron uses membrane potential to represent the neuron state and uses differential equations to dynamically update the membrane potential. The neuron state is jointly determined by the current membrane potential and the effect of the input synapse from the upper layer neuron to the current neuron. Furthermore, the synapses connecting the bionic neurons have different weights and two polarities. Positive polarity synapses will cause the target neuron's membrane potential to rise, while negative polarity synapses will cause the target neuron's membrane potential to fall. Therefore, different synapses have different effects on the target neuron's membrane potential.

[0037] The differential equation for updating the membrane potential is expressed as follows:

[0038]

[0039] Where x i This refers to the current state of neuron i, i.e., the membrane potential. It has leakage conductivity The time constant of neuron i, τ in different bionic neurons i The differences ensure the asynchronous nature of membrane potential changes. ij It is the synaptic weight from neuron j to neuron i. It is a film capacitor, σ i (x j ) is the neuron activation function, which is positively correlated with signal strength. It is the resting potential, E ij It is a reverse synaptic potential, E ij The polarity of the synapse is defined, and the overall coupling sensitivity, i.e., the time constant, of the bionic neuron is determined by... The time constant is defined to be variable, determining the reaction speed of the bionic neuron during the decision-making process.

[0040] In this invention, the number of sensory neurons N in the sensory layer s The length of the feature vector sequence output by the convolutional feature extraction module is equal to the length of the sequence, and the number of intermediate neurons is N. iThe number of instruction neurons is N c The number of motor neurons is N m , where N m -1 represents the types of gestures that this system can recognize; sparse synaptic connections are established between adjacent layers of bionic neurons according to a preset rule, and the establishment and polarity of synapses are random.

[0041] In the neural circuit strategy network established in this case, the number of perceptual neurons is the same as the dimension of the feature vector. The intermediate layer includes 32 intermediate neurons, the command layer includes 8 command neurons, and finally there are 11 driver neurons (corresponding to 11 types of recognizable gestures).

[0042] In this invention, the rules for synapse formation are as follows:

[0043] refer to Figure 2 The neural circuit strategy network consists of four layers of bionic neurons, of which N s N i N c N m The number of bionic neurons corresponding to the perception layer, relay layer, command layer, and driver layer, respectively.

[0044] refer to Figure 3 For two adjacent layers of bionic neurons, n is inserted into all source bionic neurons. so-t A synapse to n so-t On each of the target bionic neurons, n so-t ≤N n N n The number of bionic neurons in the next layer is n, and the random selection of the target bionic neuron follows a certain order. s-t The binomial distribution and the polarity selection of synapses follow a Bernoulli distribution.

[0045] refer to Figure 4 In any two adjacent layers, for all target bionic neurons that do not have synaptic input connections in 2), calculate the average number of synapses L received by each bionic neuron in the layer containing the target bionic neuron, and then calculate the number of synapses L from the upper layer through m. so-t A binomial distribution, with m randomly selected. so- A synapse is established between the source bionic neuron and the target bionic neuron. so-t For values ​​≤L, synaptic polarity is initialized using a Bernoulli distribution.

[0046] refer to Figure 5 For all bionic neurons in the control layers, insert l so-t One synapse, l so-t ≤N c The corresponding target bionic neuron is transmitted through l so-tThe binomial distribution is randomly selected from the control layer, and the polarity of each synapse is initialized using a Bernoulli distribution.

[0047] Accordingly, the gesture recognition method of the present invention includes the following steps:

[0048] Step 1) Capture gesture images in real time using an event camera;

[0049] Step 1.1), Gesture Image Capture

[0050] Gesture images are captured by an event camera and stored as AER format images. To prevent the image quality of gesture images from being distorted to varying degrees due to noise, after acquiring the original gesture image event stream captured by the event camera, the original gesture image can be preprocessed by smoothing and binarization to remove the influence of noise and lighting on the original image, and the processed gesture image is used as input.

[0051] Step 1.2), Feature Extraction

[0052] The image is processed through a two-layer convolutional neural network to extract local features. Through the convolutional neural network, the model can identify the more important features of the gesture in the image and encode only this part of the image information before transmitting it to the next module as input.

[0053] Step 2) Use the convolution feature extraction module to perform convolution and pooling operations on the gesture image to extract the gesture features in the input image and encode them to obtain a feature vector sequence.

[0054] Step 3) After receiving the feature vector sequence, the perceptual layer transmits inhibition or excitation signals to the next layer through synapses of two positive and negative polarities. Based on the weight and polarity of the synapses, it controls the pulse delay length and dynamically updates the membrane potential of the target neuron. The signal is transmitted from the perceptual layer to the control layer through the intermediate layer, then cyclically transmitted and output to the driving layer. The driving layer updates the state of all neurons. Each motor neuron calculates its output membrane potential through a nonlinear activation function. The gesture corresponding to the bionic neuron with the highest membrane potential is the gesture recognized by the system, thus completing gesture recognition.

[0055] Step 3.1), Feature Vector Reception

[0056] The feature vector sequence of the gesture image is converted into a pulse signal and input to the corresponding sensory neuron in the perceptual layer of the neural circuit strategy network. The perceptual layer transmits inhibitory or excitation signals to the intermediate layer through synapses of different polarities and updates the state of the intermediate neuron according to the weight of the synapse, while continuing to receive feature encoding and pass it down.

[0057] Step 3.2), intermediate layer transfer

[0058] Interneurons may receive excitation or inhibition signals. Excitation signals increase the neuronal membrane potential, while inhibition signals decrease it. During signal transmission, when the membrane potential of the positive polarity progenitor neuron is higher than the transmission threshold, the signal strength will be enhanced, while when the membrane potential of the negative polarity progenitor neuron is higher than the transmission threshold, the signal strength will be reduced. This is used to simulate a biological nervous system model.

[0059] Step 3.3), control layer loop

[0060] In addition to continuing to transmit excitation and inhibition signals to the motor neurons of the lower driver layer through synapses, the command neurons can also simultaneously receive the signal output generated by the previous time interval in their own control layer. Both work together to affect the membrane potential of the target neuron. The signal transmission mechanism and the effect of synapses are similar to step 3.2.

[0061] Step 3.4), driver layer output

[0062] After the motor neurons in the driver layer receive signals from the neurons in the upper layer, their membrane potential is converted into a probability value by a nonlinear activation function. This probability value represents the likelihood that the neuron is encoded as any of the possible gestures. The gesture corresponding to the neuron with the highest confidence level is selected as the gesture recognition result.

[0063] Ultimately, the membrane potential of motor neurons is converted into probability values ​​through a nonlinear activation function, which can then be used to determine the likelihood of all gestures, thus completing gesture recognition.

[0064] refer to Figure 6 The state quantization values ​​of all driving layer neurons are processed by the softmax normalization function and mapped to values ​​representing confidence levels. The neuron with the largest mapped value is identified by its ID, and a pre-designed gesture ID is queried to find the corresponding gesture. In this case, each dimension of the output 11-dimensional vector corresponds to the probability that the input feature vector is identified as a letter of that dimension, such as (0.96, 0.02, 0, ..., 0.01). The gesture corresponding to the dimension with the largest component of 0.96 in this vector is clapping, which means that the input event flow feature vector is most likely to be clapping for the system. Therefore, a gesture recognition result for an event flow is obtained.

[0065] In summary, the present invention comprises three modules to form an independent gesture recognition system framework based on a neural circuit strategy. Compared with other technologies, this invention proposes a novel approach that applies spiking neural networks to audiovisual assistance systems, achieving relatively low overall computational load while ensuring rapid response capabilities.

[0066] The above description is merely a preferred embodiment of the present invention, and the present invention should not be limited to the content disclosed in this embodiment and the accompanying drawings. Any equivalent or modified embodiments made without departing from the spirit of the present invention fall within the scope of protection of the present invention.

Claims

1. A gesture recognition system, characterized by, The gesture recognition system comprises: an input module for capturing gesture images in real time by using an event camera; a convolution feature extraction module, which is a convolution neural network comprising a convolution layer, a pooling layer and a full connection layer, for extracting digital structural features from pixels of the gesture images to obtain a feature vector sequence; a control system module, which is implemented by a neural circuit strategy network composed of four layers of neurons, i.e., a perception layer, a transfer layer, a control layer and a drive layer; the neurons of the perception layer are sensory neurons, the neurons of the transfer layer are intermediate neurons, the neurons of the control layer are instruction neurons, and the neurons of the drive layer are motor neurons; the intermediate neurons are connected to the sensory neurons and the instruction neurons by synapses; the instruction neurons are connected to the intermediate neurons and the motor neurons by synapses, and are connected to other instruction neurons by self-feedback synapses to form a cyclic connection structure; the sensory neurons are used to receive the feature vector sequence, convert the feature vector sequence into a pulse signal, and stimulate the intermediate neurons, i.e., send an inhibitory signal or an excitation signal to the intermediate neurons; the intermediate neurons are used to decode the feature vector sequence and send an inhibitory signal or an excitation signal to the instruction neurons; the instruction neurons are used to save timing information and make decisions, form a self-loop in the control layer, stimulate the motor neurons, and send an inhibitory signal or an excitation signal to the motor neurons and the other instruction neurons; the motor neurons output a digital code sequence representing the type of gesture according to the level of their own pulse signal, so as to determine the result of gesture recognition.

2. The gesture recognition system of claim 1, wherein, In the input module, an AER format gesture image event stream is collected by using an event camera as the input of the convolution feature extraction module.

3. The gesture recognition system of claim 1, wherein, In the convolution feature extraction module, the event stream gesture image is subjected to convolution, pooling and full connection operations to obtain a feature vector sequence corresponding to the image.

4. The gesture recognition system of claim 1, wherein, In the control system module, each bionic neuron uses membrane potential to represent the state of the neuron, and the membrane potential is dynamically updated by a differential equation. The state of the neuron is determined by the current membrane potential and the input synapses from the upper layer of neurons to the current neuron. The synapses between the bionic neurons have different weights and two polarities. The synapses with positive polarity will make the membrane potential of the target neuron rise, and the synapses with negative polarity will make the membrane potential of the target neuron fall.

5. The gesture recognition system of claim 4, wherein, When neuron j is connected to neuron i by a synapse, the differential equation for updating the membrane potential is as follows: Where x i This refers to the current state of neuron i, i.e., the membrane potential. It has leakage conductivity g li The time constant of neuron i, τ in different bionic neurons i The differences ensure the asynchronous nature of membrane potential changes. ij It is the synaptic weight from neuron j to neuron i. It is a film capacitor, σ i (x j ) is the neuron activation function, which is positively correlated with signal strength, x leaki It is the resting potential, E ij It is a reverse synaptic potential, E ij The polarity of the synapse is defined, and the overall coupling sensitivity, i.e., the time constant, of the bionic neuron is determined by... The time constant is defined to be variable, determining the reaction speed of the bionic neuron during the decision-making process.

6. The gesture recognition system of claim 5, wherein, The number of sensory neurons N of the perception layer s The number of intermediate neurons N i The number of command neurons N c The number of motor neurons N m Wherein, N m -1 represents the number of gestures that can be recognized by the system; sparse synaptic connections are established between adjacent two layers of bionic neurons according to a preset probability rule, and randomness exists in the establishment and polarity of the synapses.

7. The gesture recognition system of claim 6, wherein, The establishment rule of the synapse is as follows: 1), the neural circuit strategy network is composed of four layers of bionic neurons, wherein N s , N i , N c , N m respectively correspond to the number of bionic neurons of the perception layer, the transfer layer, the command layer and the driving layer. 2), for two adjacent layers of the biomimetic neurons, n so-t synapses are inserted on all source biomimetic neurons to n so- target biomimetic neurons, where n so-t ≤ N n , N n is the number of biomimetic neurons of the next layer, the random selection of the target biomimetic neurons is subject to n s-t times binomial distribution, and the polarity selection of the synapses satisfies Bernoulli distribution; 3) In any two adjacent layers, for all target bionic neurons that do not have synaptic input connections in 2), calculate the average number of synapses L received by each bionic neuron in the layer containing the target bionic neuron, and then calculate the number of synapses L from the upper layer through m. so-t A binomial distribution, with m randomly selected. so-t A synapse is established between the source bionic neuron and the target bionic neuron. so-t For ≤L, synaptic polarity is initialized using a Bernoulli distribution; 4), for all the control layer's neurons, insert l so- synapses, l so-t ≤ N c corresponding target neurons are randomly selected from the control layer by l so- times binomial distribution, and the polarity of each synapse is initialized using Bernoulli distribution.

8. A gesture recognition method based on the gesture recognition system of claim 1, comprising the following steps: Step 1) capturing gesture images in real time by using an event camera; Step 2) performing convolution and pooling operations on the gesture images by using the convolution feature extraction module, extracting gesture features in the input image, and encoding to obtain a feature vector sequence; Step 3) After the perception layer receives the sequence of feature vectors, it transmits inhibitory signals or excitatory signals to the next layer by means of synapses of two polarities, and according to the weights and polarities of the synapses, controls the length of the pulse delay, dynamically updates the membrane potential of the target neurons, and transmits the signals from the perception layer to the control layer through the relay layer, then circulates and outputs to the drive layer, the drive layer updates the state of all neurons, each motor neuron calculates the output membrane potential through a nonlinear activation function, and the gesture corresponding to the biomimetic neuron with the highest membrane potential is the gesture recognized by the system, thereby completing gesture recognition.

9. The gesture recognition method of claim 8, wherein, The step 1) comprises the following steps: Step 1.1), gesture image capture The gesture image is captured by an event camera and stored as an AER format image; Step 1.2), feature extraction The image is subjected to two layers of convolutional neural network to extract local features, and through the convolutional neural network, the model can identify the important feature information of the gesture in the image, and only this part of the image information is encoded and transmitted to the next module as input; The step 3) comprises the following steps: Step 3.1), feature vector receiving The sequence of feature vectors of the gesture image is converted into pulse signals and input to the corresponding sensory neurons of the perception layer of the neural circuit strategy network, the perception layer transmits inhibitory signals or excitatory signals to the relay layer through synapses of different polarities, and updates the state of the intermediate neurons according to the weights of the synapses, while continuing to receive feature encoding and transmit downward; Step 3.2), relay layer switching The intermediate neurons receive excitatory signals or inhibitory signals, the excitatory signals will increase the membrane potential of the neurons, and the inhibitory signals will decrease the membrane potential of the neurons, and in the process of signal transmission, the membrane potential of the source neuron on the synapse of positive polarity will be higher than the transmission threshold to enhance the intensity of the signal, and the membrane potential of the source neuron on the synapse of negative polarity will be higher than the transmission threshold to reduce the intensity of the signal, so as to simulate the biological neural system model; Step 2.3), control layer circulation In addition to continuing to transmit excitatory and inhibitory signals to the motor neurons through synapses, the command neurons also simultaneously receive the signal output generated by the control layer in the previous time interval, and the two work together to affect the membrane potential of the target neurons; Step 3.4), drive layer output After the drive layer motor neurons receive the signals of the upper layer neurons, the membrane potential thereof is converted into a probability value by a nonlinear activation function, which represents the possibility of the neuron corresponding to all possible gestures, and the gesture corresponding to the neuron with the highest confidence degree is selected as the gesture recognition result.

10. The gesture recognition method of claim 8, wherein, The step 3), the membrane potential of the motor neuron is converted into a probability value by a nonlinear activation function, which can judge the possibility of all gestures, thereby completing gesture recognition.

Images