Gesture recognition method, upper computer and system

Abstract

The invention provides a gesture recognition method, an upper computer and a system, and relates to the technical field of gesture recognition. The method comprises the following steps: acquiring hand pressure data; inputting the hand pressure data into a pre-trained convolutional pulse neural network model to obtain a gesture recognition result; wherein the pre-trained convolutional pulse neural network model is an improved convolutional neural network model based on a pulse neural network model; a pooling layer and a full connection layer in the improved convolutional neural network model are composed of a pulse neural network model. The traditional convolutional neural network can be improved, the spiking neural network model and the convolutional neural network model are combined, the improved model fuses the feature extraction capability of the convolutional neural network and the space-time coding characteristic of the spiking neural network model, the accuracy of data processing can be ensured, and meanwhile, the accuracy of the spiking neural network model is improved. Partial multiplication is replaced by pulse accumulation, and energy consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of gesture recognition, and in particular to a gesture recognition method, a host computer, and a system. Background Art

[0002] Gestures play a crucial role in the human information transmission system. Research data shows that language and text only account for 45% of information transmission, while gestures contribute a whopping 55%. With the rapid development of artificial intelligence (AI), gesture recognition, as a highly promising intelligent recognition technology, has become a research focus. It holds broad application prospects in a wide range of fields, including smart homes, healthcare, and sports training.

[0003] Currently, gesture recognition research is diverse and diverse. Strain-sensing-based gesture recognition, leveraging the advantages of tactile sensor technology, can effectively mitigate interference from environmental factors. Resistive pressure sensors, for example, offer both low cost and high accuracy, making them a key research area in the sensor recognition field in recent years.

[0004] Deep learning algorithms are one of the core technologies for tactile gesture recognition. Convolutional Neural Networks (CNNs), for example, utilize convolutional layers to achieve relatively good classification results when processing high-dimensional data, eliminating the need for manual feature extraction. However, CNN algorithms have certain limitations. First, they use continuous functions as activation functions, abandoning time-domain operations and retaining only layer-by-layer computation in the spatial domain. This results in a lack of information representation and makes it difficult to fully capture the time series information in the data. Second, CNN operations involve numerous matrix multiplications, which consume significant energy and can only be parallelized on graphics processing units (GPUs). This not only limits the algorithm's deployment on dedicated neuromorphic hardware but also hinders its application in embedded devices, significantly limiting its application in portable devices and scenarios with high real-time requirements. Summary of the Invention

[0005] Embodiments of the present invention provide a gesture recognition method, host computer, and system to address the problems of incomplete information representation and high energy consumption during the calculation process when implementing a tactile-based gesture recognition method using a traditional deep learning algorithm.

[0006] In a first aspect, an embodiment of the present invention provides a gesture recognition method, comprising:

[0007] Collect hand pressure data;

[0008] The hand pressure data is input into a pre-trained convolutional spiking neural network model to obtain gesture recognition results; wherein, the pre-trained convolutional spiking neural network model is a convolutional neural network model improved based on the spiking neural network model; the pooling layer and the fully connected layer in the improved convolutional neural network model are composed of the spiking neural network model.

[0009] In one possible implementation, the spiking neural network model includes an Izhikevich neuron model; the pre-trained convolutional spiking neural network model includes an encoder and a classifier; wherein the encoder includes the Izhikevich neuron model; hand pressure data is input into the pre-trained convolutional spiking neural network model to obtain a gesture recognition result, including:

[0010] According to the hand pressure data, the static component and the dynamic component are obtained;

[0011] The static component and the dynamic component are input into the pre-trained convolutional spiking neural network model respectively;

[0012] In the encoder in the pre-trained convolutional spike neural network model, the static component and the dynamic component are encoded to obtain the dynamic pulse component and the static pulse component, and the dynamic pulse component and the static pulse component are input into the classifier to obtain the gesture recognition result.

[0013] In one possible implementation, the spiking neural network model includes an integrate-and-fire neuron model; the classifier includes a first convolutional layer; the first convolutional layer is composed of an integrate-and-fire neuron model; the dynamic pulse component and the static pulse component are input into the classifier to obtain a gesture recognition result, including:

[0014] The first convolutional layer in the classifier extracts time series features from the dynamic pulse component and the static pulse component, and obtains the gesture recognition result based on the extracted time series features.

[0015] In one possible implementation, the classifier further includes a second convolutional layer and a third convolutional layer; the second convolutional layer and the third convolutional layer include a pooling layer; the first convolutional layer in the classifier extracts time series features from the dynamic pulse component and the static pulse component, and obtains a gesture recognition result based on the extracted time series features, including:

[0016] The first convolution layer in the classifier extracts time series features of the dynamic pulse component and the static pulse component to obtain time series feature data, and transmits the time series feature data to the second convolution layer and the third convolution layer;

[0017] In the second and third convolutional layers, spatial features are extracted from the time series feature data to obtain spatial feature data;

[0018] Based on the spatial feature data, the gesture recognition result is obtained.

[0019] In one possible implementation, in an encoder in a pre-trained convolutional spike neural network model, encoding the static component and the dynamic component to obtain a dynamic pulse component and a static pulse component includes:

[0020] In the encoder of the pre-trained convolutional spike neural network model, the dynamic component is encoded based on the Izhikevich neuron model in the fast discharge mode, and the dynamic pulse component is output; the static component is encoded based on the Izhikevich neuron model in the regular discharge mode, and the static pulse component is output.

[0021] In one possible implementation, obtaining the static component and the dynamic component based on the hand pressure data includes:

[0022] Convert hand pressure data into digital signals;

[0023] The digital signal is filtered to obtain the static component and the dynamic component.

[0024] In one possible implementation, after inputting the hand pressure data into a pre-trained convolutional spiking neural network model to obtain a gesture recognition result, the method further includes:

[0025] Visualize the gesture recognition results.

[0026] In a second aspect, an embodiment of the present invention provides a host computer, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method in the first aspect or any possible implementation of the first aspect.

[0027] In a third aspect, an embodiment of the present invention provides a gesture recognition system including a tactile sensor glove, a lower computer, and the upper computer provided in the second aspect above.

[0028] In a possible implementation, the lower computer includes a field editable gate array.

[0029] The embodiment of the present invention provides a gesture recognition method, a host computer and a system, which improves the pooling layer and the fully connected layer in the convolutional neural network model through the pulse neural network model because the pulse neural network model can naturally process time series information. It transmits and processes information in the form of a pulse sequence, and replaces the multiplication and accumulation in the traditional convolutional neural network with the pulse addition and accumulation of the pulse neural network model, which can reduce the energy consumption of the application during operation. That is, the improved pooling layer can effectively reduce the size of the input, and at the same time, the improved fully connected layer reduces the high energy consumption caused by the large number of parameters brought by the fully connected layer in the traditional model. Therefore, this embodiment improves the traditional convolutional neural network and combines the pulse neural network model with the convolutional neural network model. The improved model integrates the feature extraction capability of the convolutional neural network and the spatiotemporal coding characteristics of the pulse neural network model. It can replace some multiplication operations by pulse accumulation while ensuring accurate data processing, thereby reducing energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a system architecture diagram of a gesture recognition system provided by an embodiment of the present invention;

[0031] Figure 2 is a pressure characteristic curve diagram provided by an embodiment of the present invention;

[0032] Figure 3 This is a diagram of the lower computer framework provided by an embodiment of the present invention;

[0033] Figure 4 This is a diagram of the structure of the acquisition peripheral circuit provided by an embodiment of the present invention;

[0034] Figure 5 This is a connection diagram of the ADC module provided by an embodiment of the present invention;

[0035] Figure 6 This is a schematic diagram of the host computer main interface provided by an embodiment of the present invention;

[0036] Figure 7a-7b Schematic diagram of the serial communication interface and acquisition control interface provided by an embodiment of the present invention;

[0037] Figures 8a-8b This is a schematic diagram of a data storage interface and a result recognition interface provided by an embodiment of the present invention;

[0038] Figure 9 is a flowchart of an implementation of a gesture recognition method provided by an embodiment of the present invention;

[0039] Figure 10 is an architecture diagram of a pre-trained convolutional spiking neural network model provided by an embodiment of the present invention;

[0040] Figure 11is a schematic diagram of the tactile nerve response encoding process provided by an embodiment of the present invention;

[0041] Figure 12 It is a schematic diagram of the IF model structure;

[0042] Figure 13 It is a schematic diagram of the charging and discharging process of the IF model. DETAILED DESCRIPTION

[0043] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0044] Figure 1 is a system architecture diagram of a gesture recognition system provided by an embodiment of the present invention, such as Figure 1 As shown, the system includes a tactile sensor glove, a lower computer and a upper computer.

[0045] The tactile sensor gloves are equipped with pressure sensors. When the subjects wear the tactile sensor gloves, different gestures will cause the pressure sensors to generate different pressure signals. The lower computer controls the pressure sensors to collect gesture signals and obtain hand pressure data. After processing the hand pressure data, it is transmitted to the upper computer, which extracts and identifies it to determine the specific gesture action corresponding to the subject.

[0046] Figure 2 This is a pressure characteristic curve diagram provided by an embodiment of the present invention. Figure 2 The data acquisition principle of the tactile sensor gloves is explained.

[0047] In this embodiment, a flexible glove-type pressure sensor can be incorporated into a tactile sensor glove to collect hand pressure data. This type of pressure sensor is manufactured by transferring materials such as nano-force-sensitive materials and silver paste onto a flexible film through a precision printing process, followed by drying and curing. Its main parameters are shown in Table 1.

[0048] Table 1 Main parameters of pressure sensor

[0049]

[0050] As shown in Table 1, the pressure sensor has a range of 1-50kg; the sensing dimensions are 160×177mm, with a thickness of less than 0.2mm; the static resistance is greater than or equal to 1MΩ; the repeatability is ±6%; the hysteresis is less than 6%; the drift is less than 8%; the operating temperature is between -30°C and +60°C, and the response time is less than 20ms.

[0051] In order to facilitate the acquisition of pressure data for different gestures, the pressure film is designed to fit the shape and size of the hand. The overall sensor can be set to a 5×5 array with a total of 25 sensing units. Each sensing unit can be regarded as a pressure variable resistor. Each sensing point area is 100mm and is evenly distributed at different points of action on the palm.

[0052] The pressure characteristic curve of the pressure sensor is shown in the figure below. Figure 2 As shown, Figure 2 In the figure, the horizontal axis is the weight (pressure) applied by the subject on the pressure sensor, and the horizontal axis is the corresponding resistance value; Figure 2 The lines corresponding to the dots in the figure represent the relationship between the inverse of resistance and pressure, and the lines corresponding to the squares represent the relationship between resistance and pressure, that is, the piezoresistive characteristics. Figure 2 It can be seen that the piezoresistive characteristic of the pressure sensor is a power function relationship, and the inverse of the resistance is approximately linearly related to the pressure.

[0053] In actual use, the lower computer controls the tactile sensor glove to collect corresponding hand pressure data. Figure 3 This is a diagram of the lower machine framework provided by an embodiment of the present invention; Figure 3 As shown, the signal acquisition portion of the lower computer is primarily implemented by a signal acquisition module and an A / D conversion module. The core control portion is primarily implemented by a field programmable gate array (FPGA). The transmission portion utilizes a USB 2.0 interface. In actual use, the FPGA controls the signal acquisition unit, which acquires the outputs of 25 sensors at a 100 Hz frequency through array scanning. The A / D conversion module converts the collected voltage data into digital values for storage, and transmits the data to the upper computer via serial communication for display.

[0054] Figure 4 This is a diagram of the peripheral circuit structure of the acquisition provided by the embodiment of the present invention. Figure 4 The working process and structure of the acquisition peripheral circuit are explained.

[0055] The sensor elements of the acquisition peripheral circuit are distributed in an array. During the acquisition process, the FPGA sends a selection signal, which is applied to one electrode at a time, and the measurement of all electrode signals is achieved by scanning. During the acquisition, the selection signals ctl.1-6 sent by the lower computer control three multiplexers respectively. The ctl.1-3 signals control multiplexers (2) and (3) for selecting rows, and the ctl.4-6 signals control multiplexer (1) for selecting columns. Only one signal can be selected at a time. When reading the first column sensor signal, the column is connected to the voltage VCC (5V) through the selection signal, while the other columns remain grounded. That is, in the ideal state, the current between the resistors of the other columns is 0A, and the multiplexers (2) and (3) control the row selection. That is, if the control Y0 is selected, the signal at R (1,1) is connected, so that the two analog switches are used as analog multiplexing solutions to read the resistors one by one from the active columns. Compared with the traditional acquisition peripheral circuit, the embodiment of the present invention adds a voltage divider resistor Rs after the multiplexer (3), obtains the reading of the gate resistor by measuring the voltage on the voltage divider resistor, and makes the control signal of the multiplexer (3) the same as the control signal of the multiplexer (2), so that only one voltage divider resistor can be used to measure all the resistors in the array, saving the use of the A / D conversion module.

[0056] This embodiment uses a multiplexer as a row and column-selecting analog switch. This high-speed silicon-gate CMOS device is an 8-channel analog multiplexer / demultiplexer with three digital select inputs (S0 to S2), an active-low enable input (E), eight independent inputs / outputs (Y0 to Y7), and a common input / output (Z). The control circuit sends an address code to the digital select inputs and the active-low enable input. Y0-Y7 output the Z port voltage level according to the command. The control circuit then issues a command to collect the voltage across the voltage divider resistor in the corresponding path. Therefore, this embodiment, through this circuit structure, combined with the above-described method, can collect pressure distribution data for all gestures.

[0057] Since the signal collected by the acquisition peripheral circuit is a voltage signal obtained by dividing the voltage by a fixed resistor, in order to facilitate the storage, transmission and processing of information, the collected voltage signal can be converted into a digital signal through an analog to digital converter (ADC) during processing. Figure 5 This is a connection diagram of the ADC module provided by an embodiment of the present invention. Figure 5 The A / D conversion module shown performs the conversion.

[0058] like Figure 5As shown, this embodiment uses a high-speed AD chip as the ADC module. The chip is a single-chip, 8-bit, 32MSPS analog-to-digital converter with a built-in on-chip sample-and-hold amplifier and reference voltage. It adopts a multi-stage differential pipeline architecture with a data rate of 32MPSP, which can ensure that there is no code loss in the entire operating range. The AD chip is powered by a +2.7V to +5.5V power supply and has a rated temperature of -40°C to +85°C. The measured voltage signal is connected to the analog input signal terminal of the high-speed AD / DA board. The board will sample, quantize, and encode the input analog signal, and then convert the analog voltage into a digital value and transmit it to the FPGA. The ADC chip bit width of this solution is 8 bits, and the analog voltage input range is -5V to +5V, that is, the resolution is 10 / 28.

[0059] When the voltage value ad_data collected by the ADC chip is within the range of 0-127, that is, within the range of -5V-0V, the converted voltage value is:

[0060] V in =-(10 / 2 8 ×(127-ad_data))

[0061] When the voltage value ad_data is within the range of 128-255, that is, within the range of 0V-+5V, the converted voltage value is:

[0062] V in =(10 / 2 8 ×(ad_data-127))

[0063] During actual use, the FPGA is controlled to store and transmit the received voltage digital value after calculation.

[0064] After the above data collection and processing are completed, specific gesture recognition needs to be performed through the host computer.

[0065] Figure 6 This is a schematic diagram of the host computer main interface provided by an embodiment of the present invention. Figure 6 As shown, the host computer can be a laptop computer, which has the functions of data processing and page display. It mainly completes the human-computer interaction function, realizes data collection, transmission, analysis and result display. The host computer can include a USB communication module, an acquisition control module, a data storage module and a result recognition module. During use, each function can be accessed through the buttons of each module. For example, it can be accessed by clicking the corresponding button on the display screen.

[0066] When the host computer is operating normally, it creates a serial port object through the USB communication module and searches for and connects to the serial port device using the configured serial port parameters. Once connected, the acquisition control module controls the acquisition state of the slave computer to collect pressure data and reads the pressure data from the slave computer in real time through the USB module.

[0067] Figure 7a-7b 7 is a schematic diagram of the serial communication interface and acquisition control interface provided by an embodiment of the present invention. The display interface of the host computer during the acquisition process is shown in FIG7 , wherein: Figure 7a For the serial communication interface, Figure 7b It is the acquisition control interface.

[0068] In this embodiment, the serial communication interface can display the corresponding serial port device. After the connection is completed, the acquisition state of the lower computer is controlled by the acquisition control module. The acquisition state includes the acquisition mode, sampling parameters, sampling frequency, sampling gain and number of sampling points. The acquisition mode is divided into continuous sampling and single point sampling. Figure 7b In the acquisition control interface, the image displays the total pressure curve in real time during continuous sampling, and the pressure change curve of a single sampling point during single-point sampling. The right area controls the start and end of sampling, and presents the 5×5 pressure distribution. The color depth of the grayscale image represents the size of the pressure, making it easier for users to observe changes in hand pressure.

[0069] The collected data can be stored and identified in the host computer, wherein data storage can be implemented in the data storage module and identification can be implemented in the identification result module.

[0070] Figures 8a-8b Schematic diagram of the data storage interface and result recognition interface provided by the embodiment of the present invention, such as Figure 8a-8b As shown, Figure 8a For the data storage interface, Figure 8b The result identification interface.

[0071] The data storage module can display the collected pressure data in a table in real time, such as Figure 8a The table shows the pressure of each collection point over time and can be exported for storage at any time, making it convenient for users to analyze and view the data; Figure 8b As shown, after the collection starts, the recognition result module displays the pressure distribution of each collection point on the palm through pressure distribution, and displays the training results of the neural network on the right. The recognition result number will light up at the corresponding digital origin. At the same time, there is a help button under the pressure distribution. When it is turned on, it will interpret the palm collection points corresponding to the 5×5 distributed collection points, so that users can understand the recognition results of the pressure distribution.

[0072] In summary, the embodiment of the present invention acquires gesture pressure data through a tactile sensing glove, uses an FPGA as a lower computer to control data acquisition and transmission, and finally presents the gesture recognition results through a host computer page. The entire gesture recognition function is controlled by the host computer software, realizing an integrated platform for data acquisition, processing, and result display, and supporting real-time gesture recognition and feedback.

[0073] Figure 9 This is a flowchart of the implementation of the gesture recognition method provided by the embodiment of the present invention. Figure 9 The steps performed by the host computer during gesture recognition are explained. Of course, in actual use, Figure 9 The method shown can also be implemented by other devices. Figure 9 As shown, the method may include:

[0074] Step 110: Collect hand pressure data.

[0075] In this embodiment, hand pressure data can be collected by the subject wearing the aforementioned tactile sensor glove. After collection, the ADC module in the slave computer converts the electrical signal representing the hand pressure data into digital signal data. The converted digital signal data is then transmitted to the master computer via the slave computer, where it is processed to complete gesture recognition.

[0076] Step 120: Input the hand pressure data into a pre-trained convolutional spiking neural network model to obtain a gesture recognition result; wherein the pre-trained convolutional spiking neural network model is a convolutional neural network model improved based on the spiking neural network model; the pooling layer and the fully connected layer in the improved convolutional neural network model are composed of the spiking neural network model.

[0077] In this embodiment, after preprocessing hand pressure data is input into a trained convolutional spiking neural network model, the data is processed sequentially through convolutional, pooling, and fully connected layers. The convolutional layers extract preliminary features, which are then filtered and reduced in dimension by the pooling layers. The fully connected layers then map the processed features to specific gesture categories. Ultimately, the model outputs a probability distribution, with each probability value corresponding to a gesture category, indicating the likelihood that the input hand pressure data belongs to that gesture category. The final gesture recognition result is obtained by selecting the category with the highest probability value.

[0078] This embodiment improves the traditional convolutional neural network model and combines it with the spiking neural network (SNN) to obtain the convolutional spiking neural network model (CSNN). This model combines the feature extraction capability of CNN and the spatiotemporal coding characteristics of SNN. While ensuring accurate data processing, it can replace some multiplication operations by pulse accumulation to reduce energy consumption.

[0079] The pooling and fully connected layers in this model are composed of a spiking neural network model. This improves the feature extraction capabilities of the CNN model, and the repeated use of pooling layers effectively reduces the input size. It also reduces the high energy consumption caused by the large number of parameters in the fully connected layers.

[0080] In summary, the embodiment of the present invention improves the pooling layer and the fully connected layer in the convolutional neural network model through the pulse neural network model because the pulse neural network model can naturally process time series information. It transmits and processes information in the form of a pulse sequence, and replaces the multiplication and accumulation in the traditional convolutional neural network with the pulse addition and accumulation of the pulse neural network model, which can reduce the energy consumption of the application during operation. That is, the improved pooling layer can effectively reduce the size of the input, and at the same time, the improved fully connected layer reduces the high energy consumption caused by the large number of parameters brought by the fully connected layer in the traditional model. Therefore, this embodiment improves the traditional convolutional neural network and combines the pulse neural network model with the convolutional neural network model. The improved model integrates the feature extraction ability of the convolutional neural network and the spatiotemporal coding characteristics of the pulse neural network model. It can replace some multiplication operations by pulse accumulation while ensuring accurate data processing to reduce energy consumption.

[0081] In an optional embodiment, the spiking neural network model includes an Izhikevich neuron model; the pre-trained convolutional spiking neural network model includes an encoder and a classifier; wherein the encoder includes the Izhikevich neuron model; the hand pressure data is input into the pre-trained convolutional spiking neural network model to obtain a gesture recognition result, including:

[0082] According to the hand pressure data, the static component and the dynamic component are obtained.

[0083] The static component and dynamic component are input into the pre-trained convolutional spiking neural network model separately.

[0084] In the encoder in the pre-trained convolutional spike neural network model, the static component and the dynamic component are encoded to obtain the dynamic pulse component and the static pulse component, and the dynamic pulse component and the static pulse component are input into the classifier to obtain the gesture recognition result.

[0085] Figure 10 This is the architecture diagram of the pre-trained convolutional spike neural network model provided by the embodiment of the present invention. Figure 10 This embodiment will be described.

[0086] In this embodiment, the pre-trained convolutional spiking neural network model consists of an encoder and a classifier. When hand pressure data is required for gesture recognition, it is first converted into a digital signal. This signal is then filtered using a low-pass filter to simulate the low-frequency characteristics of skin pressure. Because pressure acquisition is a dynamic process, after passing through the low-pass filter, the pressure data is estimated into two components: a static component and a dynamic component. The dynamic component represents the pressure that changes over time during the pressure sensing process. These static and dynamic components are then input into the pre-trained convolutional spiking neural network model.

[0087] In the pre-trained convolutional spike neural network model, the static component and the dynamic component are first encoded separately through the encoder to obtain the dynamic pulse component and the static pulse component, and the obtained dynamic pulse component and the static pulse component are input into the classifier, and the classifier performs feature extraction to obtain the gesture recognition result.

[0088] The Izhikevich neuron model is built based on biophysical principles, taking into account the ion channel characteristics of neurons. It can simulate multiple discharge patterns and is closer to real neuronal behavior. It is used to encode tactile pressure data.

[0089] In this embodiment, the encoder encodes the pressure data into a spike behavior that conforms to the tactile neuron, that is, a pulse signal, through the Izhikevich neuron model. The emission frequency of the pulses in a certain time window is then used as the input of the classifier. The pulse addition and accumulation replaces the partial multiplication and accumulation in the traditional convolutional neural network model, which can reduce energy consumption in the application and achieve lightweight hardware integration of gesture recognition.

[0090] The following describes the encoding process of the encoder through an optional embodiment. In the encoder in the pre-trained convolutional spike neural network model, the static component and the dynamic component are encoded to obtain the dynamic pulse component and the static pulse component, including:

[0091] In the encoder of the pre-trained convolutional spike neural network model, the dynamic component is encoded based on the Izhikevich neuron model in the fast discharge mode, and the dynamic pulse component is output; the static component is encoded based on the Izhikevich neuron model in the regular discharge mode, and the static pulse component is output.

[0092] The Izhikevich neuron model is a simplification of the Hodgkin-Huxley neuron, aiming to balance computational efficiency and authenticity. Its dynamic equation is:

[0093]

[0094] The pulses of the Izhikevich neuron model are not superimposed one by one, but rise rapidly in a short period of time through a quadratic function, and a recovery variable u is added. Its formula is:

[0095]

[0096] The Izhikevich neuron model satisfies the rich dongtai characteristics of neurons without significantly increasing computational complexity. Among them, a, b, c, and d are the four parameters of the neuron. Their selection allows the model to produce a variety of different neuron firing patterns. For example, when a=0.02, b=0.2, c=-65, and d=8, it can simulate regular spiking (RS) neurons. When a=0.1, b=0.2, c=-65, and d=2, it can simulate fast spiking (FS) neurons.

[0097] Tactile perception is essentially a neural discharge activity based on the tactile receptors on the skin surface under external pressure. Therefore, the embodiment of the invention uses the Izhikevich neuron model to simulate the tactile nerve discharge behavior of hand pressure. There are four types of mechanical receptors on the hand skin, each of which senses different tactile signals: slowly adapting type 1 (SA-1) receptors, which are sensitive to continuous pressure and adapt slowly, helping to perceive details such as the shape and edges of objects; slowly adapting type 2 (SA-2) receptors, which mainly sense skin stretching and changes in joint position; rapidly adapting type 1 (RA-1) receptors, which are sensitive to low frequencies and rapid vibrations and can adapt quickly, and are mainly distributed in the palm, fingers, and other parts of the hand; and rapidly adapting type 2 (RA-2) receptors, which are sensitive to rapid vibrations and high frequencies. The four types of skin receptors SA-1 and RA-1 are mainly distributed on the skin surface and are mainly used to sense the pressure on the skin surface. Therefore, this embodiment uses the Izhikevich neuron model to simulate the discharge behavior of neurons when the two sense pressure, thereby completing the encoding of tactile pressure data.

[0098] Figure 11 This is a schematic diagram of the tactile nerve response encoding process provided by an embodiment of the present invention. Figure 11This embodiment will be described.

[0099] In the pre-trained convolutional spike neural network model, the static component is input into the Izhikevich neuron model based on the rapid discharge mode (RS) to simulate the SA-1 receptor, and the dynamic component is input into the Izhikevich neuron model based on the regular discharge mode (FS) to simulate the RA-1 receptor. Based on the Izhikevich neuron models in these two modes, the static pulse component and the dynamic pulse component are obtained.

[0100] In addition, it should be noted in this embodiment that, since this embodiment improves the convolutional neural network model through the pulse neural network model, the particularity of the pulse neural network model SNN processing signals makes its step function θ(x) non-differentiable. Therefore, during the back propagation process, the gradient descent method is very unstable during the training process. Therefore, during the training process of the convolutional pulse neural network model, θ(x) is still used during the forward propagation process, and a differentiable function g(x) is used to replace θ(x) during the back propagation process, and its derivative is the gradient proxy function: In the embodiment of the present invention, the Atan function is used as g(x) instead of θ(x), that is, the function g(x) is in the form of:

[0101]

[0102] Here, α is a compression function. By adjusting the value of α, the smoothness of the function can be controlled. A larger α results in a closer approximation to the step function θ(x). However, the gradient tends to explode near x = 0 and vanish away from zero. Therefore, this embodiment uses g(x) when α = 2 as the proxy gradient function.

[0103] The following describes the feature extraction and recognition process of the classifier in the convolutional spiking neural network model through an optional embodiment:

[0104] In an optional embodiment, the spiking neural network model includes an integrate-and-fire neuron model; the classifier includes a first convolutional layer; the first convolutional layer is composed of an integrate-and-fire neuron model; the dynamic pulse component and the static pulse component are input into the classifier to obtain a gesture recognition result, including:

[0105] The first convolutional layer in the classifier extracts time series features from the dynamic pulse component and the static pulse component, and obtains the gesture recognition result based on the extracted time series features.

[0106] In an optional embodiment, the classifier further includes a second convolutional layer and a third convolutional layer; the second convolutional layer and the third convolutional layer include a pooling layer; the first convolutional layer in the classifier performs time series feature extraction on the dynamic pulse component and the static pulse component, and obtains a gesture recognition result based on the extracted time series features, including:

[0107] The time series features of the dynamic pulse component and the static pulse component are extracted through the first convolution layer in the classifier to obtain time series feature data, and the time series feature data is transmitted to the second convolution layer and the third convolution layer.

[0108] In the second convolution layer and the third convolution layer, spatial features are extracted from the time series feature data to obtain spatial feature data.

[0109] Based on the spatial feature data, the gesture recognition result is obtained.

[0110] Figure 12 This is a schematic diagram of the IF model structure. Figure 13 This is a schematic diagram of the charging and discharging process of the IF model. Figure 12 as well as Figure 13 This embodiment will be described.

[0111] The integrate-and-fire neuron model has the advantages of a simple discharge pattern and low computational complexity. By using the integrate-and-fire neuron model, improving the pooling layer and fully connected layer in the convolutional neural network model can reduce computational costs.

[0112] There are two types of integrate-and-fire neuron models used in this embodiment, one is the leaky integrate-and-fire (LIF) model, and the other is the integrate-and-fire (IF) model. Figure 12 As shown, I1(t), I2(t), and I3(t) represent pulse signals from different front neurons. These signals are input to the current neuron in the form of discrete pulse trains. Each pulse train represents the electrical pulses emitted by the front neurons at different times, carrying the state information of the front neurons. These input pulse signals are summarized at the summing node in the neuron. It accumulates the intensities of multiple input signals, reflecting the combined effects of multiple front neurons on the current neuron. The result of the accumulation determines the total stimulus intensity received by the current neuron. The signal processed by the summing node will affect the membrane potential of the neuron. When the membrane potential accumulation reaches a certain condition, such as Figure 13 As shown, that is, when the membrane potential accumulation reaches the threshold, the neuron will output a pulse signal O(t) and pass it to the next neuron, realizing the transmission of information in the neural network.

[0113] exist Figure 13In the graph, the horizontal axis represents the acquisition time, and the vertical axis represents the neuron's membrane potential. Initially, the membrane potential is at a relatively low level. Over time, as input pulses enter the neuron, the membrane potential gradually rises. This process is called charging. Each input pulse increases the membrane potential by a certain amount, reflecting the cumulative effect of the input signal on the membrane potential.

[0114] When the membrane potential continues to rise and reaches the threshold V th When a neuron fires, it will discharge. This is the excitation process of the neuron. After the discharge, the membrane potential does not immediately return to its initial value, but instead drops to a reset potential.

[0115] After the membrane potential drops to the reset potential, the neuron begins a new round of charging, waiting for the accumulation of the next input pulse signal. When the threshold is reached again, the discharge process repeats. This cycle repeats, enabling neurons to continuously receive, process, and transmit information.

[0116] Therefore, based on Figure 12 as well as Figure 13 It can be seen that the signals transmitted between neurons are all pulse signals. The IF model receives the signal from the previous neuron and integrates the signal. When the signal value, that is, the membrane potential, reaches the threshold, the neuron discharges and outputs a pulse signal. The whole process is called integration-excitation.

[0117] Its kinetic equation is:

[0118] H t =f(V t-1 ,X t )

[0119] S t =θ(H t -V th )

[0120] V t =H t (1-S t )+V reset S t

[0121] Among them, V t is the neuronal membrane voltage; X t is external input, such as membrane potential increment; H t is the hidden state of the neuron, that is, the moment before the pulse signal is emitted; V th is the excitation threshold; V reset is the reset potential; S tis the output value with a time step of t, which is 1 if there is a pulse signal and 0 otherwise; θ(x) is a step signal, when x ≥ 0, θ(x) = 1, when x < 0, θ(x) = 0; f(x) is the state update equation, and the IF and LIF neuron models are:

[0122]

[0123] like Figure 10 As shown in Figure 5, the convolutional layers in this model are composed of IF models, and the fully connected layers and downsampling pooling layers are composed of LIF models.

[0124] The classifier of the model includes the first convolutional layer, the second convolutional layer and the third convolutional layer. The first convolutional layer is composed of the IF model, which serves as the basic receiving area and basic input processing unit of the neural network, and is used to extract time series features of dynamic pulse components and static pulse components to obtain time series features.

[0125] The second convolution layer and the third convolution layer are deep convolution layers, which include a fully connected layer and a downsampling pooling layer composed of the LIF model, and are used to extract spatial features from the time series feature data to obtain spatial feature data, and obtain gesture recognition results based on the spatial feature data.

[0126] This example uses three convolutional layers to extract spatiotemporal features from the training data, ensuring data accuracy. Furthermore, pooling layers are added to the second and third convolutional layers to ensure that only excited neurons pass information on. Finally, after each convolutional layer, a spiking neuron model is used to enable network interaction between layers. Finally, after the fully connected layer, a LIF neuron model is used to enable temporal inter-layer accumulation across the entire network, demonstrating the dynamic characteristics of the network in both the spatiotemporal and temporal domains.

[0127] Optionally, after inputting the hand pressure data into a pre-trained convolutional spiking neural network model to obtain gesture recognition results, the following steps may also be included:

[0128] Visualize the gesture recognition results.

[0129] The purpose of visualizing gesture recognition results is to enable users and researchers to quickly and intuitively understand the model's output. Compared to simple numbers or categorical labels, visualization can present information in a more literal way, lowering the barrier to understanding and making it easier to evaluate recognition results and identify potential issues, such as model misjudgments.

[0130] In this embodiment, the visual display method may include a graphical display; wherein the graphical display may include a gesture indication or a dynamic demonstration. The gesture indication may be realized by drawing a schematic diagram corresponding to the recognized gesture. For example, if the recognized gesture is a "thumbs-up" gesture, a simple hand thumbs-up graphic is presented on the interface. The dynamic demonstration may use animation to dynamically display the changing process of the gesture. For some continuous gestures, such as writing gestures, the recognized actions are demonstrated step by step in a time series, which can clearly show the trajectory and morphological changes of the gesture.

[0131] Through visual display, we can evaluate model performance, analyze the recognition accuracy of the model in different gestures and different environments, identify the model's weaknesses, and provide a basis for further optimization of the model.

[0132] In summary, compared with traditional methods, the method provided by the embodiment of the present invention has the following advantages:

[0133] 1. Reduced energy consumption. Traditional CNN models rely on a large number of matrix multiplication operations, resulting in high energy consumption and making them difficult to deploy in embedded or mobile devices. The method provided in this embodiment of the present invention adopts a CSNN network architecture and replaces some multiplication and accumulation with the pulse accumulation of SNN, making it more suitable for low-power scenarios.

[0134] 2. Tactile nerve response characteristics. When encoding pressure data, the embodiment of the present invention uses Izhikevich neurons to form a tactile nerve morphology model, simulating the neuronal discharge behavior of skin receptors when sensing pressure, and encoding continuous pressure data into pulse signals with gesture data characteristics.

[0135] 3. Enhanced spatiotemporal information processing capabilities. Traditional neural networks (such as ANNs and CNNs) can only process spatial domain data and ignore dynamic features in the temporal domain. The embodiments of the present invention combine the spatiotemporal processing capabilities of SNNs with the feature extraction capabilities of CNNs. This not only enables adaptive coding of the network model but also simultaneously captures the static pressure distribution and dynamic changes of gestures, improving the recognition accuracy of complex gestures.

[0136] 4. Expanded application scenarios. Traditional solutions are often limited to specific fields or fixed scenarios. The embodiments of this invention, combined with low power consumption, high robustness, and lightweight hardware, can be widely used in mobile or embedded scenarios such as smart homes, AR / VR interaction, industrial control, and sports training.

[0137] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0138] In the above embodiments, the descriptions of each embodiment have their own focus. For parts not described or recorded in detail in one embodiment, please refer to the relevant descriptions of other embodiments. Unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other. The technical features of different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0139] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A gesture recognition method, characterized in that: include: Collect hand pressure data; The hand pressure data is input into a pre-trained convolutional spiking neural network model to obtain a gesture recognition result; wherein the pre-trained convolutional spiking neural network model is a convolutional neural network model improved based on the spiking neural network model; the pooling layer and the fully connected layer in the improved convolutional neural network model are composed of the spiking neural network model.

2. The gesture recognition method according to claim 1, characterized in that: The spiking neural network model includes an Izhikevich neuron model; the pre-trained convolutional spiking neural network model is composed of an encoder and a classifier; wherein the encoder is composed of the Izhikevich neuron model; the hand pressure data is input into the pre-trained convolutional spiking neural network model to obtain a gesture recognition result, including: Obtaining a static component and a dynamic component according to the hand pressure data; Inputting the static component and the dynamic component into a pre-trained convolutional spiking neural network model respectively; In the encoder in the pre-trained convolutional pulse neural network model, the static component and the dynamic component are encoded to obtain a dynamic pulse component and a static pulse component, and the dynamic pulse component and the static pulse component are input into a classifier to obtain a gesture recognition result.

3. The gesture recognition method according to claim 2, characterized in that: The spiking neural network model includes an integrate-and-fire neuron model; the classifier includes a first convolutional layer; the first convolutional layer is composed of the integrate-and-fire neuron model; the dynamic pulse component and the static pulse component are input into the classifier to obtain a gesture recognition result, including: Time series features are extracted from the dynamic pulse component and the static pulse component through the first convolution layer in the classifier, and a gesture recognition result is obtained based on the extracted time series features.

4. The gesture recognition method according to claim 3, characterized in that: The classifier further includes a second convolution layer and a third convolution layer; the second convolution layer and the third convolution layer include a pooling layer; the first convolution layer in the classifier extracts time series features from the dynamic pulse component and the static pulse component, and obtains a gesture recognition result based on the extracted time series features, including: Performing time series feature extraction on the dynamic pulse component and the static pulse component through the first convolution layer in the classifier to obtain time series feature data, and transmitting the time series feature data to the second convolution layer and the third convolution layer; In the second convolution layer and the third convolution layer, spatial feature extraction is performed on the time series feature data to obtain spatial feature data; A gesture recognition result is obtained based on the spatial feature data.

5. The gesture recognition method according to claim 2, characterized in that: The encoder in the pre-trained convolutional pulse neural network model encodes the static component and the dynamic component to obtain a dynamic pulse component and a static pulse component, comprising: In the encoder in the pre-trained convolutional pulse neural network model, the dynamic component is encoded based on the Izhikevich neuron model in the fast discharge mode, and the dynamic pulse component is output; based on the Izhikevich neuron model in the regular discharge mode, the static component is encoded and the static pulse component is output.

6. The gesture recognition method according to claim 2, characterized in that: Obtaining the static component and the dynamic component according to the hand pressure data includes: converting the hand pressure data into a digital signal; The digital signal is filtered to obtain a static component and a dynamic component.

7. The gesture recognition method according to claim 1, wherein: After inputting the hand pressure data into a pre-trained convolutional spiking neural network model to obtain a gesture recognition result, the method further includes: The gesture recognition result is visually displayed.

8. A host computer, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

9. A gesture recognition system, characterized in that: The device comprises a tactile sensor glove, a lower computer and the upper computer as claimed in claim 8.

10. The gesture recognition system according to claim 9, characterized in that: The lower computer includes a field editable gate array.

Images